Compressing Lua scripts to use less memory

reyalp

14035

Re: Compressing Lua scripts to use less memory

« Reply #10 on: 07 / December / 2014, 19:44:58 »

Advertisements

Quote from: philmoz on 07 / December / 2014, 16:31:28

Would this works? Would it give much memory saving?

Something like that should work. Memory saving would be the size of the source file. Doing something like this has been on my todo list for a long time, but hasn't been a priority since there is a usable workaround, and dealing with the params looked ugly.

Quote

We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.

IIRC we don't scan the whole file anyway, certainly requiring the params be in the first few k of the file doesn't seem unreasonable.

Quote

This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?

It would probably be worth checking, but I suspect it's not a big deal.

Logged

Don't forget what the H stands for.

msl

1280
A720 IS, SX220 HS 1.01a

Re: Compressing Lua scripts to use less memory

« Reply #11 on: 08 / December / 2014, 05:50:26 »

Quote from: waterwingz on 07 / December / 2014, 16:55:13

Actually, during script development it would be a bonus. Just update the script file via ptp and run it instead of update the file, reload the file via the CHDK menus, and run.

We've been doing this for a long time with hostlua, chdkptp and notepad ++. The script is started in notepad++ and runs on the camera.

=> http://chdk.wikia.com/wiki/Execute_Lua_scripts_with_hostlua_%26_notepad%2B%2B

rudi has written a very nice function for the transfer of the script parameters.

Code: [Select]

-[[
********************************
REMOTE LIBRARY "lremote.lua"
set script defaults and run script remote

Project "hostlua" for CHDK
Licence: GPL
(C)2013 rudi, Version: 2
********************************
usage:
    luar require("lremote").exec(FULL_SCRIPTNAME, OUTPUT_TO_PTP)
    OUTPUT_TO_PTP = true or false

]]

lremote = {}

lremote.version= 2

local function init_defaultvars(filename)
    for line in io.lines(filename) do 
        local var, val=line:match("%s*@default%s*([A-Za-z])%s*([0-9]+)")
        if var and val then rawset(_G,var,tonumber(val)) end
    end
end

local function ptp_print(...)
    local str=""
    for i=1,select("#",...) do
        if #str>0 then str=str.." " end
        str=str..tostring(select(i,...))
    end
    cam_print(...)
    write_usb_msg(str)
    --write_usb_msg() need a little bit time for write, otherwise it lost lines
    sleep(10)
end

function lremote.exec(filename, output)
    init_defaultvars(filename)
    if output then cam_print, print = print, ptp_print end
    set_record = switch_mode_usb
    local script, loaderr=loadfile(filename)
    if script then script() else print(loaderr)
    end
end

return lremote

msl

Logged

CHDK-DE: CHDK-DE links

philmoz

3450

Re: Compressing Lua scripts to use less memory

« Reply #12 on: 09 / December / 2014, 13:40:43 »

Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose - for example (borrowed from the kap loader script):

Code: [Select]

@param     i Shot Interval (sec)
  @default i 15
  @range   i 2 120
@param     s Total Shots (0=infinite)
  @default s 0
  @range   s 0 10000
@param     j Power off when done?
  @default j 0
  @range   j 0 1
@param     e Exposure Comp (stops)
  @default e 6
  @values  e -2.0 -1.66 -1.33 -1.0 -0.66 -0.33 0.0 0.33 0.66 1.00 1.33 1.66 2.00

This could be reduced down to something like:

Code: [Select]

#i=15 "Shot Interval (sec)" [2 120]
#s=0 "Total Shots (0=infinite)" [0 10000]
#j=0 "Power off when done?" [0 1]
#e=6 "Exposure Comp (stops)" {-2.0 -1.66 -1.33 -1.0 -0.66 -0.33 0.0 0.33 0.66 1.00 1.33 1.66 2.00}

The syntax is just an example; but it definitely is a lot smaller.
(I replaced '@' with '#' to allow both methods to be supported for compatibility with existing scripts).

Scripts that used this would no longer run on old versions of CHDK, ~~so this would need to be a 1.4 change~~. Thinking more about this, it would be better if both 1.3 and 1.4 supported the changes (less issues once 1.3 is stable and people have updated).

Phil.

« Last Edit: 09 / December / 2014, 16:03:34 by philmoz »

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4451

Re: Compressing Lua scripts to use less memory

« Reply #13 on: 09 / December / 2014, 15:46:15 »

Quote from: philmoz on 09 / December / 2014, 13:40:43

Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose

Could we pre-parse the script header and store it more efficiently, separated from the script body? Verbosity (and keeping the old format) is good for the users and space-saving only matters when the script is loaded.

Logged

philmoz

3450

Re: Compressing Lua scripts to use less memory

« Reply #14 on: 09 / December / 2014, 16:07:25 »

Quote from: srsa_4c on 09 / December / 2014, 15:46:15

Quote from: philmoz on 09 / December / 2014, 13:40:43
Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose
Could we pre-parse the script header and store it more efficiently, separated from the script body? Verbosity (and keeping the old format) is good for the users and space-saving only matters when the script is loaded.

Not sure I follow. People using the script never see this, they see the menu items built from the parameter info - this would not change.

This really only affects people writing scripts.

I thought about separating the parameter stuff into a new file; but felt this would make things harder for script authors.

Supporting the existing format is a given, we can't force everyone to update all scripts.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4451

Re: Compressing Lua scripts to use less memory

« Reply #15 on: 09 / December / 2014, 16:18:04 »

Quote from: philmoz on 09 / December / 2014, 16:07:25

This really only affects people writing scripts.

By "users" I meant "users who write scripts".
I have to admit that I know very little about the CHDK scripting implementation. I thought you were concerned about RAM occupied by the verbose script header, thus my suggestion.

Logged

eponymous

38
Canon A4000

Re: Compressing Lua scripts to use less memory

« Reply #16 on: 10 / December / 2014, 10:26:56 »

I find the proposed change very easy to understand.

Logged

John

philmoz

3450

Re: Compressing Lua scripts to use less memory

« Reply #17 on: 19 / December / 2014, 19:21:08 »

Quote from: philmoz on 07 / December / 2014, 16:31:28

Looking at the 'loader' script for kap.lua, it seems this technique could be used to run all Lua scripts within CHDK.

Instead of calling libscriptapi->script_start() with the pre-loaded script as a string could we do something like:
Code: [Select]
char loader[100]; sprintf(loader,"local sub = loadfile(%s); collectgarbage(); sub()"); libscriptapi->script_start(loader,0);
Would this works? Would it give much memory saving?
This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?
It would not work while video is recording unless luaL_loadfile was re-written to use 'open' instead of 'fopen'.

We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.

Phil.

Looking at this again, the parameter issues should be fixable now since the parameters can be loaded once from the file.

The next complication is the 'lua_script_default' script hard wired into the CHDK source, if no script file is selected or it cant be loaded it uses this built in script. This script allows language selection from English, German and Russian.

Having a built in script makes using a loader for Lua scripts more difficult. It's probably doable; but the code is going to get messy.

I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.

Any thoughts?

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

waterwingz

12138

Re: Compressing Lua scripts to use less memory

« Reply #18 on: 19 / December / 2014, 19:44:06 »

Quote from: philmoz on 19 / December / 2014, 19:21:08

I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.

Hard to see any downside to this approach.

Logged

Ported : A1200 SD940 G10 Powershot N G16

srsa_4c

4451

Re: Compressing Lua scripts to use less memory

« Reply #19 on: 19 / December / 2014, 20:01:36 »

Quote from: philmoz on 19 / December / 2014, 19:21:08

The next complication is the 'lua_script_default' script hard wired into the CHDK source, if no script file is selected or it cant be loaded it uses this built in script. This script allows language selection from English, German and Russian.

Having a built in script makes using a loader for Lua scripts more difficult. It's probably doable; but the code is going to get messy.

I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.

Any thoughts?

Since changing the language from the default needs an existing language file anyway, moving that script into a file won't make the user experience any worse. So, IMO it's a good idea.

Logged