Compressing Lua scripts to use less memory

  • 28 Replies
  • 3416 Views
*

Offline reyalp

  • ******
  • 10944
Re: Compressing Lua scripts to use less memory
« Reply #10 on: 07 / December / 2014, 19:44:58 »
Advertisements
Would this works? Would it give much memory saving?
Something like that should work. Memory saving would be the size of the source file. Doing something like this has been on my todo list for a long time, but hasn't been a priority since there is a usable workaround, and dealing with the params looked ugly.
Quote
We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.
IIRC we don't scan the whole file anyway, certainly requiring the params be in the first few k of the file doesn't seem unreasonable.

Quote
This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?
It would probably be worth checking, but I suspect it's not a big deal.
Don't forget what the H stands for.

*

Offline msl

  • *****
  • 1223
  • A720 IS, SX220 HS 1.01a
    • CHDK inside
Re: Compressing Lua scripts to use less memory
« Reply #11 on: 08 / December / 2014, 05:50:26 »
Actually, during script development it would be a bonus.  Just update the script file via ptp and run it instead of update the file,  reload the file via the CHDK menus, and run.
We've been doing this for a long time with hostlua, chdkptp and notepad ++. The script is started in notepad++ and runs on the camera.

=> http://chdk.wikia.com/wiki/Execute_Lua_scripts_with_hostlua_%26_notepad%2B%2B

rudi has written a very nice function for the transfer of the script parameters.
Code: [Select]
-[[
********************************
REMOTE LIBRARY "lremote.lua"
set script defaults and run script remote

Project "hostlua" for CHDK
Licence: GPL
(C)2013 rudi, Version: 2
********************************
usage:
    luar require("lremote").exec(FULL_SCRIPTNAME, OUTPUT_TO_PTP)
    OUTPUT_TO_PTP = true or false

]]

lremote = {}

lremote.version= 2

local function init_defaultvars(filename)
    for line in io.lines(filename) do
        local var, val=line:match("%s*@default%s*([A-Za-z])%s*([0-9]+)")
        if var and val then rawset(_G,var,tonumber(val)) end
    end
end

local function ptp_print(...)
    local str=""
    for i=1,select("#",...) do
        if #str>0 then str=str.." " end
        str=str..tostring(select(i,...))
    end
    cam_print(...)
    write_usb_msg(str)
    --write_usb_msg() need a little bit time for write, otherwise it lost lines
    sleep(10)
end

function lremote.exec(filename, output)
    init_defaultvars(filename)
    if output then cam_print, print = print, ptp_print end
    set_record = switch_mode_usb
    local script, loaderr=loadfile(filename)
    if script then script() else print(loaderr)
    end
end

return lremote

msl

German CHDK pages:  CHDK forum | CHDK inside | CHDK Twitter News by msl | Download CHDK-DE (Autobuild)
Note: SDM violates the GPL rules!

*

Offline philmoz

  • *****
  • 3049
    • Photos
Re: Compressing Lua scripts to use less memory
« Reply #12 on: 09 / December / 2014, 13:40:43 »
Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose - for example (borrowed from the kap loader script):
Code: [Select]
@param     i Shot Interval (sec)
  @default i 15
  @range   i 2 120
@param     s Total Shots (0=infinite)
  @default s 0
  @range   s 0 10000
@param     j Power off when done?
  @default j 0
  @range   j 0 1
@param     e Exposure Comp (stops)
  @default e 6
  @values  e -2.0 -1.66 -1.33 -1.0 -0.66 -0.33 0.0 0.33 0.66 1.00 1.33 1.66 2.00

This could be reduced down to something like:
Code: [Select]
#i=15 "Shot Interval (sec)" [2 120]
#s=0 "Total Shots (0=infinite)" [0 10000]
#j=0 "Power off when done?" [0 1]
#e=6 "Exposure Comp (stops)" {-2.0 -1.66 -1.33 -1.0 -0.66 -0.33 0.0 0.33 0.66 1.00 1.33 1.66 2.00}

The syntax is just an example; but it definitely is a lot smaller.
(I replaced '@' with '#' to allow both methods to be supported for compatibility with existing scripts).

Scripts that used this would no longer run on old versions of CHDK, so this would need to be a 1.4 change. Thinking more about this, it would be better if both 1.3 and 1.4 supported the changes (less issues once 1.3 is stable and people have updated).

Phil.
« Last Edit: 09 / December / 2014, 16:03:34 by philmoz »
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)

*

Offline srsa_4c

  • ******
  • 3474
Re: Compressing Lua scripts to use less memory
« Reply #13 on: 09 / December / 2014, 15:46:15 »
Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose
Could we pre-parse the script header and store it more efficiently, separated from the script body? Verbosity (and keeping the old format) is good for the users and space-saving only matters when the script is loaded.


*

Offline philmoz

  • *****
  • 3049
    • Photos
Re: Compressing Lua scripts to use less memory
« Reply #14 on: 09 / December / 2014, 16:07:25 »
Another way to save some space would be to redo the way parameters are defined in the script.

This current method is somewhat verbose
Could we pre-parse the script header and store it more efficiently, separated from the script body? Verbosity (and keeping the old format) is good for the users and space-saving only matters when the script is loaded.

Not sure I follow. People using the script never see this, they see the menu items built from the parameter info - this would not change.

This really only affects people writing scripts.

I thought about separating the parameter stuff into a new file; but felt this would make things harder for script authors.

Supporting the existing format is a given, we can't force everyone to update all scripts.

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)

*

Offline srsa_4c

  • ******
  • 3474
Re: Compressing Lua scripts to use less memory
« Reply #15 on: 09 / December / 2014, 16:18:04 »
This really only affects people writing scripts.
By "users" I meant "users who write scripts".
I have to admit that I know very little about the CHDK scripting implementation. I thought you were concerned about RAM occupied by the verbose script header, thus my suggestion.

*

Offline eponymous

  • *
  • 38
  • Canon A4000
Re: Compressing Lua scripts to use less memory
« Reply #16 on: 10 / December / 2014, 10:26:56 »
I find the proposed change very easy to understand.
John

*

Offline philmoz

  • *****
  • 3049
    • Photos
Re: Compressing Lua scripts to use less memory
« Reply #17 on: 19 / December / 2014, 19:21:08 »
Looking at the 'loader' script for kap.lua, it seems this technique could be used to run all Lua scripts within CHDK.

Instead of calling libscriptapi->script_start() with the pre-loaded script as a string could we do something like:
Code: [Select]
    char loader[100];
    sprintf(loader,"local sub = loadfile(%s); collectgarbage(); sub()");
    libscriptapi->script_start(loader,0);

Would this works? Would it give much memory saving?
This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?
It would not work while video is recording unless luaL_loadfile was re-written to use 'open' instead of 'fopen'.

We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.

Phil.

Looking at this again, the parameter issues should be fixable now since the parameters can be loaded once from the file.

The next complication is the 'lua_script_default' script hard wired into the CHDK source, if no script file is selected or it cant be loaded it uses this built in script. This script allows language selection from English, German and Russian.

Having a built in script makes using a loader for Lua scripts more difficult. It's probably doable; but the code is going to get messy.

I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.

Any thoughts?

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)


Re: Compressing Lua scripts to use less memory
« Reply #18 on: 19 / December / 2014, 19:44:06 »
I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.
Hard to see any downside to this approach.
Ported : A1200  SD940  G10  Powershot N  G16*

*

Offline srsa_4c

  • ******
  • 3474
Re: Compressing Lua scripts to use less memory
« Reply #19 on: 19 / December / 2014, 20:01:36 »
The next complication is the 'lua_script_default' script hard wired into the CHDK source, if no script file is selected or it cant be loaded it uses this built in script. This script allows language selection from English, German and Russian.

Having a built in script makes using a loader for Lua scripts more difficult. It's probably doable; but the code is going to get messy.

I'm thinking to do away with any built in default script altogether - the current built in code could be moved to an actual script file and set as the default filename if the user has not selected one. This will also save a bit of memory.

Any thoughts?
Since changing the language from the default needs an existing language file anyway, moving that script into a file won't make the user experience any worse. So, IMO it's a good idea.

 

Related Topics