Compressing Lua scripts to use less memory

waterwingz

12134

Compressing Lua scripts to use less memory

« on: 06 / December / 2014, 23:25:06 »

Advertisements

As some of the better Lua scripts continue to grow in size, we have occasionally needed to resort to a "loader" (code courtesy of rudi) to get them to fit into memory.

philmoz pointed out another option on the freenode IRC channel today : https://mothereff.in/lua-minifier

(source code here : https://github.com/mathiasbynens/luamin )

This little utility strips the size of my CHDKplus.lua script from about 40K to 22K. You need to add the CHDK header stuff back once the utility finishes as it strips off all comments.

Of course, the resulting file is essentially unreadable and it loses useful things like line numbers when there are errors. But it can let scripts load and run without a separate loader script.

Edit : a few other options ( although not available as a simple online utility )

« Last Edit: 07 / December / 2014, 00:03:54 by waterwingz »

Logged

Ported : A1200 SD940 G10 Powershot N G16

reyalp

13947

Re: Compressing Lua scripts to use less memory

« Reply #1 on: 07 / December / 2014, 00:46:57 »

FWIW, if you don't want to have to re-add the header every time, you could put it in a "long string" like

Code: [Select]

x=[[
@title lua lib test
@param a skip io
@default a 0
...
]]
x=nil

Since the newlines are part of the string, the minifier won't mess with them, and setting x=nil will allow the string to be garbage collected. CHDK itself doesn't care if the @params are in comments

A loader will still be substantially more memory efficient, since the entire source is discarded after loading.

The luamin page lead me to https://github.com/stravant/LuaMinify which is a Lua minifier written in lua. This could be quite useful for chdkptp remote code. Possible also useful for people who just want a command line workflow.

Thanks

Logged

Don't forget what the H stands for.

waterwingz

12134

Re: Compressing Lua scripts to use less memory

« Reply #2 on: 07 / December / 2014, 10:49:10 »

Quote from: reyalp on 07 / December / 2014, 00:46:57

The luamin page lead me to https://github.com/stravant/LuaMinify which is a Lua minifier written in lua. This could be quite useful for chdkptp remote code. Possible also useful for people who just want a command line workflow.

Did some testing and Luamin produces compression results almost as good as lua-minifier (40K->23K vs 40K->22K) but leaves "newlines" in the code. This is a big advantage from a error message / debugging perspective !

And being written in Lua, the code ran on the command line "as is" from the github, once I converted :

Code: [Select]

-require'Strict'
+require'strict'

in ParseLua.lua

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3445

Re: Compressing Lua scripts to use less memory

« Reply #3 on: 07 / December / 2014, 16:31:28 »

Looking at the 'loader' script for kap.lua, it seems this technique could be used to run all Lua scripts within CHDK.

Instead of calling libscriptapi->script_start() with the pre-loaded script as a string could we do something like:

Code: [Select]

    char loader[100];
    sprintf(loader,"local sub = loadfile(%s); collectgarbage(); sub()");
    libscriptapi->script_start(loader,0);

Would this works? Would it give much memory saving?
This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?
It would not work while video is recording unless luaL_loadfile was re-written to use 'open' instead of 'fopen'.

We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

waterwingz

12134

Re: Compressing Lua scripts to use less memory

« Reply #4 on: 07 / December / 2014, 16:55:13 »

Quote from: philmoz on 07 / December / 2014, 16:31:28

We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.

AFAIK, when this has been discussed in the past, needing to load the title & parameters has always been the reason it has not been attempted. Having said that, finding a way to overcome that issue would be very worthwhile.

Quote from: philmoz on 07 / December / 2014, 16:31:28

This would also need to re-load the script file from the SD card every time the script is run, would this be a significant performance hit?

Actually, during script development it would be a bonus. Just update the script file via ptp and run it instead of update the file, reload the file via the CHDK menus, and run.

« Last Edit: 07 / December / 2014, 17:03:15 by waterwingz »

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3445

Re: Compressing Lua scripts to use less memory

« Reply #5 on: 07 / December / 2014, 18:02:05 »

Quote from: waterwingz on 07 / December / 2014, 16:55:13

Quote from: philmoz on 07 / December / 2014, 16:31:28
We would still need to do an initial parse of the script for the title and parameter info; but this could probably be done without loading the entire file.
AFAIK, when this has been discussed in the past, needing to load the title & parameters has always been the reason it has not been attempted. Having said that, finding a way to overcome that issue would be very worthwhile.

I think the script parameter handling could stand some improvement

In addition to only parsing the parameter list once when the script is selected, I think the following could be done:
- Don't save the last used set of all 26 parameter values in the CHDK config file.
I never really understood this, the values are loaded from the saved param file for the script so don't need to be in the config file as well (perhaps it's historical).
- Use dynamically allocated structures to store the parameter info instead of fixed arrays.
- Allow arbitrary parameter variable names for Lua (instead of just 'a' - 'z'). Could also support 'A' - 'Z' for uBasic, although this is less useful.

Anything else to add to the wish list?

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

waterwingz

12134

Re: Compressing Lua scripts to use less memory

« Reply #6 on: 07 / December / 2014, 18:18:12 »

Quote from: philmoz on 07 / December / 2014, 18:02:05

In addition to only parsing the parameter list once when the script is selected, I think the following could be done:
- Don't save the last used set of all 26 parameter values in the CHDK config file.
I never really understood this, the values are loaded from the saved param file for the script so don't need to be in the config file as well (perhaps it's historical).
- Use dynamically allocated structures to store the parameter info instead of fixed arrays.

I'm sorry to say that I still don't trust the whole script parameter saving mechanism. It's a confusing mess that you basically have to use "blind". I know you made a lot of stability fixes but it just does not do anything for me. Scrap it all and start again would be my suggestion.

One thing that might help is named parameter sets (rather than 0-9) and the ability to edit them (rather than hope you saved the current values correctly and then got the menu reset properly so they stay saved and don't get overwritten and forget about reviewing the values later so you better remember what you set ..)

Quote

Allow arbitrary parameter variable names for Lua (instead of just 'a' - 'z'). Could also support 'A' - 'Z' for uBasic, although this is less useful.

The first bunch of lines at the top of almost all my scripts are simply long variable names being set to the single variable parameter values. This would be a great improvement - although it would mean script written for 1.4.0 won't run on older versions. Unless we add some sort of hacked up backwards compatibility mode.

I would not change a thing in uBASIC. To those who like to use it, it's apparently perfect just like it is

Quote

Anything else to add to the wish list?

I'll noodle on it a bit more but I think the suggestions listed above would be a huge step forward.

Logged

Ported : A1200 SD940 G10 Powershot N G16

srsa_4c

4451

Re: Compressing Lua scripts to use less memory

« Reply #7 on: 07 / December / 2014, 18:37:37 »

Quote from: philmoz on 07 / December / 2014, 18:02:05

Anything else to add to the wish list?

Perhaps allowing 32bit integer parameters? I had a proposal once.

Logged

philmoz

3445

Re: Compressing Lua scripts to use less memory

« Reply #8 on: 07 / December / 2014, 19:10:16 »

Quote from: srsa_4c on 07 / December / 2014, 18:37:37

Quote from: philmoz on 07 / December / 2014, 18:02:05
Anything else to add to the wish list?
Perhaps allowing 32bit integer parameters? I had a proposal once.

The script parameter values are already 32 bit.

The limitation on input values comes from:
- the @range setting limited to 16 bit values
- the menu input system only allows values from -9,999 to 99,9999 (except for the SD override hack)

Fixing @range would probably not be too hard, fixing the menu input may be a bit more work.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

waterwingz

12134

Re: Compressing Lua scripts to use less memory

« Reply #9 on: 07 / December / 2014, 19:25:02 »

Quote from: srsa_4c on 07 / December / 2014, 18:37:37

Quote from: philmoz on 07 / December / 2014, 18:02:05
Anything else to add to the wish list?
Perhaps allowing 32bit integer parameters? I had a proposal once.

The idea you had there of allowing hexidecimal inputs was pretty good too.

Logged

Ported : A1200 SD940 G10 Powershot N G16