LUA script Vs precompiled C in speed

geoffire

22

« on: 06 / August / 2011, 09:05:11 »

Advertisements

Hi, I'm new here, but not to programing and scripting. I haven't touched a CHDK script yet (in the research phase), but I'm looking to make some custom modes to replace the ones I don't use in my S95 (low light, scenes, auto,) while making the workings of CHDK transparent to the user. Some things, like change the flash menu into a one press on/off, I'd like online at all times. I've never worked on a complex project like this, so it will take me some time to get up to speed, but I do have a lot of down time at work and I need to entertain myself.

I'll have to refresh myself on C, and LUA looks like much more pleasant to program in, but I really want to make things as responsive as possible.

The question is, if I have complex code is there much of a speed difference? looking at projects like sunsetF16 the main script is very simple with rather complex libs, why not make those in C and patch them in instead of having to compile them in camera? I understand responsiveness isn't exactly a requirement for a intervalometer, but it is for me.

Also, Is it upon loading the scripts are compiled or on the fly?

Thanks,
Jeff

Logged

Microfunguy

4037

Re: LUA script Vs precompiled C in speed

« Reply #1 on: 06 / August / 2011, 10:12:35 »

Hi Jeff and welcome.

The kbd.c code for a camera calls the \core\kbd.c every 10 msec normally (but there is no guarantee).
This is where a single line of uBasic is processed (I do not use Lua).
The scripts are not compiled.
In CHDK, I think multiple rem's are jumped-over to avoid wasting time.

SDM tends to implement many commands in the 'C' code.
For example, here is the time-lapse scipt :-

time_lapse a, b, c, d, e, f, g, h, i, j, k, l, m, n, p, q, r, s, t, u
end

That is it !

For photography from a balloon swinging around in the stratosphere, member Bonomi suggested taking 100 brightness readings and performing certain calculations on them.

His Lua code is here http://www.francescobonomi.it/ballon-photography-black-sky-metering

SDM uses this to get the 100 readings :-

meter_brightness 1

and this to compute a fixed brightness from those readings :-

meter_brightness

I prefer the approach of doing the 'heavy lifting' in 'C' code, it makes the minimal script easier to read.

David

Logged

reyalp

14039

Re: LUA script Vs precompiled C in speed

« Reply #2 on: 06 / August / 2011, 17:55:19 »

Quote from: geoffire on 06 / August / 2011, 09:05:11

Hi, I'm new here, but not to programing and scripting. I haven't touched a CHDK script yet (in the research phase), but I'm looking to make some custom modes to replace the ones I don't use in my S95 (low light, scenes, auto,) while making the workings of CHDK transparent to the user.

This would be a very major project, and speed is the least of your concerns. A good understanding of ARM assembler and reverse engineering are probably more relevant.

Quote

I'll have to refresh myself on C, and LUA looks like much more pleasant to program in, but I really want to make things as responsive as possible.

It depends exactly what you are doing, but having your logic in lua should not be a problem. Lua is very fast. If you are doing stuff that require say, manipulating large portions of the frame buffer or raw buffer, you probably would want to add C functions (or ARM assembler) to do that.

There is really no point in worrying about this until you are further down the road. It is very easy to expose C functions in lua if you have the need. The bigger difficulty is where things run: Lua code always runs in the kbd task at a specific point. The gui runs in spytask, and raw processing runs in the capt_seq task. To safely interface lua with the other tasks requires additional effort.

Quote

The question is, if I have complex code is there much of a speed difference? looking at projects like sunsetF16 the main script is very simple with rather complex libs, why not make those in C and patch them in instead of having to compile them in camera?

For stuff like timelapse, script execution time is irrelevant. Putting this stuff in C code would bloat the binary, whether you need this functionality or not. Putting it in lua gives you a lot more flexibility.

Quote

Also, Is it upon loading the scripts are compiled or on the fly?

Scripts are compiled to lua bytecode when loaded. Of course, you can load / compile strings at runtime as well.

A quick test:

Code: [Select]

con 2> =t1=get_tick_count(); for i=1,100000 do j = i + i; end return get_tick_count() - t1;
3:return:3000

So, 100k iterations of a simple loop in lua took 3 seconds. That looks bad, but lua code yields (for a minimum of 10ms) every 1000 lua bytecode instructions. This could be adjusted if needed, although you do have to be careful not to starve other tasks or hold up kbd task for too long.

Reducing the yield to every 10000k instructions gives us:

Code: [Select]

con> =t1=get_tick_count(); for i=1,100000 do j = i + i; end return get_tick_count() - t1
1:return:340

Looks like the yields are dominating

Eliminating it completely only brings it down to 330ms.

edit:
Above was on d10. a540 does the no yield version in 540 (!) ms.

« Last Edit: 24 / January / 2020, 21:53:57 by reyalp »

Logged

Don't forget what the H stands for.

geoffire

22

Re: LUA script Vs precompiled C in speed

« Reply #3 on: 06 / August / 2011, 23:39:01 »

Quote from: reyalp on 06 / August / 2011, 17:55:19

The bigger difficulty is where things run: Lua code always runs in the kbd task at a specific point. The gui runs in spytask, and raw processing runs in the capt_seq task. To safely interface lua with the other tasks requires additional effort.

Where do I find more info about this? I see lots of functions (most of which I can grasp), but as to how everything works is quite a mystery to me thus far. Different tasks are different threads?

Quote

lua code yields (for a minimum of 10ms) every 1000 lua bytecode instructions. This could be adjusted if needed, although you do have to be careful not to starve other tasks or hold up kbd task for too long.

How do you adjust the yields? C yields also? This have to do with tasks?

What I want to do is this: Wait for a button press, run a bit of code for some buttons, pass the button press through for others - depending on which mode is selected on the dial (auto, P, etc) and if I'm in a menu of some sort what code is ran and what's just a button press (passed though) will change. This little bit doesn't sound that hard to implement by a script in alt mode, but I'm guessing modifying kbd.c is where I should do it.

Logged

reyalp

14039

Re: LUA script Vs precompiled C in speed

« Reply #4 on: 07 / August / 2011, 17:21:38 »

Quote from: geoffire on 06 / August / 2011, 23:39:01

Quote from: reyalp on 06 / August / 2011, 17:55:19
The bigger difficulty is where things run: Lua code always runs in the kbd task at a specific point. The gui runs in spytask, and raw processing runs in the capt_seq task. To safely interface lua with the other tasks requires additional effort.
Where do I find more info about this? I see lots of functions (most of which I can grasp), but as to how everything works is quite a mystery to me thus far.

In the code of course, probably with some experimentation in addition to reading

IMO, this is the only way to really understand it.

Quote

Different tasks are different threads?

Tasks in the camera OSes is similar threads/processes in a desktop OS. Since the cameras have a flat, unprotected memory model (think DOS real mode, but with preemptive multitasking), the distinction is marginal.

A brief overview of how it works in CHDK
The camera has many native tasks. Most of them sit in loop either polling something and sleeping or waiting on some kind of synchronized object like a message queue.

CHDK modifies the OS startup, so it can replace some of these tasks with it's own modified tasks, and also create entirely new tasks. This happens starting in your cameras platform/<camera>/sub/<version>/boot.c (technically it starts in loader/<camera>/entry.S, but you probably don't have to worry about that), and for vxworks cameras, platform/generic/main.c

For what we are discussing, the the following are are the main areas of interest:
kbd_task: (aka PhySw)
This is a canon task that polls the MMIO associated with various buttons and switches etc, and updates a global variable (physw_status) from them, along with a few other things. It sits in a tight loop, polling the MMIOs at a 10ms interval. CHDK modifies this task, to intercept and possibly override the values. All CHDK script also runs in this task. See your platform kbd.c (handles hooking / overriding the hardware values, uses generic/kbd.c on some cameras) and core/kbd.c (handles alt mode input, runs scripts)

spy_task:
This is an entirely new task created by CHDK. It handles (almost) all the ui and most non-script processing. This runs in a loop, sleeping for 20ms and updating the gui every 3rd iteration (see core/main.c). It also handles raw saving.

capt_seq_task
This is a canon task (modified by CHDK) that sits in a loop waiting waiting on messages. The messages relate to various actions in the shooting process. CHDK hooks various points here to handle overrides, raw saving (in cooperation with spy_task), bracketing etc.

It should be noted that CHDK plays fast and loose with inter-task synchronization and communication. Rather than doing this the proper way with semaphores etc, it does various dangerous things with global variables.

Quote

How do you adjust the yields? C yields also? This have to do with tasks?

Lua yielding is done in core/luascript.c, via the lua_sethook function. You could quite easily make it so the yield interval was adjustable from lua. Essentially, set the hook to run at the maximum rate you'd ever want, and then inside the hook check a variable which can be set from lua.

The hook does a lua yield, meaning the script is left in a state where it can be resumed later from C code. This only works if the lua state isn't tangled up in the C stack, which is why it can only happen when not inside a C function that was itself called from lua (thus, the yield hook does not run inside pcall or other C functions called from lua. Various other functions that yield, such as wait_click suffer the same limitation) Yielding from lua returns control to the main kbd.task, which will eventually reach the end of it's iteration and yield with it's normal 10ms sleep.

Quote

What I want to do is this: Wait for a button press, run a bit of code for some buttons, pass the button press through for others - depending on which mode is selected on the dial (auto, P, etc) and if I'm in a menu of some sort what code is ran and what's just a button press (passed though) will change. This little bit doesn't sound that hard to implement by a script in alt mode, but I'm guessing modifying kbd.c is where I should do it.

The mode dial position is current something CHDK exposes. We can get the actual shooting mode, but if you override it, you can't recover the original dial position. On cameras with a physical dial, the values probably appears somewhere in physw_status, but because Canon has so many variations in their interface (some cameras have a slider, some modes are selected in menus etc), it's not practical for CHDK to expose it in a general way.

CHDK does not interact with the Canon menus at all, aside from trying to detect if the func or main menu is open.

Script currently takes over input almost completely, meaning that you can't really have a script that sits in the background intercepting specific key presses but leaving the rest of the Canon UI functioning normally. You could pass keys on using scripted key presses, but I suspect this would end up being pretty hokey without some additional work in the C code.

Logged

Don't forget what the H stands for.

geoffire

22

Re: LUA script Vs precompiled C in speed

« Reply #5 on: 10 / August / 2011, 08:19:31 »

Thank you reyalp for the thorough reply! it helped me piece together enough understanding that I have a good idea where to begin. The problem is looking at the code all you see is functions and its not obvious what's calling them! (I think I'll start a new thread for that) As for the original question, I think I can do everything in LUA, but I think I'll modify at least kbd.c so my script isn't needing to check for button presses constantly when kbd.c is already doing that.

Logged

reyalp

14039

Re: LUA script Vs precompiled C in speed

« Reply #6 on: 11 / August / 2011, 00:52:29 »

Quote from: geoffire on 10 / August / 2011, 08:19:31

Thank you reyalp for the thorough reply! it helped me piece together enough understanding that I have a good idea where to begin. The problem is looking at the code all you see is functions and its not obvious what's calling them!

I tend to keep a grepwin ( http://tools.tortoisesvn.net/grepWin.html ) window open just for that

Logged

Don't forget what the H stands for.