help: how to debug script interrupting?

philmoz

3445

Re: help: how to debug script interrupting?

« Reply #210 on: 07 / May / 2013, 09:22:55 »

Advertisements

Quote from: srsa_4c on 07 / May / 2013, 09:08:24

Quote from: philmoz on 07 / May / 2013, 04:27:17
Edit: I looked at the optimisations enabled by O2 and tried the ones that did instruction re-ordering - got lucky and found that the dodgy optimisation appears to be ' -fschedule-insns2'.
Good find, congrats to you & reyalp. Perhaps we should put a note about the issue when appending -fno-schedule-insns2 to CFLAGS.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=38644

Quote from: reyalp on 06 / May / 2013, 22:27:25
If the script is hitting a lua error after running many exposures, it would be good to know why. Do you see any errors in the console?
I forgot to look after this. Yes, "not enough memory". The script eats up memory somehow (note that this cam is using exmem).

I've seen Lua use up to 150K or RAM while a script is running - this is on top of the 100K needed to load the Lua module (if you use 1.2) and the memory needed to load the script code.

The garbage collector will eventually reclaim it; but if you are severely memory limited it can run out.

You can try adding 'collectgarbage()' in your script loop to force Lua to reclaim memory and see if it helps.
Reyalp can probably provide a more elegant method of doing this than forcing garbage collection all the time; but I'm not familiar enough with Lua yet.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4451

Re: help: how to debug script interrupting?

« Reply #211 on: 07 / May / 2013, 13:06:56 »

Quote from: philmoz on 07 / May / 2013, 09:22:55

I've seen Lua use up to 150K or RAM while a script is running - this is on top of the 100K needed to load the Lua module (if you use 1.2) and the memory needed to load the script code.

The garbage collector will eventually reclaim it; but if you are severely memory limited it can run out.

~~The camera~~ CHDK has over 1MB free RAM at start. The script is running in no-shoot mode, 1...2s/cycle, AF lock. Free memory (as displayed by the script) shows decreasing tendency (GC periodically reclaims most memory, but not all), and eventually Lua encounters an out of memory situation. I have already seen something similar, when I was testing remote shooting on the ixus870_sd880. Putting more pause between shots helped there, haven't tried that on this cam.

Logged

reyalp

13944

Re: help: how to debug script interrupting?

« Reply #212 on: 07 / May / 2013, 13:10:08 »

Quote from: philmoz on 07 / May / 2013, 04:27:17

Is that the maximum amount of task stack space it is using?

As far as I can tell, it's consistent. But per later posts -fno-schedule-insns2 sound like the right thing, the gcc bug you linked appears to be a dead ringer for our problem. It's also explicitly fixed in 4.6.3+, so we could only disable the optimization for earlier GCCs, if desired. We already have GCC version specific -march vs -mtune

-fno-schedule-insns2 does indeed prevent my build environment from generating the bad code.

Quote

Quote
on vxworks (a540) it hangs the camera, presumably because it doesn't want a task sitting in a tight loop like that.
Also shows vxworks is not doing pre-emptive task switches.

I think so, but it would be good to work out the details a bit better. vxworks certainly can be configured to be preemptive. IIRC a reading something by one of the early CHDK devs (grand ? ewar?) saying the Canon configuration was essentially cooperative multitasking, but I haven't experimented with it much myself.

FWIW, on DryOS when I set the priority of my test task lower (higher numerically) than spytask it never got scheduled at all.

Logged

Don't forget what the H stands for.

reyalp

13944

Re: help: how to debug script interrupting?

« Reply #213 on: 07 / May / 2013, 13:17:03 »

Quote from: srsa_4c on 07 / May / 2013, 13:06:56

Quote from: philmoz on 07 / May / 2013, 09:22:55
I've seen Lua use up to 150K or RAM while a script is running - this is on top of the 100K needed to load the Lua module (if you use 1.2) and the memory needed to load the script code.

The garbage collector will eventually reclaim it; but if you are severely memory limited it can run out.
~~The camera~~ CHDK has over 1MB free RAM at start. The script is running in no-shoot mode, 1...2s/cycle, AF lock. Free memory (as displayed by the script) shows decreasing tendency (GC periodically reclaims most memory, but not all), and eventually Lua encounters an out of memory situation. I have already seen something similar, when I was testing remote shooting on the ixus870_sd880. Putting more pause between shots helped there, haven't tried that on this cam.

This sounds like there is a memory leak somewhere. In my tests there was no significant increase in memory over 10k shots.

The "mem" lines produced by my modified version of the script are.
lua allocation in KB (from collectgarbage('count')) followed by free mem from get_meminfo (in bytes)
These are also logged as Alloc and free

From one of my 10k shooting runs, start was
Alloc: 57 free: 373960
and the end was
Alloc: 76 free: 346784
In between, lua allocation bounced around between ~125 and ~60

The script is pretty garbage happy (it creates a lot of unique strings) but the collector appears to keep up. This shouldn't vary between cameras much, although differences in shooting times could lead to more or less stuff being logged.

« Last Edit: 07 / May / 2013, 13:19:15 by reyalp »

Logged

Don't forget what the H stands for.

outslider

762

Re: help: how to debug script interrupting?

« Reply #214 on: 07 / May / 2013, 13:52:25 »

I'm not sure whether this is the value you are talking about, but on my cam the command:

Code: (lua) [Select]

collectgarbage("count")

returned values between 50-100 KB on sx130 when I was performing up to 1k shots. No matter whether I was using collectgarbage() itself or not.

Logged

if (2*b || !2*b) {
cout<<question
}

Compile error: poor Yorick

waterwingz

12134

Re: help: how to debug script interrupting?

« Reply #215 on: 07 / May / 2013, 14:01:01 »

Quote from: reyalp on 07 / May / 2013, 13:10:08

-fno-schedule-insns2 does indeed prevent my build environment from generating the bad code.

So the obvious question is what does it do in hacki's autobuild environment?

Logged

Ported : A1200 SD940 G10 Powershot N G16

reyalp

13944

Re: help: how to debug script interrupting?

« Reply #216 on: 07 / May / 2013, 15:46:59 »

Quote from: waterwingz on 07 / May / 2013, 14:01:01

Quote from: reyalp on 07 / May / 2013, 13:10:08
-fno-schedule-insns2 does indeed prevent my build environment from generating the bad code.
So the obvious question is what does it do in hacki's autobuild environment?

Given the gcc bug ~~phil~~ srsa found, it's a safe bet it will fix the problem. However, disassembling a .flt isn't terribly hard so I'll make sure after the change is in.

@outslider
Yes, collectgarbage('count') shows how much lua thinks it has allocated, in KB.

collectgarbage() or collectgarbage('collect') does a full gc run. You can also do collectgarbage('step'), if the time required for a full run is undesirable.

« Last Edit: 07 / May / 2013, 15:50:28 by reyalp »

Logged

Don't forget what the H stands for.

philmoz

3445

Re: help: how to debug script interrupting?

« Reply #217 on: 07 / May / 2013, 15:59:14 »

Quote from: waterwingz on 07 / May / 2013, 14:01:01

Quote from: reyalp on 07 / May / 2013, 13:10:08
-fno-schedule-insns2 does indeed prevent my build environment from generating the bad code.
So the obvious question is what does it do in hacki's autobuild environment?

The change works for GCC 4.4.3 (anywhere I mention 4.3.6 earlier I meant 4.4.3 - momentary memory blip).

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

philmoz

3445

Re: help: how to debug script interrupting?

« Reply #218 on: 07 / May / 2013, 16:05:11 »

Quote from: reyalp on 07 / May / 2013, 13:10:08

As far as I can tell, it's consistent. But per later posts -fno-schedule-insns2 sound like the right thing, the gcc bug you linked appears to be a dead ringer for our problem. It's also explicitly fixed in 4.6.3+, so we could only disable the optimization for earlier GCCs, if desired. We already have GCC version specific -march vs -mtune

I'd just add it for all versions rather than having version tests.

The description of the optimisation says it is for reducing pipeline stalls by re-ordering instructions - how much would we really gain from having it enabled?

~~(I'm going to commit a patch that just disables the optimisation - if you want to make it GCC version specific feel free.)~~ Reyalp beat me to it

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

13944

Re: help: how to debug script interrupting?

« Reply #219 on: 07 / May / 2013, 16:05:31 »

Checked in trunk 2762, release 2763

Now we know the answer to the Q posed in the title "how to debug script interrupting?"
A: With difficulty.

Logged

Don't forget what the H stands for.