rawopint.lua: Fast, accurate intervalometer with raw exposure metering

c_joerg

1248

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #140 on: 01 / May / 2021, 03:08:34 »

Advertisements

Quote from: reyalp on 30 / April / 2021, 14:11:58

As always, adding more complexity will make it harder to test.

Yes, of course, but this case is the absolute exception. I haven't had this case in a long time.

Quote from: reyalp on 30 / April / 2021, 14:11:58

An idle thought: With something like the chdkptp remoteshoot glue, it would be possible to create a GUI that lets you visualize what the script is doing and change settings in real time.

Situations where clouds keep pushing in front of the sun are extremely difficult. I had already considered another option: to deactivate the exposure control for a certain time using an external USB switch or to only allow a control for overexposure.

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

reyalp

14080

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #141 on: 01 / May / 2021, 15:31:27 »

Quote from: c_joerg on 01 / May / 2021, 03:08:34

Situations where clouds keep pushing in front of the sun are extremely difficult.

Agreed. Reflection of that on water too. But I think there is room for improvement. I have a few other changes I should finish now that ND control is in CHDK stable, so maybe I'll play with it a bit. I think slowing down the change if a lot of pixels are near the over-exposure threshold could work quite well. Right now (at least if I'm interpreting my old code correctly

) it doesn't distinguish if the "overexposed" is just over the threshold or totally saturated. If it did, it could be less aggressive in the first case.

Quote

I had already considered another option: to deactivate the exposure control for a certain time using an external USB switch or to only allow a control for overexposure.

Yes, my thought of interactive control was more for development/testing than for actual shoots.

edit:
I think I may have come up with an approach to deal with the overexposure problem properly.

One of the difficulties with overexposure is you can't predict how much a given exposure change will affect it: That is, if you have saturated pixels, you can't predict what exposure change is needed to bring them below saturation. The Sun may barely change with 5 stops, where a cloud might be completely unsaturated with a quarter stop.

However, this isn't true in the other direction: For a given positive exposure change N, all pixels with values above (overexposure threshold - N) will pass the threshold, and this number can be easily obtained from the histogram. So the exposure algorithm can take the exposure change calculated from other inputs, calculate how many additional pixels would be pushed into over-exposure, and adjust accordingly to keep it below the limit.

In practice, some knowledge in the other direction should be possible too: The script uses a range (default 1/4 stop below white level) for overexposure, so (+/- uncertainty how the sensor behaves very near saturation) it can estimate a minimum number of pixels that would be brought below the threshold, or even calculate it exactly if the exposure change significantly smaller than the range.

This won't be trivial to integrate into the existing algorithm, but I think it's the right approach, and may allow eliminating some of the other hacks I introduced trying to stop it from bouncing around the limits.

« Last Edit: 02 / May / 2021, 02:56:54 by reyalp »

Logged

Don't forget what the H stands for.

c_joerg

1248

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #142 on: 02 / May / 2021, 06:35:47 »

Quote from: reyalp on 01 / May / 2021, 15:31:27

However, this isn't true in the other direction: For a given positive exposure change N, all pixels with values above (overexposure threshold - N) will pass the threshold, and this number can be easily obtained from the histogram. So the exposure algorithm can take the exposure change calculated from other inputs, calculate how many additional pixels would be pushed into over-exposure, and adjust accordingly to keep it below the limit.

That sounds like a very good idea.
That would definitely also help with a time lapse from the moon ...

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

dolomiti_timelapse

96

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #143 on: 25 / May / 2021, 12:57:51 »

Hallo c_joerg, hello reyalp,

I have tested now for some days a Canon G1X (100.e) with CHDK (g1x-100e-1.5.1-5908-full). CHDK works very well. Also CHDKPTP works flawlessly. The camera seems quite a good improvement over the Canon SX100IS.

I do have a problem which I'm not able to solve.

When I start the rawopint v0.25 (downloaded last version form this side: https://chdk.fandom.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer) on the camera or via CHDKPTP with a command like:

Code: [Select]

rs "D:/sunset/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=3 -int=12the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series. There is a little noise when the camera goes off which is the IS unit dropping. No lens retracting, no AV change, no "shutter" movement. The camera simply goes off.
I tried a lot of different settings but nothing changed this misbehaviour at all. If the shutter speed is quite high, e.g. 10 sec the camera does the same misbehaviour but starting at a interval of 10+12=22 seconds.
On the Canon SX100IS this phenomenon could not be observed.

If I use the normal

Code: [Select]

rs "D:/sunset/input/" -shots=3 -int=12 command everything works fine.

Maybe c_joerg has a script that works on his G1X he can share with me, so at least I would now it is not the script.

I would really appreciate any help or ideas to find the "bug"!

Logged

If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw

reyalp

14080

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #144 on: 25 / May / 2021, 13:21:30 »

Quote from: dolomiti_timelapse on 25 / May / 2021, 12:57:51

the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series. There is a little noise when the camera goes off which is the IS unit dropping. No lens retracting, no AV change, no "shutter" movement. The camera simply goes off.
I tried a lot of different settings but nothing changed this misbehaviour at all. If the shutter speed is quite high, e.g. 10 sec the camera does the same misbehaviour but starting at a interval of 10+12=22 seconds.

The camera shutting off usually means it crashed. Can you get a romlog: https://chdk.fandom.com/wiki/Debugging#Camera_crash_logs_.28romlog.29 ?

Note you can use chdkptp to get the romlog if you don't have physical access to the camera:

Code: [Select]

!require'extras/devutil'.init_cli() -- add debug commands, only needed once per session
dromlog

I suspect this crash probably has something to do with waiting in the remote hook. You can try putting some of the delay in the raw hook instead, by setting ui_raw_hook_sleep. The menu currently only allows up to 100ms, but if you set it from the glue script, you should be able to use values up to about 9000 (it needs to be less than the raw hook timeout, which is currently set to 10s).

If this works, I can adjust the script to do the majority of the wait in the raw hook and only use the remote hook to keep the interval precise.

Note that ui_raw_hook_sleep doesn't add to the interval. It effectively makes the script act like shooting took longer, so in order to maintain a the interval set with -int, it must be shorter than the interval and all the overhead of shooting, metering etc.

So if the theory is correct, you could use something like -int=20 and ui_raw_hook_sleep=9000

Logged

Don't forget what the H stands for.

c_joerg

1248

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #145 on: 25 / May / 2021, 13:37:39 »

Quote from: dolomiti_timelapse on 25 / May / 2021, 12:57:51

the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series.

I’m actual running 1.5.0 5399 on my G1x 100e. I never run rawopint together with CHDKPTP.
What does it mean ‘on camera’? Running the script without CHDKPTP and starting script from SD card?
If I run rawopint v0.25 with 12s interval (value 120) I have no problems. All other params are default.

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

dolomiti_timelapse

96

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #146 on: 25 / May / 2021, 13:54:52 »

Quote from: c_joerg on 25 / May / 2021, 13:37:39

I’m actual running 1.5.0 5399 on my G1x 100e. I never run rawopint together with CHDKPTP.
What does it mean ‘on camera’? Running the script without CHDKPTP and starting script from SD card?
If I run rawopint v0.25 with 12s interval (value 120) I have no problems. All other params are default.

Thank you for your replay!

‘on camera’ mean starting the script from SD card.
Interesting. Maybe I should try 1.5.0 5399...

Logged

If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw

dolomiti_timelapse

96

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #147 on: 25 / May / 2021, 13:59:38 »

Quote from: reyalp on 25 / May / 2021, 13:21:30

The camera shutting off usually means it crashed. Can you get a romlog: https://chdk.fandom.com/wiki/Debugging#Camera_crash_logs_.28romlog.29 ?

Thank you very much for your fast replay!

I did 2 identical runs with 12 sec (the script was started with CHDKPTP)
Attached both ROMLOG since the they are quite different.

ROMLOG_int_12_test_1.txt (3.33 kB - downloaded 13 times.)

ROMLOG_int_12_test_2.txt (3.37 kB - downloaded 13 times.)

Logged

If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw

dolomiti_timelapse

96

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #148 on: 25 / May / 2021, 14:05:31 »

Quote from: reyalp on 25 / May / 2021, 13:21:30

I suspect this crash probably has something to do with waiting in the remote hook. You can try putting some of the delay in the raw hook instead, by setting ui_raw_hook_sleep. The menu currently only allows up to 100ms, but if you set it from the glue script, you should be able to use values up to about 9000 (it needs to be less than the raw hook timeout, which is currently set to 10s).

If this works, I can adjust the script to do the majority of the wait in the raw hook and only use the remote hook to keep the interval precise.

Note that ui_raw_hook_sleep doesn't add to the interval. It effectively makes the script act like shooting took longer, so in order to maintain a the interval set with -int, it must be shorter than the interval and all the overhead of shooting, metering etc.

So if the theory is correct, you could use something like -int=20 and ui_raw_hook_sleep=9000

Yes, thank you! It kind of worked.

I tried out with ui_raw_hook_sleep=8000 in the glue script.

-int=12 worked fine but -int=15 again crashed the camera. I repeated the run for both intervals and the behaviour remained the same.
I attached the ROMLOG of one of the 2 runs with -int=15

ROMLOG_int_15_ui_raw_hook_sleep_8000.txt (3.38 kB - downloaded 16 times.)

« Last Edit: 25 / May / 2021, 14:12:41 by dolomiti_timelapse »

Logged

If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw

c_joerg

1248

Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering

« Reply #149 on: 25 / May / 2021, 14:09:00 »

Quote from: dolomiti_timelapse on 25 / May / 2021, 13:54:52

Interesting. Maybe I should try 1.5.0 5399...

I update now to latest 1.6.0. Also, no problem with 12s.
Which Mode do you use? Flash?

« Last Edit: 25 / May / 2021, 14:10:38 by c_joerg »

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd