rawopint.lua: Fast, accurate intervalometer with raw exposure metering

reyalp

This is a script I developed while working on the raw shooting hooks.

It is still not really finished, but is usable in many situations.

The basic idea is to shoot rapidly without invoking Canon firmware auto-focus or auto-exposure between shots. The exposure for each subsequent shot is calculated from the previous image data. Shooting is done either using Canon continuous mode with delays inserted to maintain the desired interval, or by simulating holding the shutter button at half press and repeatedly clicking full press. In continuous mode, the script can achieve a frame rate close to the Canon continuous shooting specification.

This script intended for situations where a relatively high frame rate, smooth exposure change and fixed focus are desired. Because exposure is calculated from the raw image data, it performs well in low light but may not perform well in situations where the light changes rapidly. It is well suited for daytime landscapes, clouds, sunsets, sunrises etc.

Wiki page with details:
http://chdk.wikia.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer

Download the current version from the github releases page:
https://github.com/reyalpchdk/chdkscripts/releases
(also attached to this post)

See the github repo for in-development changes and additional info
https://github.com/reyalpchdk/chdkscripts

Please post bug reports, suggestions and questions in this thread.

If you do something cool with it, posting in creative uses of CHDK section is encouraged.

Changes for v0.26 - (Mar 8, 2025)
* CHDK 1.5.1 or later required
* ND value and initial ND state now automatically detected for most ports
* ND should work on most cameras in "quick" mode
* Canon RAW/JPEG setting can be set on cameras with native raw support
* JPEG size setting will avoid unsupported values on some propset > 6 cams
* Fixed some cases where some cameras would take an extra shot at script exit
* Removed non-working simulation option
* Fixed bug which caused Ev shift to result in incorrect Bv / Ev shift values, even when Bv / Ev shift was not enabled
* Add meter96_tgt column, records target meter96 with Ev shifts applied
* Add options to control smoothing, based on suggestion by dolomiti_timelapse https://chdk.setepontos.com/d/12697/170
** Also fixes issue in "Smooth overshoot" logic which caused flickering in some situations
* Convert ev_change to imath so final change values are rounded
* Make log columns consistently refer to the current shot, except for d_ev* which refer to the next shot
* Log limit related values in columns rather than desc. These values are only logged when the corresponding condition is present
** nd_tv_tr = When ND active, amount over ND trigger threshold, negative if hysteresis active, formerly "tv over nd" if positive, "tv nd hist" if negative
** tv_l1 = amount calculated tv is over tv96_short_limit or under tv96_long_limit, formerly "tv over long" if positive, "tv under short" if negative
** tv_sv_tr = Amount of calculated tv moved to ISO due to being over tv96_sv_thresh, formerly "tv iso adj"
** sv_l1 = Amount calculated sv is over sv96_max, formerly "iso over limit"
** sv_tv_tr = Amount of sv over limit put back on tv, formerly "iso over tv"
** bv_ev_l1 - Original calculated bv/ev shift value if over threshold or limit, formerly "bv ev limit", "bv ev thesh"
* Meter position is now adjustable
* Improved support for chdkptp tethered shooting
* "date", "time" columns now refer to the start of the shooting loop, rather than the end. "tick" column removed.
* Focus mode is restored to original value if changed by SD override option.

Changes for v0.25 (May 6, 201😎
* Add workaround to prevent aperture changes on cameras incorrectly defined to have ND even when ND threshold is disabled. Thanks c_joerg for reporting
* Added options to start at a particular clock time
* Added options for zoom and focus distance

Changes for v0.24 (Mar 4, 201😎
* Fixed bug that prevented intervals > ~10 seconds from working. Thanks SkepticaLee for the report.
* Removed "Force initial ND" option. Cameras formerly thought to have a "hidden" ND without detectable state are now known to operate on the aperture. To use ND control on these cameras (not recommended), set the aperture to full open in the Canon firmware. If ND state isn't detected correctly on a camera with manual ND, set it open in the Canon firmware.

Changes for v0.23 (Aug 27, 2017)
* Add support for ND filter control on cameras with both ND and iris.
* Add options "ND value", "Force initial ND" and "ND hysteresis". See option descriptions on the wiki for details and caveats.

Changes for v0.22 (Aug 6, 2017)
* If "ISO Adj Tv" is set longer than "Max Tv", it is now set to "Max Tv", meaning the ISO adjustment will start when shutter adjestment ends. This fixes a bug where the "Max Tv" limit would be ignored. To disable ISO adjustment, set "Target ISO" and "Max ISO" to the same value. Thanks SkepticaLee for the report.
* Add battery temperature to log.

Changes in v0.21 (Apr 2, 2016)
* Fixed a bug that caused exposure to stop increasing when Max ISO was hit, regardless of Max Tv. Thanks udo for the report.

Changes in v0.20 (Jan 8, 2016)
*Add USB remote trigger option, based on suggestion in https://chdk.setepontos.com/d/12722

Changes in v0.19
* Added interval warning LED and beep options, suggested by c_joerg in https://chdk.setepontos.com/d/11081/289
* Changed some defaults. Meter now defaults to 90%, meter step 15.
* Log mode now has options for "none", "replace" and "append", defaults to append.
* Fix delay on script exit reported by c_joerg in https://chdk.setepontos.com/d/11081/285

rawopint-0.26.zip

36.14KB

c_joerg

Itâ€™s a good idea to open a new thread with a more understandable title. 🧑‍🎄 🧑‍🎄
The description, including the summary is really great. 🧑‍🎄 🧑‍🎄

What would be the easiest way, to port the existing param files to the newest script version? What would be happened with the missing parameters, when I would be used an older param file?

reyalp

c_joerg
What would be the easiest way, to port the existing param files to the newest script version? What would be happened with the missing parameters, when I would be used an older param file?

I haven't looked at the code to see what the exact logic is, but normally, everything is reset when the parameter definitions in the script change.

I guess you could do something like this:
* save your existing file from the previous version to your PC
* load the new the new version of the script, and save a parameter set
* copy that the new file to your PC
* copy over any identically named parameters, except for ones where the meaning or range of valid values has changed.

The script saved parameter function was never really mean to support this kind of thing. It might be easier and safer to just go through the list on screen and write the settings down.

c_joerg

I updated rawopint V0.19 on my G1X, SX230 and SX50. I checked the warning LED and Beep. It works very well. Good that you gave the hint with the mute setting on the canon menu, because I set all my cams to mute.

There is one think, which I notice on the SX230. I still have a delay on the end in continuous mode. But only with very small interval values. I notice this, when I tested the warnings with an interval from 0.2s (Interval of 0s gives no warning). The delay is less (around 8s) than as it was without your modifications. This is not happens on G1X and SX50. It is OK from my side, because this isnâ€™t a typical setting. I want only report it.

I converted the last parm file which you post for moon setting to version 0.19. I still think, this is the best setting which we have right now for this. I did it like you descried with a smart merge tool â€˜Beyond Compareâ€™. Attached is a snapshot with differences. Feel free to remove the attachment when you canâ€™t agree this setting.

RAWOPINT_V019_Moon.0

1KB

c_joerg

I did a couple of investigation, to see, which influence has the memory card on the timing for rawopint.
I did it with 5 different SD Cardâ€™s in 3 steps:

1) On a PC with different tools and different block sizes
2) With CHDK benchmark test
3) Running rawopint in continuous mode on G1x with an interval of 2s (Canon RAW + JPG)

The table shows the results from the PC and CHDK benchmark.

The plot shows the sleep from the log file. As you can see, there is not much influence from the memory card. The only thing which I notice is the peek from the 64GB SanDisk Ultra Card. This was the card, where I had the massive problems with timing on my last trip. So I will observe this Card.

c_joerg

I have now also a IXUS160. I canâ€™t get enough and for that priceâ€¦

Iâ€™m not sure, if this is a point for rawopint or the IXUS160

I started rawopint on a fresh installation and I got the Error Message, that rawopsim.csv was missing. I have not changed any parameter and I have not switched on â€˜Run simulationâ€™. But I have not checked itâ€¦
After I loaded default parameters everything runs well. But I have not changed any parameterâ€¦

I remembered, some time ago, we had problem with some parameter mismatchesâ€¦

Update:
It is not reproducible. I formatted the SD and installed every think again. No problemsâ€¦

rawopint.csv

8.77KB

reyalp

In 0.20, I added support for USB remote triggering. Yes, there's more important stuff still unfinished, but it seemed easy and fun 🤣

When this option is enabled, the script waits for a usb remote pulse instead of shooting on a fixed interval. The "interval" option specifies how long it will wait for the pulse. If no pulse is received within the timeout, it will shoot anyway.

If the pulse arrives before the camera is finished with the previous shot, it should shoot as soon as it can.

I haven't tested this much, only switching a powered usb hub on and off.

There is currently no way for the thing controlling the remote to end the script. This could be done with pulse width or on timeout.

For synchonizing with a slider, I think having the camera signal the slider is probably a better approach, since it would allow the maximum possible shooting rate.

@c_joerg
I don't have any idea what would cause the rawopsim error, this could really only happen if you checked the simulation option.

c_joerg

Iâ€™m still festinated to make time-lapse from the moon, especially since I have the SX50. Unfortunately the weather was not very good this winterâ€¦

I made 2 runs. I donâ€™t know why the first run is overexposed. I used nearly the same parameters as from the second run. But the focal length was different from the first (efl=161mm) and the second run (efl=720mm). This might be a big difference.

http://youtu.be/nTeLvI6ce50

In the log file I notice â€˜WARN:over_thresh histo_samplesâ€™ this might be a problem as well. May be the histo_step was the big.

As well I used a modified version of the script which. I might be that I forgot something.
What's Changed?

I add an additional parameter for max max_ev_change (one for up and one for down) and additional a 0 for both parameters.

#ui_max_ev_change_e=4 "Max Ev change" {0 1/16 1/8 1/4 1/3 1/2 1}
#ui_max_2_ev_change_e=4 "Max Ev 2 change" {0 1/16 1/8 1/4 1/3 1/2 1}

and added the following code:

-- clamp to ui max
 if ev_change > 0 then
    ev_change = self:clamp(ev_change,self.ev_change_2_max)
 else
    ev_change = self:clamp(ev_change,self.ev_change_max)
 end

Why I did this?

I donâ€™t want that the script goes with the exposure up and down when the Moon goes behind trees and clouds. So I looked one direction with the 0. If ui_max_2_ev_change_e = 0, the script only controls overexposure. Basically this works very well, as long the pre shoot is not to dark. On the second run you see, exposure time goes only shorter. But I donâ€™t know why this not happens on the first run.

Another reason to do this is why I want to make time-lapse from growing stuff on tungsten light. To avoid flicker I also want only use the overexposure control of the script. Because in this case I know, it canâ€™t go darker.
I know, for this case I could do it in a much easier script, but a really like all the logging stuff in the scriptâ€¦

rawopint.csv

266.01KB

Steenoe

😎Just a quick thanks, for a terrific script 😀
Here is a testrun, showing how smooth the exposure transition can be with this script. From sunset to starry night is handled very well:
https://www.youtube.com/watch?v=Gl4qGcUynBw
Rendered from small jpeg's, straight out of cam, its impressively smooth (exposure wise).
The camera sadly cant keep up with the interval at the end, caused by the G1X ISO issue 🙁
It would be great if that problem could be solved (together with handling the ND filter). The G1X shoots pretty nice files up to 1600-3200 ISO, so that would open up a ton of possibilities.

reyalp

Steenoe
From sunset to starry night is handled very well:

Nice, glad it worked for you.

The camera sadly cant keep up with the interval at the end, caused by the G1X ISO issue 🙁
It would be great if that problem could be solved (together with handling the ND filter). The G1X shoots pretty nice files up to 1600-3200 ISO, so that would open up a ton of possibilities.

Unfortunately, I don't hold much hope for resolving the G1X ISO issue any time soon 🙁

I do want to get back to work on this script in the relatively near future, including finishing ND and aperture support. I'm keen to see what the g7x can do when I get the port far enough along to run it 😀

Steenoe

I'm keen to see what the g7x can do when I get the port far enough along to run it 😀

Me too 😎I have a G7X too, and its a great little camera. Getting rawopint up and running on that, would be just... 🧑‍🎄
Hope it doesnt have any weird quirks, like the G1 X which certainly does. Rawopint did solve the powersaving issue of the G1x though.
The LCD can be turned off, without having the G1x switch to autofocus. Nice! It can actually shoot approx 3 hours (in cold conditions) on one full charge now, so thats cool.....
BTW. let me know if there is anything I can do to help porting the G7X. I'm not a coder or anything, but I do have the camera.....

c_joerg

Unfortunately, I don't hold much hope for resolving the G1X ISO issue any time soon

Yes unfortunately 🙁 and you did already a lot of work on thisâ€¦

But I still donâ€™t understand, what lapser is doing, when he runs with his G1x from sunset to great Milky Ways.

I'm keen to see what the g7x can do when I get the port far enough along to run it

Yes I have seen you are really busy, every day a least one releaseâ€¦

I do want to get back to work on this script in the relatively near future, including finishing ND and aperture support.

ND and aperture would be a great improvement, because I never run timelapse against the sun without ND filter. At least the 3 steps from the ND will really helpful when you go from sunset to night.

Just now, I have to process another 10000 pictures made with rawopintâ€¦

reyalp

Here is a first, rather boring run on the g7x

From the log it looks like there might be some problem with ISO control, but it basically works.

Edit:
Slightly more dynamic run, still doesn't exercise exposure control much

udo

Hello,

I posted some sky timelapses in the examples rawopint thread.
If you see any issues I may be able to post the csv's.
The camera went to ISO400 and 20 seconds of exposure.

In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)

Udo

waterwingz

udo
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made. Only during the day the round shape is restored. (over exposure effects?)

This is probably due to the moon moving during a long exposure.

udo

waterwingz
This is probably due to the moon moving during a long exposure.

It gets more or a star-shaped form with rays extending from the points of the star.
See

c_joerg

udo
Hello,

I posted some sky timelapses in the examples rawopint thread.
If you see any issues I may be able to post the csv's.
The camera went to ISO400 and 20 seconds of exposure.

In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)

Udo

Have you seen the settings for the moon above in this thread?
The moon is definitely overexposed with 20sâ€¦
For the moon I got typically IS0100 and 1/50s.
You canâ€™t have a neutral exposed moon and stars togetherâ€¦.

Where is the problem to attach csv log files?

udo

c_joerg

udo
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)

Have you seen the settings for the moon above in this thread?
The moon is definitely overexposed with 20sâ€¦
For the moon I got typically IS0100 and 1/50s.
You canâ€™t have a neutral exposed moon and stars togetherâ€¦.

I do understand the choice between stars or moon exposed well.
But the starry appearance of teh moon was new to me.

c_joerg

udo
I do understand the choice between stars or moon exposed well.
But the starry appearance of teh moon was new to me.

Which aperture do you use?
When you close the aperture you have this effect with the sun and moon when itâ€™s overexposed. Also seen on my videos with the sun.

udo

c_joerg
Which aperture do you use?

This very timelapse appears to have been done with av=4.
Others with av=2.
I did not set aperture, so probably I should set it to the lowest value for sky timelapse?