CHDK Forum
Using CHDK => Script Writing => Completed and Working Scripts => Topic started by: reyalp on 14 / December / 2015, 00:41:56
-
This is a script I developed while working on the raw shooting hooks (http://chdk.setepontos.com/index.php?topic=11081.msg119265#msg119265).
It is still not really finished, but is usable in many situations.
The basic idea is to shoot rapidly without invoking Canon firmware auto-focus or auto-exposure between shots. The exposure for each subsequent shot is calculated from the previous image data. Shooting is done either using Canon continuous mode with delays inserted to maintain the desired interval, or by simulating holding the shutter button at half press and repeatedly clicking full press. In continuous mode, the script can achieve a frame rate close to the Canon continuous shooting specification.
This script intended for situations where a relatively high frame rate, smooth exposure change and fixed focus are desired. Because exposure is calculated from the raw image data, it performs well in low light but may not perform well in situations where the light changes rapidly. It is well suited for daytime landscapes, clouds, sunsets, sunrises etc.
Wiki page with details and download link:
http://chdk.wikia.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer (http://chdk.wikia.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer) (current version is also attached to this post)
Please post bug reports, suggestions and questions in this thread.
If you do something cool with it, posting in creative uses of CHDK (http://chdk.setepontos.com/index.php?board=20.0) section is encouraged.
Changes for v0.25 (May 6, 2018)
* Add workaround to prevent aperture changes on cameras incorrectly defined to have ND even when ND threshold is disabled. Thanks c_joerg for reporting
* Added options to start at a particular clock time
* Added options for zoom and focus distance
Changes for v0.24 (Mar 4, 2018)
* Fixed bug that prevented intervals > ~10 seconds from working. Thanks SkepticaLee for the report.
* Removed "Force initial ND" option. Cameras formerly thought to have a "hidden" ND without detectable state are now known to operate on the aperture. To use ND control on these cameras (not recommended), set the aperture to full open in the Canon firmware. If ND state isn't detected correctly on a camera with manual ND, set it open in the Canon firmware.
Changes for v0.23 (Aug 27, 2017)
* Add support for ND filter control on cameras with both ND and iris.
* Add options "ND value", "Force initial ND" and "ND hysteresis". See option descriptions on the wiki for details and caveats.
Changes for v0.22 (Aug 6, 2017)
* If "ISO Adj Tv" is set longer than "Max Tv", it is now set to "Max Tv", meaning the ISO adjustment will start when shutter adjestment ends. This fixes a bug where the "Max Tv" limit would be ignored. To disable ISO adjustment, set "Target ISO" and "Max ISO" to the same value. Thanks SkepticaLee for the report.
* Add battery temperature to log.
Changes in v0.21 (Apr 2, 2016)
* Fixed a bug that caused exposure to stop increasing when Max ISO was hit, regardless of Max Tv. Thanks udo for the report (https://chdk.setepontos.com/index.php?topic=12528.msg127553#msg127553).
Changes in v0.20 (Jan 8, 2016)
*Add USB remote trigger option, based on suggestion in http://chdk.setepontos.com/index.php?topic=12722.0 (http://chdk.setepontos.com/index.php?topic=12722.0)
Changes in v0.19
* Added interval warning LED and beep options, suggested by c_joerg in http://chdk.setepontos.com/index.php?topic=11081.msg125794#msg125794 (http://chdk.setepontos.com/index.php?topic=11081.msg125794#msg125794)
* Changed some defaults. Meter now defaults to 90%, meter step 15.
* Log mode now has options for "none", "replace" and "append", defaults to append.
* Fix delay on script exit reported by c_joerg in http://chdk.setepontos.com/index.php?topic=11081.msg125651#msg125651 (http://chdk.setepontos.com/index.php?topic=11081.msg125651#msg125651)
-
It’s a good idea to open a new thread with a more understandable title. :xmas :xmas
The description, including the summary is really great. :xmas :xmas
What would be the easiest way, to port the existing param files to the newest script version? What would be happened with the missing parameters, when I would be used an older param file?
-
What would be the easiest way, to port the existing param files to the newest script version? What would be happened with the missing parameters, when I would be used an older param file?
I haven't looked at the code to see what the exact logic is, but normally, everything is reset when the parameter definitions in the script change.
I guess you could do something like this:
* save your existing file from the previous version to your PC
* load the new the new version of the script, and save a parameter set
* copy that the new file to your PC
* copy over any identically named parameters, except for ones where the meaning or range of valid values has changed.
The script saved parameter function was never really mean to support this kind of thing. It might be easier and safer to just go through the list on screen and write the settings down.
-
I updated rawopint V0.19 on my G1X, SX230 and SX50. I checked the warning LED and Beep. It works very well. Good that you gave the hint with the mute setting on the canon menu, because I set all my cams to mute.
There is one think, which I notice on the SX230. I still have a delay on the end in continuous mode. But only with very small interval values. I notice this, when I tested the warnings with an interval from 0.2s (Interval of 0s gives no warning). The delay is less (around 8s) than as it was without your modifications. This is not happens on G1X and SX50. It is OK from my side, because this isn’t a typical setting. I want only report it.
I converted the last parm file which you post for moon setting to version 0.19. I still think, this is the best setting which we have right now for this. I did it like you descried with a smart merge tool ‘Beyond Compare’. Attached is a snapshot with differences. Feel free to remove the attachment when you can’t agree this setting.
-
I did a couple of investigation, to see, which influence has the memory card on the timing for rawopint.
I did it with 5 different SD Card’s in 3 steps:
1) On a PC with different tools and different block sizes
2) With CHDK benchmark test
3) Running rawopint in continuous mode on G1x with an interval of 2s (Canon RAW + JPG)
The table shows the results from the PC and CHDK benchmark.
The plot shows the sleep from the log file. As you can see, there is not much influence from the memory card. The only thing which I notice is the peek from the 64GB SanDisk Ultra Card. This was the card, where I had the massive problems with timing on my last trip. So I will observe this Card.
-
I have now also a IXUS160. I can’t get enough and for that price…
I’m not sure, if this is a point for rawopint or the IXUS160
I started rawopint on a fresh installation and I got the Error Message, that rawopsim.csv was missing. I have not changed any parameter and I have not switched on ‘Run simulation’. But I have not checked it…
After I loaded default parameters everything runs well. But I have not changed any parameter…
I remembered, some time ago, we had problem with some parameter mismatches…
Update:
It is not reproducible. I formatted the SD and installed every think again. No problems…
-
In 0.20, I added support for USB remote triggering. Yes, there's more important stuff still unfinished, but it seemed easy and fun :haha
When this option is enabled, the script waits for a usb remote pulse instead of shooting on a fixed interval. The "interval" option specifies how long it will wait for the pulse. If no pulse is received within the timeout, it will shoot anyway.
If the pulse arrives before the camera is finished with the previous shot, it should shoot as soon as it can.
I haven't tested this much, only switching a powered usb hub on and off.
There is currently no way for the thing controlling the remote to end the script. This could be done with pulse width or on timeout.
For synchonizing with a slider, I think having the camera signal the slider is probably a better approach, since it would allow the maximum possible shooting rate.
@c_joerg
I don't have any idea what would cause the rawopsim error, this could really only happen if you checked the simulation option.
-
I’m still festinated to make time-lapse from the moon, especially since I have the SX50. Unfortunately the weather was not very good this winter…
I made 2 runs. I don’t know why the first run is overexposed. I used nearly the same parameters as from the second run. But the focal length was different from the first (efl=161mm) and the second run (efl=720mm). This might be a big difference.
http://youtu.be/nTeLvI6ce50 (http://youtu.be/nTeLvI6ce50)
In the log file I notice ‘WARN:over_thresh histo_samples’ this might be a problem as well. May be the histo_step was the big.
As well I used a modified version of the script which. I might be that I forgot something.
What's Changed?
I add an additional parameter for max max_ev_change (one for up and one for down) and additional a 0 for both parameters.
#ui_max_ev_change_e=4 "Max Ev change" {0 1/16 1/8 1/4 1/3 1/2 1}
#ui_max_2_ev_change_e=4 "Max Ev 2 change" {0 1/16 1/8 1/4 1/3 1/2 1}
and added the following code:
-- clamp to ui max
if ev_change > 0 then
ev_change = self:clamp(ev_change,self.ev_change_2_max)
else
ev_change = self:clamp(ev_change,self.ev_change_max)
end
Why I did this?
I don’t want that the script goes with the exposure up and down when the Moon goes behind trees and clouds. So I looked one direction with the 0. If ui_max_2_ev_change_e = 0, the script only controls overexposure. Basically this works very well, as long the pre shoot is not to dark. On the second run you see, exposure time goes only shorter. But I don’t know why this not happens on the first run.
Another reason to do this is why I want to make time-lapse from growing stuff on tungsten light. To avoid flicker I also want only use the overexposure control of the script. Because in this case I know, it can’t go darker.
I know, for this case I could do it in a much easier script, but a really like all the logging stuff in the script…
-
8) Just a quick thanks, for a terrific script :)
Here is a testrun, showing how smooth the exposure transition can be with this script. From sunset to starry night is handled very well:
https://www.youtube.com/watch?v=Gl4qGcUynBw (https://www.youtube.com/watch?v=Gl4qGcUynBw)
Rendered from small jpeg's, straight out of cam, its impressively smooth (exposure wise).
The camera sadly cant keep up with the interval at the end, caused by the G1X ISO issue :(
It would be great if that problem could be solved (together with handling the ND filter). The G1X shoots pretty nice files up to 1600-3200 ISO, so that would open up a ton of possibilities.
-
From sunset to starry night is handled very well:
Nice, glad it worked for you.
The camera sadly cant keep up with the interval at the end, caused by the G1X ISO issue :(
It would be great if that problem could be solved (together with handling the ND filter). The G1X shoots pretty nice files up to 1600-3200 ISO, so that would open up a ton of possibilities.
Unfortunately, I don't hold much hope for resolving the G1X ISO issue any time soon :(
I do want to get back to work on this script in the relatively near future, including finishing ND and aperture support. I'm keen to see what the g7x can do when I get the port far enough along to run it :)
-
I'm keen to see what the g7x can do when I get the port far enough along to run it :)
Me too 8) I have a G7X too, and its a great little camera. Getting rawopint up and running on that, would be just... :xmas
Hope it doesnt have any weird quirks, like the G1 X which certainly does. Rawopint did solve the powersaving issue of the G1x though.
The LCD can be turned off, without having the G1x switch to autofocus. Nice! It can actually shoot approx 3 hours (in cold conditions) on one full charge now, so thats cool.....
BTW. let me know if there is anything I can do to help porting the G7X. I'm not a coder or anything, but I do have the camera.....
-
Unfortunately, I don't hold much hope for resolving the G1X ISO issue any time soon
Yes unfortunately :( and you did already a lot of work on this…
But I still don’t understand, what lapser is doing, when he runs with his G1x from sunset to great Milky Ways.
I'm keen to see what the g7x can do when I get the port far enough along to run it
Yes I have seen you are really busy, every day a least one release…
I do want to get back to work on this script in the relatively near future, including finishing ND and aperture support.
ND and aperture would be a great improvement, because I never run timelapse against the sun without ND filter. At least the 3 steps from the ND will really helpful when you go from sunset to night.
Just now, I have to process another 10000 pictures made with rawopint…
-
Here is a first, rather boring run on the g7x
From the log it looks like there might be some problem with ISO control, but it basically works.
Edit:
Slightly more dynamic run, still doesn't exercise exposure control much
-
Hello,
I posted some sky timelapses in the examples rawopint thread.
If you see any issues I may be able to post the csv's.
The camera went to ISO400 and 20 seconds of exposure.
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)
Udo
-
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made. Only during the day the round shape is restored. (over exposure effects?)
This is probably due to the moon moving during a long exposure.
-
This is probably due to the moon moving during a long exposure.
It gets more or a star-shaped form with rays extending from the points of the star.
See
-
Hello,
I posted some sky timelapses in the examples rawopint thread.
If you see any issues I may be able to post the csv's.
The camera went to ISO400 and 20 seconds of exposure.
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)
Udo
Have you seen the settings for the moon above in this thread?
The moon is definitely overexposed with 20s…
For the moon I got typically IS0100 and 1/50s.
You can’t have a neutral exposed moon and stars together….
Where is the problem to attach csv log files?
-
In one of the timelapses the moon is visible but this objects gets a non-round shape at night in the exposures it made.
Only during the day the round shape is restored. (over exposure effects?)
Have you seen the settings for the moon above in this thread?
The moon is definitely overexposed with 20s…
For the moon I got typically IS0100 and 1/50s.
You can’t have a neutral exposed moon and stars together….
I do understand the choice between stars or moon exposed well.
But the starry appearance of teh moon was new to me.
-
I do understand the choice between stars or moon exposed well.
But the starry appearance of teh moon was new to me.
Which aperture do you use?
When you close the aperture you have this effect with the sun and moon when it’s overexposed. Also seen on my videos with the sun.
-
Which aperture do you use?
This very timelapse appears to have been done with av=4.
Others with av=2.
I did not set aperture, so probably I should set it to the lowest value for sky timelapse?
-
This very timelapse appears to have been done with av=4.
Yes, that’s the reason of the stars. Overexposure and an aperture more than the lowest…
I did not set aperture, so probably I should set it to the lowest value for sky timelapse?
I think, there is no reason for sky timelapse to close the aperture. Typical, the first think what I am doing when I have too much light I use the internal ND.
-
Attached a log file from my SX50 with an Interval from 2.5s.
The run was only made with Canon RAW.
It looks like that the get_exp_count does not work correct at this run. I didn’t notice this before. Since I use the log file for deflicker, I had more notice about this.
For example, exp 46 (or 64, …) is twice in the log file and exp 47 is missing. But RAW File 47 is there. No RAW File is missing. This is happening about 30 times in around 300 pictures.
Update:
I attached another run from the SX50 without Errors (Moon run, also only RAW and Interval = 2,5s). This run was made in continues mode. I thought this also from the other run. EXP ist OK.
So the problem may be having something to do with the quick mode….
-
It looks like that the get_exp_count does not work correct at this run. I didn’t notice this before. Since I use the log file for deflicker, I had more notice about this.
This sounds like the typical file counter problems we see on a number of cameras. The CHDK "pause for filecounter" workaround is only enabled if CHDK raw or remote capture is enabled.
You may be able to work around this using the "raw hook sleep" option in the script. This adds a fixed delay before the log line (with the image number) is written to the log.
No RAW File is missing. This is happening about 30 times in around 300 pictures.
Missing files would only happen for CHDK raws: if the number isn't incremented in time, a file gets overwritten.
-
I would like to take a time-lapse with the M3 and my EF-S 10-18. Unfortunately, the ISO does not work yet.
If I just want to change only the exposure time, does the following workaround would help?
ISO manually to 100
ui_sv_target_mkt=100 "Target ISO"
ui_sv_max_mkt=100 "Max ISO"
and in the code:
if (ui_sv_target_mkt ~= ui_sv_max_mkt) then
set_sv96(self.sv96)
end
and as well for the get_prop(props.SV) stuff to avoid error messages.
-
I would like to take a time-lapse with the M3 and my EF-S 10-18. Unfortunately, the ISO does not work yet.
The fix in the port is probably very simple.
If I just want to change only the exposure time, does the following workaround would help?
ISO manually to 100
ui_sv_target_mkt=100 "Target ISO"
ui_sv_max_mkt=100 "Max ISO"
and in the code:
if (ui_sv_target_mkt ~= ui_sv_max_mkt) then
set_sv96(self.sv96)
end
and as well for the get_prop(props.SV) stuff to avoid error messages.
Without trying it, I think that should work. The exposure might be off somewhat, but it will be by a fixed amount, probably 2/3 of stop.
-
I'm still trying to get rawopint on the M3 to run. Since the Black Level = 2047 is, I get however about 1EV over-exposed images.
I keep my G1x on a white monitor and start rawopint, then I get already with the first recording a neutral image (exposure metering medium-weighted). All other pictures will not be changed. So as I would expect.
If I do the same with the M3 (presumably also with the M10), then the first picture is also neutral. However, rawopint then regulates the following images to + 1EV.
I guess the problem is due to the neutral value of the M3 (rawop.get_raw_neutral). I have not really understood the calculation yet.
The cameras have the following values
M3 Black Level = 2047 White = 16K Neutral = 4197
G1x Black Level = 511 White = 16K Neutral = 2891
If I look into the rawopint log file, then it looks to me as if the neutral value of the M3 (around 3100) Close to the G1x.
I could probably work around the problem if I would set ev_shift to -1EV. However, I would like understand the stuff.
What I also do not understand how the M3 from a range of 2K - 16K more dynamics than the G1x with 512 - 16K. But this is a very different issue
-
ISO override discussion split to https://chdk.setepontos.com/index.php?topic=13191.msg133698#msg133698
-
@reyalp
I still want a log for the battery temperature ;)
This is easy to add, but out of curiosity, what is it useful for?
-
This is easy to add, but out of curiosity, what is it useful for?
What is the optical temperature useful for?
I find it interesting to know how the temperatures change over time. If I operate my cameras in the underwater housing they are already very hot. I have a run since the sensor temperature is above 80 degrees Celsius. During this run I would have already been interested in how warm the battery was. Maybe there is a battery temperature at which the camera turns off? I have some batteries that do not have a temperature sensor. Could it be dangerous to use these batteries at high temperatures?
-
I looks like, that my G1x, S110 und SX50 can run with an Interval from 1s (Only RAW, Tv < 0.01s) in continuous mode. I used my fastest SD card. It differs from card to card by around 200ms (All cards minimum class 10). For save condition I would not go faster than 1.1s Interval.
-
I have been using this script for fast action time lapse while kayaking. The camera is a Powershot A 470 and can shoot at about 2 shots/s.
I would ideally like to restrict the longest shutter speed to 1/20 or less. But there are two problems.
One is that the method for setting the maximum exposure is by 1/1000 s by steps of 1. Why not just have a list of regular exposure times to click through. As it is clicking from the default value of 1000 down to 50 is a bit of a pain. I don't know much about scripting in lua, but surely there must be a way of defining an array, e.g. in pseudo code:
shutter_speeds = [16 13 10 8 6 5 4 3.2 2.5 2 1.6 1.3 1 0.8 0.6 0.5 0.4 0.3 1/4 1/5 1/6 1/8 1/10 1/13 1/15 1/20 1/25 1/30 1/40 1/50 1/60 1/80 1/100 1/125 1/160 1/200 1/250 1/300 1/400 1/500 1/600 1/800 1/1000 1/1250 1/1600 1/2000]
in which case it would only take about 20 clicks from either end to reach the desired value.
The other problem is that setting the limit at 50/1000 s (= 1/20) doesn't seem to take effect (shots were taken at 1/5 s, which was far too slow to be of much use). Is there anything I am missing here?
I include the log file. The series where the limit was ignored was taken on June 20, 2017.
-
One is that the method for setting the maximum exposure is by 1/1000 s by steps of 1.
On most CHDK ports, you can change the "multiplier" using the zoom lever. On cameras without a zoom lever, the disp button should cycle through them. The A470 has neither, but according to the wiki, you can set it to use the power button (!) in alt mode as a substitute http://chdk.wikia.com/wiki/A470
Why not just have a list of regular exposure times to click through. As it is clicking from the default value of 1000 down to 50 is a bit of a pain. I don't know much about scripting in lua, but surely there must be a way of defining an array, e.g. in pseudo code
I use that approach for some of the other settings, but supporting the full range of exposure one might want to use takes a lot of values, which would be less convenient to scroll through on cameras with a working multiplier key.
The best solution would be to specifically have an "shutter" script parameter type that uses the same interface as overrides.
The other problem is that setting the limit at 50/1000 s (= 1/20) doesn't seem to take effect (shots were taken at 1/5 s, which was far too slow to be of much use). Is there anything I am missing here?
Thanks for reporting that, it looks like a bug in the script. I haven't debugged it in detail, but I suspect it is due to having "ISO adj TV Sec/1000" set to longer value than "Max Tv Sec/1000". Try setting that to a shorter or equal value. I'd definitely suggest doing a quick test run to verify that before you take it on a trip.
-
I had just a quick look into it… plot looks really funny…
Thanks for reporting that, it looks like a bug in the script.
My first estimation was a wrong propcase in the A470…
but I suspect it is due to having "ISO adj TV Sec/1000" set to longer value than "Max Tv Sec/1000".
Could be….
This parameter sometimes driving me nuts ;). Whenever I change Max Tv, I must also change this value. Sometimes I forget that then. We had already other user with the problem with this parameter have..
My suggestion would be to remove this parameter
"ISO adj TV Sec/1000" = "Max Tv Sec/1000"
or if you think it is necessary do it with an offset
"ISO adj TV Sec/1000" = "Max Tv Sec/1000" + "ISO adj TV offset"
The default offset should be 0
Not sure if interval=0 would be a problem…
I'd definitely suggest doing a quick test run to verify that before you take it on a trip.
I would do a quick test with default parameter (may be 50 shoots) under constant light condition.
-
Can the raw exposure metering section of this script be coded into CHDK firmware, as in LUA2C? Best of both worlds
-
Can the raw exposure metering section of this script be coded into CHDK firmware, as in LUA2C? Best of both worlds
The CPU intensive parts of raw meter are already done in C code, so IMHO it's already the best of both worlds.
-
Where do I find raw metering in the CHDK menus?
-
Where do I find raw metering in the CHDK menus?
Raw metering just means the ability to measure values in the in the raw buffer. It is a building block that can be used for things like the rawopint script. Putting in the menu wouldn't make any sense, it's only useful if you do something with the measured value.
If you want to request a feature, please start a new thread, and describe what you are actually asking for.
-
New version uploaded, see first post or wiki. The bug SkepticaLee reported with Tv limit being ignored should be fixed. This was triggered when the "ISO Adj Tv" was set longer than "Max Tv". In the new version, the script will just act as if "ISO Adj Tv" is set to "Max Tv" in the case, meaning ISO adjustment will start when shutter adjustment ends. If you don't want any ISO adjustment at all, set "Target ISO" and "Max ISO" to the same value.
I also added battery temp to the log at c_joerg's request.
I'm hoping to do some more serious work in this script again soon.
-
I also added battery temp to the log at c_joerg's request.
Thanks :)
I'm hoping to do some more serious work in this script again soon.
Great! :)
The biggest point on my wish list is to reduce the oscillation. I think that would be not only for me a big improvement of the script. I think it’s not a big think. What I descript here works very well (Reply #259).
https://chdk.setepontos.com/index.php?topic=11081.msg124334#msg124334
or
I noticed those small oscillations in brightness and fixed them in the script. Basically, I don't adjust exposure until it's around 5 ev96 off. I gradually increase the amount of each adjustment from 0 to 3 ev96 until I'm 48 ev96 off, then adjust to 48 off after that. It's like driving a car. You turn the wheel more and more towards the center of the road as you near the edge. Then you turn the wheel as much as needed to keep from going off the road.
-
I uploaded version 0.23.
This adds ND filter control for cameras that have both and adjustable aperture and ND filter. I also added a hysteresis option for the ND filter control, to reduce the risk of "flapping" around the the ND threshold.
Note I haven't done much real world testing with this, so I'd encourage you to test before going out to take a real timelapse.
Unfortunately, some CHDK limitations make ND control more complicated than I would have liked, so there are two new ND confusing options that may need to be used for best results:
1) ND value.
This is the APEX96 value of the ND filter. Unfortunately, this cannot be determined by the script on cameras that have both an ND and adjustable aperture, and can only be determined on other cameras if the Canon firmware used the ND on the first shot. The value is always around 3 stops, but varies enough between cameras that using 3 will cause a noticeable jump in exposure. In my experience, if the correct value is used, ND transitions are virtually undetectable.
Known values
* Ixus 140 / Elph 130 276
* D10 283
* G7 X 293
Finding the correct value
* On cameras with only an ND filter (most ixus/elph/SD, many A series) subtract the value of the MIN_AV propcase from the value of the AV propcase after half pressing in a scene were the Canon firmware would use the ND. You can also get it from the with / without ND values of the AV table in platform/shooting.c, if they are set up correctly.
* On cameras with an ND filter exposed in the Canon UI (many G and S series), there are one or more currently unnamed (not defined in propsetN.h) propcases that gives the value. This can be detected by comparing propcases values with / without the ND filter enabled and looking for one that changes by ~3 stops. On G7 X, propcases 85, 93 and 94 contain the value. 85 is associated with the string GetDeltaNDResu and 93 is associated with PCH_DELTAND_MEASU
* For cameras with a "hidden" ND (see description in the following option), I don't know if the value can be obtained from propcases. You could compare raw meter values with/without under stable lighting. It's possible there are propcases that expose these values, but I don't have a camera like this to test.
2) Force initial ND.
On cameras like sx260 which have a "hidden" ND filter (adjustable aperture + an ND filter which is NOT controllable using the Canon UI), CHDK script cannot determine the initial state of the filter. That means that if the canon firmware initially had the ND filter in, ND control will not work. The "Force initial ND" option can be used to set the ND to a known state, however, the initial Canon auto exposure does not account CHDK overrides, so if the Canon firmware chose a different ND state than the force option, the initial exposure will be off by ~3 stops. The script should settle down to normal exposure values after a few tens of shots.
I will start a separate thread in the development forum about addressing some of these issues in CHDK code.
-
This adds ND filter control for cameras that have both and adjustable aperture and ND filter.
If I remember correctly, it wasn’t possible to change the ND Filter while you are in ‘quick’ or ‘continuous’ running mode. Would this work on my G1x now?
Another think about rawopint:
I have some presets for rawopint. If I use them on EOS M3, I got sometimes the error
error("meter step too small")
Yes I know, that has to do with the larger senor size.
But do you really have to make an error here? It’s not enough to make a warning or just increase the value to a minimum?
Does this value only affect the run time or does this makes internal overflows?
-
If I remember correctly, it wasn’t possible to change the ND Filter while you are in ‘quick’ or ‘continuous’ running mode. Would this work on my G1x now?
Changing the ND works in continuous and quick on the cameras that I have with ND (g7x, d10, elph130).
rawopint previously didn't support controlling the ND on cameras with both ND and adjustable aperture, but that was because I hadn't come up with code to deal with the complications mentioned in my post, not because of a camera limit.
I have some presets for rawopint. If I use them on EOS M3, I got sometimes the error
error("meter step too small")
Yes I know, that has to do with the larger senor size.
The 14 bit sensor is the biggest factor. See http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations#rawop.meter
rawopint could do something smarter, for example, break the meter up into smaller regions and average the results. A function that does this automatically would be a good addition to rawoplib, or perhaps the C code. I hadn't worried about this much because there was only on 14 bit camera, but now there are more...
edit:
It seems ND control may fail in quick mode on some cameras.
-
Just for understand how ND works on my cams with IRIS and ND filter (G1x and S110)
1) Sunset
In this case, I want that ND goes in after start
- I expect that TV goes to longest exposure time.
- Than ND goes out.
- Tv goes shorter
- Tv goes again to longest exposure time.
- Than ISO will increase
Script settings:
ui_nd_force_e= 1 => ND in
ui_nd_value = 284 => (G1x) because auto detected does not work
ui_tv_nd_thresh_s10k = ui_tv_max_s1k*10
If this is correct?
Why (ui_tv_max_s1k = Sec/1000) and (ui_tv_nd_thresh_s10k= Sec/10000) are different?
-
Just for understand how ND works on my cams with IRIS and ND filter (G1x and S110)
1) Sunset
In this case, I want that ND goes in after start
- I expect that TV goes to longest exposure time.
- Than ND goes out.
- Tv goes shorter
- Tv goes again to longest exposure time.
- Than ISO will increase
Script settings:
ui_nd_force_e= 1 => ND in
Since these cameras have manual ND, setting it in the Canon UI should also work, and allow the initial exposure to be correct.
ui_nd_value = 284 => (G1x) because auto detected does not work
ui_tv_nd_thresh_s10k = ui_tv_max_s1k*10
If this is correct?
I haven't tested the case where ND thresh is exactly the same as Tv max. It might work, but I'd suggest testing. You might need to make the ND thesh at least slightly less then the tv max.
Why (ui_tv_max_s1k = Sec/1000) and (ui_tv_nd_thresh_s10k= Sec/10000) are different?
Because normal menu inputs only accept 5 digits, and I expect most people will want the ND at the short end of the exposure range (need to be able to enter something like 1/2000th), while max tv needs to allow the longest exposure anyone would reasonably use (> 10 seconds)
When I set this up, I wasn't thinking about people wanting to keep the ND for motion blur.
I know the x/n format and different scales is annoying, but I haven't come up with another way that allows sufficient flexibility within the limits of the script menu system. There is an option for "long" inputs, which would allow more consistent units but keeping track of the digits gets even more annoying.
-
First run with ND on G1x. Looks like Tv goes to 2s and then ISO increases to 1000 (at 186). Than ISO goes to 160. Different as I expected but from motion blur it is better to keep the 2s.
But why ISO goes from 1000 to 160 and not from 800 to 100? Is this the Hysteresis?
You might need to make the ND thesh at least slightly less then the tv max.
I set tv_nd_thresh to 2s and tv_max to 2s? Did this make the difference?
At the point where ND goes out, there is no jump in meter. So, ND value from propset looks correct.
When I go to the picture review, the cam gives me the information, if ND is used. All pictures have the ND info so I guess that the cam doesn’t know that ND was switched out.
I'm happy with the run.... :)
-
First run with ND on G1x. Looks like Tv goes to 2s and then ISO increases to 1000 (at 186). Than ISO goes to 160. Different as I expected but from motion blur it is better to keep the 2s.
But why ISO goes from 1000 to 160 and not from 800 to 100? Is this the Hysteresis?
You're right, I totally forgot about the hysteresis.
The effect is that the ND is put in when Tv would otherwise be shorter than Tv ND thresh, and after it is in, will only be put out when Tv is longer than Tv ND thresh + hysteresis value (in consistent units, which they aren't in the script UI.)
Since this means the ND should have only been removed when the exposure was below the limit, how did your run use the ND at all? In the calculation, all the exposure change starts on Tv, even if it's outside limits, and the calculation for ISO happens after the ND calculation. So the ND calculation still saw the Tv go longer than the ND limit, and it looks like it did the right thing, although I didn't really design it for this.
To avoid this, you could make ND thresh 1 second, hysteresis 1 stop, and then the ND should be removed at 2 seconds, and only put back in if tv goes shorter than 1 second. I chose 1 stop hysteresis here only because it makes the math simple, other values should be fine.
You might need to make the ND thesh at least slightly less then the tv max.
I set tv_nd_thresh to 2s and tv_max to 2s? Did this make the difference?
I said this because I'm lazy and didn't want to try to figure out which calculations use > or >= etc and how they might interact. In the example above, if you set ND thresh to 0.9 seconds then the ND will come out slightly above the absolute tv limit of 2 seconds (including the 1 stop hysteresis) and there should be no ambiguity. From an exposure point of view, the difference between changing the ND at ~1.9 or 2 sec is insignificant.
-
Just a second run. Was made before I read you quote.
Different to first run is that this time v_nd_thresh is set to 1.9s instead 2s. And I run the script a little bit longer so ND goes out and in again.
Both tests were made with my test video which decreases my monitor from 255 to 0 and increases back to 255 again. I know that it is not linear from the viewpoint of exposure levels. Also the area around 0 (Monitor black) is not really accurate. But the video gives me reproducible results.
how did your run use the ND at all?
Before I start the script, I set ND in the Canon UI.
and it looks like it did the right thing, although I didn't really design it for this.
Of course…
To avoid this, you could make ND thresh 1 second, hysteresis 1 stop, and then the ND should be removed at 2 seconds,
I think its great how works now… :)
For example:
Sunset over the ocean. The exposure goes to 2s and smooths the waves. If now goes ND out and exposure would go to 0.5s that would be probably more noticeable as the ISO jump from 800 to 100. Since G1x has no problem at ISO 320 anymore it’s perfect (Not sure how the status is for S110 and SX50).
I do not know if that makes sense:
Sometimes (when I’m traveling) I’m using the ‘HOSTLUA’ environment. Unfortunately the 'rawoplib' will not be supported. I tried to implement these functions with simply returns but haven’t finished yet. Would the
Simulation mode run in this environment ?(ui_sim=true "Run simulation")
-
how did your run use the ND at all?
Before I start the script, I set ND in the Canon UI.
Sorry, that was a poorly worded rhetorical question. What I meant (and explained in the following text) was: Given that the Tv (+ hysteresis value) where the ND was supposed to be put out was longer than the maximum, why did the ND go out at all?
Sometimes (when I’m traveling) I’m using the ‘HOSTLUA’ environment. Unfortunately the 'rawoplib' will not be supported. I tried to implement these functions with simply returns but haven’t finished yet.
That would be a lot to implement...
Would the
Simulation mode run in this environment ?(ui_sim=true "Run simulation")
I haven't kept the code up to date, so right now it doesn't work at all. If it was fixed, you would need less of rawop, but (without looking) I think you might still need the raw/APEX conversion functions, which might be difficult without floating point.
-
Sorry, that was a poorly worded rhetorical question. What I meant (and explained in the following text) was: Given that the Tv (+ hysteresis value) where the ND was supposed to be put out was longer than the maximum, why did the ND go out at all?
Ok understood. You think ND should never go out because Tv goes never over Tv + hysteresis….
The whole stuff gets really complicated also with tv_sv_adj which is also set to 2s…
That would be a lot to implement...
Just an idea … I wrote my fist simple time-lapse script with basic histogram complete in the ‘HOSTLUA’ environment by reading values from a file…
-
Hi,
A question to the developers:
I tried setting the Interval to 600 (* 1/10 s -> one picture every 60 seconds), with Shots set to 0 (unlimited), but the camera only took one picture.
I take plenty of time lapse in continuous mode with the Interval set to 0, so it 's not the camera, and possibly not me, but what might be standing in the way between wanting a shot/minute and getting it done?
BTW is there a way of reading/setting menu values on the card?
Cheers, Lee
-
I tried setting the Interval to 600 (* 1/10 s -> one picture every 60 seconds), with Shots set to 0 (unlimited), but the camera only took one picture.
Thanks, you've found a bug. The script was written to take pictures as fast as possible, so I haven't really tested long intervals. It will probably fail if the interval is a longer than 10 seconds + the time require to shoot.
If you want to try a workaround, you can change the line (around line 1758 in the file)
hook_shoot.set(10000)
to something like
hook_shoot.set(10000+interval)
However, there might be other problems with long intervals.
Also, you should know the method this script uses for exposure calculation is not really suited to long intervals. It uses the image data from the previous shot, so with a 60 second interval, there's a lot more likely to be major changes. For long intervals like this, you might be better of using an intervalometer that does a half press and camera auto-exposure every shot.
BTW is there a way of reading/setting menu values on the card?
The settings are stored in CHDK/DATA/rawopint.N where N is the number of the parameter set in the menu. You could edit this manually if you are careful.
-
Thanks, reyalp, I'm using the program to monitor my super battery pack to try to understand what is going wrong with the connections between the battery and the camera (A470). Some sets of wires (jumper leads with croc clips, 30 cm) seem to work just fine and I can get the predicted 11 hours of shooting out of the batteries; with others (DC power output lead, 2 m) the camera shuts down after much less time. Even inserting a standard 9V battery clip shortens the shooting time considerably. Resistance doesn't seem to be an issue (all weigh in at about 0.5 - 1.0 ohms).
Cheers, Lee
-
First post is updated with version 0.24. This fixes the issue with > 10 sec intervals discussed above. The script is still not optimal for long intervals since metering is done using the previous shot.
I also removed the initial ND state option. With the "hidden" ND now understood to be aperture control (https://chdk.setepontos.com/index.php?topic=13228.msg135727#msg135727), there is little need for it. If you want to use the ND settings on one of these cameras (sx170is sx220hs sx230hs sx240hs sx260hs sx280hs sx40hs sx50hs sx510hs sx60hs), make sure the aperture is set to full open before running the script. If you have a camera with ND + iris where the ND state isn't correctly detected (propset 1 or 3), set it to OUT in the Canon firmware before starting the script.
-
Hello all together,
let me first thank you, reyalp, for this wonderful script.
I'm testing it for the last two weeks now with a G1X and G16 and must say, that I'm very impressed from the results. But what I still see is a jump in the brightness after sunset, when the scene gets darker. With the G1X as well as the G16. I did a test shooting with some new settings this evening with both cameras and checked the exposures afterwards. I didn't had time to process the movies, but I attach both log files. So interested people may have a look at it.
Look at image 789 and 790 with G1X and 5165 and 5166 with G16.
Please let me know if there are some more information needed.
Thank you and many greetings
Harald
-
But what I still see is a jump in the brightness after sunset, when the scene gets darker. With the G1X as well as the G16.
Thanks for posting. I don't see an obvious explanation in the logs, from the script point of view, both looks like the scene suddenly got darker for one frame.
On G1x, meter96 goes from -177 in image 788 to -279 in 789
On G16 it goes from -140 in image 5164 to -240 in 5165
There's some effect in the later frames because the script tried to adjust.
On G1 X, this happened right when the ISO switched to 400, which might be related to the ISO weirdness on this camera (thread https://chdk.setepontos.com/index.php?topic=13191.0). I think there are some fixes for this in the 1.5 tree that aren't in 1.4, so you might try that.
That does explain G16 though, it isn't known to have this kind of problem, and the glitch happens going from ISO 147 to 151.
If you can upload the the original image files 788 to 790 from g1x, 5164-5166 from g16, that might be interesting. The forum doesn't allow large files or many in post, you can use dropbox, google drive etc.
-
On G1 X, this happened right when the ISO switched to 400, which might be related to the ISO weirdness on this camera (thread https://chdk.setepontos.com/index.php?topic=13191.0). I think there are some fixes for this in the 1.5 tree that aren't in 1.4, so you might try that.
Yes that’s looks exactly like the ISO Problem of G1x. He is using g1x-100g-1.4.1-4988 an the fix is only in 1.5..0. With g1x-101a-1.5.0-4876 on my G1x it’s works fine. But only on cont mode. The quick mode still doesn’t work.
https://chdk.setepontos.com/index.php?topic=13191.50
Reply #57
That does explain G16 though, it isn't known to have this kind of problem, and the glitch happens going from ISO 147 to 151.
Looks like the G16 has the same ISO problem. We fixed it only on G1x, SX50 and S110.
Update:
If have seen 1.5 for G1x is only for Firmware 1.01a available :( Why?
-
Hey there,
c_joerg is absolutly right, v1.5 is only available for firmware 1.01a and that isn't on my camera. I do another test right now with the hack that lapser did and will see if it works or not. Alternatives are firmware update on the G1X, a fix of the problem in 1.4 or build a 1.5 for 1.00g.
To check the images as you request it, reyalp, I opened a folder in dropbox (https://www.dropbox.com/sh/dpqit4ssm074qzn/AADzJ2Eqm1OjqBrAQBvNBvz6a?dl=0) where I put the images in. Feel free and have a look at it. By the way: I took the potos in cont mode.
Let me know if you need something more.
-
c_joerg is absolutly right, v1.5 is only available for firmware 1.01a and that isn't on my camera.
Oops, thanks for pointing that out, looks like I accidentally removed the others when I added g16. All g1x builds should be available 1.5 autobuild 5000.
-
Oops, thanks for pointing that out, looks like I accidentally removed the others when I added g16. All g1x builds should be available 1.5 autobuild 5000.
Wow, that's quick! Thanks a lot. I`ll update tomorrow.
I stopped my testing some minutes ago. Seems as if the G16 had the same jumps at ISO 160 as yesterday.
Thanks again
Harald
-
Well, there were a lot of cloudy nights during the last time and therfore no chances to test the new versions. But now I did it and some things more.
First: the updates worked very well now on my G1X. As there is no version 1.5 available for a G16 I implemented lapsers hack and it seems to work fine, too.
Additionally I had a chance to test a EOS M3 from a friend. I tested it in combination with a Genie Mini, which is a panning device, that I own. The result is attached. There is no afterwork done so far, only rendering with virtualdub.
I would say I started wrong with a Ev shift of -1/4 and a Bv Ev shift of 25% was too much, too.
But my main problem was the outside temperature. So I stopped after one hour.
Many greetings
Harald
-
Additionally I had a chance to test a EOS M3 from a friend.
Take care about continuous mode of M3. The camera limits this mode to 1000 shoots!
Ev shift of -1/4
I would not do that at sunsets. I always limit overexposure. A negative shift can also be done in post processing.
Bv Ev shift of 25% was too much, too.
Even with this parameter I have always been wrong. I only do that in post processing
I’m wondering the interval was set to 20s but delta between shoots is 15s…
-
Take care about continuous mode of M3. The camera limits this mode to 1000 shoots!
Thank you for that hint!
I would not do that at sunsets. I always limit overexposure. A negative shift can also be done in post processing.
I agree! But I mixed + and - ::)
I’m wondering the interval was set to 20s but delta between shoots is 15s…
You don't need to wonder. I set the interval to 20s but the shots were triggered by the Genie Mini via USB port. I used the Genie in a move-shot-move mode and I set the interval there to 15s. And as far as I can see it worked pretty well.
So some more tests regarding the settings have to follow...
-
Hi everyone, especially reyalp,
I`m still experimenting with this script. Mainly for sunsets.
What comes next will be hiding the camera and shooting in the evening and during night. I think it might be a good idea to delay the shooting. What I mean is that I start the script and the shooting will start an hour later or so.
I'm not very familiar with programming, but that problem might be easy solvable with a sleep-routine that I found that routine in another script (YASS 4), but rawopint is rather complex and I don't know where I should place the delay-code. Can anyone give me some hints where to place it best?
Thanks ahead
-
What comes next will be hiding the camera and shooting in the evening and during night. I think it might be a good idea to delay the shooting. What I mean is that I start the script and the shooting will start an hour later or so.
I have recently been working on similar functionality for fixidint.lua. I still need to finish a few things, but I'll probably integrate it into rawopint as well when it's done.
I'm not very familiar with programming, but that problem might be easy solvable with a sleep-routine that I found that routine in another script (YASS 4), but rawopint is rather complex and I don't know where I should place the delay-code. Can anyone give me some hints where to place it best?
If you want to keep it really simple, putting it at the start (i.e. just above rawpoint_version) should be OK. Assuming you want to add a menu option, it needs to added to the section above with the # lines, like
#ui_wait_minutes=0 "Minutes to wait"
... license stuff etc...
]]
sleep(ui_minutes*60*1000)
rawopint_version="0.24"
This just gives you a number of minutes to delay from when you started the script. If you want to start at a particular clock time, you need more complicated code.
If battery life is a concern, you might want to stay in playback mode or allow the Canon firmware to shut off the screen and sensor while waiting, which could be a little more complicated.
The script will switch to rec mode if you start in playback, but you'd lose control of zoom and focus, and the LCD would still be on.
A better approach would probably be to use the Canon firmware power saving mode functionality. To do use this, in the Canon menu power saving section:
Set "Auto Power Down" to "OFF" and "Display Off" to some reasonable time.
In the CHDK menu, under CHDK settings, set "Disable LCD Off" to "never".
Now you should be able to set up the camera in rec mode, with your focus, zoom etc and set, start the script, and have the sensor and display stay off until the first half press. You should test this before hand. On some very old cameras, the display going turns of MF.
-
Mainly for sunsets.
What comes next will be hiding the camera and shooting in the evening and during night. I think it might be a good idea to delay the shooting. What I mean is that I start the script and the shooting will start an hour later or so.
I would like to give something else to consider:
If you turn the camera on before sunset, the sun is usually even more intense. I would not put the camera in the sun without a lens cap for a long time. For cameras with ND filter, I would always activate.
In sunrises usually the problem of condensation occurs.
If battery life is a concern, you might want to stay in playback mode or allow the Canon firmware to shut off the screen and sensor while waiting, which could be a little more complicated.
I would definitely put the camera in the playback mode, but I'm not sure if the sun does not come on the sensor. Presumably, then the aperture is full open.
The script will switch to rec mode if you start in playback, but you'd lose control of zoom and focus, and the LCD would still be on.
I already noticed that…
-
If you turn the camera on before sunset, the sun is usually even more intense. I would not put the camera in the sun without a lens cap for a long time. For cameras with ND filter, I would always activate.
...
I would definitely put the camera in the playback mode, but I'm not sure if the sun does not come on the sensor. Presumably, then the aperture is full open.
I'm quite sure the edit P&S cameras close the mechanical shutter when they are in sensor-off power saving mode or playback. The shutter might get quite hot, but it should be somewhat outside the focal point at least.
-
I'm quite sure the cameras close the mechanical shutter when they are in sensor-off power saving mode or playback.
Sure?
Not on my M3. The Senor is always visible. With a manual lens I can see the senor. Aperture is closed to minimum with EF lens. I looks like that same on G1x but I’m not sure.
-
Sure?
Not on my M3. The Senor is always visible. With a manual lens I can see the senor. Aperture is closed to minimum with EF lens. I looks like that same on G1x but I’m not sure.
Good point, I have only looked at P&S, which generally have a simple leaf shutter. I'm not surprised EOS is different.
On all my P&S, there's a click which sounds like the shutter when the display turns off. On the cameras I checked (eph130, sx160, sx710 and g7x) the entire aperture is matte black when the sensor is off. When it's on, there's an additional glassy surface (presumably the IR cut filter) visible further behind the aperture.
-
Hi, first I want to say thanks for the simple solution for a time delay! It is working pretty well. At least at home. I didn't test outside it till now.
Sure?
Not on my M3. The Senor is always visible. With a manual lens I can see the senor. Aperture is closed to minimum with EF lens. I looks like that same on G1x but I’m not sure.
I can confirm that, too. The shutter is always open with manual lenses as well as with EF lenses. That's one reason why I didn't tested it in a sunset till now.
-
Version 0.25 uploaded to the first post.
This adds options to set a start time on the camera clock and set initial focus and zoom from the script. The camera can wait in playback mode until the specified time.
It also has a workaround for the issue reported by c_joerg in https://chdk.setepontos.com/index.php?topic=13191.msg137008#msg137008 where aperture would be changed on cameras with incorrect ND definition, even if ND control threshold was disabled.
I have also added a "glue" script to chdkptp svn which allows tethered shooting with chdkptp remoteshoot. See here https://chdk.setepontos.com/index.php?topic=13386.msg136688#msg136688 for details, and thanks Bluestone_7 for the suggestion.
-
Version 0.25 uploaded to the first post.
Is there any limitation about number of parameters? ;)
Would be interesting to have an option in parameters, start_hour=-1 than start_min and start_sec are not seen.
For beginners it might be helpful to have a rawopint version with a simple UI...
-
Is there any limitation about number of parameters? ;)
CHDK only looks at the first 4 KB of the script. So rawopint is getting fairly close at ~3.3 KB
Would be interesting to have an option in parameters, start_hour=-1 than start_min and start_sec are not seen.
So, a scripting language for the script menu? :D
One thing I would like to do is to add more types to the CHDK menu system. So you could have a shutter speed input with the same UI as the CHDK override, zoom that knows the cameras actual zoom range, time in one field of hh:mm:ss etc.
That would make it easier to understand and more compact.
For beginners it might be helpful to have a rawopint version with a simple UI...
Yes, but I'm not sure which ones to remove, and I don't want to maintain even more versions...
The other thing I've thought about is making a tool to generate the configuration offline.
-
So, a scripting language for the script menu? :D
Just a sub menu…
Yes, but I'm not sure which ones to remove,
I have some rawopint versions with different UI. I never removed the option; I put them as constants in beginning of script. For me that's okay.
I couple of parameters I calculate like Max Tv
Interval >= 4s; Max Tv = Interval - 2s
Interval < 4s Max Tv = Interval /10
The most important parameter which I normally use is shots, interval, max ev change, ISO min/max, over threshold / weight, display mode and waring LED.
For threshold, limit and weight I use in 99% defaults.
I don't want to maintain even more versions...
I can understand that well…
-
Duplicate HDMI related post removed. See https://chdk.setepontos.com/index.php?topic=13481.0
-
Here's one of my attempts at the last lunar eclipse. Unfortunately, the moon is much overexposed in the end. Since the parameters probably did not set correctly…
Especially for such recordings, there should be a simpler solution. For example, I would allow -1 for the Overexposure threshold. In this case, the overexposure would have the highest priority and the exposure would always have to be reduced. That would be similar then what lapser has suggested with his regulation on pmax.
-
edit:
It seems ND control may fail in quick mode on some cameras.
From https://chdk.setepontos.com/index.php?topic=13228.msg138067#msg138067 ND control in "quick" should work on ND-only cameras (because rawopint sets the Av propcase), but not cameras with ND + Iris.
-
Additionally I had a chance to test a EOS M3 from a friend.
Take care about continuous mode of M3. The camera limits this mode to 1000 shoots!
It occurs to me that it would be possible to release shoot_full every 999 shots. There might be a relatively long gap if it spends time flushing the buffer.
-
It occurs to me that it would be possible to release shoot_full every 999 shots.
I have already thought about that. That would be a good idea precisely because the M3 always opens and closes its aperture in quick mode.
There might be a relatively long gap if it spends time flushing the buffer.
The time at the M3 is quite long. But better a big gap at a break off. I would not do that in general, but over an option.
Using the aperture as well for time-lapse would be a great idea. Including ND we had a 4 way holy grail!
The question is whether you first open the aperture or the ND (at a sunset). Here is the procedure for a 3 way holy grail.
https://lrtimelapse.com/tutorial/true-holy-grail-auto-ramping/
-
I just tested this script and I am really impressed with the quality of the night shots, they look much crisper than with canon's settings.
Is it possible to use the same reference shot for every time the camera starts or does it always start with making a reference shot and optimise from there?
-
Is it possible to use the same reference shot for every time the camera starts or does it always start with making a reference shot and optimise from there?
I'm not really sure what you mean.
The first exposure is based on whatever the Canon firmware would have used, either auto or manual. I think CHDK menu overrides should work too, but not sure it has been tested.
For every subsequent shot, it analyzes the raw data of the preceding shot and calculates a target exposure that would result in the scene being "perfectly" exposed as defined by the script options.
The target exposure calculation is essentially deterministic, i.e. target = f(scene), independent of Canon settings or anything that happened before. However, the shot-to-shot change is limited to keep the results smooth. This based on the change in the preceding exposures and how large a change is needed to achieve the target.
The result of this is that the script takes some time to "settle" from the initial, Canon firmware calculated shot to the script calculation. For Canon auto exposure under normal lighting, the change usually isn't big. If you start at night, or used manual settings, it could be quite large.
Because the exposure is calculated based on the previous shot, the script works best for fast intervals and/or scenes that don't change too rapidly.
It would be possible to make the script use a throw-away initial shot to meter the scene, but I'm loath to add more options unless someone has a real use for them.
-
I was just wondering why there was this exposure difference from first shot, and if you use a standard frame that would probably increase.
I have no use for adittions, keep it as is.
but like you wrote, when you start at night it might take a few shots before the script has settled.
-
I was just wondering why there was this exposure difference from first shot, and if you use a standard frame that would probably increase.
This happens, for example, when starting the script in AV mode at night. The longest exposure time is then through the Canon firmware 1s. If the longest exposure time in the script is 4s, then it takes just a while increase to 4s. You can work around this by starting the script in M with 4s.
Greater differences after the first shots are also there when the over and underexposure controls take care.
-
Greater differences after the first shots are also there when the over and underexposure controls take care.
And that is probably what i noticed, my first shot at night was seriously overexposed (according to the log iso400/1sec/f3 and that can't be correct), consecutive shots were timed 1.3sec (same iso/f) and higher, but were not overexposed and still very crisp.
-
I'm making outdoor shoots and ran into this snag with the script that my limited knowledge can't solve.
If i start (daytime) I usually first have a series that start darker, gets lighter and drops after 5-10 shots to a stable result. If i did a test shoot before the 10th frame is then close to the same exposed picture as where the previous series ended. My interval times are mostly between 10 seconds and 1 minuut.
For a standalone project this is no problem, i just throw away the first few pictures.
However, could I create settings in rawopint that mimic the canon firmware as close as possible?
The reason I ask is because I sometimes want to exchange the sd card mid-shoot or have to restart for some reason and i try to avoid losing 5-10 minutes if possible.
I have been using Ultimate intervalometer like that and, because it relies on the canon firmware there is no real difference in exposure between 2 series.
And although i'm very pleased with that script I've been testing rawopint mainly because of the capability to define max and min exposure.
It gives me the possibility to get better night shots with less noise from small sensor camara's that cap off at 1 second in canon's firmware and to block long exposures (30 sec) on the M10.
I have been trying to minimize the lighting differences by changing ND_value_apex96, BV/EVshift% BV/EV shift base and meter height and wide without getting the effect I was looking for...namely a picture that start with almost the same settings as the last one of the previous series.
Or am I chasing an impossible thing here, because the whole metering concept is different?
-
If i start (daytime) I usually first have a series that start darker, gets lighter and drops after 5-10 shots to a stable result. If i did a test shoot before the 10th frame is then close to the same exposed picture as where the previous series ended. My interval times are mostly between 10 seconds and 1 minuut.
You should post a log file that make it easier to understand what’s going on…
In normal cases I never have this problem. In the following cases I had the situation:
1) I used spot measuring in AV mode but script used the whole scene (90%) for metering.
2) If I take pictures against the sun and then overexposure control in the script reduces exposure.
On witch mode you are running and which exposure method you are using?
What about underexposure control? In my case is always disabled…
-
I have the cam in P setting so i can fix color tone/iso and a few other things
1)I use center weiged average
2)Testing only with the sun (or without at night)
I have not changed underexposure control, as soon as it gets dark enough I hit the iso/max exposure limit and images are getting underexposed. that is what i am looking for.
oh, i renamed the csv file to doc, because i got a flag stating i could not upload a log.
sorry, most logs of the past days were already gone, but i can make a new one if this one is not helpful.
I only have a lot of UI logs from a project i'm running.
night restart frame 8-9: from 0.6 to 1/3..if i would have let it run, it would have settled at the same 0.6 after some frames, but that setting didn't do it. so i stopped it after 2 frames. (last is with my hand in front of lens)
the other breaks have more time inbetween, so might be influenced a bit by changing light.
i have attached a new log now, so i can show what i mean properly. I stop the script by pressing the shutter(the only way on the m10), so that's why there are missing images in the log
-
I understand your problem. The fist picture is around 1EV underexposed. Meter96 starts with -96 (-EV). A neutral picture should have around 0.
Have you tried to start the script in M Mode? Default Script setting?
The same thing happens on your other cams?
-
most my camera's are limited to auto or P..only the s95 has an M setting, the m10 has no M setting, but a seperate Tv and Av function.
default script setting on all models i have results in this behaviour, camera's with smaller sensors have it a bit worse though
that subsides once the camera is running a while.
if you want i can run the same thing again with default settings on an a490 tomorrow.
the only constant in my tries might have been the fixed iso
-
the m10 has no M setting, but a seperate Tv and Av function.
Mine does, it's on the same (menu) page as Av and Tv. Check again.
-
you're right, sorry, looked over that. will test the same set tomorrow in M
-
Just tried it with my M3, see the same think, never notice this before…
Can’t see it in some older runs…
I have also problems with upload csv files with worked in the past…
-
I have also problems with upload csv files with worked in the past…
ah, but an image says more than a thousand words. you did one run, so no restart. 5 frames, looks the same to me too. if you would have restarted that whole floating away proces would happen again.
setting: M, same script settings as last csv
so i did 4 runs 2 with night shots (restarted at 26), then 2 in a cabinet with a light. (some 75 in total..guess where i restarted) i hope the slight variance in the "flat" period is due to 50hz lightsource..but can't be sure.
-
Or am I chasing an impossible thing here, because the whole metering concept is different?
There will probably always be some difference.
Have you tried the "Use initial Ev as target (https://chdk.wikia.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer#Use_initial_Ev_as_target)" option? This should make the script try to keep the overall brightness of the scene similar to the first shot. There will still likely be some variation, but in scenes that aren't too extreme (e.g lots of under/over exposure, or big variation across the scene) it should be pretty close.
Edit: If you use BV/Ev shift with this setting, it should probably be set to "First"
Regarding posting CSVs, forum settings were changed a while back to only allow whitelisted extensions. I've asked acseven to add .csv.
-
i tried bv/ev set at first and/or initial ev as target, but it did not help with this.
There will probably always be some difference.
it took me a bit longer, but i suppose you are right.
thanks for the help, could have been mucking with settings for days.
-
Have you tried the "Use initial Ev as target (https://chdk.wikia.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer#Use_initial_Ev_as_target)" option?
I always used this as default “false”.
This should make the script try to keep the overall brightness of the scene similar to the first shot.
There will still likely be some variation,
I never notice a big variation between a pre-shot and the first image with rawopint.
I made just a test with G1x and M3 with a gray image.
The pre-shot and the first image with rawopint have the same exposure, so no differences even with “Use_initial_Ev_as_target” = false.
But on M3 meter96 is around -96 and on G1x around 0. On the M3 the script increase now exposure to bring meter96 to 0.
If I watch the Histograms in Lightroom, the Histogram from G1x is always around the center. On M3 the Histogram of the first picture is in the center and on the last picture more to the right.
If I do an auto exposure in Lightroom, then Lightroom sets the exposure to around -1EV for the last image.
I thing, the neutral value on the M3 is around on step to high (may be also on M10).That makes the pictures 1 step higher exposed.
Regarding posting CSVs, forum settings were changed a while back to only allow whitelisted extensions. I've asked acseven to add .csv.
I would be very grateful for that…
-
i noticed this behaviour first time on my s95, and i also have this with my a480 and a490. so it is not limited to the m series alone.
and it depends on light conditions how obvious it is, if i do a night shoot that is very or a bit more than very underexposed you can hardly tell that it gets brighter during the first few shots. for me it is most visible with clouded days and not overly sunny days.
-
i noticed this behaviour first time on my s95, and i also have this with my a480 and a490. so it is not limited to the m series alone.
On my S110 it’s a little bit different. If I do a pre-shot in AV (I only use AV) then the exposure is around 1/2EV higher. That can be seen on the histogram which is a little bit right. The script brings the histogram exact to the middle. This is different to what I descript in Reply #94.
But on all my cams: if I do a pre-shot in AV, the first picture with rawopint has the same exposure as the pre-shot. Then the script changes exposure.
You should run the script in AV mode. Then I would change the exposure correction in the Canon Menu.
and it depends on light conditions how obvious it is, if i do a night shoot that is very or a bit more than very underexposed you can hardly tell that it gets brighter during the first few shots.
Of course, if you set underexposure or overexposure control, the script has to change it to your settings.
-
Of course, if you set underexposure or overexposure control, the script has to change it to your settings.
that is not what i wanted to point out, it still corrects itself. it is just less visible because it stayes fairly dark.
you can see that on the jpg a few posts back, if there is underexposure correction takes 4-5 frames. at proper exposure that correction takes 9-10 frames
-
I stop the script by pressing the shutter(the only way on the m10)
You should be able to use MENU to stop the script gracefully. The M10 has a menu key, so if this doesn't work, there's a bug somewhere.
edit:
csv upload should be fixed.
-
You should be able to use MENU to stop the script gracefully. The M10 has a menu key, so if this doesn't work, there's a bug somewhere.
after some testing, it does work, with ultimate if i press the menu key it can take up to a few seonds before it's processed, with rawopint it seems to only respond just after a shot has been made.
-
with rawopint it seems to only respond just after a shot has been made.
It also takes a long time for my M3 to finish rawopint. When I stop the recordings with menu on the M3 it takes a long time until the script is really finished. Feels like that for 20 seconds. On my G1x it finished much faster.
-
i was wondering with rawopint would it be possible to create a set of settings that would allow the following diagram in exposure time to be created?
I've found that most Canons set 1 sec exposure at around bv-400, here on the diagram at about bv-380.
in this diagram bv range runs from 380 to -620 (the red line), exposure from 1/100 to 1 sec.
Iso400 is fixed for this camera to keep the relation Tv/bv clear.
Idealy i would like it to reach 1 second exposure at bv-300, if i understand my math that is 1 full stop earlyer.
and i woudn't want it to overexpose at daylight. so that is why I drew the new set of exposure times from 1/60 to 1 sec.
but if it can be done -to keep it smooth- as a 1/3 stop change at 1/10sec-1/8sec (bv:0) or some other idea i would still be interested.
tested it and it's the worst idea i had in years.
What would you consider to be the best match in parameters/settings to create this smooth increase in exposure as things become darker?
-
Idealy i would like it to reach 1 second exposure at bv-300, but I'm guessing that is 3 stops earlyer.
Assuming you are using APEX*96 values, there's ~1 stop between -400 and -300.
The exposure parameters required to achieve "correct" exposure at a given Bv is defined by the APEX equation (http://dougkerr.net/Pumpkin/#APEX) : Av + Tv = Bv + Sv
Setting shutter to 1s (Tv 0) ISO 400 (Sv 7), and Bv to (BV96 = -300)/96 =~ -3
Av + 0 = 7 - 3
Av = 4 = F/4 (this is a coincidence, APEX Av and F number are not *generally* interchangeable, although the mid range is pretty close)
So if you want correct exposure at 1s to be Bv96 -400, set your aperture to F/4. If CHDKs idea of raw neutral is off on your camera (as it appears to be on the EOS Ms) you might want to shift it accordingly.
If you want rawopint to stop at 1s and 400 ISO, just set those limits. Without Bv/Ev shift, this should act a lot like scripts based on Canon AE, which the scene suddenly gets darker once the limits are hit. If you want it to fade more gradually, you should use Bv/Ev shift, which makes this more complicated, or manage the fade in post.
-
yes, sorry, i redid that 3 stop, but was too late for your answer.
but reading in on the link (thank you, will take some days to work it out.
what i'm thus asking is... ok, this is going to sound silly.
in my log it states that the first 1sec/iso400 shot was made at f3.1 bv-367 and that sound right according to that formula.
so my question should actually be, can we throw away that equation ::) when it gets dark and replace it with Av+1+Tv = bv +sv...although if it's that crude it wont work so the 1 in the equation should be in 1/3 steps :P Oh, and get it back as soon as it turns day again....
ok, ok thank you for letting me understand what i'm asking.
-
in my log it states that the first 1sec/iso400 shot was made at f3.1 bv-367 and that sound right according to that formula.
Sorcery! :D
so my question should actually be, can we throw away that equation ::) when it gets dark and replace it with Av+1+Tv = bv +sv...although if it's that crude it wont work so the 1 in the equation should be in 1/3 steps :P Oh, and get it back as soon as it turns day again....
Now I'm confused. What do you want to happen when it gets dark?
Adding 1 to the Av + Tv side is a negative exposure compensation, meaning darker than the "correct" exposure that the equation is defined for.
If you want the scene to get darker smoothly as the light level falls, that's what Bv / Ev shift is meant to do. However, the difference from your idea is that it's always active, in proportion to the difference between the "base bv" and the actual bv, rather than only kicking in at "night". It would probably be possible to make an option to only apply it below a certain Bv.
You could set the Bv / Ev shift base low, and let the meter high and overexposure limits control the high exposure.
-
I want the taken pictures to get dark more slowly between something like bv200 to bv-300....for now limited to 1 sec, that could be 4sec, 2 stops further if the camera and I permit that.
If that works i would expect then that s going from night to day 1 sec exposure would still be used until it reaches bv-300 (approx) and that it would change back to the original calculations between 1 sec and 1/60.
since the original calculation Av+Tv=bv+sv results in a specific exposure my first thought was to change that formula so it would allow for a longer exposure.
sort of cheating the camera in thinking it is darker(not true bv values stay the same) uses a lower ISO than it actually is thus moving the exposure to a longer shot earlyer than expected.
using a multiplier on the calculated exposure time at set timings before shoot would probably also do it.
after some tests I can do the other way around, so cheating the calculated exposure by using an available higher iso and so shifting 1sec exposure to bv-500 or even lower bv values (bv-768 or darker @ max 1/4sec exposure @ any ISO you set before you started)....but that is completely opposite to what i want. This way in the images it gets darker more quickly. (alas a lower iso is lost in the dark)
and, more importantly, for my purpose it is too much to change 1 stop or more during a shoot (1/3 stop iso values, like 250 and 320 didn't gave stable results in my tests, but maybe i should do more).
it creates a too big a shift in how the image is illuminated..the actual shift should be no more than 1/3 stop per exposure time (so f.i. 1/3 shift at exposure 1/60, 1/30 and at 1/10, creating 1 full stop difference between 1/60 and 1 sec.)
btw this doesn't help astrophotography since the camera is still bound to it's absolute maximum exposure/iso values
and if you are bothered with +1 on the left, i can always shift it to the right side as -1 :D
-
I want the taken pictures to get dark more slowly between something like bv200 to bv-300....for now limited to 1 sec, that could be 4sec, 2 stops further if the camera and I permit that.
That makes sense. This is what I created Bv/Ev shift for, but it operates somewhat differently: If you set the faction to say, 33% and base to 10 (daylight) then for every stop change in Bv away from broad daylight, there will be a 1/3 stop change in exposure in the same direction, subject to the limits imposed by other settings.
In practice, the scene won't get much brighter if Bv goes over the "base" value, because "correct" exposure is quite close to the point where the sensor clips, and the over exposure limits are set to keep things below that. However, rawopint might be more prone to flickering if it's bouncing off the limits.
since the original calculation Av+Tv=bv+sv results in a specific exposure my first thought was to change that formula so it would allow for a longer exposure.
The equation describes the values will produce "correct" exposure. If you want to diverge from "correct" exposure by some amount, you just have to add or subtract whatever you want from the terms you can control. Adding to Tv or Av is a negative "ev shift", i.e. shorter exposure or smaller aperture. Adding to Sv is positive shift (higher ISO).
sort of cheating the camera in thinking it is darker(not true bv values stay the same)
To be clear, the APEX equation isn't code that we control in the camera, it's just a convenient way of expressing the relationship between the values. rawopint derives the logged "Bv" from the exposure parameters and how much the "meter" value diverges from the "neutral" value.
Depending on your workflow, you might consider different approach:
Instead of trying to make the script capture what you want to see in the video, expose everything as "correctly" (i.e. as bright as possible without clipping) and manage brightness in post processing based on logged Bv. Making a well exposed image darker in post will give you a better SNR than an under-exposed one, and you can choose how much you want to fade for a real brightness change after the fact. This would likely only make sense if you are shooting raw and have a workflow that can be programmed to digest data from the CSV files.
-
Depending on your workflow, you might consider different approach:
Instead of trying to make the script capture what you want to see in the video, expose everything as "correctly" (i.e. as bright as possible without clipping) and manage brightness in post processing based on logged Bv. Making a well exposed image darker in post will give you a better SNR than an under-exposed one, and you can choose how much you want to fade for a real brightness change after the fact. This would likely only make sense if you are shooting raw and have a workflow that can be programmed to digest data from the CSV files.
you are absolutely right in this but it will take up some time to realize that. luckely i've got some 3 month before i have a purpose for all i'm doing now...although i am also eager to use your snippet in another script.
so i just put on a test cam to see how it handels dawn(an a480, i hate to wear down the m10 for naught)
for rawopint my settings are 30% and base 10 at this moment and that seemed to work fairly well....
-
Instead of trying to make the script capture what you want to see in the video, expose everything as "correctly" (i.e. as bright as possible without clipping) and manage brightness in post processing based on logged Bv.
That is also my way I go. I do it similarly as described here
https://chdk.setepontos.com/index.php?topic=12790.0
Making a well exposed image darker in post will give you a better SNR than an under-exposed one, and you can choose how much you want to fade for a real brightness change after the fact.
This is the right way
-
Making a well exposed image darker in post will give you a better SNR than an under-exposed one, and you can choose how much you want to fade for a real brightness change after the fact.
This is the right way
Being slightly pedantic, I would say it's the way with the highest potential image quality. Whether that is "right" depends on the user requirements and capability.
Slapping together a timelapse of medium res jpegs with no post processing at all is perfectly reasonable for some users, and rawopint can give pretty decent results in this case.
-
Being slightly pedantic, I would say it's the way with the highest potential image quality. Whether that is "right" depends on the user requirements and capability.
Slapping together a timelapse of medium res jpegs with no post processing at all is perfectly reasonable for some users, and rawopint can give pretty decent results in this case.
I didn't want to go there, but just assumed Right Way was spelled with capitals......if you are a believer this is the Thruth.
it all depends, yes.
I've been making timelapse, stop-motion and animation for over 10 years now and my methods change sometimes per project depending on what is feasable at that moment.
at first i used an external trigger.
but after chdk 0.7 or so i placed it on my a480, everybody thought i was a nutter leaving the Right Way behind..I still have that a480 today
although i really had to update that software to 1.5 last year, i still have the 0.7 so i can always revert back ;)
and i actually don't have an external trigger anymore..it's on my "todo" list for a few years now: building an arduino based trigger.
my projects might include raw, but especially for longer, high volume shoots, i don't consider that an option.
it's not that i don't like the quality or possibilitys, the data is just too much...with mostly jpg I already have >10TB dedicated for storage and throw away old/rendered material weekly because i need the space
furthermore, if it ends up in a hd movie the added value of raw vs jpg is minimal in my opinion.
about processing csv. well, actually i use UI mostly because it suits my need best and that has a ssv instead of csv. and i haven't really found looked for a way to script process those logs reliable. i just haven't gotten around to that in the past 10 years.....but as soon as i find time...i'll do something else first
About post-processing I feel the same..it is unavoidable. but I really like to minimize post processing for things I could have solved previously on the cam.
-
Being slightly pedantic, I would say it's the way with the highest potential image quality. Whether that is "right" depends on the user requirements and capability.
That's what I meant by that. I probably did not express myself correctly.
Slapping together a timelapse of medium res jpegs with no post processing at all is perfectly reasonable for some users, and rawopint can give pretty decent results in this case.
Of course, in 80% of the cases, I do not need any further post-processing. The script already delivers fantastic results.
furthermore, if it ends up in a hd movie the added value of raw vs jpg is minimal in my opinion.
I am not convinced. Especially when shooting directly into the sun, the differences are already very large. Either the sun is burned out or the shadows are too dark. Since the differences are enormous if you take RAW or JPG. Even a totally wrong white balance cannot be resolved with JPG.
it's not that i don't like the quality or possibilitys, the data is just too much...with mostly jpg I already have >10TB dedicated for storage and throw away old/rendered material weekly because i need the space
The memory was not going to increase if you save the developed JPG's from the RAW's. That's how I do it in most cases. The RAW data are usually only temporary. Only my best scenes are saved as RAW.
-
i just don't have the equipment or resources to spend. but if you are willing to sponsor me...send those 1tb sd cards my way! 8)
-
i just don't have the equipment or resources to spend. but if you are willing to sponsor me
I would first start clean up at the 10TB :D
...send those 1tb sd cards my way! 8)
This is all pretty off topic now here ...
I just bought a 128G card for 15 €. Since fit with the G1x about 6000 RAW's on it. I would like to see a time lapse, where this limit is exceeded and it does not get boring.
I do not guess a lapser with his 600 videos has exceeded this limit.
https://www.youtube.com/user/DrLapser
-
I just bought a 128G card for 15 €.
that is a good price.
-
I lowered the Nd value apex 96 to 192 for the M10 and the floating is gone....for some reason it took a few tries before it sticked.
-
I lowered the Nd value apex 96 to 192 for the M10 and the floating is gone....for some reason it took a few tries before it sticked.
I'm not sure what happened, but that really shouldn't work. The M10 does not have a camera controllable ND filter, and the script should not attempt to do anything with the ND on ports that are configured without one.
-
I'm not sure what happened, but that really shouldn't work. The M10 does not have a camera controllable ND filter, and the script should not attempt to do anything with the ND on ports that are configured without one.
I'm not sure either, but you can see it between the first restart and second restart in the log. at first restart i just stopped the script and restarted. the second restart i booted the camera (nd changed back to 288)
But i am also not sure why rawopint would make my sensor so much hotter than canon's firmware or why nobody else sees that...to me those temp levels look disturbing.
-
But i am also not sure why rawopint would make my sensor so much hotter than canon's firmware or why nobody else sees that...to me those temp levels look disturbing.
I'm not sure what you mean by this, did you post more detail somewhere?
Shooting rapidly is expected to run hotter than idling.
-
maybe I'm mistaken and it is normal for an aps-c sensor, but the log i posted a while back had tsens of over 50C, the log from a few days back still had 45C. although iso was only 200.
The interval was 30 sec, so i can't imagine that is quick enough to heat it up that much.
-
The interval was 30 sec, so i can't imagine that is quick enough to heat it up that much.
It does not matter which interval you take. The sensor is heated by the constant reading of the sensor. Even if you do not take pictures, the sensor gets hot.
https://chdk.setepontos.com/index.php?topic=13319.0
-
thanks
-
maybe I'm mistaken and it is normal for an aps-c sensor, but the log i posted a while back had tsens of over 50C, the log from a few days back still had 45C. although iso was only 200.
The interval was 30 sec, so i can't imagine that is quick enough to heat it up that much.
FWIW, I wouldn't be surprised to see >50c in warm ambient conditions with a script like rawopint. (lack of surprise should not be taken as any judgement on whether it's healthy or not ;))
A user reported problems around 80c https://chdk.setepontos.com/index.php?topic=13501.0 with cameras in an enclosure.
-
FWIW, I wouldn't be surprised to see >50c in warm ambient conditions with a script like rawopint. (lack of surprise should not be taken as any judgement on whether it's healthy or not ;))
I have logs from past summer stating that for another cam, so i wouldn't be either. but it is winter overhere and my other cams are all below 30C. So i'll just keep an eye on it and hope for the best.
-
Just for info:
My SX50 crashes with rawopint when I attach a LED lamp in the hot shoe. This happens only with an interval > 10s.
-
My SX50 crashes with rawopint when I attach a LED lamp in the hot shoe. This happens only with an interval > 10s.
I would not be surprised if that turned out to be a firmware bug. Try reproducing it without CHDK, if you're interested.
You might be able to work it around by enabling hotshoe override in the enhanced photo operations menu.
-
Try reproducing it without CHDK, if you're interested.
How?
A simple interval script or one that stays in Half Press has no problems with 20s interval.
You might be able to work it around by enabling hotshoe override in the enhanced photo operations menu.
It works. Thanks…
-
On my EOS M100 I can shoot in continuous mode with an interval from 0.1s (up to 20 images).
After 20 images it goes down to 0.35s.
With rawopint the maximum interval is only 0.7s.
I try to optimize the parameter but that didn’t help.
Any idea where the time remains?
-
On my EOS M100 I can shoot in continuous mode with an interval from 0.1s (up to 20 images).
After 20 images it goes down to 0.35s.
With rawopint the maximum interval is only 0.7s.
I try to optimize the parameter but that didn’t help.
Any idea where the time remains?
Not really. Unless you've accidentally enabled CHDK raw (the script option only controls CHDK raw, not native Canon raw), I guess something about the way CHDK hooks work doesn't allow it to run at the full continuous speed.
If you haven't already, you could try "quick" mode. Normally it's slower, but if cont isn't getting the full speed who knows...
-
Unless you've accidentally enabled CHDK raw (the script option only controls CHDK raw, not native Canon raw),
CHDK RAW is disabled ;)
If you haven't already, you could try "quick" mode. Normally it's slower, but if cont isn't getting the full speed who knows...
It is nearly the same…
I guess something about the way CHDK hooks work doesn't allow it to run at the full continuous speed.
It looks like that.
The M3 is usually significantly slower. Also only has a buffer for 3 RAW. But M3 manages the 3 RAWs in 0.3s
-
I guess something about the way CHDK hooks work doesn't allow it to run at the full continuous speed.
Not only that. I once took a simple test script.
The quick mode is faster than the continuous mode.
Quick Mode => 0.53s
Continuous Mode => 0.65s
This is only the case with the M100. With all my other cameras the continuous mode is faster or the same as the quick mode.
-
Not only that. I once took a simple test script.
The quick mode is faster than the continuous mode.
Quick Mode => 0.53s
Continuous Mode => 0.65s
This is only the case with the M100. With all my other cameras the continuous mode is faster or the same as the quick mode.
Two general possibilities:
1) The hooks aren't placed equivalently to where they are in other ports, making the cont technique not work equivalently
2) Digic 7 is different in a way that makes the technique not work equivalently.
It might be possible to distinguish between them with reverse engineering, but either way, it likely has more to do with the port than the script.
Or, the cause could be something totally unexpected ;)
-
it likely has more to do with the port than the script.
I understand that ...
I am also not unhappy with a 0.8s interval. I wouldn't expect my camera to do this for long either.
-
I'm just thinking about an interval ramping.
With a starry sky you need an interval of about 30s (with Tv = 25s). At this interval, however, the sunset becomes too fast.
My idea:
If Tv Max > interval, the interval ramping should be active.
If the current exposure time > (interval - 2s), then the interval should start ramping.
The interval should then increase continuously up to Tv Max + 2s.
Once started, the ramping should only go in one direction (e.g. with 0.1s per recording).
The other direction (reduction) is only started when the maximum interval has been active for a certain time and the exposure time is below the start interval.
If I've seen that correctly, then you can increase the interval in the following loop, right?
for i=1,ui_shots do
What do you do with the following command before the loop? Set on the maximum interval?
hook_shoot.set(2000+interval)
I would deactivate interval_warn_led and interval_warn_beep in this case.
-
If I've seen that correctly, then you can increase the interval in the following loop, right?
for i=1,ui_shots do
Yes, I think that should be fine.
What do you do with the following command before the loop? Set on the maximum interval?
hook_shoot.set(2000+interval)
That's just to avoid timeouts waiting in the hooks, so it should be fine. You could also set it inside the loop (after hook_shoot.continue()), but there shouldn't be any need.
-
I now had a situation where the script really started swinging. I couldn't even fix that in Lightroom anymore.
-
:blink: Maybe something like "lens flare corruption" ?
-
I now had a situation where the script really started swinging. I couldn't even fix that in Lightroom anymore.
It seems like it's driven over exposure. Starting at row 111 (exp 1043), over_frac is 0.9, so overall exposure is pushed below the normal target (meter96 is negative)
Next frame it's 0, and d_ev is +17 (because meter is below target, and over exposure is no longer balancing), next frame, over_frac is still 0, d_ev is +21. Next frame, over_frac is 1.6, d_ev is 0
Same pattern repeats a few times.
Why did over-exposure drop to 0? I don't know, would probably have to look at the individual frames. But from the scene, there could have been quite rapid real changes in overexposure.
I'm not sure what settings would mitigate this. Setting the ev range considered over exposure wider (Overexp Ev range) and allowing a larger fraction might smooth the apparent changes.
Or maybe the code should be changed to smooth more aggressively.
-
Why did over-exposure drop to 0? I don't know, would probably have to look at the individua frames.
I could upload RAW if you want...
I have now marked all parts of the image in red, where the RAW values are> 15000. As you can see, there is a large plateau there. Small EV changes ensure that the area is sometimes overexposed and sometimes not. Maybe ev_change_max = 32 was too big.
-
I have now marked all parts of the image in red, where the RAW values are> 15000. As you can see, there is a large plateau there. Small EV changes ensure that the area is sometimes overexposed and sometimes not.
Yeah, that seems likely to be the explanation.
Maybe ev_change_max = 32 was too big.
IIRC, I didn't pick that as the default based on any real analysis, it was a guess and it seemed to work most of the time. The minimum does need to be big enough to keep up with the fastest sustained change, and obviously smaller will make it react more slowly to real changes.
That said, I think the script has always had a tendency to get into oscillations in cases like this, and probably needs different logic to handle it well. In fact, I see a comment
-- TODO
-- to avoid flapping as limits approached over / under weights
:-[
Perhaps a way to deal with this would be to account for the fraction near over exposure somehow or apply varying weight depending how deep into exp_over_margin it is, effectively dividing the "overexposed" bucket in the histogram into multiple buckets with different weight. Or perhaps the "smoothing" should give more weight to the longer term trend, but I think this problem is specific to how the histogram based limits work.
As always, adding more complexity will make it harder to test.
An idle thought: With something like the chdkptp remoteshoot glue, it would be possible to create a GUI that lets you visualize what the script is doing and change settings in real time.
-
As always, adding more complexity will make it harder to test.
Yes, of course, but this case is the absolute exception. I haven't had this case in a long time.
An idle thought: With something like the chdkptp remoteshoot glue, it would be possible to create a GUI that lets you visualize what the script is doing and change settings in real time.
Situations where clouds keep pushing in front of the sun are extremely difficult. I had already considered another option: to deactivate the exposure control for a certain time using an external USB switch or to only allow a control for overexposure.
-
Situations where clouds keep pushing in front of the sun are extremely difficult.
Agreed. Reflection of that on water too. But I think there is room for improvement. I have a few other changes I should finish now that ND control is in CHDK stable, so maybe I'll play with it a bit. I think slowing down the change if a lot of pixels are near the over-exposure threshold could work quite well. Right now (at least if I'm interpreting my old code correctly :-[) it doesn't distinguish if the "overexposed" is just over the threshold or totally saturated. If it did, it could be less aggressive in the first case.
I had already considered another option: to deactivate the exposure control for a certain time using an external USB switch or to only allow a control for overexposure.
Yes, my thought of interactive control was more for development/testing than for actual shoots.
edit:
I think I may have come up with an approach to deal with the overexposure problem properly.
One of the difficulties with overexposure is you can't predict how much a given exposure change will affect it: That is, if you have saturated pixels, you can't predict what exposure change is needed to bring them below saturation. The Sun may barely change with 5 stops, where a cloud might be completely unsaturated with a quarter stop.
However, this isn't true in the other direction: For a given positive exposure change N, all pixels with values above (overexposure threshold - N) will pass the threshold, and this number can be easily obtained from the histogram. So the exposure algorithm can take the exposure change calculated from other inputs, calculate how many additional pixels would be pushed into over-exposure, and adjust accordingly to keep it below the limit.
In practice, some knowledge in the other direction should be possible too: The script uses a range (default 1/4 stop below white level) for overexposure, so (+/- uncertainty how the sensor behaves very near saturation) it can estimate a minimum number of pixels that would be brought below the threshold, or even calculate it exactly if the exposure change significantly smaller than the range.
This won't be trivial to integrate into the existing algorithm, but I think it's the right approach, and may allow eliminating some of the other hacks I introduced trying to stop it from bouncing around the limits.
-
However, this isn't true in the other direction: For a given positive exposure change N, all pixels with values above (overexposure threshold - N) will pass the threshold, and this number can be easily obtained from the histogram. So the exposure algorithm can take the exposure change calculated from other inputs, calculate how many additional pixels would be pushed into over-exposure, and adjust accordingly to keep it below the limit.
That sounds like a very good idea.
That would definitely also help with a time lapse from the moon ...
-
Hallo c_joerg, hello reyalp,
I have tested now for some days a Canon G1X (100.e) with CHDK (g1x-100e-1.5.1-5908-full). CHDK works very well. Also CHDKPTP works flawlessly. The camera seems quite a good improvement over the Canon SX100IS.
I do have a problem which I'm not able to solve. :(
When I start the rawopint v0.25 (downloaded last version form this side: https://chdk.fandom.com/wiki/Lua/Scripts:_Raw_Meter_Intervalometer) on the camera or via CHDKPTP with a command like:
rs "D:/sunset/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=3 -int=12the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series. There is a little noise when the camera goes off which is the IS unit dropping. No lens retracting, no AV change, no "shutter" movement. The camera simply goes off.
I tried a lot of different settings but nothing changed this misbehaviour at all. If the shutter speed is quite high, e.g. 10 sec the camera does the same misbehaviour but starting at a interval of 10+12=22 seconds.
On the Canon SX100IS this phenomenon could not be observed.
If I use the normal rs "D:/sunset/input/" -shots=3 -int=12 command everything works fine.
Maybe c_joerg has a script that works on his G1X he can share with me, so at least I would now it is not the script. ;)
I would really appreciate any help or ideas to find the "bug"! :)
-
the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series. There is a little noise when the camera goes off which is the IS unit dropping. No lens retracting, no AV change, no "shutter" movement. The camera simply goes off.
I tried a lot of different settings but nothing changed this misbehaviour at all. If the shutter speed is quite high, e.g. 10 sec the camera does the same misbehaviour but starting at a interval of 10+12=22 seconds.
The camera shutting off usually means it crashed. Can you get a romlog: https://chdk.fandom.com/wiki/Debugging#Camera_crash_logs_.28romlog.29 ?
Note you can use chdkptp to get the romlog if you don't have physical access to the camera:
!require'extras/devutil'.init_cli() -- add debug commands, only needed once per session
dromlog
I suspect this crash probably has something to do with waiting in the remote hook. You can try putting some of the delay in the raw hook instead, by setting ui_raw_hook_sleep. The menu currently only allows up to 100ms, but if you set it from the glue script, you should be able to use values up to about 9000 (it needs to be less than the raw hook timeout, which is currently set to 10s).
If this works, I can adjust the script to do the majority of the wait in the raw hook and only use the remote hook to keep the interval precise.
Note that ui_raw_hook_sleep doesn't add to the interval. It effectively makes the script act like shooting took longer, so in order to maintain a the interval set with -int, it must be shorter than the interval and all the overhead of shooting, metering etc.
So if the theory is correct, you could use something like -int=20 and ui_raw_hook_sleep=9000
-
the script works fine (on camera and via CHDKPTP) for less than or equal to -int=11 but for greater or equal than -int=12 (12seconds, value 120 in camera) the camera simply goes off in that moment where the camera should take the second picture of the series.
I’m actual running 1.5.0 5399 on my G1x 100e. I never run rawopint together with CHDKPTP.
What does it mean ‘on camera’? Running the script without CHDKPTP and starting script from SD card?
If I run rawopint v0.25 with 12s interval (value 120) I have no problems. All other params are default.
-
I’m actual running 1.5.0 5399 on my G1x 100e. I never run rawopint together with CHDKPTP.
What does it mean ‘on camera’? Running the script without CHDKPTP and starting script from SD card?
If I run rawopint v0.25 with 12s interval (value 120) I have no problems. All other params are default.
Thank you for your replay! :)
‘on camera’ mean starting the script from SD card.
Interesting. Maybe I should try 1.5.0 5399...
-
The camera shutting off usually means it crashed. Can you get a romlog: https://chdk.fandom.com/wiki/Debugging#Camera_crash_logs_.28romlog.29 ?
Thank you very much for your fast replay!
I did 2 identical runs with 12 sec (the script was started with CHDKPTP)
Attached both ROMLOG since the they are quite different.
-
I suspect this crash probably has something to do with waiting in the remote hook. You can try putting some of the delay in the raw hook instead, by setting ui_raw_hook_sleep. The menu currently only allows up to 100ms, but if you set it from the glue script, you should be able to use values up to about 9000 (it needs to be less than the raw hook timeout, which is currently set to 10s).
If this works, I can adjust the script to do the majority of the wait in the raw hook and only use the remote hook to keep the interval precise.
Note that ui_raw_hook_sleep doesn't add to the interval. It effectively makes the script act like shooting took longer, so in order to maintain a the interval set with -int, it must be shorter than the interval and all the overhead of shooting, metering etc.
So if the theory is correct, you could use something like -int=20 and ui_raw_hook_sleep=9000
Yes, thank you! It kind of worked. :D
I tried out with ui_raw_hook_sleep=8000 in the glue script.
-int=12 worked fine but -int=15 again crashed the camera. I repeated the run for both intervals and the behaviour remained the same.
I attached the ROMLOG of one of the 2 runs with -int=15
-
Interesting. Maybe I should try 1.5.0 5399...
I update now to latest 1.6.0. Also, no problem with 12s.
Which Mode do you use? Flash?
-
I update now to latest 1.6.0. Also, no problem with 12s.
Which Mode do you use? Flash?
Hmm.. strange..
I did the last view test in Av Mode (set in CHDKPTP, the Camera was in M Mode). I use AV mode for sunset and M mode for sunrise.
EDIT:
I changed the ui_raw_hook_sleep=9000 and now I was able to shoot up to exactly 20 without shut down and with 21 the camera again went off.
also with latest (g1x-100e-1.6.0-5910-full) same problem.
-
I changed the ui_raw_hook_sleep=9000 and now I was able to shoot up to exactly 20 without shut down and with 21 the camera again went off.
I am also wonder that I doesn't have to change ui_raw_hook_sleep.
Maybe another parameter (IS, Driving mode, RAW+JPG,…)
-
I update now to latest 1.6.0. Also, no problem with 12s.
Which Mode do you use? Flash?
Hmm.. strange..
I did the last view test in Av Mode (set in CHDKPTP, the Camera was in M Mode). I use AV mode for sunset and M mode for sunrise.
EDIT:
I changed the ui_raw_hook_sleep=9000 and now I was able to shoot up to exactly 20 without shut down and with 21 the camera again went off.
It looks like waiting ~10s in the remote hook is the problem. The script calculates how long it needs to wait based on how much time all the work (shooting, processing the raw data, saving the file) took, so you probably want to leave some margin for variation.
But, this seems to confirm moving long waits to the raw hook probably addresses the problem, so the script could be updated to allow arbitrary lengths, as long as there's not some similar problem in the raw hook. I think I actually considered this early on, because blocking for a long time in the remote hook is quite sketchy, but didn't bother because it seemed to work.
You should be able allow arbitrary ui_raw_hook_sleep by changing the line (1991 in rawopint 0.25)
hook_raw.set(10000)
to
hook_raw.set(10000 + ui_raw_hook_sleep)
That said, rawopint metering will degrade with long intervals, since it bases exposure on the previous shot. For this kind of interval, it might be better to use camera AE, or take (and ideally, discard) a metering shot immediately before the real shot.
The first two romlogs refer to CntFlashTask0, which I don't recall seeing before. The name suggests it's either related to flash (strobe) mode, as c_joerg suggested, or possibly flash memory.
IIRC the MotionVector.c one has been seen before in connection with the remote hook. A quick search found https://chdk.setepontos.com/index.php?topic=13893.msg141682#msg141682 but I'll have to do some digging to find the details.
-
The first two romlogs refer to CntFlashTask0, which I don't recall seeing before. The name suggests it's either related to flash (strobe) mode, as c_joerg suggested, or possibly flash memory.
IIRC the MotionVector.c one has been seen before in connection with the remote hook. A quick search found https://chdk.setepontos.com/index.php?topic=13893.msg141682#msg141682 but I'll have to do some digging to find the details.
Strange... I just tried it now but I can't access the flash settings (via menu or func.set) no matter what I try: flash retracted, popped up, with/without CHDK?! In Auto, the flash is not used either. Since I'm not using the flash for time-lapse, I don't care, but could that be why the camera is crashing?
What does CntFlashTask0 mean? is that a CHDK setting? By flash mode, do you mean when the flash is popped up?
I will try this out:
hook_raw.set(10000 + ui_raw_hook_sleep)
Thank you for this "fast fix" ;)
I think the metering is not an issue as I don't want big ev changes to avoid flicker. normally ui_max_ev_change_e is set to 1.
EDIT
:o :o
OMG: the reason was: the little plastic cover plate of the hot shoe... after I took it of I was able to access the flash settings... LOL
HAHA
I will try everything from scratch again... :haha
-
Strange... I just tried it now but I can't access the flash settings (via menu or func.set) no matter what I try: flash retracted, popped up, with/without CHDK?
Is there something in the hot shoe? Using the hot shoe is known to affect CHDK in some cases, IIRC including where the Canon firmware thinks there's something there because of a cover or stuck switch.
CHDK on g1x has a hot shoe override setting to force the camera to see particular state if you need it.
edit: :lol
What does CntFlashTask0 mean? is that a CHDK setting?
It's a task (like thread or process) in the Canon firmware.
I think the metering is not an issue as I don't want big ev changes to avoid flicker. normally ui_max_ev_change_e is set to 1.
It will be behind, so if there's like clouds moving in front of the sun, the script will react to that 20 seconds later, which could easily be completely the wrong direction. Of course this can happen with short intervals too, but it's much less likely in this kind of scene.
-
Is there something in the hot shoe? Using the hot shoe is known to affect CHDK in some cases, IIRC including where the Canon firmware thinks there's something there because of a cover or stuck switch.
CHDK on g1x has a hot shoe override setting to force the camera to see particular state if you need it.
edit: :lol
What does CntFlashTask0 mean? is that a CHDK setting?
It's a task (like thread or process) in the Canon firmware.
I think the metering is not an issue as I don't want big ev changes to avoid flicker. normally ui_max_ev_change_e is set to 1.
It will be behind, so if there's like clouds moving in front of the sun, the script will react to that 20 seconds later, which could easily be completely the wrong direction. Of course this can happen with short intervals too, but it's much less likely in this kind of scene.
Yes, a little plastic cap covering the contacts of the hot shoe (micro switch) :-[
Everything now works as it should. Both using CHDKPTP and directly running the script on the camera now works with intervals higher than 11 seconds.
I owe you both a beer, do you have paypal?
At least today's sunset was one of the best. :D I reduced the quality of the video to bring it below the 2MB limit (and then converted it from mp4 to mpg, which completely destroyed the remaining quality).
shot with Canon SX100IS and rawopint.lua, slightly overexposed for my taste.
-
Yes, a little plastic cap covering the contacts of the hot shoe (micro switch) :-[
Everything now works as it should. Both using CHDKPTP and directly running the script on the camera now works with intervals higher than 11 seconds.
Just to confirm, it works *without* the ui_raw_hook_sleep modification?
I owe you both a beer, do you have paypal?
Honestly, seeing people do cool things with it is the best reward for working on this stuff.
At least today's sunset was one of the best. :D I reduced the quality of the video to bring it below the 2MB limit (and then converted it from mp4 to mpg, which completely destroyed the remaining quality).
shot with Canon SX100IS and rawopint.lua, slightly overexposed for my taste.
Very nice. Have you thought about posting these outside IG?
The moon + clouds definitely gets bright at the end. May be able to offer suggestions if you post the log file.
-
OMG: the reason was: the little plastic cover plate of the hot shoe... after I took it of I was able to access the flash settings... LOL
Interesting…
Yes, a little plastic cap covering the contacts of the hot shoe (micro switch) :-[
But it's not an original part of the G1x, is it?
Could be a problem with my timelapse dolly. I haven't used it with 12s yet. https://chdk.setepontos.com/index.php?topic=13726.0
-
But it's not an original part of the G1x, is it?
Could be a problem with my timelapse dolly. I haven't used it with 12s yet. https://chdk.setepontos.com/index.php?topic=13726.0
Yes it is. You should have this part too.
Attached is a photo with the plastic cap in place. And a detail of the microswitch that may have caused the problem.
Maybe you can try this out too on your G1X. You may use just a toothpick if you can't find the original plastic cap to trigger the micro switch, then we could confirm this behavior on a second G1X. :)
-
Yes it is. You should have this part too.
I have two G1x and no cap. But I bought them used.
The original cap of my M3 has a notch so that the switch is not switched.
I don't think your cap originally belongs to the G1x. It makes no sense for me to removing the cap when using the internal flash.
-
Just to confirm, it works *without* the ui_raw_hook_sleep modification?
Yes, new firmware (g1x-100e-1.6.0-5910-full) and new rawopint v0.25 script copied to the Canon G1X.
And the script works just fine out of the box, without the little plastic cover plate.
Honestly, seeing people do cool things with it is the best reward for working on this stuff.
Nice to hear that. And nice to see that this little time lapse project also finds favor here.
Very nice. Have you thought about posting these outside IG?
I actually don't like the facebook company... I don't use Whatsapp or Facebook nor have a private IG account but I found a cool python Instagram Private API (https://github.com/adw0rd/instagrapi) and so I started with IG.
What other platforms would you suggest?
The moon + clouds definitely gets bright at the end. May be able to offer suggestions if you post the log file.
Yes I will post the log. I think the "Moon flare" comes from the optically bad window the camera has to photograph trough... yeah.. still room for improvements, such as removing the window :lol
Some background info:
The sunrise is in M mode. this is because in that way the script is able to start whit tv ~ 6 seconds
The sunset is in AV mode. this is because the amount of daylight can be very different. The script always start at 0 EV and than goes to the -1.1/4 i have set in the glue script (ui_ev_shift_e=5) which si not much of an Issue. I could just skip the first 5 images, but in the videos its not quite visible.
-
For better understanding, attached the rawopint_rs.lua file and the chdkptp scripts.
sunset:
clock -sync
luar set_lcd_display(0) --switch of display during taking photos
luar set_mf(1) --set focus to manual
luar set_focus(1495) --set focus distance
luar set_capture_mode(4) --set to AV mode
luar set_user_av96(320) --set AV value
#AV modes: 3.2 3.5 4.0 4.5 5.0 5.6 6.3 7.1 8.0
# 320 350 380 410 440 470 500 530 560
luar set_ev(-96) --set EV value
#EV modes: -192 -160 -128 -96 -64 -32 0 32 64 96 128 160 192
# -2 -12/3 -11/3 -1 -2/3 -1/3 0 1/3 2/3 1 11/3 12/3 2
luar set_iso_mode(80) --set ISO
luar set_raw_nr(0) --auto dark frame
rs "D:/sunset/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=600 -int=15
luar sleep(2000)
d A/rawopint.csv D:/sunset/
sunrise:
clock -sync
luar set_lcd_display(0) --switch of display during taking photos
luar set_mf(1) --set focus to manual
luar set_focus(1495) --set focus distance
luar set_capture_mode(5) --set to manual mode
luar set_user_av96(320) --set AV value
#AV modes: 3.2 3.5 4.0 4.5 5.0 5.6 6.3 7.1 8.0
# 320 350 380 410 440 470 500 530 560
luar set_tv96_direct(-256) --set TV value to 6.3s
luar set_iso_mode(200) --set ISO
luar set_raw_nr(0) --auto dark frame
rs "D:/sunrise/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=600 -int=15
luar sleep(2000)
d A/rawopint.csv D:/sunrise/
-
I have two G1x and no cap. But I bought them used.
The original cap of my M3 has a notch so that the switch is not switched.
I don't think your cap originally belongs to the G1x. It makes no sense for me to removing the cap when using the internal flash.
You might be right. At least what you say makes more sense to me.
-
I actually don't like the facebook company... I don't use Whatsapp or Facebook nor have a private IG account but I found a cool python Instagram Private API (https://github.com/adw0rd/instagrapi) and so I started with IG.
What other platforms would you suggest?
Hmm, youtube would let people see full resolution, without the square letter box. Even 4k once you get the g1x set up. But probably not something people would regularly watch unless it was embedded somewhere else.
2x daily short videos could work well as a twitter bot, and code for those should be readily available, but the video quality is pretty bad.
Yes I will post the log.
I took a look... and found a bunch of bugs in my code :-[
You may have noticed the log is spammed with "bv ev thresh:0" and the bv_ev_shift column is -12, even though you have bv_ev_pct 0, which should disable it. This is related to using the initial ev shift (ui_ev_shift_e=5, = -1.25 stops) being larger magnitude than the meter high threshold (ui_meter_high_thresh_e=2, = 1 stop)
I *think* the net effect in this case is the target exposure will be reduced by a further 1/8th stop, but it's been a long time since I wrote that code.
With that out of the way, a few other observations on the 2021_05_25_sunset log:
Despite that big blown-out looking area around the moon at the end, the over_weight column is all zero, so overexposure as measured by the script never got to the point where it influenced exposure. The highest over exposure was at the very end, around 0.17%. If I've done the math right that would be about ~120 pixels square, or roughly 3x the size of the moon at the reported 43.8 EFL.
If you want the moon to trigger overexposure protection, you'd need a much lower over_thresh_frac, 0.1% or lower. Beware that setting this very low can trigger oscillations and run into precision issues. And of course, if you did correctly expose the moon at night, the rest would get very dark.
If you want more of the bright area to be considered overexposed, you could increase over_margin_ev
Exposure was mostly driven by underexposure, with under_frac ranging from 8% to 40%, and under_weight never below 65. Given that the bottom ~half of your scene is going to be very dark a lot of the time, you probably want to keep the influence of underexposure fairly low. You could turn it off completely by setting under_thresh_frac to 0, or some combination of increased under_margin_ev and under_thresh_frac.
For a scene like this, it might be nice to have the meter not measure the bottom, to bias the exposure for the sky and snowy peaks, but the script currently only allows setting size, not position.
Updated notebook and htmlized output for the sunset log attached.
-
Hmm, youtube would let people see full resolution, without the square letter box. Even 4k once you get the g1x set up. But probably not something people would regularly watch unless it was embedded somewhere else.
Yes, an upload to YouTube will be a future project too. I believe hardly anyone will be intersted in this timelapse but I want to use it as cloud backup ;) . Actually a cloud backup is also on my list. The problem here is, that the internet is at the moment way too slow for uploading ~100MB of videos. So I think I will have to delay this too. (sometimes the time lapse computer has to deal with 0,2-0,4MBit/s of upload speed, also remote access is a mess :-[ there is a copper wiring issue... and the street will not get fiber before 2023...)
Despite that big blown-out looking area around the moon at the end, the over_weight column is all zero, so overexposure as measured by the script never got to the point where it influenced exposure. The highest over exposure was at the very end, around 0.17%. If I've done the math right that would be about ~120 pixels square, or roughly 3x the size of the moon at the reported 43.8 EFL.
If you want the moon to trigger overexposure protection, you'd need a much lower over_thresh_frac, 0.1% or lower. Beware that setting this very low can trigger oscillations and run into precision issues. And of course, if you did correctly expose the moon at night, the rest would get very dark.
I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment...
Since three exposure times (inkl dark frame) would get too long for HDR, I think it is good enough for the moment. Also HDR would be quite difficult with the flare of the window glass. The G1X should make a difference though. The real problem I see is (since the over_weight parameter always remains the same.) in daylight condition the overexposure could trigger too easily with some bright clouds or snow reflection.
Maybe the highlight clipping in clouds and snow are a better way to define over_thresh_frac and over_margin_ev.
If you want more of the bright area to be considered overexposed, you could increase over_margin_ev
Exposure was mostly driven by underexposure, with under_frac ranging from 8% to 40%, and under_weight never below 65. Given that the bottom ~half of your scene is going to be very dark a lot of the time, you probably want to keep the influence of underexposure fairly low. You could turn it off completely by setting under_thresh_frac to 0, or some combination of increased under_margin_ev and under_thresh_frac.
For a scene like this, it might be nice to have the meter not measure the bottom, to bias the exposure for the sky and snowy peaks, but the script currently only allows setting size, not position.
hmm.. I already did that: the time lapse in the final videos includes only the upper 2/3 of the frame. (I cut the village out of the picture.) And because the missing part did influence too much the metering I excluded it.
line 703ff
-- meter rectangle, centered in active area
self.meter_left = rawop.get_active_left() + rawop.get_active_width()/2 - self.meter_width/2
self.meter_x_count = self.meter_width/self.meter_step
self.meter_top = rawop.get_active_top() + rawop.get_active_height()/2 - self.meter_height/2
self.meter_y_count = self.meter_height/self.meter_step/3 --only upper 2/3 counts
Updated notebook and htmlized output for the sunset log attached.
Thank you very much. I haven't had time to really get into it yet, but would like to create some jupyter notebook as well in the near future.
-
I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment...
Yes, it's always a trade.
Since three exposure times (inkl dark frame) would get too long for HDR, I think it is good enough for the moment.
FWIW, without darkframe might look fine with the kind of exposures you're using. I'd certainly expect it to be OK on G1x. You can also use pre-made darkframes and subtract them on the PC, but that works better with raw and takes a lot of processing.
The real problem I see is (since the over_weight parameter always remains the same.) in daylight condition the overexposure could trigger too easily with some bright clouds or snow reflection.
Maybe the highlight clipping in clouds and snow are a better way to define over_thresh_frac and over_margin_ev.
I'm not sure I follow. over_weight was 0 in the sunset run because of the particular conditions. It was triggered quite a bit in the sunrise (https://www.instagram.com/p/CPUtoF8HpuA/ i think), but underexposure still pushed the exposure quite high.
hmm.. I already did that: the time lapse in the final videos includes only the upper 2/3 of the frame. (I cut the village out of the picture.) And because the missing part did influence too much the metering I excluded it.
line 703ff
-- meter rectangle, centered in active area
self.meter_left = rawop.get_active_left() + rawop.get_active_width()/2 - self.meter_width/2
self.meter_x_count = self.meter_width/self.meter_step
self.meter_top = rawop.get_active_top() + rawop.get_active_height()/2 - self.meter_height/2
self.meter_y_count = self.meter_height/self.meter_step/3 --only upper 2/3 counts
That makes sense. It should probably be a script option, but it already has so many :-[
You could also adjust the area measured by the histogram, if you haven't already.
-
I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment...
That's what I usually do. I try to expose the moon as brightly as possible but so that it isn’t overexposed. In post-production, I then pull the shadows up and the lights down. RAW is of course a must. The problem is the dynamic range of the old G1x. Then there is not so much in it. A reason for me to switch to the M100.
-
I'm not sure I follow. over_weight was 0 in the sunset run because of the particular conditions. It was triggered quite a bit in the sunrise (https://www.instagram.com/p/CPUtoF8HpuA/ i think), but underexposure still pushed the exposure quite high.
OK thanks. It will take a few weeks and several changes to the settings and analysis of the result until it fits.
I will wait now and see how it performs with the G1X and match the parameters for the G1X.
-
Hello reyalp,
I want to share my experience and some code changes I did to rawopint.lua in this and the next post.
For a better understanding of the settings/parameters ui_bv_ev_shift_pct, ui_bv_ev_shift_base_e, ui_ev_shift_e I did a bit of reverse engineering by gradually changing one parameter and then recording a sunset and so forth.
I attached a pdf with the plots of the ui_bv_ev_shift_pct changes from 2% to 80%, which I think could be also interesting to others for the understanding of this functionality.
For my kind of timelapse a value of about 16-24% is best depending on the ui_bv_ev_shift_base_e and ui_ev_shift_e parameters. At the beginning it was quite strange that high values would overexposed the image in daylight. But in the end it makes sense since it lowers the exposure when it's dark and rises the exposure when it is higher than the "neutral" exposure. The ui_bv_ev_shift_base_e did basically the same as the ui_bv_ev_shift_pct. It should the shift starts later for a sunset or earlier for a sunrise but the result is minimal and very similar to the ui_bv_ev_shift_pct changes even when you use the extreme values.
An other topic was the under_expo_weight, over_exp_weight. As far as I understand for my type of sunset it is best to switch this two parameters off and only relay on the meter exposure, because I think that if a bright cloud comes into the frame the hole frame should not darken. Because the trade off would be the video to flicker if the clouds are changing too fast. Also in the night the under expo tried to raise the exposure up but if it is dark outside it is ok if parts of for me if in the frame are completely dark areas. On the other hand if the clouds are too overexposed I would just underexpose the whole image and if the image in the night is too dark in general lower the ui_bv_ev_shift_pct value a bit.
What does actually mean a rise of the meter weight from 100 to say 200? I attached a plot where exactly that happens but I could not see any changes in the timelapse video itself.. ???
The last part I'm not sure is, if whether the ui_meter_width_pct and ui_meter_height_pct are dependent on the selected image format. E.g. if I choose 1:1 or 16:9 image format stays the area that rawopint uses to calculate the exposure always the same? ???
-
I found this line in the rawpint code:
1148 -- smooth out rapid changes
1149 -- TODO smoothed fraction should be configurable
What I did was to implement a simple exponential moving average.
https://en.wikipedia.org/wiki/Exponential_smoothing (https://en.wikipedia.org/wiki/Exponential_smoothing)
Disclaimer:
I actually don't know nothing about lua and I just took vscode to implement the changes. I also didn't format the code because it could be that the auto formatting of vscode would apply an other format that you did.
Also I have no experience which code performance in lua and I could not test the script property with some proper test but just loaded it to the camera and compared the d_ev value before and after the change :lol
But it actually works:
All changes are done in the function exp:calc_ev_change() function.
I always included the line before so it is easier to find, since the line number could vary.
930 local last_ev_change
931 local last_ev_change_decimal
932 local median_filter_intensity = 5 -- this is a magic number!! it has to become an input parameter
933 -- allowed values: 0: no filter, 9: max filter (>10 and <0 not allowed here) expample:
934 -- 1: ev_change = 10% * last_ev_change + 90% * ev_change -> low moving average filter intensity
935 -- 8: ev_change = 80% * last_ev_change + 20% * ev_change -> high moving average filter intenisty
940 last_ev_change = self.ev_change
941 last_ev_change_decimal = self.ev_change_decimal
949 last_ev_change = 0
950 last_ev_change_decimal = 0
1126 -- smooth out rapid changes exponential moving average implemented
1127 if self.smooth then
1128 local ev_change_smooth = (last_ev_change * 10 + last_ev_change_decimal) * median_filter_intensity +
(10 - median_filter_intensity) * ev_change * 10
1129 ev_change = ev_change_smooth / 100
1130 last_ev_change_decimal = ev_change_smooth % 100 / 10 -- not sure if modulo works this way in lua
1131 end
1132 self.ev_change = ev_change
1133 self.ev_change_decimal = last_ev_change_decimal
After I wrote this lines of code I asked my self why I did not used a float or double. I actually was not sure if there is a performance or memory issue using doubles since I could only find Integers in the code.
In the end I simulated the first decimal number after the decimal point with an Int which should be ok.
the magic is simply this line of code:
ev_change = ev_change * ( 1 - moving_average_fiter_intensity) + last_ev_change * moving_average_fiter_intensity
the value for the moving_average_fiter_intensity goes from 0 to 1
since I used only int for this calculation I multiplied the new ev_change by 10 and also the moving_average_fiter_intensity goes form 0 to 9 (10 would mean no change at all)
after the calculation the result has to be divided by 100 and to save also the first decimal number line 1130 comes into place.
The two pdfs shoes the d_ev value with the moving average filter (set to 9) and without the moving average filter.
What is still missing: the moving_average_fiter_intensity is a magic number at the moment and has to be declared as parameter at the beginning.
I also deleted the smooth overshoot and smooth sign switch as the moving average does a better job.
If the filter is set to high it will get quite sluggish.
EDIT:
What I forgot to mention: the d_ev was scaled by a factor of 10 in the plot. (because it is very small compared to the other values)
-
Thanks for posting this, it seems like a good idea. I'll take a closer look later, but just one comment for now:
After I wrote this lines of code I asked my self why I did not used a float or double. I actually was not sure if there is a performance or memory issue using doubles since I could only find Integers in the code.
CHDK Lua numbers are integers. To use real numbers, you need imath fixed point (https://chdk.fandom.com/wiki/Lua/Lua_Reference#Mathematical_functions_.28imath_library.29 ) or fmath (https://chdk.setepontos.com/index.php?topic=14305.msg145763#msg145763 added in CHDK 1.6 )
Performance for mathematical operations with regular integers or either of the above shouldn't be a concern unless you're doing hundreds or thousands of calculations in tight loop (see https://chdk.setepontos.com/index.php?topic=14305.30 and post linked from there)
-
The filter parameter is now an input parameter so I can control it with CHDKPTP remotely.
Also I added the raw d_ev and the filtered d_ev to the log file.
The Plot is attached. (with ND filter kicking in at around 450. frame, it works flowlessly! before I checkt the plot I was not able to find the moment where it drops in, very pleasant :D)
I think the filtering effect is quite visible, but maybe not necessary, since the camera changes the exposure only every 1/3 stop...(?)
I will try the next days to increase the value from 0 to 9 and then upload the plots here.
-
I want to share my experience and some code changes I did to rawopint.lua in this and the next post.
Thanks for doing this and sharing it.
For a better understanding of the settings/parameters ui_bv_ev_shift_pct, ui_bv_ev_shift_base_e, ui_ev_shift_e I did a bit of reverse engineering by gradually changing one parameter and then recording a sunset and so forth.
As I mentioned earlier ui_ev_shift_e is a bit broken (https://chdk.setepontos.com/index.php?topic=12697.msg146241#msg146241). This could be a confounding factor trying to understand the other values.
I attached a pdf with the plots of the ui_bv_ev_shift_pct changes from 2% to 80%, which I think could be also interesting to others for the understanding of this functionality.
For my kind of timelapse a value of about 16-24% is best depending on the ui_bv_ev_shift_base_e and ui_ev_shift_e parameters. At the beginning it was quite strange that high values would overexposed the image in daylight. But in the end it makes sense since it lowers the exposure when it's dark and rises the exposure when it is higher than the "neutral" exposure. The ui_bv_ev_shift_base_e did basically the same as the ui_bv_ev_shift_pct.
bv_ev_shift_base and bv_ev_shift_pct shouldn't be equivalent in the general case. base should define the Bv value (derived from meter and exposure) where the effect of bv_ev_shift is 0, regardless of the bv_ev_shift_pct value. Bv is an absolute measure of the amount of light, where a scene in full daylight would typically be around 10 APEX.
Looking at your plots, I guess bv_ev_shift_base_bv is APEX 4? I would generally set it around the brightest you expect the scene, since you don't want to overexpose much, but meter and overexposure limits should stop it from going too far.
In your examples, bv_ev_shift crosses the origin right around where bv96 crosses 400. When Bv is higher, it pushes exposure brighter, and when it's lower, it pushes it darker, subject to the meter thresh/limit values (and I think a bit confused by the ui_ev_shift values, but it's hard to say without the actual log).
An other topic was the under_expo_weight, over_exp_weight. As far as I understand for my type of sunset it is best to switch this two parameters off and only relay on the meter exposure, because I think that if a bright cloud comes into the frame the hole frame should not darken.
I would set the "thresh_frac" values to 0 if you want to completely disable under or over protection. This should completely disable all the related calculations.
For sunrise/sunset, I might disable under (or set very large % and low ev limit), but I would probably not disable over. If the scene is half bight sky and half shadowed land, you probably want the sky not to be totally blown out, even when the land is dark.
What does actually mean a rise of the meter weight from 100 to say 200? I attached a plot where exactly that happens but I could not see any changes in the timelapse video itself.. ???
The weights are arbitrary numbers that determine the relative influence of the meter and over / under exposure, subject to various limits. The key line is
ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)
ev_change is the difference between the meter value and the target value, clamped to +/-ev_change_max.
over_weight and under_weight are calculated from the percentage of over and under exposure, such that they reach 100 when "thresh" is reached, and quickly ramp up to 200 after that. You can see if over and under are 0, then ev_change is unaffected. Meter weight starts at 100, and ramps up to 200 between from thresh to limit.
The intent is that the three factors will tend balance in the range defined by the thresh and meter limit values, without any sudden transitions triggered by if/else logic. For example, if overexposure is at the "thresh" value, it will balance with meter the thresh value in the opposite direction, and if that is exceeded, push meter to the limit value without allowing much more overexposure.
The actual change in any given frame is limited to ev_change_max.
I haven't tested much with the weights at non-default values. I'd generally use the thresh / limit and over / under fractions instead.
The last part I'm not sure is, if whether the ui_meter_width_pct and ui_meter_height_pct are dependent on the selected image format. E.g. if I choose 1:1 or 16:9 image format stays the area that rawopint uses to calculate the exposure always the same? ???
Anything related to size/position in rawopint refers to the full raw buffer. So if you use a different jpeg aspect ratio than the native sensor resolution, you'll have to account for that yourself.
-
Thank you very much for all the information and clarifications of my misunderstandings. It really helps a lot!
bv_ev_shift_base and bv_ev_shift_pct shouldn't be equivalent in the general case. base should define the Bv value (derived from meter and exposure) where the effect of bv_ev_shift is 0, regardless of the bv_ev_shift_pct value. Bv is an absolute measure of the amount of light, where a scene in full daylight would typically be around 10 APEX.
Looking at your plots, I guess bv_ev_shift_base_bv is APEX 4? I would generally set it around the brightest you expect the scene, since you don't want to overexpose much, but meter and overexposure limits should stop it from going too far.
In your examples, bv_ev_shift crosses the origin right around where bv96 crosses 400. When Bv is higher, it pushes exposure brighter, and when it's lower, it pushes it darker, subject to the meter thresh/limit values (and I think a bit confused by the ui_ev_shift values, but it's hard to say without the actual log).
thank you very much! I think this is a very good explanation of how bv works and how it should be setup! You should really add it to the wiki of the rawopint script! The explanation there is not so comprehensively explained.
I actually tried to use the bv value to try to influence the bv_ev_shift in a way that the timelapse get darker later in a sunset (by lowering the bv value as the amount of light decreases over time and so should a lower value activate the ev_shift later. I don't like it, when it gets dark too early in the timelapse.
I have done also a series where I change only the bv_base but the effect is not so prominent as I would like.
I attached the pdf with the bv_base change series.
I think this plot series shows quite well how the bv_ev_shift_base_bv parameter influences the algorithm.
The point where the curve of the ev_shift crosses the zero line shifts depending on how high bv_base is set to be.
I would set the "thresh_frac" values to 0 if you want to completely disable under or over protection. This should completely disable all the related calculations.
That seems like a good idea!
The weights are arbitrary numbers that determine the relative influence of the meter and over / under exposure, subject to various limits. The key line is
ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)
ev_change is the difference between the meter value and the target value, clamped to +/-ev_change_max.
over_weight and under_weight are calculated from the percentage of over and under exposure, such that they reach 100 when "thresh" is reached, and quickly ramp up to 200 after that. You can see if over and under are 0, then ev_change is unaffected. Meter weight starts at 100, and ramps up to 200 between from thresh to limit.
The intent is that the three factors will tend balance in the range defined by the thresh and meter limit values, without any sudden transitions triggered by if/else logic. For example, if overexposure is at the "thresh" value, it will balance with meter the thresh value in the opposite direction, and if that is exceeded, push meter to the limit value without allowing much more overexposure.
The actual change in any given frame is limited to ev_change_max.
I haven't tested much with the weights at non-default values. I'd generally use the thresh / limit and over / under fractions instead.
This makes sense to me. Thanks.
Anything related to size/position in rawopint refers to the full raw buffer. So if you use a different jpeg aspect ratio than the native sensor resolution, you'll have to account for that yourself.
Thanks, that is great to know! With this I can adjust the ui_meter_width_pct and ui_meter_height_pct according to the crop of the final video without any doubt ;)
-
Here are the results of the test run I did with rawopint.lua + G1X + exponential smoothing code mod (moving average filter), I presented earlier. All settings can be found in the pdf.
Attached are 2 pdf with some plots. One with 4 runs from 0 (no filter) to 9 (max filter) in steps of 3.
shoot all 4 in the same afternoon.
In the other pdf you can see the sunrise plots with filter value set from 0 to 9.
2 Observations:
Without smoothing the ev change value some flickering in the video appears, most noticeable when light condition are stable and don't change at all.
I would suggest a minimum value of 3-4 to get rid of the "flickering".
Also notice that I used max ev change of 1/8 which kind of prevent flickering form its own.
At 0 the flickering is noticeable but not a problem. At 3 it is not more or less gone.
With max filtering some transient response of the filtered ev change value can be observed, especially in the beginning as rawopint has to lower the overall ev a bit (as I underexpose the image a bit).
But the timelapse after the settle in is perfectly smooth.
I think, I will keep the filtering functionality and choose for my application a value of 4 to 7.
Bottom line:
It works as it should. Edge values to be avoided imho.
-
Yesterday I did an M3 time lapse with rawopint. When heavy rain set in, I unfortunately switched off the camera via Power Off. In the logfile there are now about 130 entries of 8s each missing. I wondered why so many lines were lost.
-
Yesterday I did an M3 time lapse with rawopint. When heavy rain set in, I unfortunately switched off the camera via Power Off. In the logfile there are now about 130 entries of 8s each missing. I wondered why so many lines were lost.
The file is only closed when the script ends, so if the camera is powered off, anything in the OS buffer is lost. There may be additional layers where things can be lost if the file isn't closed.
The log module can write every line if you want (commented line buffer_mode = 'sync'), but that may make the shooting interval less consistent.
-
2 Observations:
Without smoothing the ev change value some flickering in the video appears, most noticeable when light condition are stable and don't change at all.
I would suggest a minimum value of 3-4 to get rid of the "flickering".
So I went to add exponential smoothing, and realized I already implemented it without knowing ;)
The original smoothing algorithm was (ev_change + last_ev_change)/2, or effectively the same as yours with a factor of 0.5.
In the old code, this is followed by additional logic:
* Avoid "overshoot", where the calculated ev_change would land exactly on the target exposure. That is, if meter == target, then ev_change is 0, and including the previous value would make the next value different from the target. In this case, ev_change is left at 0, so the next shot should end up exactly at the target value. However, this isn't necessarily optimal if there's a trend...
* Avoid smoothing pushing the exposure in the wrong direction. That is, if the sign of the calculated ev_change and the smoothed ev_change are opposite, the change is set to 0.
* If the sign of ev_change changes (after smoothing), reduce the change by half.
So my new plan is to make the factor an option and add some control for the additional logic.
This all begs for some kind of controlled, reproducible testing, but I haven't figured out a manageable way to do it. Maybe a video playing on a monitor?
-
The file is only closed when the script ends, so if the camera is powered off, anything in the OS buffer is lost.
Do you know the size of the buffer?
The log module can write every line if you want (commented line buffer_mode = 'sync'), but that may make the shooting interval less consistent.
Thanks for the hint.
This all begs for some kind of controlled, reproducible testing, but I haven't figured out a manageable way to do it. Maybe a video playing on a monitor?
At least that's how I do it for simple cases. It works very well with that. But I can't test the oscillation with it.
-
Do you know the size of the buffer?
I think it's usually 32K.
At least that's how I do it for simple cases. It works very well with that. But I can't test the oscillation with it.
Yeah, could be tricky. What kind of exposure range do you get from that video?
-
What kind of exposure range do you get from that video?
I can't say exactly because my tests are always limited by the set parameters.
In this example with the M3, ISO goes from 100 to 1600 (4 steps) and the exposure time from 0.13s to 3s (4.5 steps).
In this video I also change the values linearly from 255 to 0. This means that the dark times are less represented.
-
The file is only closed when the script ends, so if the camera is powered off, anything in the OS buffer is lost.
Where to place "log:flush()" without disturbing intervall? :-[
function restore()
...
log:flush()
log:close()
end
-
Where to place "log:flush()" without disturbing intervall? :-[
Why should such a place exist?
The csv log module used in rawpoint accepts 3 buffer_mode values
'os' - output is buffered by the OS, written to the card when the OS thinks it needs to, or when log:close() or log:flush() is called. This is the default.
'table' - output is kept in a Lua table until log:flush() or log:close() is called. May use all available RAM unless the script takes steps to avoid this.
'sync' - output file is opened, written and closed each time log:write() is called. log:flush() is ignored.
The assumption is that 'os' will generally minimize disturbance, since the OS will probably try to write optimally sized chunks. Scripts that do the equivalent of 'sync' have been reported to have much less consistent intervals, which isn't surprising since it would likely involve a lot of writes smaller than an allocation unit on the SD card.
If you have enough memory and don't care about losing everything in a crash, then 'table' would have the minimum impact.
-
On perhaps a slightly different topic:
I was thinking of using rawoptint to photograph a sunset/moonrise on a nice white mountain. It would just be important here that the interval is kept close to 15s, the ISO at 200. With that I could take well exposed single shot with the shutter speed going from about 1/1000 s in daylight to 15 s in moonlight, if my calculations are correct.
What parameters would I have to set to what values for this to work?
Best
-
On perhaps a slightly different topic:
I was thinking of using rawoptint to photograph a sunset/moonrise on a nice white mountain. It would just be important here that the interval is kept close to 15s, the ISO at 200. With that I could take well exposed single shot with the shutter speed going from about 1/1000 s in daylight to 15 s in moonlight, if my calculations are correct.
What parameters would I have to set to what values for this to work?
Max Tv Sec/1000 = 15000 (or less to allow for image saving while keeping 15s exactly, perhaps 14000 for jpeg, or 11000 for raw, depending on cam)
Min Tv Sec/100K = 100 for 1/1000th
Target ISO = minimum your camera supports in the Canon UI (usually 80 or 100)
ISO adj TV Sec/1000 = the value you used for Max Tv Sec/1000 (if you want ISO to only adjust once the longest shutter speed is reach) or less (if you want ISO and shutter to adjust together)
Max ISO = 200
Of course, if the moon will be in the scene, you won't be able to expose it and the mountain correctly, so you need to decide which to prioritize.
Some relevant settings
"Overexp thresh x/100k"
Controls how much of the scene is allowed to be over exposed. If it's larger than the fraction of pixels the moon takes up, then the moon will be over-exposed, and the scene will be more correctly exposed. The fraction depends on zoom. You can calculate from the FOV and knowing the moon is ~1/2 degree, but if you aren't using a lot of zoom, a few percent (values of a few thousand) should be plenty to ignore it. If you do want to properly expose the moon, I suggest calculating the actual fraction. If the fraction is very small, you may need to reduce "Histogram step"
"Underexp thresh x/100k"
Controls how much of the scene is allowed to be under-exposed. This will resist the effect of the overexposure protection. Generally it should be a larger fraction for over-exposure. Higher values allow over-exposure protection to have more effect. If you want to allow the moon to over expose, the default of 10% should be OK, but if you want to minimize how much of the scene gets blown out, you can use a higher value.
"Underexp -Ev"
How many stops below correct exposure the scene can be before it counts as under exposed. Larger values will allow the scene to get darker when there's over exposure in the scene.
"Meter low thresh -Ev" and "Meter low limit Ev"
Work similarly to the previous items, but for the average value of the scene, rather than a fraction of pixels. The "thresh" value controls what exposure value it starts to take effect (resisting further exposure change from bright objects) and "limit" sets where it reaches full effect. Again, if you want bright objects to allow the scent to get dark, use larger values. If you want to keep the scene well exposed at the expense of having bright areas over-exposed, use lower.
FWIW, I've been meaning to put out a new release of this script for a while, but for various reasons haven't quite got around to it. The in development version is at https://github.com/reyalpchdk/chdkscripts (use "download" next to "development build" for the latest)
Some bugs that caused flickering in certain situations are fixed. New bugs are also undoubtedly introduced, but identifying them is left as an exercise ;)
-
Thanks for the info. Just for clarification, I will have the moon rising to the left of the scene and going behind the camera, setting to the right. In case that doesn't make sense, the mountain is Titiroa in New Zealand, and my shooting position is to the north of the mountain looking south. It is a lovely white mountain, so I am trying to get the foreground illuminated, with stars in the background sky. Unfortunately I can't carry enough lighting up the mountain with me, so I am getting the moon to do the job for me.
I'll see what the new version does in a close simulation in a couple of weeks.
Cheers, Lee
-
I was thinking of using rawoptint to photograph a sunset/moonrise on a nice white mountain. It would just be important here that the interval is kept close to 15s, the ISO at 200.
I find a sunset with 15s interval too fast (with 30fps)
-
I find a sunset with 15s interval too fast (with 30fps)
Ah, but it's not just the sunset/moonrise, I want to get the mountain illuminated by the moon with the stars behind. Without too much noise. That's what the 15 s are for. But thanks anyway.
-
Experiment done and it was much better than I hoped for. Wow! Video is at I was using the old version of rawopint (perhaps I will get a clear night on Monday or Tuesday of this week and can repeat with the new version).
Camera was an SX 520 powered by a 20Ah powerbank - 14 hours of shooting used up about half of the power.
The only fly in the ointment was that I had raw shooting turned on and that slowed down the shooting rate, but that was my mistake.
-
Experiment done and it was much better than I hoped for. Wow! Video is at
Nice. There's a fair amount of overexposure near the start and end. Throwing it in the notebook (https://github.com/reyalpchdk/chdkscripts/blob/main/tools/rawopint-analysis.ipynb) we can see (frac-weight-20230205.png) it hit about 4.2%. Over thresh is 3%, with the additional amount driven by under-exposure, which has thresh 10%. Since under exposure was more than 20% for must of the run, it dominated exposure control as seen in the weights. So this was working as designed, but would probably look better with more over exposure control.
For a scene like this, you probably prefer to allow the landscape to get dark to favor of more correctly exposing the sky, so I'd suggest a lower over thresh, maybe 1%, and higher under thresh, maybe 30% or more, or even turn it off, and rely on average scene brightness (meter) to handle it. Meter low limit set to 1 1/4 stops, which is quite strict. I'd probably increase that to 3 or 4.
Beware that the old version is probably more prone to oscillation with tight overexposure limits.
(note I had to trim the log to the final run to get it to load, there were other runs in the log, one of which had a corrupted row, likely to due to a crash.)
Camera was an SX 520 powered by a 20Ah powerbank - 14 hours of shooting used up about half of the power.
Nice. I guess this is a custom hardware setup?
-
So:#ui_meter_low_limit_e=4
#ui_exp_over_thresh_frac=1000
#ui_exp_under_thresh_frac=20000
Have I understood that correctly?
-
So:#ui_meter_low_limit_e=4
#ui_exp_over_thresh_frac=1000
#ui_exp_under_thresh_frac=20000
Have I understood that correctly?
Yeah, something like that should handle similar scenes with less overexposure.
-
So:#ui_meter_low_limit_e=4
#ui_exp_over_thresh_frac=1000
#ui_exp_under_thresh_frac=20000
Have I understood that correctly?
Overexposure is always a big problem. Overexposure and underexposure cannot be controlled at the same time. That's why the underexposure is always switched off (ui_exp_under_thresh_frac=0) for me.
#ui_exp_under_thresh_frac=0 "Underexp thresh x/100k (0=Off)"
-
OK, still using the old rawopint (have one more experiment running with the latest version and will post). This time I stopped the dark frame subtraction and switched off raw shooting, and the interval was a very solid 15s. In my real life shoot neither the sun nor the moon will be making an appearance so please keep that in mind when reviewing the rawopint data. Attached is also a picture of the set up. Had the camera running for 26 hours and the powerbank still didn't run out of juice. This shoot as a video:
I'm very happy with the result, and am looking foreard to the next full moon which is when I hope to do the real shoot.
-
This time I stopped the dark frame subtraction and switched off raw shooting, and the interval was a very solid 15s.
From the log, the minimum value "sleep" is 220 ms, so you've got pretty close to the minimum interval with your chosen settings. It's pretty stable so you could probably use ~100 ms shorter interval or longer exposure without exceeding the interval, but neither will make a big difference.
In my real life shoot neither the sun nor the moon will be making an appearance so please keep that in mind when reviewing the rawopint data.
That should certainly make things easier.
Looking at the most recent run, most of the night is on the exposure limits (max ISO, longest shutter) so in this segment the effect of the under exposure and low meter limits is to make the script more resistant to moving exposure in the other direction. In other words, making the under exposure limits stricter wouldn't brighten the scene. Relaxing them or making the over exposure limits stricter would potentially allow the scene to get darker in response to areas of overexposure.
If you want to get more exposure at night, you might get away with upping the ISO a bit. Assuming your target is 1080P youtube playback or similar, downscaling from 16 MP reduces the noise you see quite a bit compared to a full res still. If you do, you should test in advance, because the Canon firmware introduces various levels of noise reduction (other than dark frame) at various ISO levels, which add to processing time. So you might find that setting ISO 360 breaks your interval, while ISO 359 does not (numbers made up).
c_joerg may disagree, but I think these settings are a reasonable compromise. The moon would look better with less over exposure, but not at the expense of making the foreground totally dark. Letting the scene get a little darker might be better compromise, but it's a subjective call, and if you want to see the landscape, the moon itself is going to be blown out.
Similarly, the bright lit windows in the nearby building are over exposed, but the rest of the scene getting darker when they go on would likely be distracting.
Fortunately, it sounds like neither of these should be issues in the real shoot.
For me, the worst over exposure in this run is probably right around sunrise (starting around 2:30, corresponding to the over exposure peak of ~2.78% around frame 3918). Again may not be an issue for your real shoot if the sun won't be rising directly in view, but it's a case where controlling overexposure more would probably look better. Further relaxing the under exposure limits (increasing under_thresh_frac or turning it off) would probably help there.
The flip side of this is, once the sun actually comes into view, it's going to over expose no matter what, so you still have to decide how dark you want to allow the rest to get. But again, allowing it to get a little darker would probably be fine. However, in this case, when the sun is in view, you hit the short shutter limit of 1/10000, so making the limits stricter wouldn't make the scene darker anyway. You could allow even shorter shutter speeds, but 1/10000 is already pretty extreme, and pushing the limits may result in flickering (though it seems fine here)
Note the meter limits are not active (meter weight == 100) in the sunrise section. Roughly half the scene is bright and half is dark, so the average exposure is great :haha. In a case like this, you could bias the meter to favor the sky or ground by adjusting the height and position. E.g. if you want to focus on sky colors, metering only the top half of the scene, or if you want the landscape, only the bottom. Or 2/3 if you want favor one but still have some influence from the other. Not suggesting this applies for your current project.
Overall, my suggestions if you want to tune further form this run would be:
1) Explore increasing ISO, probably to something under 400, before the extra noise reduction kicks in
2) Increase under_thresh_frac to maybe 40%, or turn it off entirely (0) as c_joerg suggested.
-
c_joerg may disagree, but I think these settings are a reasonable compromise. The moon would look better with less over exposure, but not at the expense of making the foreground totally dark.
But that's exactly how I do it when shooting in the sun. The foreground is then quite dark. I then increase it in the RAW processing. That's the only way it works properly from my point of view. Of course, this only works with RAW and not with JPG. With a 12-bit RAW you are certainly more limited.
-
Does anyone have any idea why this shoot stopped prematurely? The log file stops @ 20:18 and the shooting itself @ 20:44. It doesn't appear to be the battery. Thanks.
-
Does anyone have any idea why this shoot stopped prematurely? The log file stops @ 20:18 and the shooting itself @ 20:44. It doesn't appear to be the battery. Thanks.
The fact the the log ends with an incomplete line suggests either a crash or a script error, but it doesn't provide any information beyond that.
Did the camera shut down? If so, was the lens extended or retracted?
If the camera crashed, there may be a romlog https://chdk.fandom.com/wiki/Debugging#Camera_crash_logs_(romlog), please post it if so.
Can you see what the number of the last actual image taken was? The log ends at 666 (:blink:) but it's buffered, so if the camera crashes some portion will be missing.
Was the camera on battery or external power?
edit:
If this happens and the camera doesn't shut down, it's likely a script error. In that case you should check Miscellaneous -> Console -> Display last console to check for script errors. But this is lost if the camera has been shut down.
-
The kkast picture taken was #769 and it appears normal. Camera was being run on an extrenal power supply. As I was expecting the camera to have shutdown after another 10 hours it was only after I had turned everything off that and removed the card that I noticed that something was amiss, so I have no idea what state the camera was in. Romlogging will be enabled the next time around. Sorry I can be of any further help.
-
Romlogging will be enabled the next time around.
The Canon firmware saves the romlog to internal flash memory when the crash occurs, so nothing needs to be enabled. You can get the most recent romlog by following the instructions on the linked wiki page.
-
Found the rom.log
-
Found the rom.log
Hmm, if the clock on the camera is set correctly, the most recent crash is in 2021
ASSERT!! CaptureAKBAD.c Line 537
Occured Time 2021:07:11 22:00:34
That suggests a script error.
-
Can confirm that the time was set correctly.
-
Can confirm that the time was set correctly.
OK, that strongly points to a script error. Since this is on the new version of the script, there's a good chance it's a bug, so if we don't track it down before your shoot, you might want to stick to the old version.
You can try to reproduce it using the same settings and (if possible) similar scene, and then check the console for script errors without shutting down.
You can also make the log save after every shot, which might narrow down the conditions when the error occurs, but this will likely reduce shooting right, and could potentially affect whatever was triggering the bug in the first place. To do this, remove the initial -- from the line (around line 1811, under log = xsvlog.new)
-- buffer_mode='sync', -- for crash debugging, save every line
so it looks like
buffer_mode='sync', -- for crash debugging, save every line
That said, I'd suggest trying to reproduce without doing that first, since I suspect the console error will be more informative.
Camera was being run on an extrenal power supply.
While I suspect this is a script bug, one thing that is notable is that your previous runs were rock solid at 4.769 volts, while this one was bouncing around between 4.4 and 4.7. Was it the same powerbank based power supply, or something different?
I also see a partial run at the start of the log, which starts at 4.769 and then drops slightly.
If it's the same supply, that might be worth investigating. A powerbank will likely try to provide a fixed voltage until it's low, and then cut off, so the script (and Canon firmware) low batter detection would not function. Also, if the Canon firmware thinks it's on external power, the Canon low battery logic may be disabled in any case (I'm not sure if cameras like sx520 detect external power specifically, some cameras do.)
-
I can do some more experiments when I get off the plane in Auckland on Sunday and I will try the detailed logging.
I was using a different power bank to the one I used previously (10 Ah instead of 20), as I wanted to see how long it would run for. The 20 is much newer and possibly better technology (it can switch voltage by recognising what the source wants), but it ran for 26 hours and was only half-empty at the end. The one I used last night was also not used up completely - again half-empty, but it is 4+ years old.
-
I can do some more experiments when I get off the plane in Auckland on Sunday and I will try the detailed logging.
Note the change I suggested isn't more detailed per se, it just ensures that the values up to (but almost certainly not including) the shot cycle that crashed will be saved to the file. So it might provide additional clues but it's unlikely to point directly to the cause.
I was using a different power bank to the one I used previously (10 Ah instead of 20), as I wanted to see how long it would run for. The 20 is much newer and possibly better technology (it can switch voltage by recognising what the source wants), but it ran for 26 hours and was only half-empty at the end. The one I used last night was also not used up completely - again half-empty, but it is 4+ years old.
That would explain the voltage difference. My first guess would still be a script error but these cameras can draw quite a lot of current, so it's conceivable the older pack wasn't quite up to it once the battery dropped below some level of charge.
If the power supply cuts out suddenly, I'd expect you to find the camera off with the lens extended. Note this can also lead to filesystem corruption, so checking the card in a PC before the next run is a good idea.
And again, if we haven't found the cause before your shoot, I'd definitely suggest using the configuration you used on those long test runs, with the old script and the 20 Ah supply.
-
OK, that strongly points to a script error. Since this is on the new version of the script, there's a good chance it's a bug, so if we don't track it down before your shoot, you might want to stick to the old version.
Where is the download for rawopint v:0.26-dev?
The build is from sx520hs-100b-1.5.1-5783 Mar 14 2021
-
Where is the download for rawopint v:0.26-dev?
https://github.com/reyalpchdk/chdkscripts development build download link