CHDK Forum
CHDK Development => General Discussion and Assistance => Topic started by: reyalp on 31 / December / 2013, 01:36:57
-
This is inspired by lapsers continuous mode intervalometer stuff from http://chdk.setepontos.com/index.php?topic=8997.0 (http://chdk.setepontos.com/index.php?topic=8997.0)
The basic idea is to let script pause the shooting process at various points, do something, and then tell shooting it can resume. So to shoot a fixed interval in continuous mode, you pause somewhere right before the exposure starts. To analyze the raw buffer, you pause in the raw hook.
The attached patch is a preliminary implementation of this. Unless there are major objections, I will check it into the trunk. The interface will still be subject to change, and I haven't tested it thoroughly yet.
There are currently 3 hooks defined (in order of execution)
'hook_override' - executes in shooting_expo_param_override, before overrides are applied. Can be used to adjust exposure settings after the camera has calculated it's own values.
'hook_remote' - executes in wait_until_remote_button_is_released. Can used to control the start of the exposure.
'hook_raw' - executes in raw_process, just after shot_histogram. It could be used to analyze or modify the raw buffer.
Each of the hooks provides a standard interface:
hook.set(timeout)
Enables the hook, so it will block the shooting process for up to timeout milliseconds. 0 disables the hook.
hook.is_ready()
returns true when the shooting process is in the hook
hook.continue()
allows the shooting process to exit the hook
hook.count()
tells you how many times the hook has been reached for this script. This allows you to tell if a particular point has been reached without actually blocking the process.
The attached script is a very simple continuous mode intervalometer using the remote hook to enforce the interval and the raw hook to adjust exposure. The exposure calculation is purely as an example. Lapsers code has much more sophisticated metering, which is something that can and should be added separately. Another thing that should be added is functions to read and write the raw buffer in the raw hook.
some issues and questions
A few ports may call shooting_expo_param_override more than once. The current code will just call the hook multiple times. Affected cameras include A1000 (but I'm not sure if it's in the same code path) and ixus1000 which does it for two subs out of four :-[
I vaguely remember there being some cameras without the remote hook too.
The names are confusing, especially remote. I'm open to suggestions.
Would it be desirable to have a post-raw save hook?
I originally intended to have a hook.wait_ready() which would do the equivalent of
while not hook.is_ready() do sleep(10) end
in the action stack.
Should there be a way to detect timeouts?
-
The basic idea is to let script pause the shooting process at various points, do something, and then tell shooting it can resume. So to shoot a fixed interval in continuous mode, you pause somewhere right before the exposure starts. To analyze the raw buffer, you pause in the raw hook.
This is very cool. The possibilities this opens up are going to be a lot of fun. Too bad we can't go back over all the forum posts were we had to tell people what they wanted was not scriptable and needed changes to the core C code. :(
The attached patch is a preliminary implementation of this. Unless there are major objections, I will check it into the trunk.
Yes please.
A few ports may call shooting_expo_param_override more than once. The current code will just call the hook multiple times. Affected cameras include A1000 (but I'm not sure if it's in the same code path) and ixus1000 which does it for two subs out of four :-[
Porting error or valid reason? I'd guess the former - I wonder if a PM to the original porting person would work. Those are both pretty old cameras. I'm thinking anyone using these functions can probably safely ignore handling the multiple hook calls and we fix the actual port if someone complains?
I vaguely remember there being some cameras without the remote hook too.
IIRC correctly, there were originally four and srsa_4c has fixed three of those? I'll check.
The names are confusing, especially remote. I'm open to suggestions.
I agree that hook_remote is confusing (unless you know the hook is where the USB remote pauses for sync) but the other two seem okay to me. How about changing hook_remote to hook_shot or hook_shoot ?
Would it be desirable to have a post-raw save hook?
Not sure it would help with deleting the jpg (?) but you never know what it might be useful for if it's not a big deal to add.
I originally intended to have a hook.wait_ready() which would do the equivalent of while not hook.is_ready() do sleep(10) end in the action stack.
Seems useful - why did you give up on it?
Should there be a way to detect timeouts?
That would be nice ... especially during the script testing phase. (assuming this is a seperate question than the previous quote).
Thanks for doing this!
-
hook_shot or hook_shoot
I thought about that, but they are all kind of shot related.
Hmm. Maybe hook_shutter or hook_preshutter.
I originally intended to have a hook.wait_ready() which would do the equivalent of while not hook.is_ready() do sleep(10) end in the action stack.
Seems useful - why did you give up on it?
Because I had a working implementation without it, and then wasn't sure if it was worth adding.
However, I think there's value because the script side should also probably have a timeout. Using the above code, script would just wait forever if shooting wasn't triggered for some reason. The timeout in set() refers to the hook waiting for script to release it. Obviously you could timeout the script side that in lua, but it would be start to be a lot of code for every wait. I guess it could be a lua lib. Running it in the action stack might be slightly faster.
A few notes:
The "remote" hook runs before the actual wait for the remote signal. I'm not sure if it would ever make sense to use the actual remote with this hook, but if the script wanted to do something after the USB signal was sensed, it could just do it using get_usb_power(). Anything involving script would negate the fast response anyway.
Similarly, I put the "override" before CHDK normal overrides, so the script could adjust them. With the get_shooting / set_now logic this doesn't matter for most settings, but I don't see any good reason for it to run after.
I haven't included ubasic support, but there's no reason it couldn't be added.
-
A few notes:
The "remote" hook runs before the actual wait for the remote signal. I'm not sure if it would ever make sense to use the actual remote with this hook, but if the script wanted to do something after the USB signal was sensed, it could just do it using get_usb_power(). Anything involving script would negate the fast response anyway.
The only reason the remote hook exists if for shot synchronization between two or more cameras. The last thing needed after the hook is anything that would vary the time from that point to shutter release. I can't think of any reason someone would want to insert script code between the 5V USB falling and the shutter opening.
Similarly, I put the "override" before CHDK normal overrides, so the script could adjust them. With the get_shooting / set_now logic this doesn't matter for most settings, but I don't see any good reason for it to run after.
That whole priority of overrides probably needs to be clarified somewhere. Right now I believe script overrides beat out menu ones?
I haven't included ubasic support, but there's no reason it couldn't be added.
Seem like anyone writing something that needs a shooting hook is not going to be using uBASIC by choice ...
-
That whole priority of overrides probably needs to be clarified somewhere. Right now I believe script overrides beat out menu ones?
Right, script beats menu.
But in this context, "overrides" would also includes previous script settings with SET_LATER. Hmmm. In this case it could be confusing, if the script did say
set_tv96 before half shoot (creating a set_later)
hook does a set_tv96 (which becomes a set_now, since get_shooting is true at that point)
previous set_later gets applied right after the hook.
One could argue that the first set_tv96 was dumb, but it's unintuitive that an earlier call would beat a later one.
So maybe it makes more sense to do it after? Would there be any disadvantage? It would let you manipulate CHDK bracketing or auto-iso values, if you wanted.
Adding hooks is very easy and doesn't cost much so we could have both, but that might just be more confusing.
The above applies to the current code as well, if you set both outside and inside half shoot. Or a SET_NOW should clear any SET_LATER values, but I'm not sure I want to get into that.
ISO also needs investigation for cameras that have a standalone shooting_expo_iso_override. It looks like this gets called before the regular shooting_expo_param_override, so probably won't work from the hook.
edit:
The ISO issue also probably applies to the existing logic for using SET_NOW when get_shooting() is true.
-
Checked in, changeset 3301 (http://trac.assembla.com/chdk/changeset/3301)
Note this is in development, functionality and interfaces may change. That's why we have a development branch. It should not impact stability of the trunk when the hooks are not used.
Changes from the initial post:
Hooks renamed
hook_override -> hook_preshoot
hook_remote -> hook_shoot
hook_preshoot now executes after the CHDK override code.
Updated test script attached.
-
So I guess a CHDK wiki page (http://chdk.wikia.com/wiki/CHDK_Scripting_Cross_Reference_Page#All_Scripting_Functions) write-up linked to the xref page (http://chdk.wikia.com/wiki/CHDK_Scripting_Cross_Reference_Pag) is in order?
Added to my "round-tuit" list. Just before the one about "merge all the scripting references into sub-pages below the xref page, sorted by section in the xref page to avoid any one page being too long".
Only so many hours in a day .. sigh.
-
I added some documentation http://chdk.wikia.com/wiki/Script_Shooting_Hooks (http://chdk.wikia.com/wiki/Script_Shooting_Hooks)
-
I added some documentation http://chdk.wikia.com/wiki/Script_Shooting_Hooks (http://chdk.wikia.com/wiki/Script_Shooting_Hooks)
Thanks for that. Every so often I think about these hooks and try to come up with a cool idea for what to do with them. Your recent improvements to the HDR script is good. But not a lot of coolness factor. 8)
-
Hadn't had time to look at this before; but seems like a really cool feature.
Random thought - would it be possible to register a function name in the script that the hook code could then 'call'.
Not sure if it's possible; but it might make the scripting easier and avoid the state tests in the main loop.
Phil.
-
Random thought - would it be possible to register a function name in the script that the hook code could then 'call'.
I think I asked at one time if the script engines are re-entrant? I was thinking at the time about assigning individual scripts to camera buttons or events to make it easy to mod camera performance without a big honking master script (http://chdk.wikia.com/wiki/CHDKplus). The answer IIRC was "no" - things like the action stack come to mind immediately as a problem.
Would that not be a requirement for hook "callbacks" to work?
-
Random thought - would it be possible to register a function name in the script that the hook code could then 'call'.
I think I asked at one time if the script engines are re-entrant? I was thinking at the time about assigning individual scripts to camera buttons or events to make it easy to mod camera performance without a big honking master script (http://chdk.wikia.com/wiki/CHDKplus). The answer IIRC was "no" - things like the action stack come to mind immediately as a problem.
Would that not be a requirement for hook "callbacks" to work?
Yes, that would cause problems.
Calling script functions from a different task would most likely be problematic as well.
Phil.
-
Yes, that would cause problems.
Calling script functions from a different task would most likely be problematic as well.
Yes, calling from a different OS task would be bad. Calling a separate Lua "thread" from kbd_task might be possible in theory, but I think it would be counter productive.
What I do plan to do to simplify usage is to add a wait_ready() function that uses the action stack to wait until the hook is reached, with a timeout. This should be straightforward, I just haven't got around to do doing it, and the current code is useable.
Note there are cases (like chdkptp rsint) where you want script to continue doing stuff while it waits for the hook to be ready, so both options should be available. wait_ready could be done as a lua module without any C code, but it might be more convenient to have it standard.
-
wait_ready could be done as a lua module without any C code, but it might be more convenient to have it standard.
I somehow didn't end up posting here when I did it, I added a module called hookutil in 1.3 that includes wait_ready().
I have now started working on the next phase of this project, which is allowing lua to analyze and modify the contents of the raw buffer. With the shooting hooks, the approach is somewhat different than the old shot histogram or lapsers code.
Attached patch and test script includes some initial work. This is very preliminary, but in the spirit of keeping the dev branch unstable, I'll probably check it in if no one screams. Function names, parameters, code organization all subject to change.
The code is currently a C lua module called "rawop". This isn't dynamically loadable (yet).
rawop currently provides the following functions (as rawop.whatever)
fb_info
Returns a table containing information about the raw framebuffer, e.g for d10
={
width=4104,
height=3048,
white_level=4095,
black_level=127,
cfa_pattern=16777729,
bits_per_pixel=12,
active_area={
x1=0,
y1=10,
x2=4072,
y2=3040,
},
jpeg_area={
x1=36,
y1=25,
x2=4036,
y2=3025,
},
}
raw_to_ev96
converts a raw value to a pseudo APEX96 value. This is very primitive, it doesn't know anything about different color channels, but it does give a rough approximation of how many stops of exposure the value is from neutral.
raw_ev96 = log2((raw - blacklevel)/(raw_neutral-blacklevel))
where raw_neutral is based on the average value of a DNG shot uniform grey subject with auto exposure on my d10.
The remaining functions are only valid inside the raw hook, although this isn't currently enforced.
value=get_pixel(x,y)
set_pixel(x,y,value)
These should both be self explanatory
meter(x1,y1,xcount,ycount,xstep,ystep)
This takes the average raw value of the defined region.
x1 and y1 are the upper left corners
xcount and ycount are how many pixels to sample in each direction. They are NOT width and height unless the step is 1.
xstep and ystep are how many pixels to skip between in each direction, so width is xcount*xstep
To prevent overflow, the total number of pixels must less unsigned_max / white_level. For a 12 bpp camera, this is about 1 million. For 14 bpp, it's only 256k. You could of course meter multiple areas and average the results.
On my d10, it takes about 0.5 second to sample 1m pixels. Doing the same with get_pixel in a Lua for loop (optimized and using set_yield) takes about 11 seconds.
meter isn't directly aware of the cfa pattern, but you could sample particular channels choosing your start and step appropriately.
The test script is a simple rapid shooting intervalometer. It uses meter and tries to adjust the tv so that the next shot will be "neutral". If the "draw" option is enabled, it draws a box around the metered area. A real application would probably want a smoothing function for the exposure adjustments. I will try to come up with a more realistic script a bit later.
There's a bunch more I could talk about but this post is already long enough ;)
-
There's a bunch more I could talk about but this post is already long enough ;)
Really nice work. Can't play with it (much) this week but its a big addition to 1.4.0.
-
Very nice, great job :D
Phil.
-
I haven't decided how histogram fits into this yet. Some notes (mostly for my own reference, feel free to tl;dr)
As far as I understand it, the current shot_histogram operates as follows:
* The whole active area is sampled
* The step size of 31 ensures the all the colors are sampled, and keeps the total counts down
* Bit depths higher than 10 are shifted to give a 10 bpp value
* Counts are stored as unsigned shorts
* The histogram can only be queried as % pixels in a range.
* memory for the histogram is allocated for as long as the histogram is in use (included in the module in 1.4)
This is sufficient to get an idea if there are a lot of pixels near over or under exposure. By repeatedly querying, you can get an idea where the peak is and how much of the dynamic range is used.
Drawbacks:
* Can't set sampled area
* Sample density is fixed, and relatively sparse
* Can't sample specific color channels
* Using percentages reduces precision.
* Shifting to 10 bpp reduces precision
* 10 bpp values aren't directly compatible with the meter/get_pixel values, fb_info black level, whitelevel.
* get_histo_range is limited due to ubasic limitations. In lua it would be nice to be able to access the histogram as an array
* If the megapixel war goes over 60 MP, ushorts will be at risk of overflow :-[
Other:
Average of histogram value*count is equivalent to meter average, but a dedicated meter is faster and more memory efficient if you don't need histogram.
-
I've checked in the initial code, not changed too much from the patch a couple posts back. Still very much a work in progress.
I also added two lua files
LUALIB/rawoplib.lua
This has some helper functions for drawing and metering in RGB. Some of this will probably go in C code.
SCRIPTS/TEST/drtest.lua
A dynamic range tester. It takes the exposure selected by canon auto exposure, and then shoots a selectable number of stops over and under. This can be used to check the raw to ev approximation.
To use it, you should aim at a uniform, neutral color subject.
A detailed log is recorded in A/drtest.csv. The m, m96, r, r96 etc columns recorded the "metered" value and apex96 ev equivalents. For exposures that aren't clipping, the m96 value divided by 96 should give approximately the number of stops of over or under exposure.
"peak" is the peak value detected in shot_histogram. bl% and wl% are the percent of pixels near black level and white level respectively.
If you enable "draw" it draws a box around the metered area and bars showing the relative brightness of each channel. This mostly a test of the drawing stuff.
-
Looking at this, it seems we might be close to being able to play with "in camera" HDR algorithms now. (scripting or embedded C)
-
Looking at this, it seems we might be close to being able to play with "in camera" HDR algorithms now.
I think you could actually do a fair job of this with just shot_histogram, but it would be inconvenient. I originally wrote the drtest script using only histogram, when I was trying to figure out the how to do the meter values.
Getting back to the project at hand:
I haven't been doing much with the C code, because I need a real application figure out how it should really work. So I've been working on an intervalometer script that uses the new feature.
obsolete see later posts
Current version attached. This is not completed, in particular, it doesn't handle aperture or ND filter adjustments yet, and the way metering is specified is not very clear.
The basic idea is the script keeps the camera in half_press and repeatedly clicks full shoot. The exposure for each subsequent shot is calculated from the raw of the previous.
The metering is done using an adjustable sized rectangle in the middle of the screen, and the script normally tries to keep this at the "neutral" value. Additionally, shot_histogram is used to limit the % of over and under exposed pixels to certain values.
I think this is working pretty well, but I encountered an oddity with D10. In one run, about 1 in 20 shots ends up with ~1/5 stop lower exposure even though no exposure parameters have changed.
I don't think this is a real lighting change (like a bird or airplane shadow), since it is apparent in the sky. According to the log, the tv and sv are unchanged. I've done a couple more runs without seeing it so :-[
Script options
shots:
number of shots
Draw:
Draw metering information on the raw buffer. This consists of
1) a box around the meter area
2) A bar above the meter box, representing the metered value as a % of white level (log scale).
3) A bar under the meter box, representing the change in exposure as a as a fraction of the max EV change value described below. The origin is at the center, with longer exposure to the right, and shorter exposure to the left.
Drawing can take a significant amount of time, especially if the meter area is large (e.g. 0.5 sec for a 1200 px on d10). The log contains this information and much more.
Interval sec/10:
time between shots, in 10ths of a second.
Meter size:
Size of metered area in pixels. This should be even and a multiple of step. Currently, it is always square and centered.
Meter step:
Ever step-th pixel is measured. To sample all colors, it should be odd.
To prevent overflow, the total number of pixels (size/step)2 must less unsigned_max / white_level. (~1M for 12 bpp, 250K for 14 bpp). Larger numbers of pixels will be slower.
Max ev change 1/x:
Maximum exposure change in a single step, in fractions of a stop. The actual exposure change is smoothed, so the max value will be only used if there is a sufficiently large, consistent trend in scene brightness.
Max Tv:
Longest shutter speed to use, in 1/100ths of a sec. If required exposure is longer, ISO will be used up to ISO limits.
Target ISO:
ISO value to use if shutter limits (described below) not reached. Should normally be lowest ISO
ISO adj TV Sec/100:
Shutter value (in 1/100ths of a sec) at which ISO will start to be increased. If the exposure time exceeds this value, exposure will changes are split 50/50 between shutter and ISO, subject to limits below
Max ISO:
Absolute max ISO to be used.
Av Pref:
Not implemented
Overexp thresh %:
The percent of pixels that must be near to white level to trigger over exposure protection. This uses shot histogram, so includes the whole frame not just the meter area. If this threshold is hit, the script will prevent further exposure increase from the meter. If it is exceeded, exposure will be reduced.
Underexp thresh %:
Similar to the above, but for pixels near black level. Underexposure is only checked if over is not triggered. Under exposure adjustment is also not triggered if it would push the metered exposure more than 1 stop over the "neutral" value.
Display:
Display power saving mode. On = always on, Off use set_lcd_display off. Bklt_Off use set_backlight every shot. Pressing set while the script is running will turn the display on for 30 sec.
Append log:
If unchecked, the logfile "A/rawopint.csv" is overwritten every time the script is run.
The is a CSV with a bunch of columns. The most relevant to the metering process are
sv, tv
exposure values for this shot (image number indicated in the exp column)
meter, meter96
the average value of the metered area. *96 indicates this as an APEX96 value where the target exposure is 0. Negative values indicate an underexposure, positive is overexposure.
over_pct, under_pct
shot histogram percentages in the over and under limits.
d_ev
exposure change requested for the next shot. Negative (confusingly) means increased exposure.
desc
various cases in the metering process trigger messages here, e.g if over exposure protection is triggered, shutter limits hit etc.
Umm.... if you made it this far, help yourself to another :xmas cookie and beverage :haha
edit:
The "sleep" column indicates how long the script slept between shots. If it's negative, it means the script wasn't able to maintain the requested frame rate.
edit:
Might help if I attached the script
-
A successful, if boring example. At the start exposure is driven by under exposure protection, pushing the metered area almost 1 stop over neutral but not quite far enough to test that limit. By about the middle the sky darkens enough for the metered area to be neutral. Looking at the log I think there might be a bug in the transition, but it didn't affect the results much in this case.
Total exposure change is about 1.8 stops. Individual exposure changes are almost always ~2 APEX96 units at a time.
http://youtu.be/hivtoQU7xJQ (http://youtu.be/hivtoQU7xJQ)
Aforementioned glitches
http://youtu.be/RUMe06EQFd8 (http://youtu.be/RUMe06EQFd8)
-
Updated script attached. This should fix the bug with the transition from over/under protection and regular metering. I also switched the sign of d_ev, so now positive is more exposure (longer shutter speed, higher ISO) and added indicators for the over / under protection in the draw function.
A readme with full details is included in the zip.
-
On the road right now. Would you mind posting an S100 101a build for me? I'll probably have a bit of time to play with it if so.
TIA
-
On the road right now. Would you mind posting an S100 101a build for me? I'll probably have a bit of time to play with it if so.
The C code is in the trunk autobuild.
edit:
If for some reason you can download here an not the autobuild, I'd be happy to post one. You also need rawoplib.lua, which is included in the trunk CHDK/LUALIB
Also, as noted earlier I haven't added Av or ND support. Since the script keeps half press down, it should stick with whatever the camera used for the first shot, and exposure calculations should not be affected.
I'd suggest using a larger, sparser meter than the script default, something like 1200 step 5 maybe
-
I've got 3843 with me. Is that current enough? Otherwise I can update.
-
I've got 3843 with me. Is that current enough? Otherwise I can update.
Need to update, C code went in at 3848. Suggest 3859 or later.
-
Another boring test. Where are my clouds >:(
http://youtu.be/IGG8NOj89m4 (http://youtu.be/IGG8NOj89m4)
-
I think you need to get some skis & a toboggan and head up to Spenser's Butte?
-
I think you need to get some skis & a toboggan and head up to Spenser's Butte?
Yeah, I should totally bill the customer for travel to appropriate testing locations... oh, right :P
Meanwhile... Clouds!
http://youtu.be/NzLj45IcL_0 (http://youtu.be/NzLj45IcL_0)
This worked pretty well, still didn't challenge the metering code much. It looks like the value I consider over-exposed (currently hard coded 1000-1023) is a bit too high. The over expose % stayed <= 1 even though the brightest clouds were blown out in the Canon jpeg. OTOH, in a scene like this letting them be blown out may be the better choice (the sun is just out of frame on the right).
-
In trunk build 3861, I added rawop.ev96_to_raw which takes an APEX96-ish value (0 = neutral) and converts it to a raw value. Among other things, this lets you get shot_histogram ranges in terms of EV. Additionally, I changed the definition of raw_neutral exposed to Lua to include blacklevel, so it represents an actual raw value.
Updated script attached. Trunk build 3862 or later is required.
This update fixes a nasty flipped sign bug in under exp handling, plus the following changes:
Improved transition from under exposure limiting.
Exposure ranges considered over or under exposed are now user selected, in terms of exposure stops. Note they are specified in slightly different ways:
Over exposure is fractions of a stop below white-level (so 1 = 1 stop below white level, 96 = 1/96th of stop). On my cameras there are ~2.5 stops between neutral and white level.
Under exposure is stops below neutral. This is more useful than specifying over black level, since on a log scale there are several stops over black level that have negligible significance. On my cameras you tend to start getting (actual) black level pixels ~5 stops below neutral (shooting a uniform neutral subject.)
Shutdown options now split into individual booleans, and now includes and option to shut down when SD space is low.
I'm not really satisfied with the way the exposure logic is put together at the moment. I need a general way to smoothly transition between the different regimes. The current code has special cases to reduce flapping at various transitions, but doesn't catch them all and is hard to verify. That said, it does a pretty good job some of the time :)
Here's and example hitting the over exposure limits
http://youtu.be/QiddcAOax8o (http://youtu.be/QiddcAOax8o)
edit:
12/30 attached zip containing correct files :-X
-
Here's and example hitting the over exposure limits
Wow - quite the amazing job of ignoring the sun and keeping everything else constant !
-
An update which contains no end user improvements.
I mentioned trouble testing earlier... it's hard to get a scene that exercises the different scenarios, and if I see it do something bad it can be hard to reproduce and verify fixes.
Thinking about this last night, I realized if I refactored the metering code a bit, I could feed it fake meter and over/under values. Then I realized that the log contains enough information to get back out the scene brightness. So I added a simulation mode that takes the normal log output as input (named rawopsim.csv), and does all the exposure calculation without actually shooting. The log can also be edited to create specific scenarios.
Here's a sunset with the current version:
http://youtu.be/KJ3d-fk6ZFI?list=UUvBOkM9SkswQao0KN7Dj0jA (http://youtu.be/KJ3d-fk6ZFI?list=UUvBOkM9SkswQao0KN7Dj0jA)
This behaved pretty much perfectly (aside from the camera getting bumped and the crud flapping in front of the lens :haha), but aesthetically it would have been better if the under exposure protection had been set to be more aggressive (this was 10% / neutral -5).
-
I noticed you are not managing the ND filter (you mentioned that originally I know). When I was working on the Lua version of the yass script I ran into issues where the camera would insert the filter and ruin the exposure. IIRC it was usually when the sun came out from behind the clouds and intervening tree branches.
A set_nd_filter(2) at the start of the script solved that.
-
When I was working on the Lua version of the yass script I ran into issues where the camera would insert the filter and ruin the exposure. IIRC it was usually when the sun came out from behind the clouds and intervening tree branches.
Since rawopint holds down halfshoot for the entire sequence, the Canon firmware won't change the ND (or focus, or aperture) after the first shot. It could still start with the ND in, if the sun happened to be in the FOV.
Setting the ND out for the first is probably a good idea. Originally, I was thinking about having try to maintain the initial Canon exposure value, but just using the neutral value (and perhaps an EV comp option later) seems fine. This does mean that it takes several frames for the metering to settle.
I would like to have proper ND support to be able to put the ND in over specific shutter speeds, but one sticking point is getting the exact value of the ND. If the Canon firmware puts it in, you can compare the Av propcase and the min av propcase (at least for cams that don't have both iris and ND), but with CHDK control, the propcases don't seem to be affected by ND.
-
Then I realized that the log contains enough information to get back out the scene brightness. So I added a simulation mode that takes the normal log output as input (named rawopsim.csv), and does all the exposure calculation without actually shooting.
I was feeling pretty smart when I did this. Of course, now I realize the under / over pct from the log won't respond to any adjustment, so (with additional hacks) it only "works" in the normal metering regime. :-[
-
Update:
I wrote a new exposure calculation based on weights, which should solve most of the flapping issues.
There is still a little bit due to the low precision of shot_histogram, e.g. if you set over exposure to limit to 3%, you can only "ramp in" the change over ~3 distinct values. This gives me motivation to work on the histogram code...
There are now options to limit how far under/over exposure protection can drive the value of the metered area. These are the Meter high/low thresh and limit described in the readme.
I also made simulate do something useful and fixed number of other bugs.
Sunset:
http://youtu.be/zlQaTRbmn_M (http://youtu.be/zlQaTRbmn_M)
-
In trunk revs 3866 and 3866 I updated the C code and rawoplib.lua support module. Scripts for previous versions will be incompatible.
I moved the rgb rectangle function to C, which speeds up drawing > 10x. I also implemented the cfa aware get and set pixel functions in C.
I have renamed these functions to *_rgbg. Functions that write take four color arguments in r,g,b,g order with the second g optional. Getters return in the same order, so lua callers can just ignore the last g if they want.
A added a test script CHDK/SCRIPTS/TEST/rawdraw.lua which tests some of the drawing functions. This will be expanded later. drtest.lua is also update for the changes.
Updated rawopint.lua attached. Not many functional changes: ND filter is forced out per waterwingz suggestion, and the debug display is reworked.
Boring sunset showing the previously mentioned histogram precision issue
http://youtu.be/Moji_gNpm6o (http://youtu.be/Moji_gNpm6o)
Edit:
Updated scripts fixes off by one in the Underexp EV menu option, changed video to a slightly less boring one showing the same problem. This run did 4118 shots before low battery shutdown, starting on less than a full charge. Time was 1:49. Target was 1.3 sec/frame, but it got pushed over by shutter time around 3400 shots in.
I think toward the end we see another issue which IIRC lapser noted before, that ISO changes only take effect in some increment greater than a single APEX96 unit (3?) (edit: nope, see http://chdk.setepontos.com/index.php?topic=12165.0 (http://chdk.setepontos.com/index.php?topic=12165.0))
Happy new year!
-
In trunk 3873, I added histogram functionality to rawop. I'm not sure how this will end up fitting with shot_histogram, right now it's something to play with so I can figure out how it really should work.
The following are added
edit: see following posts.
histo=rawop.new_histogram([shift])
Creates a new histogram object, with the number of slots optionally reduced by shift. All other histogram operations are done on the object, like histo:foo() rather than from rawop. The memory used by the histogram is tied to this object, and garbage collected like other lua values.
histo:update(top,left,width,height,xstep,ystep)
Builds a histogram of the specified area. You can use the CFA offsets to build R, G, B histograms.
frac=histo:range(min,max[,'count'])
return number of values in range, either as a fraction in parts per 1000, or total count
total=histo:total_pixels()
returns total number of pixels sampled
Comments / rationalizations / issues:
1) The way shift is specified isn't clear. It should probably specify an absolute number of bits.
2) I will probably change the bit depth to be specified in update, with the memory (re-)allocated if needed. This would also allow optimizing to use shorts if the number of samples is sufficiently low, and only holding on to the memory during the actual use.
3) Does it makes sense to allow multiple simultaneous histograms? IMO, it probably does. You can update the same one as many times as you like, but it may be useful to have R G B all active at the same time, or have an overall scene histogram and separate one for a meter area or something.
4) It's easy to get overflow if you work with raw counts on a significant fraction of the frame buffer. (e.g 0.5 MP, * 4095 white level will just about wrap negative)
5) Are raw counts useful at all? I think so, you can do things like finding peaks more efficiently.
6) Is it useful to allow different bit depths? I think it is, since it allows allows you to trade memory and precision. It also gives you a really quick way to bin down to say 8 levels.
7) If you could average the histogram, you could meter the histogram area without the cost of reading the frame buffer again. However, it's easy to run into overflow problems if you don't do it in fp or 64 bit.
8) Is there any reason to allow range() to produce a fraction other than parts per thousand? I don't immediately see a need for more precision, and getting to % in lua is trivial.
Updated version of rawopint is attached. This uses the new histogram, and the parts per thousand precision eliminates some of the bad over/under protection behavior.
-
Video with the new code. I didn't get the sun close enough to the FOV to trigger overexposure. There's some drops of melting frost falling on the lens toward the end.
http://youtu.be/u1iaoaYVEvI (http://youtu.be/u1iaoaYVEvI)
I'm still seeing occasional glitches where the exposure change without any change commanded by the script or apparent change in external lighting. E.g. around 2:14
The log looks like
sv sv96 tv tv96 meter meter96 d_ev
75 371 0.000676 1011 724 0 100 0
75 371 0.000676 1011 722 0 100 0
75 371 0.000676 1011 695 -6 100 3
75 371 0.000691 1008 734 2 100 0
75 371 0.000691 1008 739 3 100 -2
For the first two frames, the shutter and ISO are constant, and the script hasn't requested any changes, since the scene is within rounding error of neutral. On the third frame, the meter value drops 6 (1/16th stop), and the exposure code requests +3 to start compensating. Smoothing prevents it from requesting the whole 6. On the next frame, the meter is over by 2, meaning the scene brightness has returned to essentially where it was before.
I don't think this is a real lighting change, since it affects the whole scene (although maybe shadow -> less glare on lens?)
I mostly been using the D10 for testing, I need to see if this happens with my other cams.
-
Another application I've had in mind for raw metering is a psuedo auto exposure in continuous mode. So I wrote a script called contae.lua which does this.
It borrows a bunch of code from rawopint, but instead of being an intervalometer, it's meant to be used while shooting bursts of shots manually. While the script is running, keys other than full shoot and menu are just passed to the canon firmware. So you can open the func menu, adjust mf etc without leaving alt or exiting the script. Menu is currently set as the exit key.
In continuous mode, shoot_full is basically passed through as well, making the camera shoot for as long as you have it held. If not in continuous mode, it simulates continuous with the "hold half press and click shoot" strategy for as long as you hold the shutter down.
In either case, shots after the first shot in a burst are fed though the the same exposure system used by rawopint, with smoothing disabled and a large default ev step.
This isn't the same as real auto exposure, since it only reacts to the previous frame, but it should do a lot better than fixed exposure if you track a subject from sunlight to shade or something like that.
The interface is still in proof of concept state, but it works well enough to use for basic shooting. It doesn't know anything about jog dials etc, and menu isn't a great choice for the exit key.
For both scripts, I added an EV shift option, which lets you set the target meter level instead of the default neutral value. I also added an option to use the exposure level from the first shot as the target. They can be combined to get e.g. +1 ev from whatever the Canon firmware used for the first shot.
Attached zip contains both contae and rawopint, along with readme files for each.
I have shot a bunch of test time lapses with the current code (see my youtube channel from the previous links if you haven't had enough ;)). I think it's working pretty well, but I still see the glitches mentioned earlier, both on d10 and a540.
-
Updated scripts for trunk r1911 param changes. Also includes a few other minor changes.
both scripts:
Also changed the defaults for over and under ev ranges based on test results.
rawopint:
Added options to set canon image resolution and CHDK raw for duration of the script.
Reduce EV change if sign flipped.
contae:
Buffer log until burst is done.
Pretty sunset:
http://youtu.be/87Sj8nzbmas (http://youtu.be/87Sj8nzbmas)
Edit:
Oops, I was supposed to attach a file here, wasn't I?
-
1) The way shift is specified isn't clear. It should probably specify an absolute number of bits.
2) I will probably change the bit depth to be specified in update, with the memory (re-)allocated if needed. This would also allow optimizing to use shorts if the number of samples is sufficiently low, and only holding on to the memory during the actual use.
I did this in trunk 3939.
The functions are now
histo=rawop.create_histogram()
Creates a new histogram object. All other histogram operations are done on the object, like histo:foo() rather than from rawop. Memory used by this object, and any histogram data will be garbage collected like other lua values.
(I incorrectly documented this as new_histogram before, but it was named create_histogram from the start)
histo:update(top,left,width,height,xstep,ystep,bits)
Builds a histogram of the specified area and bit depth. You can use the CFA offsets to build R, G, B histograms.
Bits must be >=1 and <= sensor bit depth. Memory for the histogram is allocated based on the bit depth.
This must be called before the histogram can be queried with histo:range() or histo:total_pixels()
frac=histo:range(min,max[,'count'])
return number of values in range, either as a fraction in parts per 1000, or total count
total=histo:total_pixels()
returns total number of pixels sampled
histo:free()
Immediately free any histogram data. histo:update() must be called before the histogram can be queried again. This is not required, the data will be garbage collected normally. This function just gives you the option of explicitly freeing the data as soon as you are done with it.
Updated versions of rawopint and contae are attached. The only update is for compatibility with the API changes.
Sunrise:
http://youtu.be/tkIVDoslzK4 (http://youtu.be/tkIVDoslzK4)
-
Updated scripts
* Fixed under exposure protection. In some previous versions, it always used black level rather than the Underexp -Ev value.
* Meter area is now specified as percentages of sensor active area width and height
* Histogram uses jpeg area rather than active area, to avoid possible dark borders
* The first exposure (start of timelapse for rawopint, start of burst for contae) now applies the same Tv/ISO/ND limits as metered exposures. It currently does not use Ev shift.
* Initial ND support. The "ND Tv Sec/10000" value sets the shutter speed at which the ND is put in (0 disables) This only works for cameras without an adjustable aperture, and the exposure value for the ND will only be accurate if the Canon firmware would have used the ND in the first shot.
* Some changes to the log format and drawing code.
Storm clouds:
http://youtu.be/pOBCfgiNqTw (http://youtu.be/pOBCfgiNqTw)
The shifting exposure as the sun moves through the tree and clouds in the meter area is as designed, but not very nice looking.
Attached plot shows the exposure (tv96, everything else was constant), scene brightness l (shifted to = tv when meter matches target), meter value (1/2 stop ev shift was in effect, also affected by over/under protection) and over and under protection weights.
-
I'll insert a post here to say "very nice work - watching this with great interest". As I'm sure several other people are (and more will once they realize what you have here)!
-
I am most impressed with the results there.. and am also watching this thread with much interest.
-
Updated scripts
* ....
reyalp : all versions of these script have had a note in the readme file that says :
"Doesn't handle aperture adjustments yet."
Which is pretty much self-explanatory.
If I understand the effect of this correctly, on cameras with an adjustable aperture, the f-stop value used is locked at the "half press'" by the Canon firmware and as the script cycles through shooting images, that value is held and the exposure calculations work because that was the f-stop used when the exposure was checked after the half-press?
tl:dr : cameras with adjustable apertures will use the f-stop value selected during the half-press?
-
tl:dr : cameras with adjustable apertures will use the f-stop value selected during the half-press?
Exactly.
-
Mostly had my head buried in the keyboard code, but here's a sunset with ev shift and over exposure protection working.
http://youtu.be/JebE_jawnJc (http://youtu.be/JebE_jawnJc)
Almost wish I'd turned off the debug displays...
-
Hallo,
my name is Joerg and first I must say I’m am really impressed what you are doing.
My intention is to make flicker free (glitch free) time laps, special in the wadden sea (mudflat, tideland). I´m using the S110 and G1X
http://youtu.be/ZyOeVs3Ha6g (http://youtu.be/ZyOeVs3Ha6g)
I hope my English is not too bad and you can understand me.
What I learned after 50000 shots was:
1) Do not use the automatic exposure of the camera
2) Underexposure can be better corrected in post processing than Overexposure
3) Closed aperture makes bad flicker in time laps
https://www.youtube.com/watch?v=PVnqd4f3Kmk (https://www.youtube.com/watch?v=PVnqd4f3Kmk)
What I did before:
I always make time laps with open aperture to avoid aperture flicker.
Then I wrote my own Script with a Tv gain control. I got the Tv Value witch the camera calculated after half press and analyzed this value. Then I overwrite the Tv camera value with my Tv value. I tried to keep my Tv value in an area from +- 1/3 Ev of the camera Tv value. When the Tv value from the camera was outside this values, I start a counter. When the value was inside, I set the counter to zero. After a specified counter value, I changed my Tv value in 1/3 Ev steps. Of course, I have to run it with –Ev to avoid Overexposure. I also planned, to use the histogram to avoid Overexposure. This algorithm is very easy and works very well.
What I really miss are long Tv values to smooth the waves. When I close the aperture with my script, the camera always opens the aperture for measure, which makes bad flicker.
No I found your script rawopint, which have for me the advantage, that the camera does not change the aperture. It was not easy to find it with a title like ‘proposal - script shooting hooks’. I have not known was hooks before….
Yesterday a tried to run the script. Well I have not really understand every parameter yet (which effect has a larger meter area), so I run mostly with defaults. I make a Test against the sun. Basically the result was Ok.
The first thing, which I was wondering was, that the script change the ISO after first shoot from ISO 80 to ISO 160. The Tv values where around 1/30s. Any idea?
Then all pictures had too much Overexposure. Ok the default was 3%. That’s already a lot. I analyzed the pictures (values = 255) and find more the 4%. The different was, I take pictures in 16:9. The JPG includes fewer pixels. This is also the reason, why I can’t see the bar in the debug info near the top.
Next time I will run it with 1% Overexposure. But why I can’t set Overexposure to zero? Especially for time laps with the moon it should be a nice option to have no Overexposure and see structures is the moon. I tried it with my script but unfortunately the moon was less than 0.5% of the picture and the normal histogram delivers only steps in 1%.
What would be also nice to have?
ui_shots=0 for endless shoots.
And a safety mode for the auto focuses like:
get_focus_state() == 0 -> set_focus(get_dofinfo().hyp_dist)
This help you really, when you start a night or with fog. I lost a few series because of focus problems.
I’m looking forward to make more tests with your script….
Greets Joerg.
-
My intention is to make flicker free (glitch free) time laps, special in the wadden sea (mudflat, tideland). I´m using the S110 and G1X
I hope my English is not too bad and you can understand me.
Very nice video, and your English is fine.
The first thing, which I was wondering was, that the script change the ISO after first shoot from ISO 80 to ISO 160. The Tv values where around 1/30s. Any idea?
Were you using G1X or S110?
It sounds like CAM_MARKET_ISO_BASE could be wrong for your camera, see http://chdk.setepontos.com/index.php?topic=12170.0 (http://chdk.setepontos.com/index.php?topic=12170.0)
This value is not set for S110 and probably should be. It should be correct on G1x.
If you can upload the log (rawopint.csv on the root of the SD card) that may also be helpful.
The exposure calculation is only supposed to increase ISO if the shutter speed goes below "ISO adj Tv Sec/100". This value is given in 1/100ths of second, so with the default value ISO should only increase at 1/4 second. Otherwise, it will it should be set at "target ISO".
Note that the default "Target ISO" is 80, but some cameras only go down to 100. Trying to use 80 on these cameras that only go to 100 might give incorrect results. Also, some cameras that have a minimum display value of 80 actually need the input set to ~75 to get 80 in the exif.
Then all pictures had too much Overexposure.
If your camera has the ISO base bug, it would cause bad overexposure.
Next time I will run it with 1% Overexposure. But why I can’t set Overexposure to zero? Especially for time laps with the moon it should be a nice option to have no Overexposure and see structures is the moon. I tried it with my script but unfortunately the moon was less than 0.5% of the picture and the normal histogram delivers only steps in 1%.
Zero would be hard because there's always a few hot pixels, and to smoothly adjust exposure you need need a range to blend the correction in over.
In my script, the Overexp Ev range is the range of values (in exposure stops) treated as over exposure. If you increase this, you can trigger over exposure protection before the pixels are actually saturated.
ui_shots=0 for endless shoots.
Good idea.
And a safety mode for the auto focuses like:
get_focus_state() == 0 -> set_focus(get_dofinfo().hyp_dist)
Yes, I want to include focus control at some point.
-
Hallo,
I have used the S110.
Very nice video,
Yes, thanks, one of my first ones. But unfortunately too much Overexposure and I don’t like the turbulent waves. The waves have to smooth with longer Tv values.
The exposure calculation is only supposed to increase ISO if the shutter speed goes below "ISO adj Tv Sec/100".
That is what I expect.
Note that the default "Target ISO" is 80, but some cameras only go down to 100.
ISO 80 is my camera default for time –laps to get longer Tv values. That is also what I found in the first Picture of EXIF Data. I have never changed it, also not in Scripts. I will run the isobase.lua to see what I get on the S110.
May be some other interesting thing: Another default on my cam is, that I switch always the ND Filter on, to get longer Tv values. But you work on the RAW Buffer. This should not have an influence on your control or?
If your camera has the ISO base bug, it would cause bad overexposure.
Ok, for the first pictures, but then you use the RAW Buffer and you don’t look about the ISO value anymore or?
In my script, the Overexp Ev range is the range of values (in exposure stops) treated as over exposure.
If you increase this, you can trigger over exposure protection before the pixels are actually saturated.
OK, default was ¼. Maybe I go to ½ or 1. But I don’t think it helps for the moon, because there is a big jump from the moon to the background. Maybe zero on ui_exp_over_thresh_pct ist not absolute 0 assorted 0.1% or 0.01%.
-
Here is the log file. It’s a really got idea to have logged al this information… :)
S110 ==> CAM_MARKET_ISO_BASE 200
-
Here is the log file. It’s a really got idea to have logged al this information… :)
S110 ==> CAM_MARKET_ISO_BASE 200
Thanks for confirming. I've checked the change in for both the trunk and 1.3
At one point, having ISO_BASE wrong resulted in a feedback loop, but I think the current code doesn't do that, so it may not explain the over-exposure problems.
The logs are very helpful to see what the code is doing. Most of the fields relevant to the exposure calculation are documented in the readme.
In your log, the under_weight 200 shows that under exposure protection was maxed out. Over exposure protection was triggered too, but not maxed out, so the net result was a positive ev shift of ~1 stop.
You can make under exposure protection less aggressive by increasing "Underexp -Ev" and/or "Underexp Thresh %". The default value for -Ev may be high, there was a bug in this area for a while.
There isn't currently a way to adjust the relative priority of under/over protection. This is done in exp:calc_ev_change() if you want to play with the code.
The sv 380 != cam 476 lines show the ISO_BASE bug, but I don't think this ends up affecting the overall exposure.
I would be interested to see the output from drtest.lua on your cameras. This is included in the scripts/test directory of the 1.4 packages. To use it, aim the camera at a neutral subject with constant lighting, and run the script. The default settings should be OK. It outputs to drtest.csv and on the images. I only need the csv, but the images may help if something is weird.
OK, default was ¼. Maybe I go to ½ or 1. But I don’t think it helps for the moon, because there is a big jump from the moon to the background. Maybe zero on ui_exp_over_thresh_pct ist not absolute 0 assorted 0.1% or 0.01%.
This is a good point, I'll have to do some experimenting and see how it behaves. Up to now, I've mostly been thinking in terms of ignoring the sun or moon and having the rest of the correctly exposed.
Currently the new histogram code only works in 0.1% increments, but perhaps this is an argument for greater precision.
The over exposure protection actually ramps in below the specified percent. If the "ramp in" range doesn't have enough precision, you end up with flickering as at flaps between values.
-
Hallo,
At one point, having ISO_BASE wrong resulted in a feedback loop, but I think the current code doesn't do that, so it may not explain the over-exposure problems.
I make a new test with Overexposure 1%, but still with wrong ISO_BASE. So maybe you can ignore this. Basically Overexposure works fine. I analyzed the pictures, correct it with factor from 4:3 to 16:9 and I got mostly 1%. But from one point it starts to increase…
-
Hallo,
You can make under exposure protection less aggressive by increasing "Underexp -Ev" and/or "Underexp Thresh %".
Doe it makes any problems when I switch it totally off? It reduces the number of rules. I would prefer when the time laps goes darker to the night. Underexposure can be also better corrected in post processing.
I did not really understood, how Underexposure works.
My understanding of your algorithms is: You try to get the box in the middle to a neutral value. If there is no Over or Underexposure every think is OK. If you have Overexposure than this has a higher priority or? But what is when you have Underexposure at the same time? May be Underexposure switching off makes understanding easier.
The sv 380 != cam 476 lines show the ISO_BASE bug
So I will update CHDK…
I would be interested to see the output from drtest.lua on your cameras
I´ll try to run it on the weekend…
Do you know dslrdashboard?
I use it sometimes to make time laps with my EOS1200. In the Module LRTimelapse they transfer pictures via WLAN or USB to a host (Tablet or PC). Then they are calculating some floating average and a Histogram. With the result, they control the DSLR. For sunset they also increasing Tv values first, and on the minimum Tv value they increase ISO to a maximum value (Max Tv and ISO adj Tv is the same). I never had much over-exposure problems with this program, so they must also have a protection. It is interesting to study the ’ lrtimelapse holy grail tutorial’.
-
Doe it makes any problems when I switch it totally off?
You can't actually turn it off, but if you set the margin to -7 it will only count pixels near black level as under exposed, and you could increase the % to 100. I think this would be pretty close to disabling it, but as I mentioned earlier, the current logic ramps in before the thresh % is reached.
Edit: Actually, you should be able to turn it of by setting Thresh % to 0.
Note that the "Meter low thresh" and "Meter low limit" discussed below also have an influence, and will eventually cancel out over exposure protection.
My understanding of your algorithms is: You try to get the box in the middle to a neutral value.
Or an ev shift if you've specified one.
If there is no Over or Underexposure every think is OK. If you have Overexposure than this has a higher priority or? But what is when you have Underexposure at the same time?
Over, under and the box each have a "weight" that controls the contribution to the final exposure. For over and under, the weight is based on square of the fraction of the threshold %.
This means that if values are much less than the threshold, the has a very small contribution, and if it's larger it has a very big contribution. The slope of these quadratics are modified so that over-exposure has more weight for a given fraction. (The /120 and /150 in the code mentioned earlier)
The box also has a quadratic behavior: As the average value is pushed past the limits defined by Meter high thresh Ev and Meter low thresh Ev, the weight is increased based on the fraction of corresponding "limit ev" reached.
The base weight for the box is 100, and all the weights are limited to 200.
The actual change in exposure is based on the combined weights, limited by Max Ev Change. So if both over and under are maxed out, they will cancel out. If over exposure is high, but meter has been pushed far below "Meter low limit", they will balance out.
The reason behind all of this is to vary the exposure smoothly between the different limits, with emphasis on keeping the meter area at a "reasonable" exposure.
Note that this is all a work in progress, and was mainly written to test the C code and provide an example of using the new fatures. I'm open to bug reports and improvements!
The exposure code is quite modular. You should be able replace exp:calc_ev_change with pretty much any logic you want as long as sets self.ev_change at the end (you also need to do something with the code at the top which ensures self.ev_target is set on the first iteration)
I would prefer when the time laps goes darker to the night.
This is currently something my script does not do well, since it always tries to maintain the target exposure at a fixed value. Things only get darker once you hit the shutter and ISO limits.
My idea to handle is adding an option to make the target Ev vary proportional to the absolute brightness of the scene. So for example, your initial shot would be neutral, and every stop of actual brightness would change the target exposure by say 0.1 stop.
-
I make a new test with Overexposure 1%, but still with wrong ISO_BASE. So maybe you can ignore this. Basically Overexposure works fine. I analyzed the pictures, correct it with factor from 4:3 to 16:9 and I got mostly 1%. But from one point it starts to increase…
The under exposure fraction is very high, and the under_weight is hitting 200.
If your scene doesn't actually have big dark areas, there might be some other problem. If the raw active area isn't defined correctly, it could include a lot of masked pixels or something like that. If you can upload a sample DNG somewhere, that might be helpful.
-
Updated scripts
Add shots=0 for unlimited as suggested by c_joerg.
Added "Bv Ev shift %" option, based on the idea I posted earlier. This shifts the target exposure as a percentage of the difference between the Bv value of the fist shot and the most recent meter. So if you set this to 50 and the scene brightness changes by 1 stop, the target exposure will change by 1/2 stop. Setting to 0 gives you the old behavior of always targeting a specific value.
This shift is applied on top of any specified Ev shift.
Note that this is based on the first exposure, because it was easy and I didn't come up with a convenient way of expressing it in absolute terms. For a sunset, it would automatically make the scene start out normally exposed and get progressively darker. For a sunrise to start dark and brighten to "normal" you would need to start with a negative Ev shift.
The shift is subject to the limits set by "Meter high thresh Ev" etc. (edit: nope, buggy in current implementation) Indicators for these values are added to the debug overlay. Note that these limits are relative to any initial Ev shift specified in the "Ev shift" or "Use initial Ev as target".
I haven't tested this extensively yet.
contae.lua is updated to use the new exposure module code, but the Bv Ev shift is hard code to 0 and other than logging, the script is unchanged.
-
Hello,
I have a busy weekend, could not much do…
Update CHDK and 380 != cam 476 is gone…
Then I run drtest.lua, but it seems that there is not much information. The S110 makes 5 pictures, G1X just 3. It looks like, there was an exception but I haven’t had the time to find out…
Ok, upload from DNG is to much. Can I try to send on the EMail which is in the LUA Script?
-
Note that this is all a work in progress, and was mainly written to test the C code and provide an example of using the new fatures. I'm open to bug reports and improvements!
Of course….
If your scene doesn't actually have big dark areas, there might be some other problem.
Pictures against the sun have mostly big dark areas. But when you don`t want to have to much Overexposure, you must let them dark. Yes this scene has big dark areas….
Pictures against the sun are the difficults ones. I think for this condition, Underexposure has to be ignored. I will try it the next time.
I not sure, if it make sense, to move the box also up and down. May be on sun set or rise to move the box to the sky.
-
Then I run drtest.lua, but it seems that there is not much information. The S110 makes 5 pictures, G1X just 3. It looks like, there was an exception but I haven’t had the time to find out…
Since the csv files only have the header row, there must have been some error. If you don't see an error on the screen after running, go to miscellaneous->console in the CHDK menu and select "Display last console"
Ok, upload from DNG is to much. Can I try to send on the EMail which is in the LUA Script?
I'd suggest uploading to a file hosting site like box.com, dropbox, google drive etc, but if you can't do that email is OK.
Pictures against the sun have mostly big dark areas. But when you don`t want to have to much Overexposure, you must let them dark. Yes this scene has big dark areas….
Pictures against the sun are the difficults ones. I think for this condition, Underexposure has to be ignored. I will try it the next time.
You aren't going to get correct exposure of the sun itself without a very heavy filter, so I prefer to just let it be over exposed and keep the rest of the scene reasonable. Maybe that's not what you meant?
-
Updated scripts. This should make the limits work for bv ev shift. This still has some rough edges and I haven't got a good test in real conditions yet, so bugs are likely.
Additionally, I added an option Bv Ev shift base Bv, which lets you specify an absolute Bv value rather than using the first shot. This should correspond reasonably well to real Bv values, so ~10 would target neutral exposure in full sun.
Finally, I added an option "Gauge Y offset %" for the vertical position of the debug display, so you can make it show up if the jpeg isn't the same aspect ratio as the sensor. The value is in % of sensor height, except zero which uses a small fixed offset from the top.
-
Here's a boring but mostly successful example of the bv ev shift setting at 30%
http://youtu.be/1JaED5mztjI (http://youtu.be/1JaED5mztjI)
This gives a ~4 stop drop in exposure over a ~16 stop drop in actual scene brightness.
There is glitch toward the end: When the clouds roll in, under exposure protection pushes the exposure up. As the Bv plot shows, the sky didn't actually get brighter. Other than that it worked pretty much exactly as I hoped.
While going through the log, another thought occurred to me:
Rather than changing the exposure, you could theoretically just expose all the frames "correctly" and use the Bv adjust the brightness at whatever scale you want after the fact. Not sure what tools you'd use to do it though.
-
Here's a boring but mostly successful example of the bv ev shift setting at 30%
Yes, I agree. Fantastic. That is how I would prefer it.
Overexposure and Underexposure are defaults?
When you make videos, I would be interest about changes to the default settings.
I’m wondering that the aperture (av96) changes.
I thought the script doesn't handle aperture adjustments yet.
I run the script in Av Mode. Is this OK? I have not seen any restriction.
I try to run drtest.lua again and look about the exception.
When it is necessary to share more files, I will install a dropbox. Until now, I don’t have something like this.
When I talking about pictures against the sun, I mean sunset and sunrise, like in your last video. Of course, I always use the ND filter from the cam to protect the cam a little bit. I also check the sensor temperature in my script and make a shutdown when the temperature exceeds 59 degree. Does this make sense in your script?
lets you specify an absolute Bv value rather than using the first shot.
The first shoots with the script have always a lot Overexposure (not seen on my plot, but it’s in the log file). After 5-10 shoots it is Ok. Can this make problems when using the first shot?
This is the time laps video from the test with 1% Overexposure where the overexposure increases after around picture 225 (9s on the video). May be its helpful.
http://youtu.be/_6X7-tTv29M (http://youtu.be/_6X7-tTv29M)
-
Run drtest with a different background again.
S110 runs OK but G1X makes following Error:
A/CHDK/SCRIPTS/TEST/DRTEST.LUA:388: bad argument #1 to 'raw_to_ev96' (number expected, got nil)
-
Yes, I agree. Fantastic. That is how I would prefer it.
Overexposure and Underexposure are defaults?
Ev shift 1/4
Meter low thresh 3
Meter low limit 4
Overexp thresh% 2
Overexp Ev 1/4
Underexp thresh% 10
Underexp -Ev 5
I’m wondering that the aperture (av96) changes.
I thought the script doesn't handle aperture adjustments yet.
This is actually the ND filter, but for exposure calculation it's treated as Av. Note that the script currently supports this only on cameras that don't have an iris. On cameras like the S110 or G1X that have both, ND is currently ignored by the script.
I run the script in Av Mode. Is this OK? I have not seen any restriction.
It shouldn't matter. As far as I know, the canon firmware won't adjust settings while it's shooting. I'd stay away from weird scene modes.
When it is necessary to share more files, I will install a dropbox. Until now, I don’t have something like this.
With most of those services you can also just upload through the web site if you don't want to install the app.
When I talking about pictures against the sun, I mean sunset and sunrise, like in your last video. Of course, I always use the ND filter from the cam to protect the cam a little bit. I also check the sensor temperature in my script and make a shutdown when the temperature exceeds 59 degree. Does this make sense in your script?
That's not a bad idea, I can put in a temperature shutdown option. I've wondered how bad it is to point the sun, but I haven't seen problems yet. The sensor and optical temperature is recorded in the logs, so far I haven't seen any crazy values. FWIW, on some cameras at least the reported "sensor" temperature is somewhere on the sensor PCB, but not part of the actual image sensor chip.
The first shoots with the script have always a lot Overexposure (not seen on my plot, but it’s in the log file). After 5-10 shoots it is Ok. Can this make problems when using the first shot?
Maybe, but I don't think it should be a big problem in most cases. Since it uses absolute brightness, the exact exposure doesn't matter, but the Bv will be underestimated if the pixels in the meter area are saturated.
This is the time laps video from the test with 1% Overexposure where the overexposure increases after around picture 225 (9s on the video). May be its helpful.
It would be helpful to see the debug displays. Is this the same sequence from the log you posted earlier? It looks like quite a large area of sky is overexposed, so a larger "Overexp Ev" might help. If you don't mind having the rest of the scene dark, you could also reduce or disable under exposure protection (larger Underexp thresh% and Underexp -Ev)
I'll have a look at the drtest stuff later.
-
Is this the same sequence from the log you posted earlier?
Yes, but without the first few pictures.
but I haven't seen problems yet.
I also not until yet... hope the ND Filter helps a little bit. but have always a bad feeling....
The sensor and optical temperature is recorded in the logs, so far I haven't seen any crazy values.
I have it also logged in my scripts, highest temperature in my logs was 56 degree Celsius.
-
When you think, the parameter from your last time lapse are good values to start with, then you should put them as defaults.
Who do you make the plot from the log file? Do you have a special tool for it?
It would be helpful to see the debug displays.
Yes of course. When I make both, JPG and RAW, than I would have the debug information probable in the RAW or? RAW has always the full size senor size.
I’m always getting a little bit scared, to lose a nice scene with this option. OK, as long I’m testing, it should be OK.
But, my understanding was, all the information is also in the log file or? So it is possible, to put this bar later on in the JPG Files. I’m using MATLAB. In MATLAB it’s very easy to open and writing JPG Files. In MATLAB you have a matrix of R G B data which can be easy modified. When I have the time, I will try it.
-
When you think, the parameter from your last time lapse are good values to start with, then you should put them as defaults.
Yes, I try to update the defaults as find better values.
Who do you make the plot from the log file? Do you have a special tool for it?
I used the chart tool in libreoffice spreadsheet program. It's not very flexible and hangs sometimes but it's easy.
Yes of course. When I make both, JPG and RAW, than I would have the debug information probable in the RAW or? RAW has always the full size senor size.
It probably would, but it's probably not worth using raw just for that. You could shoot native sensor resolution jpeg and crop to 16:9 when you make the video. You can turn off the meter box completely. I don't know what you use to make video, but with ffmpeg it's very easy to crop.
Do you know what aspect ratio the camera was set to for the DNG you sent? I'm pretty sure the camera will always do full frame raw, but it would be good to verify just in case. If the camera only read out the rows it needs, that would definitely cause trouble.
But, my understanding was, all the information is also in the log file or? So it is possible, to put this bar later on in the JPG Files. I’m using MATLAB. In MATLAB it’s very easy to open and writing JPG Files. In MATLAB you have a matrix of R G B data which can be easy modified. When I have the time, I will try it.
Yes, all the information in should be in the log. For me it's convenient to have on the jpeg so I can tell visually where something happened, but as you say it's not good if the video comes out nice.
Regarding your S110 drtest results:
I found the active area was not defined correctly for CHDK (http://chdk.setepontos.com/index.php?topic=12264.0 (http://chdk.setepontos.com/index.php?topic=12264.0)) This would make functions that work on color not correct, but shouldn't affect the exposure metering in rawopint. The colors in the debug display would have been wrong.
I haven't looked into the G1x failure much yet. I think the line number is actually 386, the 6 and 8 can be hard to tell apart in the CHDK display.
-
Do you know what aspect ratio the camera was set to for the DNG you sent?
I just checked the JPG from this shoot. It is 4000*3000 so the ratio must be 4:3.
you could shoot native sensor resolution jpeg and crop to 16:9 when you make the video..
Yes of course. The process is not the problem. The reason, why I made the picture in 16:9 was that I don’t want to lose something from the scene, when I cut the pictures later on to 16:9. May be some lines on the Display for 16:9 should be helpful. Is there any option in CHDK to do this? But I think to generate lines in a Script should be not a problem.
-
I made just a short test. Only 28 Pictures. Overexposur 2%, Underexposure 0%. Ok it was a stupid scene. Until picture 16 it goes down to 2% but then it increases again.... Frame rate of the video is only 2 frames/s. The weather is not so good for longer test…..
http://youtu.be/OIwHyH7SVfc (http://youtu.be/OIwHyH7SVfc)
-
Do you know what aspect ratio the camera was set to for the DNG you sent?
I just checked the JPG from this shoot. It is 4000*3000 so the ratio must be 4:3.
Ok, just to be 100% sure, can you can take a DNG with the jpeg aspect ratio set to 16:9, and check that the DNG has image in the whole 4:3 frame? I'm 99% sure it will but might as well make that 100%
May be some lines on the Display for 16:9 should be helpful. Is there any option in CHDK to do this?
Do you mean just for framing? CHDK lets you create custom grids. The canon firmware might have a 16:9 grid as well. The script also lets you set the resolution (which usually includes aspect ratio) in the "Image size" setting, so you could compose in 16:9 and have the script to something else. Maybe I misunderstood?
I think your last test shows is expected behavior. Overexp protection only reaches max weight a bit above the "limit" value due to the /120 "slope" factor in calc_ev_change, so the meter being below target partially cancels it out. You could set the limit down to 1%, or increase the Overexp Ev range. I'd suggest the second option, 1/4 stop below over exposure really isn't much.
You could also change the /120 to /100 to make overexposure protection stick closer to the actual limit. I originally put in these factors when histogram precision was limited to 1% increments.
-
Do you mean just for framing?
Yes, just for framing.
Ok now I understood a little bit more. The bar extending from right to left, representing overexposed pixels, represents only the meter area and not the whole frame.
The meter area in the scene is in the shadow. Not there where I want to have. Does it make sense, for scene like this, to have a much larger meter area, may be mostly 90% ? Or, as I suggest before, having a parameter for moving meter area up and down?
A meter area from 10% is something what the cam is doing in spot or partial metering mode. 90-100% is an average metering mode.
You could also change the /120 to /100 to make overexposure protection stick closer
You mean in the code...
In your test you made Ev shift +1/4. That means you made the whole scene a little bit more brightness?
-
Ok, just to be 100% sure, can you can take a DNG with the jpeg aspect ratio set to 16:9, and check that the DNG has image in the whole 4:3 frame?
I confirm, the jpeg ratio has no influence to DNG. DNG has always the same size.
A grid is Ok for 16:9. But I am wondering, there is no 16to9grid. But I think it is easy to make one.
What about native lua calls. They where not enabled.....
-
The bar extending from right to left, representing overexposed pixels, represents only the meter area and not the whole frame.
Correct, the top bar with the tick marks is the meter value.
The meter area in the scene is in the shadow. Not there where I want to have. Does it make sense, for scene like this, to have a much larger meter area, may be mostly 90% ?
Yes, there is no reason not to. You may need to increase "Meter step", there will be an error when you try to run the script if its too small. The value should be odd so it samples all the colors of the bayer pattern.
Or, as I suggest before, having a parameter for moving meter area up and down?
Yes, I'd like to make the meter area configurable. A simple position wouldn't be hard to add, I will try to put that in the next version.
You could also change the /120 to /100 to make overexposure protection stick closer
You mean in the code...
Yes.
In your test you made Ev shift +1/4. That means you made the whole scene a little bit more brightness?
This means that the target exposure is slightly brighter than the default "neutral" value. This will still be modified by the over / under exposure limits. The meter was mostly on sky, and I find the sky looks better if it's a bit brighter than the default "neutral".
What about native lua calls. They where not enabled.....
Not needed for this script.
-
Nice sunset. Meter area was on the right place. Interval time was 4s and max Tv also 4s. So the Interval increases when it gets dark. But this doesn’t matter… unfortunate I have not switch meter area of…. But I will find a way in postprocessing….
http://youtu.be/9suSmmt_d2A (http://youtu.be/9suSmmt_d2A)
-
Nice sunset.
Interval time was 4s and max Tv also 4s. So the Interval increases when it gets dark.
Yeah, not too much can be done about that. I've thought about making a setting to keep integer multiples of the interval (if it can't do 4, it does 8, 12 etc) so you could keep the same time scale by duplicating or interpolating frames, but I don't know if it would be worth it.
After trying to do some moon rises, I agree with you that better than 1% precision is needed for over exposure control.
-
Hello,
I've thought about making a setting to keep integer multiples of the interval (if it can't do 4, it does 8, 12 etc) so you could keep the same time scale by duplicating or interpolating frames, but I don't know if it would be worth it.
I think it is OK as it is.
There are two thinks which conspicuous me.
1)I run both cameras (S110 and G1X) with ‘Review Off’
In the script ui_display_mode_t=0 so I expect the Display on all the time or?
On S110 the Display is off all the time after I start the script.
On G1X I see the Review picture for the full interval time (in the Test 8s).
2) I set ui_sv_target_mkt=80 in the script. I found this value also in the EXIF data on the G1X. In technical data of the G1X I read that the minimum ISO is 100. Does the G1X works correct with ISO 100?
-
1)I run both cameras (S110 and G1X) with ‘Review Off’
In the script ui_display_mode_t=0 so I expect the Display on all the time or?
On S110 the Display is off all the time after I start the script.
On G1X I see the Review picture for the full interval time (in the Test 8s).
If display is set to ON, the script should not try to change the display state at all.
Can you tell if the display is actually off (no backlight) or black, with the backlight on?
The difference you see may be a difference in camera behavior. The script acts as if you are holding the shutter half way and clicking full shoot, without letting up on the half.
On my cameras, with review OFF it seems to show the last image between clicks, but I don't know if this is true for every camera or if it depends on some canon settings.
2) I set ui_sv_target_mkt=80 in the script. I found this value also in the EXIF data on the G1X. In technical data of the G1X I read that the minimum ISO is 100. Does the G1X works correct with ISO 100?
G1X should definitely work with 100, to find out if it works with lower values, you would need to test. You can use the script in http://chdk.setepontos.com/index.php?topic=12165.msg120951#msg120951 (http://chdk.setepontos.com/index.php?topic=12165.msg120951#msg120951) to test this.
-
Can you tell if the display is actually off (no backlight) or black, with the backlight on?
For a while I made a test with set_backlight(0), set_lcd_display(0) and both together on the S110 and measured the current. I don’t see any difference on the display form S110 if I do set_backlight(0) or set_lcd_display(0). I booth cases it looks black for me.
The script acts as if you are holding the shutter half way and clicking full shoot, without letting up on the half.
Yes, that’s the different to my tests…
On my cameras, with review OFF it seems to show the last image between clicks,
On the G1X as well. I see the last image the full interval. But when I change review to 2s, I see last picture for 2s. On S110 with review OFF I see last picture just flashing….
I will run isoinc.lua…
-
Lapser suggested (http://chdk.setepontos.com/index.php?topic=12258.msg120815#msg120815) that flickering happens at high shutters speeds. The attached plot from my last run appears to support that.
I am also starting to suspect that the black frames I mentioned in that thread are not a hardware failure but are caused by some kind of timing/scheduling issue related to the script hooks. It does not seem to occur if I have the debug display enabled (with takes > 10ms, and so makes the kbd_task yield), where it occurs ~1% of the time if I don't. In this test, I added a 10ms sleep, and the problem didn't occur in ~4300 frames. :-[
flickery video
https://youtu.be/HaxUQrLvXwE (https://youtu.be/HaxUQrLvXwE)
-
I’m still thing of the Display behavior.
I have a small test script, which holds the camera in half press, like in your script. When I run the script, I have live view between the shoots (S110 and G1X).
When I run your script
ui_display_mode_t=0, the script should not try to change the display state at all true?
On S110 I see the Review picture just flashing
On G1X I see the Review picture for the full interval time (in the Test 8s).
So where is the difference?
-
When I run your script
ui_display_mode_t=0, the script should not try to change the display state at all true?
Correct, though there could be bugs ;)
On S110 I see the Review picture just flashing
On G1X I see the Review picture for the full interval time (in the Test 8s).
So where is the difference?
I'd guess this is probably because my script uses the "shutter hook" to keep the interval exact. It presses shoot_full for the next shot, and then makes the code wait somewhere very close to where the actual exposure would start. This gives a very precise interval, but where the hook waits is likely to be implemented differently on different cameras.
I said earlier that my cams show the review, but now that I look at D10 it seems like it behaves more like you describe the S110, the image just flashes and stays black for most of the "wait" time. I didn't really notice before because I usually use an interval close to the maximum shooting speed.
-
I think toward the end we see another issue which IIRC lapser noted before, that ISO changes only take effect in some increment greater than a single APEX96 unit (3?)
I've been away from coding for awhile, but I remember tracking the ISO problem down to the set_sv96(sv) function in CHDK. The glitches happen when it changes the "base" ISO.
I fixed it by using my own code for set_sv96. This is how I tested it in Lua:
===
props=require("propcase")
pSV=props.SV -- real sv
pDELTA_SV=props.DELTA_SV
. . .
(calculate desired sv)
set_prop(pDELTA_SV,sv-get_prop(pSV)+get_prop(pDELTA_SV))
set_prop(pSV,sv)
===
Or, you can use this code in C to replace set_sv96()
===
shooting_set_prop(PROPCASE_DELTA_SV,
sv96-shooting_get_prop(PROPCASE_SV)+shooting_get_prop(PROPCASE_DELTA_SV));
shooting_set_prop(PROPCASE_SV,sv96);
===
Maybe you could re-run your ISO test using the propcase method above instead of set_sv96() and see if that fixes it?
My theory on what's happening is that the "base" ISO is calibrated to match film cameras, so base ISO 200 isn't exactly twice as fast as ISO 100. But if you leave the base ISO unchanged and just use the "delta_sv" to set the ISO speed, then everything works as expected.
-
I said earlier that my cams show the review, but now that I look at D10 it seems like it behaves more like you describe the S110, the image just flashes and stays black for most of the "wait" time. I didn't really notice before because I usually use an interval close to the maximum shooting speed.
If you use continuous mode and just hold the shutter down, it displays the pictures on the screen in real time. That's one big advantage of continuous mode. You can see what you're doing. Review should be set to off.
The other advantage is that you get a faster maximum shot rate, about 2 shots per second, in continuous mode.
-
I think toward the end we see another issue which IIRC lapser noted before, that ISO changes only take effect in some increment greater than a single APEX96 unit (3?)
I've been away from coding for awhile, but I remember tracking the ISO problem down to the set_sv96(sv) function in CHDK. The glitches happen when it changes the "base" ISO.
I verified later that this was not actually what I was seeing:
http://chdk.setepontos.com/index.php?topic=12165.0 (http://chdk.setepontos.com/index.php?topic=12165.0)
Based on the those results, I believe the current CHDK ISO override code works very well. That code was reworked significantly a while back, so it's possible the version you worked on behaved differently.
The other advantage is that you get a faster maximum shot rate, about 2 shots per second, in continuous mode.
I thought that too, but in my testing on the cameras that I have, the "hold half press / click full press" approach gives very similar performance to continuous mode (http://chdk.setepontos.com/index.php?topic=11527.0 (http://chdk.setepontos.com/index.php?topic=11527.0)).
That said, I do intend to add continuous support to my script at some point. It should be straightforward, I already have it in the contae script.
The continuous mode review is nicer.
-
I did some more testing on the D10 shutter issues over the last few days.
I have never seen the issue with the debug drawing on, > 12k shots since I first saw the issue.
Overlay off, sleep(10) before the raw hook continue: no failures in ~4300 shots
Overlay off, set_yield(-1,50). I figured if sleep made it go away, yielding less would make it happen more often. Nope. No failures in >6800 shots.
OK, maybe it's random glitchy hardware and it's not doing it any more? Go back to the original version that failed:
Overlay off, no set_yield or sleep - 21 failures in ~5400 shots.
The last run didn't use super short shutter speeds (all longer than 1/1000) and many of the failed shots were at the longer end.
???
-
The other advantage is that you get a faster maximum shot rate, about 2 shots per second, in continuous mode.
I thought that too, but in my testing on the cameras that I have, the "hold half press / click full press" approach gives very similar performance to continuous mode (http://chdk.setepontos.com/index.php?topic=11527.0 (http://chdk.setepontos.com/index.php?topic=11527.0)).
inuous mode review is nicer.
If you're saving DNG files or writing log files in continuous mode, that can slow things up. Try holding the shutter down without CHDK active and compare that to the rate your script gets. You could also try running the script without writing a log and see if that's faster.
I get a maximum of about 1 shot per second in single shot mode, and 2 shots per second in continuous. This is an example done at 2 shots per second:
https://www.youtube.com/watch?v=b_nVZbV-MR4 (https://www.youtube.com/watch?v=b_nVZbV-MR4)
-
I get a maximum of about 1 shot per second in single shot mode, and 2 shots per second in continuous. This is an example done at 2 shots per second:
Canon continuous mode spec for D10 is 1.1 FPS. The script does around 0.9 FPS. Other cameras may well behave differently.
-
Overlay off, set_yield(-1,50). I figured if sleep made it go away, yielding less would make it happen more often. Nope. No failures in >6800 shots.
I did a further test with set_yield(-1,-1): >7100 shots with no failures.
At this point I'm fairly confident that it's a timing issue related to the script rather than a hardware failure. The exact scenario isn't very clear though. The normal shooting cycle goes like
- wait_click 10 to check for key presses
- press shoot full
- sleep until remote hook is hit
- sleep for as long as needed to achieve desired interval, typically a few hundred ms in my tests
- continue from remote hook
- release shoot full
- sleep until raw hook is reached
- do some intensive processing: meter, exposure calculation, debug drawing
* meter and drawing can take 10s of ms, so will likely result in a yield under default settings.
* there is also a lot of code, so the count hook might also trigger one or more yields
- continue from raw hook
- write log line
- do lua garbage collection step
- goto 1
The above works without issue if debug drawing is enabled. This typically adds 10-20 ms on d10, so would normally cause one extra yield inside the raw hook.
Metering takes a similar amount of time, so with metering alone, there is probably at least one in the raw hook. In the cases that failed, the meter time is shown as 10 on both failed and successful shot.
Both meter and drawing will tie up the CPU pretty solidly when they are actually running, although the dryos scheduler might preempt them.
capt_seq is blocked for the duration of the hook.
The two "fixes" I have found are
1) adding an extra yield inside the raw hook, with sleep(10)
2) preventing any yield with set_yield.
???
The script yields almost immediately after the raw hook as been released, so it's not clear why 10ms inside the hook should make a difference. The obvious difference is that capt_seq has been allowed to continue, which will keep the cpu busying for a while creating the jpeg and may also interact with other tasks.
If there was a way to figure out when the shutter normally opens, that could provide some clues.
-
http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252 (http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252)
We found a timing related problem on the G1X where it appears that another task was modifying the raw buffer. It happened with my time lapse script in single shot, but not in continuous mode, and only when the exposure time was >1 second. We never figured out what caused it, but I was able to work around it by always running in continuous mode.
It's possible that some of the Canon image processing settings are triggering the other task to modify the raw buffer. In my case, I think it may have been related to high ISO noise reduction that was an advertised new feature on the G1X.
One thing I thought about, but never looked into, was if somehow the base raw buffer address was incorrect. That might be worth looking at for your problem.
I also remember a timing issue in capt_seq_hook_set_nr(), which is where I put my pre-shot delay. I had to do the delay right at the beginning, before it calls set_nrflag(). I doubt if this is related to your issue, though.
-
If you're suspecting placement issues with the raw hook, moving it to developseqtask could be a possibility (only done in the sx170 port, see this year's posts in the porting thread (http://chdk.setepontos.com/index.php?topic=11164.msg119647#msg119647)).
-
One thing I thought about, but never looked into, was if somehow the base raw buffer address was incorrect. That might be worth looking at for your problem.
I am very confident the failed shots are correct image data with the shutter closed, because I see the same hot pixels and noise as I see in a normal dark frame. The meter and histogram also see black level rather than 0.
If there's a timing issue, it's something causing the part of the firmware that controls the shutter to behave abnormally.
I initially dismissed this as hardware failure, since it is clearly connected to the mechanical shutter, but this seems untenable given that some yield and debug draw settings reliably show a 0.5%-1% failure rate, while others show no failures over tens of thousands of shots.
I could just throw in one of the workarounds that makes it go away in my script, but assuming it is really a side effect of the hook usage, I'd like to have better understanding, because it will probably affect other users too.
-
Hello,
I have not really everything understood, what you have talked about the last days. Yes I’m still a Rookie…
We found a timing related problem on the G1X where it appears that another task was modifying the raw buffer. It happened with my time lapse script in single shot, but not in continuous mode, and only when the exposure time was >1 second. We never figured out what caused it, but I was able to work around it by always running in continuous mode.
I made a test with my G1X last day. I run rawopint.lua which is not in continuous mode. All Canon RAW pictures with exposure time >1 second and >= ISO400 where OK. Can there a difference between CHDK DNG Files and Canon RAW Files?
may have been related to high ISO noise reduction that was an advertised new feature on the G1X.
Can this switch off with CHDK?
Unfortunately with exposure time from 4s the Intervall can be faster than 10s.
On my EOS1200D I can switch it off in the Camera.
For my S110 and G1X I found no Camera Menu…
About my last Test with G1X and rawopint.lua:
https://www.youtube.com/watch?v=QRid9lNPFOA (https://www.youtube.com/watch?v=QRid9lNPFOA)
1) I found differences between ISO Values in the log File and EXIF Data. The Last Picture in the log File has ISO800, which I expect, but EXIF Data has ISO1600. In the EXIF Data there is a jump between ISO400 to ISO1000. I found no ISO800 in EXIF.
2) This is the first time, where I had flicker with your script. I analyzed the JPG pictures in the following way:
I calculated A = (R*0.3 + G*0.59 + B*0.11) for every Pixel. Then I calculated a mean Value for all pictures and plot these values (flicker was around picture 360). Interesting results for me:
- Flicker was only at ISO320 (The last runs on S110 where only until ISO200)
- look at the zoom plot around picture 360, how continually the progress off the mean value is…This is not random. The steps look like 1/3 EV steps.
Do you have the possibility, to analyze your EXIF data, when you had the flicker? May be there are also concentrated on some ISO values….
May be that is stupid what I am thinking now.
When I wrote my script I notice, when I made set_tv96_direct(xxx), that the Camera rounded this to an APEX value in 32 steps. Then I had problems on some other point. But from that time I rounded the value in 32 steps before I wrote it to the camera, I had no problems anymore. May be that has nothing to do with the problem.
ISO works in steps to?
Who do you make this with the ISO values? You write not rounded values true?
But how is the ISO function realized in the Camera. I think it is just an amplifier.
Can it be that not every step is realized?
So when this is a problem, I can see it when I run isoinc until ISO400?
On my last test I stopped around ISO200…
-
The zoom plot...
-
Looking at your log, I see the "meter" starts jumping at row 351. Two possible explanations
1) ISO override doesn't behave as expected in this range. e.g. script sets 320, but the camera uses some other value. Running isoinc over this range might give some useful information. You can get the range from the sv96 values in the log.
2) The raw buffer used for the meter contains some garbage instead of raw data. Saving CHDK raw might show if this is the case. I think this is less likely, because the meter still shows a trend.
What ISO setting did you have in the canon firmware?
I've attached my own plot of the log.
-
Looking at your log, I see the "meter" starts jumping at row 351.
Did you find something like this in your logs when you have flicker?
The raw buffer used for the meter contains some garbage instead of raw data. Saving CHDK raw might show if this is the case.
I saved Canon RAW Data. Does this data comes from an other buffer as the CHDK RAW?
What ISO setting did you have in the canon firmware?
It was set to ISO100.
-
Looking at your log, I see the "meter" starts jumping at row 351.
Did you find something like this in your logs when you have flicker?
When I see flicker, the change is much smaller, only a few APEX96 units. Yours are much larger, 60+ units. On yours, the under_frac also goes to zero. Interestingly, it stays 0 after the flickering ends.
I saved Canon RAW Data. Does this data comes from an other buffer as the CHDK RAW?
It should be the same, but they aren't necessarily at saved at the same point in the process, so the buffer could be corrupted by the time CHDK sees it. Bugs in CHDK might also cause CHDK to use the wrong buffer.
-
I run isoinc from 500 to 730. I found all ISO values from ISO200 until ISO1000 in the EXIF data. No jump. I think everything is ok?
Should I run next time G1X with CHDK RAW?
-
I run isoinc from 500 to 730. I found all ISO values from ISO200 until ISO1000 in the EXIF data. No jump. I think everything is ok?
Thanks for testing. Seems like ISO control is actually working so something else odd was happening when you got the flickering.
Should I run next time G1X with CHDK RAW?
It might provide some additional clues.
-
Hallo reyalp,
i have sometimes problems with the default parameters in your script, when I run the script at the first time on a new installation. Some parameter where random and not default. Can it be that has something to do with the default values of a table?
#ui_draw_meter_t=0 " Meter area" {None Corners Box} table
I think tables be actual 1 based or?
https://github.com/c10ud/CHDK/commit/f848302f8ebe02b3c6ec9030671233a3d129d98e (https://github.com/c10ud/CHDK/commit/f848302f8ebe02b3c6ec9030671233a3d129d98e)
-
i have sometimes problems with the default parameters in your script, when I run the script at the first time on a new installation. Some parameter where random and not default.
I have noticed this too and haven't tracked it down.
Can it be that has something to do with the default values of a table?
#ui_draw_meter_t=0 " Meter area" {None Corners Box} table
I think tables be actual 1 based or?
Good idea, but looking back at the posts around the change, I don't think the #param value was changed, only the table values: http://chdk.setepontos.com/index.php?topic=12121.msg119816#msg119816 (http://chdk.setepontos.com/index.php?topic=12121.msg119816#msg119816)
In the current trunk ,=0 selects "None" and 1 selects "Corners" when you use "Load default param values"
I think the weirdness may happen when the parameter definitions in the script change and you already have saved params, but I'm not completely sure.
-
Yes only table values are changed. tables begins with 1, arrays with 0.
So when i delete the files under data and start the script, i got random values for ui_draw_meter_t (None, box or corners). I don’t understand why it switch to None when I call "Load default param values".
But when I change ui_draw_meter_t=1, then I got always None at start. Whem I then call "Load default param values" it switches to Corners….
There must be an mismatch…
-
So when i delete the files under data and start the script, i got random values for ui_draw_meter_t (None, box or corners). I don’t understand why it switch to None when I call "Load default param values".
Good idea. I did the following
1) Loaded a different script in the script menu
2) Deleted are rawopint* files in data
3) Loaded rawopint
In my current script, I have
#ui_display_mode_t=0 "Display" {On Off Blt_Off} table
#ui_draw_meter_t=1 " Meter area" {None Corners Box} table
After loading the script is
Display is Blt_Off
Meter area is None
After I run "Load default param values"
Display is On
Meter area is Corners
edit:
OK, I think I see the problem. It looks like the loaded value is assumed to be 1 based, but the default is stored before that adjustment is made. So both the script and CHDK have a bug.
edit 2:
I've checked in a change that should fix the CHDK part in trunk 4117
-
Fantastic, good job, now it works correct….. :)
-
Hello reyalp,
I made another test with G1X. The same problem as the run before. Its absolute reproducible. The result in the log file looks similar to the run before. The flicker starts around when bv96 gets negative.
The first 18s from the video are made from JPG, the second 18s are made from CHDK DNG files. Flicker around 11s. I never worked with DNG date before, but JPG and DNG data looks similar. The white balance looks a little bit crazy when flicker ends.
So it did not looks like a problem with the RAW buffer or?
http://youtu.be/1C-dMp1UTWs (http://youtu.be/1C-dMp1UTWs)
May be I switch next time noise reduction in the CHDK menu off…
-
The first 18s from the video are made from JPG, the second 18s are made from CHDK DNG files. Flicker around 11s. I never worked with DNG date before, but JPG and DNG data looks similar. The white balance looks a little bit crazy when flicker ends.
So it did not looks like a problem with the RAW buffer or?
Yes, it looks like there's valid image data in all the DNGs, so it's not a case of having a completely wrong raw buffer or something like that.
The pattern is definitely the same as the last one.
I don't have any good ideas at the moment, maybe it's a case of the script hooks interfering with normal canon code like I think is happening on D10.
The next version of the script will have some parameters to adjust the sleep and yield settings that affect this on D10. I have a couple more things to sort out before I upload it.
If you want to try this manually, you can put sleep(10) just before the line hook_raw.continue()
May be I switch next time noise reduction in the CHDK menu off…
The only noise reduce option CHDK has is dark frame subtraction. That's usually triggered by shutter speed, where your problem happens after the shutter has already been bottomed out for a while. I don't have any better ideas though...
One other thing to try might be to set the shutter speed limit different. If you allow it to go to say 2 sec, does the flicker start at the same ISO?
-
Something I missed before: In the logs from both the G1x runs with flicker there are messages like
sv 619 != cam 715
This means the script tried to set ISO ~456 but the value it read back in the next exposure was ~898
This happens after the flickering has stopped :-[
The problem appears to occur when the script requests sv96 > 571 (ISO ~317). It seems like somewhere between 571 and 574, the effective change in ISO is much larger than expected, although the set value is still read back... the fact that this doesn't happen with the isoinc script suggests this isn't just a generic issue with ISO overrides on this cam.
edit:
This explains the bouncing behavior: Because the going over 571 causes a big jump in exposure, the next iterations try to compensate, dropping below the threshold. Then it's under exposed, so it pushes over the threshold again. Eventually, even the compensated value is above the threshold and it stops flapping.
-
Since you seem to be having ISO problems, why don't you try a test where you set the ISO using the propcase method I described here:
http://chdk.setepontos.com/index.php?topic=11081.msg121063#msg121063 (http://chdk.setepontos.com/index.php?topic=11081.msg121063#msg121063)
Also, I suspect there are some timing related issues with having the script control the interval for every shot. Using my set_shot_interval(i) method, the pre-shot delay always ends after a msleep(10). This may be important. The script could still control the actual interval for each shot by setting it very long, and re-setting it to trigger the end of the delay. I've done that with my code for external sync applications.
I also found that it sometimes take a very long time to write a single line of the log file when you're writing a lot of picture data (raw files especially). Set_shot_interval() can still trigger the next shot on time, even when the script is still hung up writing the log file.
The other advantage of set_shot_interval() is that it makes time lapse scripts simpler.
-
If you allow it to go to say 2 sec, does the flicker start at the same ISO?
Yes, thought about the same, would be interesting to see if every think moves…
And also running S110 with the same parameter…..
The only noise reduce option CHDK has is dark frame subtraction.
When I make pictures with 4s the interval goes to around 10s. This is a noise reduction which the cams (S110 and G1X) makes. There is nothing on the cam, to switch it off. I thought, I can switch it off with CHDK…
I also found that it sometimes take a very long time to write a single line of the log file when you're writing a lot of picture data
I have always problems with my own simple interval script and log files, when I using the fast camera bracketing function + RAW. On every second run the log function brakes…
why don't you try a test where you set the ISO using the propcase method
Would this help? With my knowledge right now I have not really understood what’s going on… :)
I have seen, you change the scale in histo:range. For the moon ;)
Could you this also do in shot_histogram.c for get_histo_range?
This is what I using in my simple script. It was already suggest here
http://chdk.setepontos.com/index.php?topic=8997.20 (http://chdk.setepontos.com/index.php?topic=8997.20)
but I have not understood, why it was not realized. I would be a really improvement...
-
I also found that it sometimes take a very long time to write a single line of the log file when you're writing a lot of picture data (raw files especially).
FWIW, I found something quite similar and charted it here :
Re: KAP & UAV Exposure Control Script (http://chdk.setepontos.com/index.php?topic=10822.msg116123#msg116123)
The "solution" was to buffer log writes in RAM and only squirt them to the SD card occasionally when the buffer got "big" or on script termination.
-
FWIW, I found something quite similar and charted it here :
Re: KAP & UAV Exposure Control Script (http://chdk.setepontos.com/index.php?topic=10822.msg116123#msg116123)
The "solution" was to buffer log writes in RAM and only squirt them to the SD card occasionally when the buffer got "big" or on script termination.
That's a good idea. I used a 1-line buffer in my script. If it takes longer than 200 msec to write a line, I put the next line in the buffer and write 2 lines the next time. If it takes >200msec and the buffer is full, I have to write anyway. But that doesn't happen too often.
The point is, the timing is more accurate with set_shot_interval(). But I thought of a way to test the pre-shot script hook flicker problem. Try adding msleep(10) after the script signals the pre-shot delay to end. That is, always start the shot after an msleep(10).
I think that makes the shutter timing more accurate, especially at faster shutter speeds. The shutter needs to open at a consistent time relative to the tick counter (I think).
-
When I make pictures with 4s the interval goes to around 10s. This is a noise reduction which the cams (S110 and G1X) makes. There is nothing on the cam, to switch it off. I thought, I can switch it off with CHDK…
Right, this is dark frame subtraction, which you can force off in CHDK. What I was saying is that the flickering doesn't start when the shutter speed that reaches the length that triggers DFS, the shutter speed has been at 4 sec for many shots. The trigger seems to be ISO, but it is entirely possible there is some interaction between DFS and certain ISO levels. It's worth a try.
Would this help? With my knowledge right now I have not really understood what’s going on… :)
I believe the current CHDK C code already does essentially the same thing lapser is suggesting. The CHDK code has changed substantially since he wrote his timelapse mods.
The fact that isoinc works shows this isn't a case of the ISO override code being completely broken, there is some other factor involved.
I have seen, you change the scale in histo:range. For the moon ;)
Yes. histo:range now accepts a scale factor in the third parameter. This is one of the things I'm working on for the next version of the script.
Could you this also do in shot_histogram.c for get_histo_range?
I'm not inclined to change the shot_histogram code at this point. It should be fairly easy to convert your simple script to use the new raw hook code. If you want to post your code, I may be able to help.
-
I also found that it sometimes take a very long time to write a single line of the log file when you're writing a lot of picture data (raw files especially).
FWIW, I found something quite similar and charted it here :
Re: KAP & UAV Exposure Control Script (http://chdk.setepontos.com/index.php?topic=10822.msg116123#msg116123)
The "solution" was to buffer log writes in RAM and only squirt them to the SD card occasionally when the buffer got "big" or on script termination.
FWIW, I haven't seen a slowdown in logging in the rawopint script. The code for this keeps the file open and just calls write() for each line. This does mean you will lose a indeterminate amount of data if there is a crash, but the same is true if you buffer yourself in lua. You might see some old posts saying that keeping a file open is bad, but it turns this was a mistake from when we thought the FsIoNotify crash was due to having too many handles open.
The log code is a self contained CSV based log system, if anyone wants to borrow it, you can just copy everything from
-- log module
to
-- end log module
I'll put a separate post about this in the scripting forum.
-
FWIW, I haven't seen a slowdown in logging in the rawopint script. The code for this keeps the file open and just calls write() for each line.
The code that was running when I charted the "slowdowns" in the link I posted did a fopen/fclose for each line written to the log. So I guess it's understandable that it would slow down.
-
FWIW, I haven't seen a slowdown in logging in the rawopint script. The code for this keeps the file open and just calls write() for each line.
The code that was running when I charted the "slowdowns" in the link I posted did a fopen/fclose for each line written to the log. So I guess it's understandable that it would slow down.
I noticed the occasional long delays in print() with continuous mode at intervals faster than 1 second for jpg, and 2 seconds for Canon raw (SX50 and G1X). Saving DNG files with CHDK is even slower (I never do that). I use print() for logging because I discovered it was a little faster than the Lua write() statement. I worked pretty hard to get the maximum shot rate in continuous mode.
I've been doing astrophotagraphy lately with the Canon EOS-M and Magic Lantern. There are a lot of bugs in the EOS-M port, so you can't even do basic things like exposure bracketing. There isn't any scripting available either. I sure wish Phil, Reyalp, and Waterwingz could work on it! You guys are fantastic.
-
Hello,
I tried to run a new test with G1X but unfortunately the G1X starts with an E42 Error :(
Then I make a run with S110 and max Tv=4s. It looks good. May be I stopped too early. But here was no problem around sv96=576.
I used the last update from hookutil.lua and add the additional delay which you suggested.
I hope I get the G1X fixed…..
https://www.youtube.com/watch?v=p7RhF01WWCE (https://www.youtube.com/watch?v=p7RhF01WWCE)
-
I got the G1X fixed. One night without battery helps…. :)
I run a test with G1X and max Tv=2s. It looks like that flicker starts at the same sv96 values as with the 4s run. I used the last update from hookutil.lua and add the additional delay which you suggested.
Does it make sense to run isoinc with G1X with a Tv=2s? So with additional tv settings?
http://youtu.be/XhYMedSxgws (http://youtu.be/XhYMedSxgws)
-
I run a test with G1X and max Tv=2s. It looks like that flicker starts at the same sv96 values as with the 4s run.
Thanks, that's a useful data point.
One idea might be to try isoinc over this range, with a subject that is dark enough to require a similar exposure length.
Did you ever try with dark frame subtraction disabled? Skimming the manual, I don't see what exposure length triggers it on this camera, but on most I think 2 sec would.
I did see in the manual this camera has some settings related to auto-ISO: The "Max ISO Speed" "Rate of change". These might have some effect on CHDK ISO overrides, since the overrides use Canon auto mode internally.
The dynamic range corrections settings might also have some effect.
edit:
If you think it sounds like I'm grasping at straws... that would be an accurate assessment ;)
-
Did you ever try with dark frame subtraction disabled?
Not yet, will be one of the next thinks which I do.
Skimming the manual, I don't see what exposure length triggers it on this camera, but on most I think 2 sec would.
I think also as start at 2. But I would expect a jump in the log file with the sleep values when it starts or not?
I did see in the manual this camera has some settings related to auto-ISO: The "Max ISO Speed" "Rate of change". These might have some effect on CHDK ISO overrides, since the overrides use Canon auto mode internally.
Ok, I thought when I use fixed ISO100, this has no effect.
The settings at this run where
Max ISO Speed = 1600
Rate of change = Standard
High ISO NR = Standard
The dynamic range corrections settings might also have some effect.
I have not looked about this yet, but should not have an effect on RAW data or?
If you think it sounds like I'm grasping at straws... that would be an accurate assessment
Everythink is OK. I’m really interested to get it run and what’s going on and why it not happened on S110….
edit:
Run isoinc with high Tv=8s on G1X. That’s it. Meter makes a jump…
One other thing. I tried to run from 520 to 640. But isoinc stops at 603. I saw this later so I have not seen if there was any exception in lua…
I will try to run with different parameters on S110 and G1X….
-
I think also as start at 2. But I would expect a jump in the log file with the sleep values when it starts or not?
There should be a drop in the sleep time, and the raw_ready time should go up.
I have not looked about this yet, but should not have an effect on RAW data or?
I wouldn't expect it to affect the raw data, but it might affect how the ISO overrides are interpreted, or somehow affect our hooks in the shooting code.
Everythink is OK. I’m really interested to get it run and what’s going on and why it not happened on S110….
Yes I want to understand too. I think there is probably something different about the G1x, but it would be good to understand it, and fix it if we can.
Run isoinc with high Tv=8s on G1X. That’s it. Meter makes a jump…
Thanks.
edit:
Very clear. It seems like maybe the "base" ISO changes under some condition.
One thing to try would be a simple script that doesn't use shoot hooks, just like
set shutter to 4 seconds
for sv96=550,600
set sv96
shoot
The images should show the jump if it's present.
It would also be interesting to know the specific shutter speed it happens at.
-
I can reproduce this on my G1X.
Attached images are from DNG's saved from CHDK, on either side of the meter jump.
First image is at sv96 = 571 (ISO 317), second image os sv96 = 572 (ISO 320).
As you can see the second image is wrong, so this could explain why the rawop meter is wrong - the JPEG and Canon RAW images are fine.
Looks like it could be a bug in the RAW capture on the G1X port - I suspect we may be getting to the RAW buffer too late, and the JPEG conversion process has already started.
I don't have much free time at present; but if I get a chance I'll dig into it more.
Phil.
-
I think this is the same problem with the G1X I referred to here:
http://chdk.setepontos.com/index.php?topic=11081.msg121139#msg121139 (http://chdk.setepontos.com/index.php?topic=11081.msg121139#msg121139)
We never figured out the cause, but it didn't happen in continuous mode on the G1X, as described here:
http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252 (http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252)
-
I tried moving 'capt_seq_hook_raw_here' to task_DvlpTask as suggested by srsa_4c in another thread; but it made no difference - the DNG image is still wrong once the ISO hits 320.
So I took a closer look at the actual RAW data and compared the good DNG to the bad DNG.
It appears that we are actually getting the RAW data from the sensor in both cases; but the camera seems to be switching to a 12 bit mode when the ISO hits 320.
In the good DNG the image data starts at ~ level 512 as we would expect from the black point.
In the bad DNG the image data doesn't start until ~ level 2048 - 2 stops higher.
If I convert the bad DNG file using dcraw with a black point of 2048 it looks much closer to the good DNG image.
My guess is the camera is switching to a 12 bit ADC. The Canon raw files still report as 14 bits with a black point of 512. So Canon is doing some sleight of hand later in the image pipeline to convert back to 14 bits.
At this stage we don't have a way to fix this - we assume a fixed black point, not a variable one.
I don't yet know under what circumstances Canon is using this 12 bit mode - so there is no easy way to even know which black point to use.
Phil.
-
Thanks for looking into this phil. My impression from c_joerg's post was that exposure of canon jpegs also jumps at the 320 switch, which suggests something triggered by CHDK overrides, rather than just canon doing something funky under the hood with the raw data.
Also noted in http://chdk.setepontos.com/index.php?topic=11081.msg121572#msg121572 (http://chdk.setepontos.com/index.php?topic=11081.msg121572#msg121572) at somewhat higher ISO, the value reported by get_prop(props.SV) stops matching the value set with overrides, somewhat similar to what happens if ISO_BASE was wrong.
-
Thanks for looking into this phil. My impression from c_joerg's post was that exposure of canon jpegs also jumps at the 320 switch, which suggests something triggered by CHDK overrides, rather than just canon doing something funky under the hood with the raw data.
Also noted in http://chdk.setepontos.com/index.php?topic=11081.msg121572#msg121572 (http://chdk.setepontos.com/index.php?topic=11081.msg121572#msg121572) at somewhat higher ISO, the value reported by get_prop(props.SV) stops matching the value set with overrides, somewhat similar to what happens if ISO_BASE was wrong.
I'd lay odds that Canon are doing something funky with the ADC.
Attached are two more images:
- the first is 2 secs @ ISO 400 (Canon manual settings), DNG converted with black point at 2048
- the second is 2 secs @ ISO 800 (CHDK settings), DNG converted with black point at 512
As you can see they look pretty much the same.
When using CHDK ISO overrides, the DNG and JPEG ISO values do not match once the ISO goes over 320. The DNG files always shows 1/2 the selected ISO and has 1/2 the exposure of the JPEG. For some reason the image is being exposed at 1/2 the selected ISO; but the JPEG is then overexposed to compensate. This is why I used ISO 800 in the sample attached image with CHDK settings - the ISO reported in the file is 400.
If I use Canon M mode to manually set Tv >= 1.3 seconds and ISO >= 320 then I get bad DNG files as described earlier.
If I used CHDK overrides to do the same thing, the DNG files are fine regardless of the ISO selected (but I get the exposure issue above).
I have no idea why setting the Tv and Sv values in the script does not behave like setting them in the CHDK overrides menu.
Phil.
-
There should be a drop in the sleep time, and the raw_ready time should go up.
But I don’t see it on the last run….
I make a second run with isoinc and ‘Dark frame subtraction ’ = Off. Interesting on this run is the peak on sv96=603. Only one peak. On the first run the script crashes here…
Off topic:
I bought I second used G1X. This one has the firmware 100e. My first one has Firmware 101a. I have not found a firmware at canon. Did someone know where the difference is and where I find the newest firmware?
-
There should be a drop in the sleep time, and the raw_ready time should go up.
But I don’t see it on the last run….
Yes, maybe this camera is different. Usually the Canon manual says something like "With shutter speeds of 1.3 seconds or slower, there will be a delay before you can shoot again..."
I don't see anything like this in the G1x manual. Anyway, it doesn't look like the problem is related to dark frame.
I make a second run with isoinc and ‘Dark frame subtraction ’ = Off. Interesting on this run is the peak on sv96=603. Only one peak. On the first run the script crashes here…
Do you mean the script ends with an error, or the camera shuts down? If the script ends with an error, please check 'last console' and the chdk menu. If you the camera shuts down, get a romlog: http://chdk.wikia.com/wiki/Debugging#Camera_crash_logs_.28romlog.29 (http://chdk.wikia.com/wiki/Debugging#Camera_crash_logs_.28romlog.29)
Off topic:
I bought I second used G1X. This one has the firmware 100e. My first one has Firmware 101a. I have not found a firmware at canon. Did someone know where the difference is and where I find the newest firmware?
Usually the differences are very minor. We can't easily tell exactly what changed, but it's usually very little code, or some non-code area like translations strings. Canon doesn't issue a public firmware update unless there are serious bugs, and unless canon issues an an update, there is no way to update the firmware.
-
Do you mean the script ends with an error, or the camera shuts down? If the script ends with an error, please check 'last console'
When I came on the first run to the camera, it looks like that the script normally finished (sv from 520 to 640). I notice only that there was a problem, when I copied the data. But then it was too late to watch at last consol.
On the second run I tried to run it under same condition. I put an additional print_screen on the script. But on the second run it did not stopped. May be it has something to do with dark frame subtraction.
Should I run I again with dark frame subtraction again?
I was only wondering on the peak at sv96=603 on the second run that the script finished at 603 on the first run…
there is no way to update the firmware
so than I can’t run both cams with the same sd card without changes… :(
-
I made a couple of trys with isonic between sv96=590 – 610. The peak at 603 is always reproducible. After 10 runs it stopped ones (Tv=8s, Sv=590). The run was with dark frame subtraktion.
-
I made a couple of trys with isonic between sv96=590 – 610. The peak at 603 is always reproducible. After 10 runs it stopped ones (Tv=8s, Sv=590). The run was with dark frame subtraktion.
Are you sure you have the most recent version of CHDK/LUALIB/hookutil.lua installed? In r4138, I made it adjust the add shutter and dark frame time to the standard 10 second timeout. Of course, there could be bugs...
so than I can’t run both cams with the same sd card without changes…
Yes, unfortunately this is how it is.
-
Are you sure you have the most recent version of CHDK/LUALIB/hookutil.lua installed? In r4138, I made it adjust the add shutter and dark frame time to the standard 10 second timeout.
Yes, but I will check again…
I analyzed sleep and raw_ready of the last rawopint runs. I can always see a jump in the S110 runs at tv > 1s. But never in the G1X runs. May be the G1X start earlier and smoother.
rawopint8 => S110 without RAW
rawopint6 => G1X without RAW
-
Runs with Raw are very noisy.
rawopint7 => G1X with RAW
Update:
I`m using r4138....
-
I've added a fix for the G1X in revision 4153 (version 1.4 only at the moment).
This calculates the black point from the sensor masked area on each image, instead of using a fixed value.
In my testing this seems to fix the metering calculation in rawop_meter, as well as making DNG's saved with CHDK look correct.
Note, the 'raw merge' code has not been updated to handle the case where input files may have different black points. Attempting to use the raw merge functions on such files is likely to cause strange results :(
Phil.
-
Made the last update and run isoinc from sv=520 to 740. Looks good expect the peak on sv=603. Good job :) . On higher ISO values meter96 gets more noisy which I would expect…
-
Just a note the rawopint.lua script (and other rawop code) may not work correctly with this. In the C code, the raw_neutral and log2_raw_neutral_count values are calculated at script start time. The scripts also only call rawop_fb_info once at startup.
I'll have to look at the code in more detail to know what the exact implications are, but it will affect the neutral value and the ev96 <=> raw count conversions.
edit:
I should add, this is not an objection to Phil's fix, just a warning that the script may not be completely fixed.
-
Made the last update and run isoinc from sv=520 to 740. Looks good expect the peak on sv=603. Good job :) . On higher ISO values meter96 gets more noisy which I would expect…
This appears to be the point where the other behaviour I mentioned kicks in.
At Sv=603 (ISO 400), the actual ISO used by the camera is 200 and the JPEG is boosted to compensate (and shows an ISO of 400).
Above Sv 603, the camera uses the requested ISO; but is still boosting the JPEG by 1 stop.
I haven't found the cause for this yet.
Phil.
-
Just a note the rawopint.lua script (and other rawop code) may not work correctly with this. In the C code, the raw_neutral and log2_raw_neutral_count values are calculated at script start time. The scripts also only call rawop_fb_info once at startup.
I'll have to look at the code in more detail to know what the exact implications are, but it will affect the neutral value and the ev96 <=> raw count conversions.
edit:
I should add, this is not an objection to Phil's fix, just a warning that the script may not be completely fixed.
It's never easy is it ::)
Phil.
-
Lapser:
but it didn't happen in continuous mode on the G1X
maybe this is the easiest solution
reyalp:
I already have it in the contae script.
I tried to run contae.lua today but I was wondering about two things:
1) That I have to hold the shutter always pressed…
2) That I can’t change the Interval… I thougt it was put a delay in continuous mode…
A misunderstanding about my side how to use it?
-
I tried to run contae.lua today but I was wondering about two things:
1) That I have to hold the shutter always pressed…
2) That I can’t change the Interval… I thougt it was put a delay in continuous mode…
contae has a different purpose than rawopint. It is meant to provide a kind of continuous mode autoexposure for the cameras that don't have it, so you can shoot a moving subject that goes through different lighting. Many high end powershots already have a mode like this, but the older and lower end ones don't. It's not an intervalometer.
What I meant by that comment earlier is that I already implemented the logic to use continuous mode in contae, so porting it over to rawopint won't be too difficult.
Due to RL stuff, I haven't had much time for CHDK recently. I should have more time again after next week.
-
I should have more time again after next week.
It’s OK. Don’t hurry…
As you say, the release didn’t fix everything in rawopint, there is a jump at around 26s….
http://youtu.be/1P9fXrJMnQE (http://youtu.be/1P9fXrJMnQE)
-
Just for completeness.
The curves didn’t look so bad expect the peek around sv96=603.
The run was with dark frame subtraktion
There are two thinks which I was wondering:
1) After the peek the speed of the video increases (it looks like factor 2). But tv has not changed. Can it be that cam switched of dark frame subtraction?
2) After the peek the brightness looks to high. But meter96 shows a different direction.
-
Hi,
I had another use for this. I have trouble taking pictures of flowers with very saturated colours, they can be overexposed while the rest isn't, and have less shading detail.
My idea is to take 3 pictures but exposure for R,G, B separately. Then combine the 3 pics into a normal, but fake higher-bit DNG file. For example boost Blue by 2 stops, then shift it down 2 bits, and make a fake 16bit DNG from a 14bit sensor. This is like a fake floating point. It would give better accuracy in colour balance and more detail with saturated subjects. You might wonder about mismatched highlights - this can be fixed with usual highlight recovery, but there's a slight problem. You'd have to do normal highlight recovery on equalized channels first then put them back to proper colour balance, if you follow.
For now, make sure there's no overexposure in each colour. Can this be done? I don't know to construct a fake DNG in this way.
In fact for flowers, I'd like to just expose for (say) red, I don't even care if background has highlights since it's not the subject.
For another idea, I'd like to analyze raw buffer without saving between pictures. At first for noise analysis, but also for photogrammatry or looking for scene change etc. Hopefully it's much faster when I don't need to save the buffer.
-
For now, make sure there's no overexposure in each colour. Can this be done? I don't know to construct a fake DNG in this way.
The meter and histogram code can be used to measure individual channels, by using the CFA offsets and appropriate step size. There are examples of this in the drtest script.
As far as constructing a combined DNG, you might have to write some software to do that. Aside from combining the actual data, the color matrix would be need to be different.
I don't have much opinion about how viable the overall concept is. If you have to heavily over expose one channel to get "correct" exposure on another, you may run into problems with blooming.
For another idea, I'd like to analyze raw buffer without saving between pictures. At first for noise analysis, but also for photogrammatry or looking for scene change etc. Hopefully it's much faster when I don't need to save the buffer.
You can analyze the raw buffer without saving raw. You can't currently prevent jpeg saving, but it is a relatively small part of the total shooting time. We do have the ability to prevent jpeg save on some cameras for USB remote shoot but it currently isn't accessible from script, and because it only affects the final write to SD card, it doesn't save a lot of time.
Analyzing the full raw buffer is quite slow (several seconds) even in C code, if you do it pixel by pixel in Lua, it's even slower. If your analysis only needs to look at a small area or sample a small fraction of the pixels, it can be fairly quick (tens of milliseconds or less.)
-
I think, revision 4153 makes it even worse.
I got now flicker also with ISO200 (on the end of the video), not much, but this did not happens before…
http://youtu.be/BuMPmHyrO4s (http://youtu.be/BuMPmHyrO4s)
-
And the log file...
-
I am still really happy with the script rawopint. It controls everything very good. Here is an nice example made with S110:
http://youtu.be/AP2_ZMTQREA (http://youtu.be/AP2_ZMTQREA)
I’m still thinking, revision r4153 makes it worse with G1X. I get now still flicker also with lower ISO values. For me, the best solution right now, is to go back to revision r4138.
-
Here is an nice example made with S110:
That looks like it worked quite well.
I’m still thinking, revision r4153 makes it worse with G1X. I get now still flicker also with lower ISO values. For me, the best solution right now, is to go back to revision r4138.
It definitely won't work right until the script and rawop code is updated to deal with it. I should hopefully have some time to work on this again soon. (yeah, I know I've said that before :-[)
From a quick look, I think the flicker you see in post #144 is a different issue, although I'm not sure exactly what it is. Philmoz change shouldn't have any effect outside of when the black level jumps, which I don't think is happening there.
-
I should hopefully have some time to work on this again soon
Don’t get on pressure from my side …
From a quick look, I think the flicker you see in post #144 is a different issue, although I'm not sure exactly what it is.
It looks definitely different as the thing before. So r4153 should not have an effect on G1X with Tv=4s and ISO200. It is not much flicker which can be reduced with a deflicker filter in VirtulDub.
-
but in my testing on the cameras that I have, the "hold half press / click full press" approach gives very similar performance to continuous mode
This is one issue where I still thinking about. I can’t agree this for S110 and G1X.
May be it is a misunderstanding from my side.
Running rawopint with minimum interval time gives me an interval from around 1.3s.
I got the same result, with a simple script like this:
loop_stop = false
press("shoot_half")
repeat
sleep(10)
until get_shooting() == true
repeat
count = get_exp_count()
press("shoot_full_only")
sleep(200)
release("shoot_full_only")
repeat
sleep(10)
until get_exp_count() ~= count
if is_pressed "menu" then
print("Stop because MENU Click")
loop_stop = true
end
until loop_stop
A script in continues mode like iv_turbo.lua gives me on both cameras a interval from around 0.5s
-
This is one issue where I still thinking about. I can’t agree this for S110 and G1X.
My comment was only based on testing of cameras I have. It's quite likely that other cameras behave differently.
Updated scripts are attached. Changes
* rawopint now supports continuous mode. You must enable continuous mode in the Canon settings, and make sure "Use cont. mode if set" is checked (on by default). This may improve the review display, and may shoot faster. There may be other side effects.
* over / under protection can now be specified in parts per 10,000, and is measured in parts per 100k. It's logged as % (to 3 decimal places).
* Some shutter values are changed from 1/100ths to 1/1000ths.
* Meter low limits extended
* "simulation" mode is probably broken.
I have not yet updated to the code to deal with the changing black point on G1x, however from Lapser's comments it's possible that running in continuous mode will avoid the issue.
I haven't tested this a whole lot.
I had one run with the D10 which had two shutter failures and then crashed without are romlog after ~630 shots. This was with the "disable script yield" set, not using continuous mode.
I subsequently did a run of ~2700 in continuous mode with yield enabled and raw hook sleep set to 10 without any shutter failures or crashes.
-
That sounds really good :) . I am looking forward to work with the script. I’ll be out for a couple of days may be without any internet connection. So the response can be a little bit delayed….
-
Here's a boring test with the current script on D10 in continuous mode. Shooting rate was similar to the old method, ~160ms to spare with a 1.2 second interval.
No shutter errors or crash in ~6100 shots. I had script yield enabled and a raw hook sleep set to 10ms.
I think the flickering is due to overexposure thresh being set to 2 (0.02%). Histogram step was 11, so over protection would reach full strength a bit over 20 pixels. At low percentages there isn't much room for the change to ramp in. From the frames I looked at, it appears variations in the clouds and highlights on the grass were the brightest parts, although they weren't actually ever maxed out in the jpeg. Overexp Ev range was set to 1/2 stop.
I don't think the flickering is a camera issue, since the bv is nice and smooth. ND was in the whole time, and shutter speeds weren't excessive.
-
Unfortunately I have not had much time to do something. Make only a small test with S110, to see how continuous mode works. I made 4 runs with each 20 pictures.
1) Single Mode with Interval of 4s.
2) Continuous mode with Interval of 4s.
3) Continuous mode with Interval of 0s.
4) Single Mode with Interval of 0s.
As you can see on the log file, continuous mode on S110 is around 2 times faster.
I hope I have more time next week to make a longer test with G1X.
I think the flickering is due to overexposure thresh being set to 2 (0.02%).
I would think it also. May be it is a hard condition with a histogram step of 11.
In this case, probably step must be much smaller. What was the step size in the previous version?
Would be really interested, how this will be work with the moon.
But I still think about the flicker in my last video. Do you think, the whole control can be going in oscillation? In my simple script, I have a hysteresis when I change direction of control. I also don’t change any control, when changes are less than 1/3EV. On some conditions, I stay a long time with the same ISO and Tv values. Also on your last run (on the second half), you could stay probably a long time without changing values.
-
As you can see on the log file, continuous mode on S110 is around 2 times faster.
Good to know. Did you check how the images are displayed when the screen is on? On D10, continuous mode shows the last image until the next one is shot, while non-continuous goes black in between.
I would think it also. May be it is a hard condition with a histogram step of 11.
In this case, probably step must be much smaller. What was the step size in the previous version?
The old shot_histogram code has a hard coded step size of 31.
Small step sizes may have a significant impact on shooting time.
Would be really interested, how this will be work with the moon.
Me too, but weather and celestial mechanics have not cooperated ;)
But I still think about the flicker in my last video. Do you think, the whole control can be going in oscillation?
In my simple script, I have a hysteresis when I change direction of control. I also don’t change any control, when changes are less than 1/3EV. On some conditions, I stay a long time with the same ISO and Tv values. Also on your last run (on the second half), you could stay probably a long time without changing values.
Oscillations are definitely possible, although I try to limit them.
In my script, the exposure change for shot N is normally made by averaging the change needed for "correct" exposure with the change used in shot N-1. This is modified in a couple ways:
1) If it would cause the next exposure to overshoot (e.g. the current frame is correct, but the previous frame has a large exposure change, or the change for correct exposure has the opposite sign and smaller magnitude than the previous change), the change is set to 0. This generates a "smooth overshoot" message in the log desc column
2) If the sign of the requested change has switched between frames, the total change is reduced by 1/2. This generates a "smooth sign switch" message.
I find this works very well for normal metering, but over and under exposure protection have a different problem: You can't know how much the exposure should change. If 1% of pixels are over exposed on a highlight, you might only need 1/16th of a stop to correct it, but if it's the sun, you might need 10 stops.
The amount of requested change is currently only controlled by what fraction of the threshold value is overexposed. This means that the same amount of exposure change (controlled by the max ev change setting) is used when the over exposure equals the threshold, regardless of whether the threshold is 0.1% or 10%.
It might make more sense to use smaller changes when the threshold is smaller, but because the change actually starts before the threshold is reached, you can get a similar effect just by using a higher threshold. In other words, if you set the threshold to 2%, the code will start reducing exposure some when only 1% is overexposed. The threshold actually only defines when it reaches the "max ev change" value.
-
On D10, continuous mode shows the last image until the next one is shot
As I remember, it was the same, but I can check next time again.
but weather and celestial mechanics have not cooperated
I know, also days are really long right now.
The best condition for the moon are always at winter time….
I find this works very well for normal metering
Yes as I said before, I’m really happy with the script.
but over and under exposure protection have a different problem
Yes, I understand the problem.
So, I’m looking forward to get better weather und run the G1X until Tv=4s and ISO=1600.
-
Made just a small test with G1X, to see how continuous mode works.
I made 3 runs with each 20 pictures.
1) Single Mode with Interval of 0s.
2) Continuous mode with Interval of 0s.
3) Continuous mode with Interval of 4s.
Continuous mode on G1X is around 2 times faster (simular to S110).
On D10, continuous mode shows the last image until the next one is shot
It's the same on G1X...
-
Good to know you get a faster shooting rate in continuous mode on G1x. It would be interesting to know if you still still see the black point jump.
I took a closer look at the log from post 145 (http://chdk.setepontos.com/index.php?topic=11081.msg122081#msg122081)
I think this flickering is caused by over exposure protection flapping. The jumps in "over_weight" (and hence final contribution to exposure) are relatively large, because the low precision of the histogram only gives 10 steps to the threshold and over_weight goes by x^2.
In the attached chart, you can see that over_weight starts to oscillate around shot 800 and gets bad by 900. Bv stays quite smooth, which suggests the exposure controls are working correctly.
You can see the low precision of over_weight earlier, but with the sun in the scene, the over exposed fraction doesn't change nearly as much for a given amount of exposure change, so it's less prone to flapping. At the end, a large part of the sky is nearly uniform, so a small change in exposure changes the over fraction by a much larger amount.
I think the current code should handle this better, since the histogram has much higher precision, but I'm not sure if it will fix it completely. I could make more of the over/under protection factors configurable, but the script already has too many options.
The first part of this run is a good illustration of how the weight system works. At the start, having the sun in the frame maxes out over_weight. This reduces exposure until the meter limits increase meter_weight enough to balance it out. The weights stay in balance until the meter area is above the meter low threshold.
-
I made a run with G1X. There is still a jump around sv96=603 (In the video around second 22). The rest of the run looks very good. This is the first run, where I see correct ISO values in the EXIF data. The run was made with r4153. Should this change still have an effect in continues mode?
http://youtu.be/LM67ACxPQbM (http://youtu.be/LM67ACxPQbM)
-
because the low precision of the histogram only gives 10 steps to the threshold and over_weight goes by x^2
Yes, I understand the problematic…
I think the current code should handle this better, since the histogram has much higher precision
Yes I think the same. I tried to write a script, with the old histogram. This script should hold overexposure at 1%. But with 1% resolution, it was not working good...
I analyzed the jpg picture by calculating an average of every picture. I don’t understand, why after sv96=603 the level on the pictures a still very high. There is not just a peak, which I analyzed from your log file data. Can this have something to do with the black point?
-
I made a run with G1X. There is still a jump around sv96=603 (In the video around second 22). The rest of the run looks very good. This is the first run, where I see correct ISO values in the EXIF data. The run was made with r4153. Should this change still have an effect in continues mode?
Philmoz change should apply no matter what mode the camera is in. How the camera would behave was unknown, but it looks like it does something similar but not exactly the same :blink:
AFAIK, Lapser had good results with the G1x in continuous mode using his custom code, so perhaps there are some clues there.
-
So I will try it next time with r4138 again....
-
@c_joerg
A long while back (http://chdk.setepontos.com/index.php?topic=11081.msg120858#msg120858) you reported the drtest.lua script failed on G1x. I think this should be fixed in the script included with autobuild 4158, so if you can try again, that would be appreciated.
I'm still thinking about how to make the C code and the script properly deal with varying black level.
-
you reported the drtest.lua script failed on G1x. I think this should be fixed in the script included with autobuild 4158, so if you can try again, that would be appreciated.
I’m really busy this week. But I will definitely try it on the weekend or beginning next week.
I'm still thinking about how to make the C code and the script properly deal with varying black level.
Let’s see, what the run with G1X with r4138 in continuous mode shows (I’ll try as soon as possible). If this works fine, I will be happy with this solution.
-
I’m really busy this week. But I will definitely try it on the weekend or beginning next week.
Thanks, no rush.
Let’s see, what the run with G1X with r4138 in continuous mode shows (I’ll try as soon as possible). If this works fine, I will be happy with this solution.
If the jump at sv96=603 happens in continuous mode, that won't depend on the CHDK version. Philmoz change just attempts to detect if the jump happened and adjust the black level used by CHDK code accordingly. It doesn't affect the actual data, or Canon jpeg processing.
-
There are differences, which I don’t understand.
If the jump at sv96=603 happens in continuous mode
The jump is not on sv96=603. The jump (peak) is on the first sv96 value, which is >= 603. So on the last run on sv96=605. When I run isoinc before, the jump was exactly on sv96=603. 605 were on isoinc OK.
Does it help, to implement the continuous mode in isoinc? Then we could compare the runs under same conditions.
Why I see only a peek on your log data, but the level of all jpg pictures are higher after sv96 >= 603.
What is different, when lapser use continuous mode?
http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252 (http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252)
About drtest:
Would it be interesting to run drtest under different light condition?
May be one run with Tv < 1s and one run where Tv goes >2s?
-
I just run drtest.lua from r4158. I got a printf error, when I start it. Then I run isobase.lua and after drtest.lua again. Then it looks like, that it works. I think, I have seen it also on the old version.
-
I just run drtest.lua from r4158. I got a printf error, when I start it. Then I run isobase.lua and after drtest.lua again. Then it looks like, that it works. I think, I have seen it also on the old version.
Oops. This happens if the camera is not in record and P mode when you start the script. Fixed in svn 4159.
edit:
The CSV you attached is OK, but the results are a bit strange.
Was the subject a neutral, uniform scene? The neutral shot (row seven, exposure #6) seems to be as expected, but the others are unexpected. There should be ~96 difference between each shot in the m96 column, and the wl% and bl% should only be significant in the first and last shots.
The jump is not on sv96=603. The jump (peak) is on the first sv96 value, which is >= 603. So on the last run on sv96=605. When I run isoinc before, the jump was exactly on sv96=603. 605 were on isoinc OK.
What I was trying to say is that whatever crazy thing happens is camera firmware behavior, and philmoz change has no impact on that.
In addition to unknown factors within the Canon code, timing of the overrides in CHDK may have some influence.
Does it help, to implement the continuous mode in isoinc? Then we could compare the runs under same conditions.
That's a good idea, I'll do that when I have some time. (edit: done http://chdk.setepontos.com/index.php?topic=12165.msg122547#msg122547 (http://chdk.setepontos.com/index.php?topic=12165.msg122547#msg122547))
I don't think there is a need to run drtest under different conditions, at least not yet.
What is different, when lapser use continuous mode?
http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252 (http://chdk.setepontos.com/index.php?topic=7934.msg105252#msg105252)
I don't know, lapser used his own custom code.
-
Was the subject a neutral, uniform scene?
No, sorry…. On the first run yes, but then I got the error and I changed the scene. I thought, the scene has something to do with the error …
But today I had a discussion with some colleges, what a neutral, uniform scene is. We think, a gray card would be the best solution or? I’ll run it again…
I'll do that when I have some time.
This was fast … is there any reason in isoinc, why I can’t set append=true?
The next drtest run...
-
But today I had a discussion with some colleges, what a neutral, uniform scene is. We think, a gray card would be the best solution or? I’ll run it again…
The main point is just to have a similar amount of light across the scene, and have all the color channels at a similar level. A gray card would be fine, I mostly use an LCD monitor with white image on it. If you do use a monitor, make sure sure it's out of focus (e.g. focus at infinity and put the camera a few cm from the screen) so it doesn't pick up individual sub-pixels.
This was fast … is there any reason in isoinc, why I can’t set append=true?
My laziness is the only reason there isn't an option, if that's what you mean ;)
edit:
The drtest results look good, so it seems the meter code works fine with the 14 bit sensor (aside from the ISO issues)
1) The "neutral" shot m96 value is exactly 0.
2) The remaining m96 values go in steps very close to 96, except for the +3 and +4 exposures which are clipped
3) w% is only > 0 for +3 and +4 ev
4) bl% is only > 0 for -6
-
This is one of the pictures from last drtest.
Should I run this with a monitor again?
-
No, it fine, see edit on previous post.
It only looks at the area inside the white box.
edit:
The jpeg gives you a nice idea how much distortion correction and cropping the camera does in the jpeg.
Thinking out loud: By drawing a grid on the raw and using the jpeg, it should be possible to automatically generate lens calibration data to replicate the canon distortion correction.
-
By drawing a grid on the raw and using the jpeg, it should be possible to automatically generate lens calibration data to replicate the canon distortion correction.
Fantastic idea. For every zoom step…
-
Hallo reyalp.
I have seen the timelapse from the moon ;)
Looks good….
I think it’s really hard to try with condition behind the tree. On the beginning it looks a little bit overexposed. On the end you see really nice the structure of the moon. I would be interested, which parameter you are used (and focal length) and to see the plot from the run.
Is there any reason, why you limited minimum Histogram step to 9?
I know, smaller values increases CPU load but precision of histogram will be also increased.
-
I have seen the timelapse from the moon ;)
Looks good….
I think it’s really hard to try with condition behind the tree. On the beginning it looks a little bit overexposed.
At the very beginning, it takes quite a while for the algorithm to reach steady state, because the initial exposure is for the whole scene and very far from correct for the moon. When less than the whole moon is visible, there is still some over-exposure. Some issues with the exposure algorithm also make it react more slowly than it should.
On the end you see really nice the structure of the moon. I would be interested, which parameter you are used (and focal length) and to see the plot from the run.
This was some code I was testing that isn't in the last version I posted. I've spent some time working on getting better results from small over-exposure limits, but the current exposure code has some problems. There's a couple of other tests on my youtube.
In the original code, there was always a "meter limit", which eventually balances any change caused by over or under protection (by ramping up the "weight" described in http://chdk.setepontos.com/index.php?topic=11081.msg120803#msg120803 (http://chdk.setepontos.com/index.php?topic=11081.msg120803#msg120803)) This doesn't work well for the moon at night, because the meter essentially goes to black level.
In the script used for that moon rise, I made the weight not ramp up. However, the meter still has it's normal weight, meaning that when the meter is far below the target value, it will still add up to 1/2 the "max ev change" in the wrong direction. This means that it still balances out the over exposure protection, just at a lower level. It also slows down the reaction time.
I have some ideas about how to deal with this (more options!) but I haven't gotten back to working on the code yet, which is why I hadn't posted.
The settings, aside from the customization above, were
meter size: 90% in each direction
meter step: 15
max ev change: 1/3
bv ev shift: 40
bv ev shift base: 9.5
max tv: 1 sec
iso adj tv 0.25 sec
max iso: 400
overexp thresh: 5 (0.05%)
overexp ev range: 1/2
underexp: disabled
histo step: 11
Focal length is ~211 (35 mm equivalent)
Note this was done on sx160, not the D10 I usually use for testing.
Chart and CSV are attached. Note the "over_frac" (in %) is on the right hand scale. You can see it does hover around 0.05 for most of the time. Because of the slope parameter in the over exposure weight calculation, it actually balances meter at 0.055.
Is there any reason, why you limited minimum Histogram step to 9?
The histogram precision is parts per 100k. On a 12MP cam, 9 gives you ~300k samples, so I don't think going to lower step sizes will help much. The time required goes up quickly with smaller steps because it's a square. Higher precision is possible, but with the kind of calculations the current code does on the results, it is gets quite easy to run into overflow.
-
I found in the log file over_thresh_frac=50.
Any reason to log it at 50 and not as 5 or 0.05%?
May be I would be helpful, also log things like focal length, focus distance, camera and so on.
How did you focus the moon?
When I use an external ND filter on G1X, I have much problems to focus. Manual focus while staying in the water and having the sun behind you is really hard. ;) I think manual focus setting in the script to infinity or hyperfocal distance would be really helpful.
-
I found in the log file over_thresh_frac=50.
Any reason to log it at 50 and not as 5 or 0.05%?
The exposure code uses parts per 100k internally, and just logs all the raw values it uses.
May be I would be helpful, also log things like focal length, focus distance, camera and so on.
Yes, that's a good idea.
How did you focus the moon?
Just manual focus in the camera UI at maximum distance.
I think manual focus setting in the script to infinity or hyperfocal distance would be really helpful.
Agreed, but focus override in CHDK has a lot of camera specific quirks and bugs so I've left it alone for now.
-
Hello reyalp,
I’m getting Addicted making time lapse with rawopint ….
Unfortunately I haven’t found a good situation with the moon yet…
I try to make a Boat time lapse ride with a minimum interval in continues mode with G1X or S110.
Do you think, it is a good idea to set the interval to 0?
I think, I will get a big jitter in time.
Which is the best value from the log file to analyses, to see how big the jitter is?
Is there any timestamp in jpg which is more accurate then 1s?
I analyzed time tick / start tick from my lust run with 3s interval. I found a jitter from around plus/minus 20ms.
This looks really good.
Do you think it is a good idea, to set the interval a little bit higher (0.7s) as the maximum speed (0.6s) from S110 to avoid time jitter?
-
I try to make a Boat time lapse ride with a minimum interval in continues mode with G1X or S110.
Do you think, it is a good idea to set the interval to 0?
I think, I will get a big jitter in time.
If you set the interval to 0, then the script will shoot as fast as it can and the time between frames will vary a lot.
Is there any timestamp in jpg which is more accurate then 1s?
AFAIK there is not.
I analyzed time tick / start tick from my lust run with 3s interval. I found a jitter from around plus/minus 20ms.
This looks really good.
Do you think it is a good idea, to set the interval a little bit higher (0.7s) as the maximum speed (0.6s) from S110 to avoid time jitter?
Yes. I would look at the "sleep" column. This is the number of milliseconds the script waited to maintain the proper interval. If it's less than 0, then shooting took longer than the interval. I would aim for a sleep of ~100 ms, keeping in mind that exposure time can be a big factor if you aren't full sun.
As long as the sleep is greater than 0, the jitter should be very low in this script, probably something like +/-20ms.
The columns shoot_ready, exp_start, raw_ready and raw_done are all ms offsets from "start", which is the tick time the shooting iteration started.
"exp_start" is some very short time (tens of ms, probably) before the exposure started, so if you want to measure the time between exposures it would be
(start[n] + exp_start[n]) - (start[n-1] + exp_start[n-1])
but unless there is a bug it will be exactly equal to the interval in ms as long as sleep is > 0. This doesn't mean there is 0 jitter: The tick counter only has 10ms resolution, and there is some room for variation after exp_start.
-
I made just a test in continues mode with S110. I have probably not much chance for the run.
50 pictures with interval = 0s, 50 pictures with interval = 0.8s.
I wondering, that the parameter of interval is not in the log, or I have not seen ;)
I put both runs in one plot, so jumps between both runs can be ignored.
Exposure time is less than 2ms so can be ignored.
As you see something is happens with interval 0.8s around shot 80. May be SD card is blocking..
When I calculate
(start(n)-exp_start(n))-(start(n-1)-exp_start(n-1))
It looks like that jitter is still with 0.8s interval higher than +/-100ms. But better than interval = 0.
But what does
tick(n)-tick(n-1)
Says? It looks like less jitter.
So when I have the chance for the run, I will try it with interval = 0.8s.
-
Just for completeness ..
-
I wondering, that the parameter of interval is not in the log, or I have not seen
It's not, I should add it.
(start(n)-exp_start(n))-(start(n-1)-exp_start(n-1))
If this is what you are using, it's not correct.
start is the tick counter (10ms resolution clock, counted from camera boot) value at which the shooting loop iteration started.
exp_start is how many ms after that the script signaled the exposure should start.
So the tick time of a given exposure started at is start + exp_start.
If you want the number of milliseconds between two exposures, then you should use
(start(n)+exp_start(n))-(start(n-1)+exp_start(n-1))
but as I said before, this should always be exactly equal to the interval as long as sleep is >= 0, so you might as well just look at sleep.
In your log, there is one single instance where sleep is <0. As you say, this might be an SD hiccup due to garbage collection or something like that.
The rest of the sleep times are all over 300ms, so you could probably get away with a 200ms shorter interval.
It's possible that setting "disable script yield" would reduce jitter, but this would not be detectable in any of the log values. You'd need to take pictures of an millisecond resolution wall clock or something.
edit: also, it might have unspecified side effect.
tick(n)-tick(n-1)
the "tick" column just records when the log line was written.
-
Yes, sorry, I mixed the sign, now it looks good.
With the tick I was only wondering, that is still very accurate.
-
If have seen your video from the moon, yes it is full moon, great ;)
I tried it as well with G1X, but I made a couple of things wrong. My moon is overexposed :(
I forgot to switch of the ND, so cam goes to maximum exposure time and ISO. I set max ISO to 400 (I thought I had set ISO = 300). And I set overexposure to 0.1% which was probably too high. May be I get another changes tonight…
http://youtu.be/F5z95312A0g (http://youtu.be/F5z95312A0g)
-
If have seen your video from the moon, yes it is full moon, great ;)
I tried it as well with G1X, but I made a couple of things wrong. My moon is overexposed :(
G1x is going to be difficult as long as the ISO override issues aren't resolved. The over exposure logic in the version of the script you have is also not very good for the moon.
Here's an updated version of the script. I'm not really satisfied with it, but I've been sitting on it for too long and it does some things better than before.
Histogram now works in parts per million, and over / under thresh are specified in parts per 100k. However, to use low values you must set your histogram step appropriately. Very small values are also more likely to suffer oscillation, though it will depend on the scene.
Added options:
Minimum shutter speed. This should be set to shortest shutter speed your camera can physically achieve, or longer. This is especially important if you prioritize over exposure protection. Setting it close to the Canon factory limit may help avoid flicker.
Meter high and low max weight. These control how the influence of the meter area changes between the thresh and limit values. The default 200 is the same as previous versions of the script. If set to 100, the corresponding thresh and limit are effectively disabled, and the meter influence does not increase (the limits still affect Bv Ev shift.)
Overexp and underexp max weight. These control the maximum strength of histogram based limits. These don't work in quite the same way as the meter weight limits: Histogram limits always reach weight 100 at the "thresh" value. If the thresh is exceeded, they continue to increase up to the max weight. The default 200 behaves the same as previous versions.
Overexp and underep prio (=priority) values. The default 0 behaves the same as previous versions. Greater values reduce the weight of opposing meter or histogram values, proportional to the fraction of thresh used. So overexp prio 50 will reduce the weight of meter by 50 when thresh is reached, if and only if the meter would drive the exposure in the other direction. This ramps linearly with the fraction of the thresh value used. This also applies to opposing histogram limits, so if over prio is set, and there is under exposure in the scene, the influence of the under-exposure will be reduced.
Add some information to the log
* histo_time is the time required to measure the histogram
* on startup, the chdk build info, interval and some zoom + focus related values are logged
The new settings are probably as clear as mud, so here's some hints:
* The default settings will generally keep the scene within the meter limits, as long as the shutter / iso limits permit. Over and under exposure protection will balance against the meter at the limits. This is good for situations where you don't want things like the sun, moon, or areas of strong shadow to completely throw off the overall exposure of the scene.
* Reducing the meter max weights allows over / under protection to push beyond the meter limits. This is useful if you want to ensure the over or under limit is not exceeded. If meter low max weight is set to 100, then over exposure will balance the meter when the over exposure fraction is reached. However, it will still respond slowly because it will be opposing the standard 100 weight of the meter.
* Adding "prio" reduces the influence of opposing meter or histogram values. As the name suggests, this lets you prioritize the over or under inputs. However, high values are likely to lead to oscillation.
* The over / under max weights only influence the behavior when the threshold is exceeded. They control how far the weight is allowed to increase.
I'll do another post for the last moon video, which gives an example of these.
-
Example:
http://youtu.be/WBM7o9dmhpg (http://youtu.be/WBM7o9dmhpg)
This was done with a slightly different version of the script, so some of the log values are the same as the current script would produce. In particular, this was before I changed the histogram to parts per million.
Relevant settings:
Meter width & height 90%
Meter step 15
Bv Ev shift % 30
Bv Ev shift base Bv 9
Meter low thesh 5 (only relevant to Bv / Ev shift)
Meter low limit 6 (only relevant to Bv / Ev shift)
Meter low limit weight 100 (it's 0 in the log due to script differences)
Overexp ev range 1/2
Overexp thresh 0.04%
Over prio 50
Underexp thresh 0 (histogram under limit disabled)
histogram step 11 (130k pixels)
over_frac is on the right axis, other values on the left.
edit:
Focal length was ~200 35 mm equivalent.
-
Thinking about what needs to be done to wrap up the C code changes for 1.4 release. This is a long post and is partly just to get my own thoughts in order, but any feedback would be appreciated.
Code organization
Currently, the code is built into the Lua script module. It depends on the scripts hooks, which are also built into the same module.
Originally, I was going to put the raw manipulation code in it's own module, but currently a large fraction of code is Lua interface "glue".
The glue can't be in a separate .flt, because there's no convenient way to export the Lua API from the main Lua module to another. This is something I'd like to address in the future, but not for 1.4
The rawop code adds a few kb to the Lua module, which I think is acceptable. If significantly more non-glue code is added, it should probably be split out. There are also possible uses for raw manipulation outside of script, which would also argue for splitting it out.
Error handling
Currently, the code doesn't check if you are in the raw hook, or if a valid raw buffer is available in the current mode (IE cameras that don't have raw in auto, or binned modes with low res raw).
* IMO it should, since writing outside of these conditions could be fatal
* OTOH the hook could time out while a long running call is in progress so it will never be completely safe.
* If a function is called when raw is not valid, should it error(), or just ignore the call? I'm inclined to use error. This means scripts will terminate if the hook timeout is too short, but script writers should use generous timeouts.
* The overhead of error checking itself could be a concern for functions like get_pixel and set_pixel. My current plan is to set / clear a flag on hook entry / exit
* a related question is how to handle invalid coordinates etc. The current code clamps/ignores calls that would fall outside the frame buffer, and I'm inclined to leave it this way.
G1x blacklevel issue
I don't really want to change the code a lot to deal with quirks in the one camera, especially since the ISO issues aren't really resolved. However, we will probably run into other cameras with quirks later.
* values associated raw "neutral" needs to be re-calculated when black level changes.
* a callback on entry into the hook could be used to re-calculate the values.
* scripts will no longer be able to pre-calculate values before shooting, which is inconvenient
* fb_info is not well suited to values changing on the fly. It's possible that other values could change at runtime in the future so this should probably be converted to individual calls. This may make it less convenient to write efficient lua code, but C calls shouldn't be worse than a table lookup.
meter interface
The valid meter step and count values are limited by overflow, and the limits depend on sensor bit depth. It would be possible to make a higher level function that makes multiple meter calls and averages them, but in practice I haven't found it to be a problem.
shot_histogram vs rawop histogram
Since shot_histogram is now a module, and is also available in ubasic, I'm inclined to leave it alone.
combined meter and histogram
If you want both a histogram and average value from a given area of the screen, getting the average from the histogram could be much more efficient, since reading large numbers of pixels is slow. However, I haven't come up with a good way to do this without either risking integer overflow, or using FP or 64 bit math.
Additional drawing functions
Some things I originally had in mind were
* arbitrary lines
* ellipses
* text
I'm inclined to skip the first two for 1.4, since I haven't found myself needing them. If someone *really* wants they could implement them in lua with set_pixel, and if there is demand, they can be added for 1.5
Text seems potentially more useful.
* People have requested things like date stamps before, and putting text on the image could be convenient for debugging.
* Adding date stamp functionality implies that some of the code must be outside of the script module
* Reusing the CHDK font infrastructure looks like it would be complicated, especially if you don't use the current OSD font.
* For numbers, you could a 7 segment display style output pretty easily with the current code.
* It would be possible to do simple text entirely from Lua too.
Other functions
Some other stuff I've thought about, but am not planning to implement for 1.4
* some kind of color matrix interface, to allow scripts to work in roughly correct RGB
* allow script to efficiently load/save chunks of the raw buffer
rawops in binned modes
A long time ago, srsa_4c investigated the the raw buffer for some half-res low light modes: http://chdk.setepontos.com/index.php?topic=10648.10 (http://chdk.setepontos.com/index.php?topic=10648.10)
At the time, we concluded that it wasn't worth pursing since the quality was low, but having it available for rawops could be quite useful. Metering and histogram don't depend on quality, and these modes typically shoot very fast, so people making timelapse for find the quality vs speed trade acceptable. If your final output is significantly smaller than the binned resolution, the quality loss is probably not a big deal. I'm not planning to pursue this for 1.4, but it argues for structuring the code to support varying raw size in the future.
-
I made the boot ride with G1X. I did not look as I expected. It is probably a lot of post processing necessary, to stabilize the pictures.
! No longer available (http://www.youtube.com/watch?v=3W2n7EAkpXE#)
I have not update rawopint until now. The run was made with version rawopscripts-2015-05-17. I still have some other runs with this version but I have not analyzed until now.
On the run itself, I couldn’t see anything on the display with the sun. It is really hard to stabilizing the boot. If you drive to slow, the boot gets instable. If you drive to fast, the interval from 0.7s is still too long.
The video was made in two parts. The first part was made with approximately 10km/h and the second part (starting around picture 1040) with 15km/h. I don’t use all pictures for the video.
May be it is not worse to think about:
There are a lot of exceptions in sleep. I’m not sure, with effect this has on the video. I have some other runs with 3s interval, where sleep goes close to 0. I have seen this on three different SD cards.
Another interesting think about the run:
The Battery has still more than 8,4V after 1500 shoots. The specification of G1X says 700 pictures with display off.
For the rest of the stuff you wrote, it takes time for me to understand… ;)
-
This is another sunset run with G1X. The run was made with version rawopscripts-2015-05-17. It starts to get flicker on the end of the video. Not as much as on the last video in Reply #144
May be your newest version will reduce it…
http://youtu.be/RiKxICDgYJw
(http://youtu.be/RiKxICDgYJw)
G1x is going to be difficult as long as the ISO override issues aren't resolved.
The over exposure logic in the version of the script you have is also not very good for the moon.
My understanding was, as long I stay with ISO under 300, it should be OK or?
G1x blacklevel issue
I don't really want to change the code a lot to deal with quirks in the one camera
I can understand you…
-
There are a lot of exceptions in sleep. I’m not sure, with effect this has on the video. I have some other runs with 3s interval, where sleep goes close to 0. I have seen this on three different SD cards.
I'll take a closer look at how mine are behaving. My impression is that it's been pretty stable, I usually run 1.2 sec interval on D10, and see few if any negative sleep until the exposure gets long. D10 continuous shooting rate is around 1 FPS.
SD cards have their own CPU and do wear leveling and garbage collection so their performance can be quite complicated, especially if they are used in a different way than the designers expected. See this page for some background https://wiki.linaro.org/WorkingGroups/KernelArchived/Projects/FlashCardSurvey
The script logging might fall into this category. It writes one log line every shot, and the camera firmware probably buffers it up to 32 k, but exactly how isn't clear. SD cards are mostly designed for continuous write speed of large files, so this might have an impact. I could make the log buffer to RAM for the whole run, but this would probably cause errors on long runs. You could also turn off logging, but then you wouldn't know how it's performing ;)
If you haven't tried it already, formatting the card might also give more stable performance. I haven't tried this, and only reformat my cards very rarely.
The Battery has still more than 8,4V after 1500 shoots. The specification of G1X says 700 pictures with display off.
On the cameras I've used, it seems to depend more on how long the camera is running rather than how many shots, so a longer interval gives you a bit more run time, but many less shots. On D10 I've had over 7000 shots, which isn't bad for a CIPA spec of 220 (screen on)
My understanding was, as long I stay with ISO under 300, it should be OK or?
I think so, unless there are other quirks we don't know about.
-
It starts to get flicker on the end of the video. Not as much as on the last video in Reply #144
May be your newest version will reduce it…
I think this is mostly a flaw in the over exposure algorithm, which isn't really fixed in the new version :(
A big part of the problem is there is no way to know in advance how much you need to adjust exposure for given change in over exposure fraction. For something like the sun, a full stop might barely change the over exposed fraction at all, where as a large area of sky might go from completely over exposed to barely over exposed at all with a small fraction of a stop. I have some ideas for dealing with this (some kind of feedback, or additional damping, or measure of how much is near the limit), but it will take some time to develop and test.
You could change the "Max Ev change", but this will affect all exposure adjustments, and if it's too small, the script might not keep up with lighting changes. I could make the max change for "meter" and over/under separate options, but this would have side effects on the weight system, and there would still be no obvious way to pick a "good" value.
-
SD cards have their own CPU and do wear leveling and garbage collection so their performance can
be quite complicated, especially if they are used in a different way than the designers expected.
Yes I understand the problematic of SD Cards. I made a lot of test with speed on PC and in the cam. Results can be really random. I always format the SD card in the cam, hope this will be the best for the cam. On the last runs I used two different SDXC 64G cards which are formatted as eFAT. May be I should try them to format with FAT32.
I could make the log buffer to RAM for the whole run, but this would probably cause errors on long runs.
I would not change it…
You could also turn off logging, but then you wouldn't know how it's performing .
I will try it, when I do a fast run again…
A big part of the problem is there is no way to know in advance how much you need to adjust exposure
for given change in over exposure fraction. For something like the sun,
a full stop might barely change the over exposed fraction at all
The run is definitely different as on in Reply #144. On Reply #144 you really see an oscillation on meter96.
This was not seen on the last run.
Overexposure on the end of the run only exceed on two pictures.
The flicker is really not much. This can be reduced easy with deflicker filter...
You could change the "Max Ev change", but this will affect all exposure adjustments,
Does this really make a different for the last run?
Max Ev change was set to 32. Max changes on the end of the run where only 7 (calculated d_ev (n) - d_ev(n-1) or tv96(n) - tv96(n-1)). So far away from maximum.
Or a misunderstanding from my side?
The overexposure show really big jumps on the end of my last run.
I understand that adjust exposure can give unknown change in over exposure fraction.
I’m not sure, but it might be helpful, to give the overexposure a smooth before analyzing it.
From your documentation:
There are too many options, and their meanings are not clear.
I agree… specially for not experience user…
I still not try to play too much with all parameters.
On my first analog SLR I had a Switch for ‚Simple‘ and ‚Advanced‘ user.
Something like this could be helpful for beginners…
-
SD cards have their own CPU and do wear leveling and garbage collection so their performance can be quite complicated, especially if they are used in a different way than the designers expected.
For reference, there was a discussion (with data) about this here (http://chdk.setepontos.com/index.php?topic=10822.msg116507#msg116507). It is a legitimate concern with "high shot rate" intervalometers. Logging makes it worse.
I could make the log buffer to RAM for the whole run, but this would probably cause errors on long runs.
That's eventually what I did (with code from naccio) for the kap_uav.lua script, with the refinement of writing the buffer periodically to the SD card as it filled up.
-
My understanding of rawopint is, there is only one write for a line after each picture.
of writing the buffer periodically to the SD card
What does ‘periodically’ mean? Collecting 100 Lines for 100 pictures and writing them together?
For time-lapse probably not helpful, because every 100th interval will be definitely wrong (maybe).
the log buffer to RAM for the whole run
May be an option for a fast and short run…
My longest run had 5000 pictures; I want to make a 24h run with 20000 pictures (4s Interval) or more…
What happens, when the external power supply gets disconnected….
-
The run is definitely different as on in Reply #144. On Reply #144 you really see an oscillation on meter96.
This was not seen on the last run.
Overexposure on the end of the run only exceed on two pictures.
The flicker is really not much. This can be reduced easy with deflicker filter...
The oscillation in meter is small, but it's there. It's more obvious in over_frac and over_weight. I think the oscillation just got more extreme in 144, it's gets larger toward the end of 188.
Does this really make a different for the last run?
Max Ev change was set to 32. Max changes on the end of the run where only 7 (calculated d_ev (n) - d_ev(n-1) or tv96(n) - tv96(n-1)). So far away from maximum.
Or a misunderstanding from my side?
The over/under protection always use the "max" value, scaled by the "weight", which depends on close (or far over) it is to the thresh value. So adjusting the max would change how much the exposure changes for a given amount of over exposure.
This again is because there is no way to know how much adjustment is needed. The "meter" calculates exactly how much it needs to get "correct" exposure, and then limits the change to the "max"
From your documentation:
There are too many options, and their meanings are not clear.
I agree… specially for not experience user…
I agree completely. I hope to simplify it eventually, but right now it's all experimental. I don't know what will work well in a given situation.
Although I hope to make it useful to users in the end, the I wrote the script because I needed a real world test case for the features I'm adding to the C code.
-
For reference, there was a discussion (with data) about this here (http://chdk.setepontos.com/index.php?topic=10822.msg116507#msg116507). It is a legitimate concern with "high shot rate" intervalometers. Logging makes it worse.
As we discussed before (http://chdk.setepontos.com/index.php?topic=11081.msg121585#msg121585), in that case the log was being opened/closed for every line, which is not what rawopint does.
That's eventually what I did (with code from naccio) for the kap_uav.lua script, with the refinement of writing the buffer periodically to the SD card as it filled up.
Which is what the camera firmware does if you don't close or flush every line.
My understanding of rawopint is, there is only one write for a line after each picture.
It calls write() for every line, but the firmware physically writes the data to the file less often (probably, the details are mostly unspecified although I know there's a 32k buffer involved).
What does ‘periodically’ mean? Collecting 100 Lines for 100 pictures and writing them together?
For time-lapse probably not helpful, because every 100th interval will be definitely wrong (maybe).
IMO the actual problem probably isn't writing to the card per se, but that the card periodically takes much longer to write than normal, due to needing to GC erase cycles or something like that. Lots of small writes, or writes that don't line up with the cards underlying block structure probably make this worse.
There may be some optimal size that minimizes this. I went with OS buffering because it was easy, and there is at least some chance it's optimized to play well with typical cards.
-
Some updates in trunk 4191
Error handling
Currently, the code doesn't check if you are in the raw hook, or if a valid raw buffer is available in the current mode (IE cameras that don't have raw in auto, or binned modes with low res raw).
All the functions that access the raw buffer will now error() if called outside the hook, or if CHDK knows the current shooting mode does not have raw (note that many ports probably don't do this correctly in all cases)
G1x blacklevel issue
I don't really want to change the code a lot to deal with quirks in the one camera, especially since the ISO issues aren't really resolved. However, we will probably run into other cameras with quirks later.
* values associated raw "neutral" needs to be re-calculated when black level changes.
* a callback on entry into the hook could be used to re-calculate the values.
I added code to update the blacklevel based values on hook entry.
Note that rawopint and the other scripts will need to be updated to correctly handle this. It also doesn't address the other ISO related weirdness on G1x.
This raises another question:
Should blacklevel and dependent values and functions (like raw_to_ev96) be accessible outside of the hook at all? In theory, they shouldn't, since any calculation done with them could become invalid before the next shot. However, for all (known) cameras except the G1x this is a needless inconvenience. I haven't decided which way to go.
Finally, I made a subtle change to the behavior of the hooks. Previously, if you didn't set the hook, it would become "active" momentarily when the hook was reached, without blocking the hooked task. I don't think this could have been detected reliably from script, bit in any case 4191 removes the ambiguity and hooks only become active if the hook has been set. If you just want to detect the hook being reached, you can check for hook.count() incrementing. I don't think this will affect any of my scripts.
-
Note that rawopint and the other scripts will need to be updated to correctly handle this.
So an update from my side to r4191 is not useful until rawopint (or isoinc) is updated?
It also doesn't address the other ISO related weirdness on G1x.
My understanding where, there are 2 things on G1X:
1) The black point at ISO320
2) The1/2 ISO kick at ISO400
R4191wil fix the black point but not the ½ ISO kick?
But R4191 together with the ISO Set mode Script on isoinc will work?
-
Note that rawopint and the other scripts will need to be updated to correctly handle this.
So an update from my side to r4191 is not useful until rawopint (or isoinc) is updated?
rawopint needs to modified to take advantage of this change.
I think isoinc should not need any changes.
My understanding where, there are 2 things on G1X:
1) The black point at ISO320
2) The1/2 ISO kick at ISO400
R4191wil fix the black point but not the ½ ISO kick?
Right, R4191 makes it possible to write scripts that use rawop and behave correctly when the blacklevel changes.
But R4191 together with the ISO Set mode Script on isoinc will work?
The details are confusing enough that I'm not sure, but it would be worth trying.
-
* fb_info is not well suited to values changing on the fly. It's possible that other values could change at runtime in the future so this should probably be converted to individual calls. This may make it less convenient to write efficient lua code, but C calls shouldn't be worse than a table lookup.
In trunk r1493, I got rid of rawop.fb_info() and converted the members to individual function calls.
Most scripts that used rawop need to be updated for this change. This includes laulib/rawoputil.lua and the scripts under scripts/test so be sure to include those when you update you CHDK build.
Attached are updated versions of rawopint and contae. There are no changes to functionality, but they will see the correct blacklevel dependent values on G1x when it changes. I'm not sure if this will result in correct behavior, and it still doesn't account for the other ISO change at 400.
Should blacklevel and dependent values and functions (like raw_to_ev96) be accessible outside of the hook at all? In theory, they shouldn't, since any calculation done with them could become invalid before the next shot.
Still on the fence about this one. For now, I have not made them error if called outside the raw hook.
-
Attached are updated versions of rawopint and contae. There are no changes to functionality, but they will see the correct blacklevel dependent values on G1x when it changes. I'm not sure if this will result in correct behavior, and it still doesn't account for the other ISO change at 400.
So a run on G1X until ISO399 should be OK…. but together with the ‘Set ISO mode Script’ from isoinc should be handle both problems on G1X … so test this might be helpful?
-
So a run on G1X until ISO399 should be OK
Possibly. I'm not sure if the neutral and meter calculations are still valid after the blacklevel switch.
but together with the ‘Set ISO mode Script’ from isoinc should be handle both problems on G1X …
Not quite. With the "script" method, the exposure seen in the raw buffer drops after ISO400. It acts as if the actual sensor ISO was reduced back to 200 and +1 ev compensation is added back in jpeg processing. This will throw off metering, since it operates on the raw data. In theory, it should be possible to detect and compensate for this, but it would require more code changes.
One thing that would probably be worthwhile is writing a script that dumps all the propcases before and after the transitions (both blacklevel and the ISO thing). If there's something we can query to definitively know which state the camera is in, that would simplify things a lot, and perhaps even allow us to fix it generically in C code instead of script workarounds. I'll try to come up with something like this when I have some time.
-
I have problems running rawopint (also isoinc) with R4193 on S110…
-
I have problems running rawopint (also isoinc) with R4193 on S110…
Are you certain you are using the script from post #199 http://chdk.setepontos.com/index.php?topic=11081.msg123560#msg123560 (http://chdk.setepontos.com/index.php?topic=11081.msg123560#msg123560) and also the CHDK directory files from r4193?
-
Sorry, I forgot to update the script. With Version 0.17 it is running…
-
I made a new run on G1X with R4193 on really hard condition. Extremely fast moving clouds.
I think the interval from 3s was still too high.
A couple of times the run is controlled by overexposure.
Rawopint does a really good job…
http://youtu.be/7WEHsPb9ENQ (http://youtu.be/7WEHsPb9ENQ)
-
This is one of my latest runs on S110 with rawopint (rawopint_wm4) on S110. I thought the scene should be easy because not much changes, no moving clouds and nothing against the sun. I also reduced max ev changes.
I was wondering that I got a flicker (not much on it). You can see it especially when you are concentrated on the sky
https://youtu.be/UZBKzCky6wc (http://youtu.be/UZBKzCky6wc)
I analyzed the data.
I think there is no overexpose control at all.
When you see the plot of meter96, you see parts, where it gets oscillated. The amplitude is not much.
On the second plot I compared a part of meter96 with the JPG pictures. You see the same oscillation in the pictures. The changes are not much, but the oscillation made the flicker.
Where das the oscillation come from? May be from the surface of the water. The meter window is default but probably 50% of the meter window is water at the beginning. When the water goes higher and more waves coming in, flicker increases. I’m still thinking, if an option to move the meter window could be helpful.
For ui_draw_meter_t an option { None Corners Box FirstPictureBox}, to plot a box on the first window would be really helpful for analyzing.
I made a second run to the same time with G1x (rawopint_wm3). This run show also flickers but smaller as from S110. Here the meter window is on the beginning more outside from the water. When the water comes up, also here the flicker increases.
I’m not sure, if this can help, to avoid oscillation:
As long as meter96 is in a specific range (may be +-8 values or selectable EV range), rawopint should not change any exposure. Exposer should only be changed, when meter96 is outside this range (or on overexposure).
-
An the zip file...
-
This is my first run on SX230 with rawopint.
https://youtu.be/krilj4H_4lM (http://youtu.be/krilj4H_4lM)
Generally it works very well. Results looking similar to S110.
I see the same oscillation as on the other cams. It is really not much and it is not really seen on the video, because the scene is more changing as the scene on S110.
Also with the last run on S110: It is really if difference, watching the video on YouTube or seeing it with lower compression on a 55 inch flat screen. With a deflicker filter on VirtualDub it can be eliminated. But sometime this filter shows other effects.
Is somebody here who has a source code for some deflicker filter? I thought I have even to change the brightness, but this will not give very good results…
-
About Attachments:
Why there is a Restriction about 2 per post?
I can understand limitation about size. The size of the plots is only 20k.
-
Interesting script.
Did anyone test with an sx260hs yet?
If not: should I and using what values?
-
Interesting script.
Well, I worked with a lot of scripts for time-lapse. Wrote my own script, which is not bad but rawopint is absolutely great. Far away from other time-lapse script, which I have used.
should I and using what values?
Use as much as possible the defaults. Defaults give you already great results.
My suggestions about timelapse against the sun:
ui_exp_over_thresh_frac=1000 1% overexposure
ui_exp_under_thresh_frac=0 underexposure control off
Joerg
-
Thanks for the tip.
I copied rawopint.lua to the SCRIPTS directory on the SD card.
For running it appears that I also need a rawoplib.lua file.
I'll need to try the 1.4.0 CHDK build...
And yes, it runs now.
I got got over 2500 photo's from my NB6L clone battery, many more than the dpreview (http://www.dpreview.com/reviews/canon-powershot-sx260-hs) said.
I used the info from this blog (http://blog.waan.name/creating-timelapse-videos-on-linux/) to make an mp4 out of the photos.
The result is boring as the sky was tightly clouded up high.
But as a test it worked.
I did notice some flickering though.
How can I fix this?
What does it mean when the csv says 'sv 380 != cam 476 / smooth overshoot:-1' ?
PS:
I changed the Target ISO to 100 as the sx260hs doesn't know 80.
Max ISO was changed to 400 as that gives a bit of noise already. (maybe I am paranoid w.r.t. noise and we will not notice this in an mp4)
If I disable Draw Debug will that remove the meter at the top of the photo?
-
Hello,
I'll need to try the 1.4.0 CHDK build...And yes, it runs now.
Yes, the script is still under developing. You have to use the latest CHDK build...
I got got over 2500 photo's from my NB6L clone battery, many more than the dpreview said.
The same with my cams. It's amazing who long the cam works. I think zooming and focusing needs a lot of energy…
I used the info from this blog to make an mp4 out of the photos.
I do it with Virtual Dub…
I did notice some flickering though. How can I fix this?
Where we are talking about? When this is the same flicker as in my videos it is around 2/96 EV steps. Not much. Using a DLSR or most other scripts give you 1/3 EV steps. Factor 16 higher. I fix it with a deflicker filter in VirtualDub. But I’m sure, reyalp will find a solution to fix it.
It would be probably helpful, when you give us a link to the video und upload the log file.
What does it mean when the csv says 'sv 380 != cam 476 / smooth overshoot:-1' ?
I changed the Target ISO to 100 as the sx260hs doesn't know 80.
Yes, when your cam did not support ISO80…
Max ISO was changed to 400 as that gives a bit of noise already. (maybe I am paranoid w.r.t. noise and we will not notice this in an mp4)
I can’t see any noise on my S110 or G1x on ISO400. But both have a greater sensor. On my SX230 I haven’t checked it…
If I disable Draw Debug will that remove the meter at the top of the photo?
Yes, the draw might be helpful for the developer to analyze…
This is another run with my S110. I’m not sure if all this uploads useful…
http://youtu.be/SC-L8NFk-RI (http://youtu.be/SC-L8NFk-RI)
-
I did notice some flickering though. How can I fix this?
Where we are talking about? When this is the same flicker as in my videos it is around 2/96 EV steps.
It is similar to the flickering in this (https://www.youtube.com/watch?v=UZBKzCky6wc&feature=youtu.be) video.
So perhaps not bad but still noticeable.
-
When your flicker is similar, you will see it on the meter96 value in the log file.
This is one of my first videos, which I made with my own script. Go to 1 minute, where the sun goes down. There you will see, who it looks, when you have 1/3 EV steps…
! No longer available (http://www.youtube.com/watch?v=ZyOeVs3Ha6g#)
Here are some test videos with different deflicker filters…
You have to see it in HD.
! No longer available (http://www.youtube.com/watch?v=FOb2TJYtm0s#)
! No longer available (http://www.youtube.com/watch?v=whRrdYU8qTw#)
-
I got got over 2500 photo's from my NB6L clone battery, many more than the
FWIW, on D10 (also NB6L) I get over 7000 shots sometimes ;)
The result is boring as the sky was tightly clouded up high.
But as a test it worked.
I did notice some flickering though.
How can I fix this?
Flicker can happen for many reasons. If you upload the log file, I may be able to tell what's happening in your case. Uploading the video may be useful too.
There are some deflicker plugins from virtualdub that can help some.
What does it mean when the csv says 'sv 380 != cam 476 / smooth overshoot:-1' ?
In the desc column, the / separates different messages, so these are two unrelated things.
The sv message means the script tried to set ISO 80, but when it read back, it was set to 160.
I assume this is from before you changed the target ISO?
The smooth overshoot isn't a problem, it's just information about what the code was doing. The exposure algorithm tries to smooth out exposure changes by averaging the previous exposure change with next calculated exposure. "smooth overshoot" happens when the average would change exposure in the wrong direction for the current meter result. For example, if the last exposure change -3 and meter shows +1 is needed to reach the target exposure, then averaging in the -3 would would result in a -1 change. In this case, no change is made. This message will also be generated every time the exposure change becomes 0.
You should understand that this script was mainly written to test some new features in the CHDK 1.4 code. It works fairly well, but there are some rough edges.
@c_joerg
Thanks for posting your results and logs. I'm trying to focus on stuff for the 1.4 release now, so I may not respond quickly, but it should help improve the script in the future.
-
I'm trying to focus on stuff for the 1.4 release now, so I may not respond quickly
No hurry ... basically it works very well ... with the little flicker I can live ...
-
What does it mean when the csv says 'sv 380 != cam 476 / smooth overshoot:-1' ?
In the desc column, the / separates different messages, so these are two unrelated things.
The sv message means the script tried to set ISO 80, but when it read back, it was set to 160.
I assume this is from before you changed the target ISO?
Yes, I changed the 80 into 100 afterwards.
The log is attached.
-
Yes, I changed the 80 into 100 afterwards.
OK, using 100 should fix it. Unfortunately there's no way to make the menu default to the lowest ISO the camera supports.
The log is attached.
One thing I notice is that exposure is quite short, sometimes less than 0.0005 (1/2000) second. I have found very short exposure to cause some flicker on my cameras, but sx260 allows down to 1/3200 in the factory firmware, so maybe it's OK at this speed.
I can't tell if this is the cause of the flicker in your case. The meter only changes by +/- 2/96th of stop, so it's not very much. If there is more variation in light, it's easier to tell if this is happening because it smooths out when the shutter is below some value (around 1/1000th on the D10 I usually use)
Unfortunately the script doesn't yet control aperture, or ND filter on cameras that have both iris and ND, so to avoid this you would have to set a smaller aperture or ND before starting the script. You can set a minimum shutter speed (Min Tv sec/100k), but if that limit is hit the scene will be overexposed.
If the ISO had been set to 100, the exposure might have been slightly longer.
-
I have found very short exposure to cause some flicker on my cameras, but sx260 allows down to 1/3200 in the factory firmware, so maybe it's OK at this speed.
It could be interesting to have a tvinc.lua (similar to isoinc) or a script, which measures meter96 with constant tv and ISO settings…
The meter only changes by +/- 2/96th of stop, so it's not very much.
Meter96 looks similar to my runs. It is really not much but the oscillation gives you the impression, it is more. Especially when you see it on larger screens with higher bitrate. But deflicker filter can handle this very well.
Unfortunately the script doesn't yet control aperture, or ND filter on cameras that have both iris and ND, so to avoid this you would have to set a smaller aperture or ND before starting the script.
My understanding was, ND filter can’t change while running in quick or continues mode. But setting ND filter on the beginning of the script would be easy possible…
I notice on my SX230, that I can’t switch the ND filter in the canon firmware, which is different to S110 and G1x.
On S110 and G1x I switched the ND always on, when I doing timelapse.
When I turn on the ND filter in the SX230 in CHDK menu, then normal images are underexposed because presumably the Canon firmware did not notice this. I have not checked this together with rawopint, but maybe the first pre shoots are wrong.
@udo:
This might be interesting for you, why do I always turn on the ND filter .
http://chdk.setepontos.com/index.php?topic=12258.0 (http://chdk.setepontos.com/index.php?topic=12258.0)
Reply #7 '180 degree shutter rule’
-
One thing I notice is that exposure is quite short, sometimes less than 0.0005 (1/2000) second. I have found very short exposure to cause some flicker on my cameras, but sx260 allows down to 1/3200 in the factory firmware, so maybe it's OK at this speed.
I can't tell if this is the cause of the flicker in your case.
Aside from the configuration changes I mentioned, the camera was at ISO200 before starting the script.
I am unsure if this has any influence, as are the mode (P), etc.
I read through the thread etc but I might have missed a clear statement of what settings/modes in the camera do carry over into the script's operations.
I'll do a run with the correct script settings when the weather clears up a bit.
-
It could be interesting to have a tvinc.lua (similar to isoinc) or a script, which measures meter96 with constant tv and ISO settings…
Yes, either extending that script or making additional ones for tv, av and nd is on my list of things to do.
My understanding was, ND filter can’t change while running in quick or continues mode. But setting ND filter on the beginning of the script would be easy possible…
On cameras with that only have an ND, the script can change it on the fly. It should be possible to do for cameras that have both, but there is some complication around detecting the initial state and value of the ND and I haven't got around to it yet.
I notice on my SX230, that I can’t switch the ND filter in the canon firmware, which is different to S110 and G1x.
Yes, a few cameras have a "hidden" ND filter, that doesn't show up in the Canon UI. Thanks Canon ::)
I have not checked this together with rawopint, but maybe the first pre shoots are wrong.
The first may be wrong, but it should settle after a few shots. There's normally some difference between what Canon thinks the correct exposure and what the script targets anyway.
-
Aside from the configuration changes I mentioned, the camera was at ISO200 before starting the script.
I am unsure if this has any influence, as are the mode (P), etc.
I read through the thread etc but I might have missed a clear statement of what settings/modes in the camera do carry over into the script's operations.
If everything is working, the script should completely control the the Tv and ISO regardless of the canon UI settings. Settings that it doesn't control (like Av) will be whatever the Canon firmware would have used for the first shot, and shouldn't change while the script is running.
In your case, it tried to set an ISO lower than the camera can use, so the ISO ended up with a different value. I'm not sure if this would have been what you set in the Canon UI or what the camera picked for auto ISO, but it doesn't really matter.
-
@udo:
This might be interesting for you, why do I always turn on the ND filter .
http://chdk.setepontos.com/index.php?topic=12258.0 (http://chdk.setepontos.com/index.php?topic=12258.0)
Reply #7 '180 degree shutter rule’
Hmm. Thanks for the tip!
If I see shutter times approach 1/1000 (and the weather will get a bit better over here later this week) I might do a 2nd run with the ND filter 'in' to see how that influences the performance.
-
times approach 1/1000
The ND gives you 3 stops 1/1000 => 1/125.
Additional you can use the aperture…
-
I made just a run with SX230 with ND Filter in. The first picture was underexposed and the second OK. Meter96 shows a jump from around 195. settings from both pictures are the same. Can it be that the ND filter goes off after the first shoot and that the ND has only 2 stops?
-
Picture 2
-
I made just a run with SX230 with ND Filter in. The first picture was underexposed and the second OK. Meter96 shows a jump from around 195. settings from both pictures are the same. Can it be that the ND filter goes off after the first shoot and that the ND has only 2 stops?
Were you using continuous mode? Continuous or not may behave differently. (edit: looks like this was not continuous from the log)
A override in CHDK it should apply to every shot, but I think for both continuous and the hold half, click full method will probably only set it for the first shot. Normally the firmware doesn't change exposure settings in in this case, so it should work, but the "hidden" ND is weird so I wouldn't be surprised if it behaved differently.
You could test this manually by setting the ND in CHDK, and shooting a burst in continuous mode.
Normally I think the ND is closer to 3 stops, but it's possible this cam could be different. It would be easier to be sure with a controlled test similar to isoinc.
-
You could test this manually by setting the ND in CHDK, and shooting a burst in continuous mode.
I made some testing with ND on SX230. I was wondering, that the ND is always off, when I switched the camera on. Is the parameter not saved?
When the ND is in, I got following result:
Single mode: ND is active on every shot.
Continuous Mode: ND is active on every shot.
Continuous Mode with AF: ND is active only on first shoot, all the other shots the ND is inactive.
But when I do a full press, without waiting for focusing, the ND is always inactive, independent from the mode.
-
Interesting findings.
Does this mean we need ND control in the script?
(to stay `below` 1/1000 and to have control...)
-
I made 3 isoinc runs. Single, Quick and continuous mode. It looks like, that continuous mode works but the ND has less than 2 steps…
Does this mean we need ND control in the script?
You have to set your camera in continuous mode. And you have to switch ND on after every new start. May be it is different on your cam. But I look like that the ND is different to the other cams. I would use additional the aperture on minimum f=5.6.
-
I made some testing with ND on SX230. I was wondering, that the ND is always off, when I switched the camera on. Is the parameter not saved?
There is an option to clear overrides on startup, which is enabled by default.
The overrides are only actually applied on half-shoot.
Continuous Mode with AF: ND is active only on first shoot, all the other shots the ND is inactive.
Since CHDK overrides are applied in half press, this isn't really a surprise. I'd consider it a bug, but probably not easy to fix. The rawopint script may have other trouble in this mode too, since it assume the canon firmware keeps everything fixed after the first shot.
But when I do a full press, without waiting for focusing, the ND is always inactive, independent from the mode.
This is a is also a bug. This case often takes a different path in the canon code and has issues, so it isn't much of a surprise either.
Can you check whether other overrides (Tv, ISO, Av) are applied and whether raw is saved in case?
-
Hmm..
I had the sx260 in P mode and set continuous mode before starting the script.
I enabled the ND filter ('in').
Yet I see exposure times below 1/1000th when checking the EXIF of the images.
The sky looks OK.
The buildings on the ground look a bit underexposed at times.
I also see some dark parts in the foreground light up white at times...
-
Hmm..
I had the sx260 in P mode and set continuous mode before starting the script.
Is it continuous with AF, or standard continuous? c_joerg's results suggest that continuous + AF doesn't handle the ND correctly on sx230. It's likely that other cameras with the same feature suffer similar problems.
I enabled the ND filter ('in').
Yet I see exposure times below 1/1000th when checking the EXIF of the images.
Since your camera as an adjustable aperture, you could set that in canon controls (using M or Av mode) before shooting.
The sky looks OK.
The buildings on the ground look a bit underexposed at times.
I also see some dark parts in the foreground light up white at times...
It's impossible to have any useful comment on this without seeing the results.
-
There is an option to clear overrides on startup, which is enabled by default.
Interesting, I never used overrides before….
The rawopint script may have other trouble in this mode too,
Yes, it was only a test, I don’t want to use continuous AF mode with rawopint…
Can you check whether other overrides (Tv, ISO, Av) are applied and whether raw is saved in case
I checked ISO and Av override. It work on all modes and also with half and full press.
I checked RAW only on single mode (because of crashing in continuous mode). It works with half and full press (override Av + RAW).
@udo:
The sky looks OK.
The buildings on the ground look a bit underexposed at times.
Well, it is your decision what you want. The script can’t change the dynamic of your senor.
Especially against the sun: You can have a
1) neutral sky and underexposed ground
or
2) overexposed sky and neutral ground.
My decision is always neutral sky and underexposed ground. When you use RAW, you can rescue the underexposed ground. But this didn’t works with overexposed sky…
-
Hmm..
I had the sx260 in P mode and set continuous mode before starting the script.
Is it continuous with AF, or standard continuous?
No AF. Just continuous.
Less flickering in the video than before!
I enabled the ND filter ('in').
Yet I see exposure times below 1/1000th when checking the EXIF of the images.
Since your camera as an adjustable aperture, you could set that in canon controls (using M or Av mode) before shooting.
I'll do a run with lower aperture next time and hope for ND filter support in the future... ;-)
In Av mode aperture dial goes to 8 but the Av override goes to 16. (does anything over 8 work on the sx260?)
What would be the best way to use an aperture setting? Via Av mode or via CHDK overrride?
CHDK mode is easiest to do; also in there I can override the distance to infinity.
-
What would be the best way to use an aperture setting? Via Av mode or via CHDK overrride?
In theory, either should be OK. I suggested using canon controls because they are less likely have bugs or special quirks than CHDK controls.
In Av mode aperture dial goes to 8 but the Av override goes to 16. (does anything over 8 work on the sx260?)
Maybe, maybe not. In general, it's likely that you can go higher than at if you are zoomed in with the optical zoom, but the exact details are up to you to determine by experiment, if you care.
An example on sx160 http://chdk.setepontos.com/index.php?topic=12297.msg121510#msg121510 (http://chdk.setepontos.com/index.php?topic=12297.msg121510#msg121510)
CHDK mode is easiest to do; also in there I can override the distance to infinity.
Regarding the video you sent me in PM:
I don't see any obvious defects except for a very small amount of flicker. I'm not sure which "darker spots" you are referring to, I see shadows on the ground and birds (and maybe bugs or other debris close to the camera) on the sky.
As c_joerg says, you always have to trade how much over exposure you want to accept against how dark you want to allow the rest of the scene to get. The script offers you a lot parameters to adjust this, but you will probably need to do many runs to get a good feeling for how they work. I'll try to provide a few comments suggestions.
First, a general description of how it works:
* The script uses two measurements to control exposure: A "meter" area centered in the scene, and a histogram of the entire scene.
* The exposure system tries to keep the average value of the meter at some target exposure value, and uses the histogram keep over and under exposure within some limits.
* By default, the meter is 30% of the sensor width and height, something like spot metering mode in the canon firmware. Stuff outside of this area only influences exposure if it reaches the histogram limits.
* By default, the target exposure value for the meter is a "neutral" exposure, roughly what what Canon AE would give if you aimed the camera at a uniform grey subject.
* The meter target value, limits, and balance between meter and histogram control are all adjustable in the script settings. The individual settings are described in the readme.
Some notes on this specific run:
* Meter was the default 30%, which is mostly sky in this scene.
* The log shows that the exposure was almost entirely driven by meter. The meter_weight, over_weight and under_weight columns show the relative contribution of meter and histogram over and under exposure limits.
* Since you set Underexp thresh to 0, under exposure is ignored. The under_frac column shows there was substantial under exposure, with 15-20% of the scene -4 ev or below.
* There was some over exposure toward around 3/4 of the way through the run, but over_weight never exceeded 1 (except the very start before the algorithm settled), so it had very little impact.
Some things you could do differently
* In a normally composed landscape scene, the sky is bright and should be more exposed than "neutral". So if your meter mostly on the sky, you may want to use the script "Ev shift" option to increase the target exposure. I'd suggest +1/4 to +1/2.
* Alternatively, you could use a larger meter, so it would include the ground. You can use something like 90% if you want.
* If you had under exposure protection enabled, it would have pushed the exposure higher until the sky started to hit the over exposure and / or meter high limits. You can use the "max weight" and "prio" options to control the relative influence if there is both under and over exposure.
* Since the meter is mostly on the sky, clouds moving by cause the brightness of stuff on the ground to change even when the actual lighting hasn't changed. You can use the "Bv Ev shift" option to reduce this effect. This adjusts the target exposure by the absolute brightness of the scene. I find a value around 30% gives reasonable results. The "Bv Ev shift base" option controls sets the brightness at which there is no shift. "First" uses the initial exposure. A well lit subject in daylight would be around 10.
-
Did another timelapse.
This one was in Av mode with aperture at 8.
Exposure still shorter than 1/1000 at times because over time the sun was visible in the picture when the clouds did not obscure this luminous object.
I am quite happy with the results.
-
This one was in Av mode with aperture at 8.
Together with ND?
I always use ND. I never do it without against the sun and never on midday.
I made a couple of runs against the sun (you can see it on sides before) with G1x and S110.
I don’t find a run which is higher than 1/500. Ok the ND is one step higher...
-
This one was in Av mode with aperture at 8.
Together with ND?
No ND as I did not yet get the ND to work (see my previous test with continuous mode and ND 'in').
So what should I try on the sx260hs to see if ND works or not?
Soon I'll have an S110 too to see it that one behaves differently.
-
So what should I try on the sx260hs to see if ND works or not?
Try 2 runs with rawopint (one with and one without ND) each 8 shots with the same scene (maybe inside with tungsten light) in continuous mode without AF in Av Mode and see in the log file, if there is any difference.
-
Try 2 runs with rawopint (one with and one without ND) each 8 shots with the same scene (maybe inside with tungsten light) in continuous mode without AF in Av Mode and see in the log file, if there is any difference.
The ND column says 0 for both sets of eight photos yet CHDK says `ND Filter: in` in red over the display after taking the second set of photos.
Soon I can repeat this for S110.
-
The ND column says 0 for both sets
I’m not sure if this column is correct set. On my S110 it is the same, still when ND is in.
Interesting is, if there is any difference between both runs in exposure time? Do you have the log file?
-
The logfile is attached.
Maybe we can deduce the ND presence from the exposure times but then we need a very constant situation w.r.t. light.
So the ND indication in the logfile might (!) be buggy.
-
but then we need a very constant situation w.r.t. light
Yes, as I said, inside with tungsten light.
The second run is with ND in? There is a jump in tv from 2EV steps…
So the ND indication in the logfile might (!) be buggy.
I think the column is correct for cams without aperture…
-
So the ND indication in the logfile might (!) be buggy.
I think the column is correct for cams without aperture…
Yes, that column is only meaningful if the script is actively controlling the ND. As I've said, the script does not currently support ND on cams with an adjustable aperture. If you set it outside the script, you should see an effect on the exposure, and also an effect on the Bv if ND value isn't counted as part of the aperture.
On cams without an adjustable aperture, the script can detect the initial ND state (if it was set by the canon firmware) by looking at min Av and current Av, because one of the Av propcase reflects ND value. If the ND filter is set it can also detect the value. The CHDK ND overrides functions do update the propcase.
For cameras with both, AFAIK the ND setting does not affect the Av propcases, but I don't have a camera to test. There are some ND related propcases, but without cameras to test I'm not sure their exact behavior. The cameras with a "hidden" ND are likely to differ from those with manual control in the canon firmware.
-
I made a couple of moon runs.
It was a little bit frustrating. First run with G1x. Much overexposure. I don’t know why. I set overexposure to 0.02% (or only 0.2%?)
OK, script starts with ISO800. I know the problems with G1x, but then ISO goes to 100. But I got always sv 411 != cam 507. We never tried the other direction with isoinc (from higher to lower).
http://youtu.be/mKzMyddhn4c (http://youtu.be/mKzMyddhn4c)
-
Here are the runs with SX230 on full zoom.
First run looks OK.
On second run exposure decries very late. May be max_ev_change was to low?
The other runs have not focused…..
http://youtu.be/ASvK54BTY6o (http://youtu.be/ASvK54BTY6o)
-
It was a little bit frustrating. First run with G1x. Much overexposure. I don’t know why. I set overexposure to 0.02% (or only 0.2%?)
over_thresh_frac=200 => 200/1,000,000 => 0.02%
(the menu value is in parts per 100k, but the actual value is per million)
OK, script starts with ISO800. I know the problems with G1x, but then ISO goes to 100. But I got always sv 411 != cam 507. We never tried the other direction with isoinc (from higher to lower).
I would say any use of ISO over 320 is suspect on this camera, not just the transition from one range to another other. However, I don't think this is the main cause of your problems.
Looking at your log, over_weight was always 200, but meter weight also reaches 200, so it balances.
This is controlled by the following settings:
"Meter low thresh -Ev" meter_low_thresh = -168 = 1.75 ev below neutral
"Meter low limit -Ev" meter_low_limit = -264 => 2.75 ev below neutral
"Meter low max weight" meter_low_limit_weight = 200, meaning weight increases from 100 at "Meter low thresh" up to 200 when "Meter low limit" is reached.
I'd suggest setting "Meter low max weight" to 100, so there is no increase in meter weight. You may additionally want to set "Overexp prio" to something greater than 0, to further reduce influence of meter when over exposure is in effect.
edit:
For the sx230 runs, I'm not clear which section of the log matches each run, but the meter weight setting suggestion still applies.
-
For the sx230 runs, I'm not clear which section of the log matches each run, but the meter weight setting suggestion still applies.
I updated the log file for SX230, which includes now only the 4 runs from video.
I had some funny thinks, when I manual focus. On 2 runs without changing script parameters, the moon was totally overexposed. I don’t have the log for this runs anymore.
I see also on the Display
SD: 65486
Adj:Off
But only on manual focus. On auto focus not.
-
I see also on the Display
SD: 65486
Adj:Off
But only on manual focus. On auto focus not.
This is normal CHDK behavior.
Here's a moon rise taken with sx160
Settings:
Zoom 100mm (35 mm equivalent)
Focus manual at infinity
Interval 3 sec
Meter width and height 90%
Meter step 13
Bv Ev shift % 30
Bv Ev shift base Bv 9
Max Tv 1000 (1 second)
Target ISO 100
ISO adj TV 250 (1/4 sec)
Max ISO 400
Meter low thresh -Ev 5
Meter low limit -Ev 6
Meter low max weight 100 (no meter weight increase for low, but thresh and limit are still used for Bv Ev shift)
Overexp thresh 5
Overexp Ev range 1/2
Oeverexp prio 10
Underexp thresh 0 (disabled)
Histogram step 9
I think this worked pretty well technically, but even with the wisps of cloud it's pretty boring when the moon is visible. In the log it looks like there is some oscillation, but it's not really visible in the video.
It stays dark and doesn't quite hit the ISO limit after the moon leaves the frame due to Bv Ev shift. I should have either set the base value lower, set the meter low limits higher, or not used it at all.
-
I think this worked pretty well technically
Yes, I can only agree. My first two races are also very good.
but even with the wisps of cloud it's pretty boring when the moon is visible.
Yes of course. But as you can see, this is quite well so already quite complex.
I wanted to make just a couple of similar runs and only change exp_over_thresh_frac to see the effects.
Yes of course, the more interesting races are already, if clouds are in play or the moon goes through trees or buildings.
n the log it looks like there is some oscillation, but it's not really visible in the video.
The moon in your video is very small .May be you would see it when the cutting would be greater.
It stays dark and doesn't quite hit the ISO limit after the moon leaves the frame
The same in my first run, but very close...
I should have either set the base value lower, set the meter low limits higher, or not used it at all.
But it can be seen but the complexity of the whole.
Changes or one forgets only one parameter, the result is quite different.
Especially if you come from one day run into a night run, a lot of parameters have to be changed.
Also I'm not sure whether the change of a particular parameter always conceded the desired effect .
This really is very complex ...
Do you have any idea why run 3 and 4 have not properly focused ?
The log file is in all 4 races :
sd:-1 af_ok:true
So I have the runs focused :
The start I have the moon set in the center of the image.
After the start of the script, I then slightly shifted the camera.
The problem is, that can be seen only at the end of the run, if the moon was really sharp.
With manual focus I have the impression, that infinite is not really infinity at this focal length from 400mm.
(the menu value is in parts per 100k, but the actual value is per million)
That makes understanding not easier... ;)
Update:
I 'm not sure if the overexposure must be weighted. Why not just stop or reduce the exposure, when overexposure exceed?
What would happen in the following example with 1% overexposure?
Putting a hysteresis window around overexposure maybe 20%. So the window is 0.8% to 1.2%
Increasing exposure (depending on meter window) until limits or as long overexposure < 0.8%
Between 0.8% - 1.2% no changes in exposure at all to avoid oscillation.
If overexposure > 1.2% only decreasing exposure.
Maybe that's too easy. But you can estimate fairly accurately what happened.
So I did it in the past with my script, but with the old histogram with 1 % resolution, it did not work really well ...
I have already tried to integrate the new histogram in my old script , but that was a bit too complex for me . ;)
Why I play at all with my old script ?
My script is very simple. I change only the exposure time. The exposure time I calculate out from camera values by smoothing. I use Canon ISO settings. I look like that’s this works on G1x with higher ISO values than 320 (But fixed ISO :( ).
-
But it can be seen but the complexity of the whole.
Changes or one forgets only one parameter, the result is quite different.
Especially if you come from one day run into a night run, a lot of parameters have to be changed.
Also I'm not sure whether the change of a particular parameter always conceded the desired effect .
This really is very complex ...
Yes, the script has too many options. Eventually I'd like to simplify it, but I don't know in advance what will give the best results in different situations, so I have lots of options to try things out. As I've said before, the script was mostly written to prove the new features in the C code.
In the CHDK script menu, you can save parameter sets, so you could save one for moon rises, another for daytime, and so on.
Do you have any idea why run 3 and 4 have not properly focused ?
No ???
On a camera that has manual focus, if you set it using the Canon UI before running the script, it should stay at whatever you set.
So I have the runs focused :
The start I have the moon set in the center of the image.
After the start of the script, I then slightly shifted the camera.
Are you using Canon AF? If so, I suggest you don't do that. Use MF, take a test shot and look at it in review mode if you need to. On my cameras, I find just using the "focus at infinity" mode or selecting MF and setting the maximum distance works fine.
edit:
I should note that the Canon "focus at infinity" (mountain icon on the cameras that have it) mode is only treated as a suggestion by the Canon firmware. I haven't had trouble with it for timelapse, but in other situations I've seen it fail.
I 'm not sure if the overexposure must be weighted. Why not just stop or reduce the exposure, when overexposure exceed?
There are many possible algorithms, but I don't have time to code them all. The current algorithm was intended to try to keep the overall scene reasonable, rather than strictly limiting over exposure.
The rawopint script is modular, so it should be relatively easy to replace the exposure calculation. You should only need to replace exp:calc_ev_change() and set the desired exposure change in self.ev_change. You may have to set a few other values to keep the debug drawing and logging code happy (self.ev_target_base, self.ev_target...) but they can just be dummy values.
-
Do you have any idea why run 3 and 4 have not properly focused ?
No ???
On a camera that has manual focus, if you set it using the Canon UI before running the script, it should stay at whatever you set.
There's a firmware feature that can alter focus in MF mode: "Safety MF". It should be set to "off".
-
As I've said before, the script was mostly written to prove the new features in the C code.
Yes, of course and I’m really happy to use it….
In the CHDK script menu, you can save parameter sets, so you could save one for moon rises, another for daytime, and so on.
Yes, I did this before. Unfortunately, the parameter sets have only number and no name as sunset , moon rise and so on.
Where I find information, what happens when a parameter name changes or when a new parameter comes in?
Are you using Canon AF?
Yes, all 4 runs one the SX230 I made with Canon AF. That is why I was wondering…
I suggest you don't do that
After the runs I tried to play with manual focus. That brought then also unsuccessful. I ended up being quite as tired. I want to try it with MF again ...
but I don't have time to code them all
Yes of course, I understand this.
You should only need to replace exp:calc_ev_change()
Well, I will try it….
Don't forget what the H stands for.
In rawopint not for hacking… I have reviewed a lot off code in my live and rawopint is definitely no hacking (may be the C code behind ;) )
@srsa_4c
"Safety MF". It should be set to "off".
It is always off on all my cams….
-
Another moon test run
Camera elph130
Zoom 224mm (35 mm equivalent)
Focus Focus at infinity mode.
Interval 4 sec
Meter width and height 90%
Meter step 13
Bv Ev shift: off
Max Tv 2500 (2.5 second)
Target ISO 100
ISO adj TV 250 (1/4 sec)
Max ISO 400
Meter low max weight 100 (no meter weight increase for low)
Overexp thresh 5
Overexp Ev range 1/2
Oeverexp prio 10
Underexp thresh 0 (disabled)
Histogram step 7
This is similar to the last run, except Bv Ev is disabled and the zoom is higher. Due to some mistakes on my part, it doesn't include the moon entering or leaving the scene, so the bv ev doesn't actually have any impact. Some oscillation is visible.
edit:
OK, script starts with ISO800.
This is probably a quirk in CHDK, which you can see in my log too.
The first line in the log (after the column headers) is from half press, before the actual shot. When CHDK overrides ISO (from script or the menu) it actually sets the Canon firmware to auto ISO. It doesn't use Canon auto ISO, it just needs to be in this mode to make the override work. However, once you stop using the override, the Canon firmware is still in auto ISO mode, even though it might look like it is set to something else in the UI. You can reset this by changing the ISO in the Canon UI.
This means if you run rawopint more than once, without rebooting or touching the Canon ISO settings, the runs after the first one will start with the Canon auto ISO values. This shouldn't normally be a problem, because the values will set before the actual shot, but it might cause problems on g1x where high ISO has other impacts.
-
This means if you run rawopint more than once, without rebooting or touching the Canon ISO settings, the runs after the first one will start with the Canon auto ISO values. This shouldn't normally be a problem, because the values will set before the actual shot, but it might cause problems on g1x where high ISO has other impacts.
Interesting discovery. This can be very well seen in the log file of G1x. I had 2 test starts. The first starts at ISO100. The other with ISO1600. So when I use the G1x, I have to remember ...
Update:
Just for understanding, if I have understood the code correct:
I made it very simple, just for understanding. The hysteresis window is not implemented…
What I want to do:
1) Ignore underexposure at all
2) On overexposure take only care about overexposure
3) No changes in exposure when meter96 is close to 0 and no overexposure.
Your code
ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)
-- clamp to ui max
-- TODO everything above should already be in limits
ev_change = self:clamp(ev_change,self.ev_change_max)
-- smooth out rapid changes
-- TODO smoothed fraction should be configurable
if self.smooth then
local ev_change_smooth = ev_change + last_ev_change
ev_change_smooth = ev_change_smooth/2 + ev_change_smooth%2
-- but don't allow overshoot in the wrong direction
if (ev_change == 0 and ev_change_smooth ~= 0)
or (ev_change_smooth > 0 and ev_change < 0)
or (ev_change_smooth < 0 and ev_change > 0) then
logdesc('smooth overshoot:%d',ev_change_smooth)
ev_change = 0
else
ev_change = ev_change_smooth
end
-- TODO hacky hardcoded
-- sign changed, reduce by half (and trunc to 0 if 1)
if ev_change * last_ev_change < 0 then
logdesc('smooth sign switch')
ev_change = ev_change/2
end
end
self.ev_change = ev_change
My changes:
if self.over_frac > self.over_thresh_frac then
-- on overexposure take only care about the overexposure
ev_change = ( - self.ev_change_max)
else
-- avoid oscillation around meter96 = 0
if (self.mval96 > -8 and self.mval96 < 8)
-- no changes when meter96 is close to 0 and no overexposure
ev_change = 0
end
end
-- clamp to ui max
-- TODO everything above should already be in limits
ev_change = self:clamp(ev_change,self.ev_change_max)
self.ev_change = ev_change
-
Interesting discovery.
Sadly not a discovery, but remembering a very old CHDK bug :(
Just for understanding, if I have understood the code correct:
I made it very simple, just for understanding. The hysteresis window is not implemented…
What I want to do:
1) Ignore underexposure at all
2) On overexposure take only care about overexposure
3) No changes in exposure when meter96 is close to 0 and no overexposure.
Without actually testing, you code seems like it would do what you described.
A couple of notes
ev_change = ( - self.ev_change_max)
the current code scales the weight (and so, the eventual effect on ev) depending how much over exposure there is. As discussed before (http://chdk.setepontos.com/index.php?topic=11081.msg123350#msg123350 (http://chdk.setepontos.com/index.php?topic=11081.msg123350#msg123350)), you can't know how much change is required, but if you are just slightly over thresh, you probably don't want a full ev_change_max. Maybe your hysteresis will take care of this.
My plan to deal with this is to make the algorithm know something about how much the previous change affected overexposure, but I'm not sure when I will get to it, and obviously it will make the algorithm even more complex.
if (self.mval96 > -8 and self.mval96 < 8)
-- no changes when meter96 is close to 0 and no overexposure
ev_change = 0
end
For meter, the existing code calculates the exact change require to get correct exposure (so if mval96 == 9, it would give a change of -9 before smoothing) If you keep that logic, there will be a sudden jump when mval96 leaves the window. Also some settings (ev shift, bv ev) change the target exposure, so they won't work if you use mval96 directly like this.
FWIW, I originally started with lots of ifs like this. The reason I switched to the current system with weights ramping smoothly up and down is that I always had corner cases where there would be jumps or oscillation switching from one state to another. I'm not saying it can't be done, I just personally found it kept getting more complicated and harder to test.
-
the current code scales the weight (and so, the eventual effect on ev) depending how much over exposure there is.
Yes, of course. Over exposure changes has to be weighted. This was only to simply the example for understanding.
For meter, the existing code calculates the exact change require to get correct exposure (so if mval96 == 9, it would give a change of -9 before smoothing) If you keep that logic, there will be a sudden jump when mval96 leaves the window.
This was also just to simplify. I want to try a dumping curve (see plot) for ev_changes . So maybe I have not check mval96 I can just but ev_changes in translation.
settings (ev shift, bv ev) change the target exposure, so they won't work if you use mval96 directly like this.
Yes, of course. On the first run, I want only to try how my stuff works.
So an –1EV shift gives you mval96 == -96?
I originally started with lots of ifs like this.
May be it get even worse, but then I have learned a little bit more about your code…
Update:
This is the code, which i want to try:
---------------------------------------------------------------------------------------------------
-- Window is 5% of over_thresh_frac
local win_over_thresh_frac = self.over_thresh_frac / 20
if win_over_thresh_frac < 1 then
win_over_thresh_frac = 1
end
if self.over_frac > self.over_thresh_frac then
-- on over exposure take only care about the overexposure
-- for every 5% over exposure one step
local ev_change_factor = (self.over_frac - self.over_thresh_frac) / win_over_thresh_frac
if ev_change_factor < 1 then
ev_change_factor = 1
end
ev_change = ( - ev_change_factor)
logdesc("Overexposure ev_change",ev_change)
elseif self.over_frac > (self.over_thresh_frac - 2 * win_over_thresh_frac) then
-- only negative ev_change when close to overexposure
if ev_change > 0 then
ev_change = 0
end
logdesc("Only negative ev_change",ev_change)
else
--if (self.mval96 >= -10 and self.mval96 <= 10) then
-- no changes when meter96 is close to 0 and no overexposure
-- dumping ev_changes to avoid oscillation ev_change = (ev_change^3)/100
local ev_change_mod = ev_change
if (ev_change >= -10 and ev_change <= 10) then
ev_change_mod = (ev_change * ev_change * ev_change)/100
end
logdesc("Change ev ",ev_change," to ",ev_change_mod)
ev_change = ev_change_mod
end
-- clamp to ui max
-- TODO everything above should already be in limits
ev_change = self:clamp(ev_change,self.ev_change_max)
self.ev_change = ev_change
-
Yes, of course. On the first run, I want only to try how my stuff works.
So an –1EV shift gives you mval96 == -96?
Correct. The mval96 is in (approximately) the same units as the APEX*96 units used by the canon firmware for settings, with 0 set at the "neutral" value.
-
I originally started with lots of ifs like this.
According to some experiences, I now understand what you mean. The adaptation of overexposure I first abandoned.
I then concentrated only on the oscillation at constant lighting condition.
When I run rawopint with constant lighting condition (same like isoinc), it was pretty easy to generate oscillation. How do you run isoinc.lua? Also like you describe for drtest.lua against a monitor?
First I analyzed the smooth stuff in rawopint and I can say, the smooth stuff did already a good job. Without smooth it gets even worse.
Then I added only the following code on the end of the smooth stuff. I tried other functions like ev_change^2 but this was even harder. This code only reduces changes. So ev_change 3 goes to 2, 2 to 1 and 1 to 0.
if (ev_change >= -3 and ev_change <= -1) then
ev_change_mod = ev_change + 1
end
if (ev_change >= 1 and ev_change <= 3) then
ev_change_mod = ev_change - 1
end
Yes of course, this helps for oscillation at constant lighting condition.
When you see the ‘rawopint_compare3’ plot, you see 100 shots with my code changes and 100 with your original code on constant lighting condition. So you don’t see oscillation produced by the code anymore. Yes meter96 stays I little bit more far away from 0, but this should be Ok.
Then I run the changes under real condition.
http://youtu.be/T5YagfYfPA0 (http://youtu.be/T5YagfYfPA0)
The plot ‘JCrawopint_test3’ shows you the result. Meter96 is now above or under 0. There is no up and down more in d_ev but you can still see parts of oscillation. Around picture 420 you see even oscillation, when rawopint made no changes. But this might be come from the scene. I think the changes are not really a improvement.
I saw in your comments, to make smoothed fraction be configurable. This might be a good idea.
My understanding about meter and meter96 was, that meter has a high resolution. So it may be helpful, not to work with a meter96 value, and to work with a meter96*100 internal. Also to work with a smoothed fraction it would be helpful, to work with an ev_change *100 value internal, and only change this value when it is set to the cam.
At the end of the day I definitely now much more understood by the algorithm in rawopint...
-
When I run rawopint with constant lighting condition (same like isoinc), it was pretty easy to generate oscillation. How do you run isoinc.lua? Also like you describe for drtest.lua against a monitor?
Yes. Note an LCD monitor can have some flicker, but under normal conditions I think exposure will be long enough that it doesn't matter.
I don't think there should be oscillation in rawopint in constant lighting if over or under protection isn't active, unless shutter speed is high enough to run into precision issues. However, I haven't actually tested like that, so maybe there is.
edit: You should be able to tell if it's real change or the algorithm by looking at the meter value. Of course, noise an thermal effects will cause some changes in sensor values even if the actual scene isn't changing.
edit 2: of course, meter will change *after* the exposure changes. Bv should tell you the real scene brightness, assuming there is no clipping and exposure controls are accurate.
I saw in your comments, to make smoothed fraction be configurable. This might be a good idea.
Now you understand why the script has so many options ;)
There are probably some more sophisticated smoothing algorithms that could be used too.
My understanding about meter and meter96 was, that meter has a high resolution. So it may be helpful, not to work with a meter96 value, and to work with a meter96*100 internal. Also to work with a smoothed fraction it would be helpful, to work with an ev_change *100 value internal, and only change this value when it is set to the cam.
Meter is the raw sensor count, meter96 is converted to log scale like APEX values. Working with a higher precision version of meter96 an interesting idea. A simple way to do this might be to use imath http://chdk.wikia.com/wiki/Lua/Lua_Reference#Mathematical_functions_.28imath_library.29 (http://chdk.wikia.com/wiki/Lua/Lua_Reference#Mathematical_functions_.28imath_library.29) (but you have to be careful of overflow)
The C code would need a function to return raw_to_ev96 as a scaled value, which is probably a good idea anyway.
Another note, in theory we can set the shutter speed directly in microseconds. The actual precision of the hardware is unknown (a shutter testing script like isoinc would be useful), but it should be possible to adjust exposure in steps smaller than 1/96th ev for some of the range. This isn't currently possible through script though.
-
In trunk 4241, I changed the rawop functions raw_to_ev96 and ev96_to_raw. They are now called raw_to_ev and ev_to_raw, and take an optional scale factor as the second argument. The default scale factor is 96, giving the same results as the previous functions.
Other scale factors that might make sense are 1000, if you wanted to work in standard APEX using imath, or 96000, if you want to work in APEX*96 using imath.
By way of example, drtest now reports the meter96 values to 3 decimal places.
I temporarily added the ev96 names in rawoplib.lua, so the current version of rawopint etc will continue to work. I plan to remove these before 1.4 is released.
I also noticed that the double/int conversions in the current code truncate rather than rounding. I will probably change this.
edit:
Done in 4243. I don't think this will have a big effect, but the meter96 values should follow the meter values a bit more accurately. Note the the meter function itself (taking the average raw values) is all integer, so it still truncates.
-
I started a documentation page for the rawop stuff on the wiki http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations (http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations)
This is mostly based on comments in the rawhookop.c source, so nothing really new.
-
I started a documentation page for the rawop stuff on the wiki http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations (http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations)
This is mostly based on comments in the rawhookop.c source, so nothing really new.
Nice.
For reference, the actual hooks were there in 1.3.0 though? The additional functionality is related to manipulating the raw data once you hit the callback in the hooks?
-
For reference, the actual hooks were there in 1.3.0 though? The additional functionality is related to manipulating the raw data once you hit the callback in the hooks?
Right "Script Shooting Hooks" were added in 1.3, and have their own page http://chdk.wikia.com/wiki/Script_Shooting_Hooks (http://chdk.wikia.com/wiki/Script_Shooting_Hooks)
The new page only describes "rawop", the system for interacting with the raw buffer while you are in the raw hook.
I'll add a link from the raw hook section of the shooting hooks page to the rawop page after I have the rest of the the content up. I put rawop on it's own page, because it's distinct from the general hook system, and pretty long all by itself.
-
I have a mismatch between the log file and what I have calculated from the pictures on over fraction.
The data from log file shows me a jump in over fraction between line 635 and 636 from 0.1013 to 0.984. When I calculated over fraction from the picture, I don’t have the jump. Yes of course, my threshold is different so the curves have an offset. But I don’t find the jump.
For me it looks like that the correct value on Line 636 is not 0.984 rather 0.0984. So it might be an error of conversion or may be my tooling.
Don’t hurry about this…
-
c_joerg: That jump sounds like a bug in the logging code, I'll have a look.
Another moon test with elph130
Settings
Zoom 224mm (35 mm equivalent)
Focus at infinity (canon UI)
Interval 2 sec
Meter width and height 90%
Meter step 13
Bv Ev shift % 20
Bv Ev shift base Bv 8
Max Tv 4000 (4 second)
Target ISO 100
ISO adj TV 250 (1/4 sec)
Max ISO 400
Meter low thresh -Ev 2 1/2
Meter low limit -Ev 3 3/4
Meter low max weight 100 (no meter weight increase for low, but thresh and limit are still used for Bv Ev shift)
Overexp thresh 5
Overexp Ev range 1/2
Oeverexp prio 5
Underexp thresh 0 (disabled)
Histogram step 9
It gets dark suddenly at the end because of the streetlight triggering overexposure. Oops :haha
Now I have a reason to make the histogram area adjustable.
Aside from that and the camera being on a slightly unstable surface, I think it worked pretty well. (edit: but the moon left the frame before it was really dark, so it wasn't a really challenging scene)
A few frames at the start and end of the run aren't included in the video since I was moving the camera around.
-
It looks good. I have seen you have some new information in the log file.
Aside from that and the camera being on a slightly unstable surface,
I had a couple of runs which get a Image registration …
I would be helpful, if you could post the saved parameter sets, so I have only to copy it on my SD card to test the same set of parameters.
Tonight we have a full lunar eclipse. Hope the weather clears up. All my cams a prepaid…
-
It looks good. I have seen you have some new information in the log file.
Only the script version, I think.
I would be helpful, if you could post the saved parameter sets, so I have only to copy it on my SD card to test the same set of parameters.
Not a bad idea. I've attached the one from this run. It's for newer version of the script than the last one I posted here, bit I don't think I've changed any of the menu parameters.
Tonight we have a full lunar eclipse. Hope the weather clears up.
Same here, the above was a test run for the eclipse. Good luck.
edit:
Note attached has #ui_image_size_e=3 => M2 image size, which is 1600x1200 on this camera, but may be a different resolution on others. I usually shoot at this size since I render the videos at 720 lines anyway, and it keeps the disk space down.
-
I would be helpful, if you could post the saved parameter sets, so I have only to copy it on my SD card to test the same set of parameters.
Not a bad idea. I've attached the one from this run. It's for newer version of the script than the last one I posted here, bit I don't think I've changed any of the menu parameters.
One of the things on my longer term "todo" list - named parameter sets.
-
Unfortunately the weather was bad at the Lunar Eclipse.
The first video was created with the SX230.
Video 2-4 with G1x.
Video 2 and 3 only have a frame rate of 5 frames per second
One of the things on my longer term "todo" list - named parameter sets.
Would be one of my thinks one the wish List…
http://youtu.be/CmcKxZXxI_U (http://youtu.be/CmcKxZXxI_U)
-
Unfortunately the weather was bad at the Lunar Eclipse.
Same here :(
https://www.youtube.com/watch?v=EcuzbTGSF8s (https://www.youtube.com/watch?v=EcuzbTGSF8s)
rawopint has trouble keeping over exposure under control with clouds like this.
-
rawopint has trouble keeping over exposure under control with clouds like this.
This is a run from last night. There is also an overexposure at beginning, when the moon comes out. I’m not sure if a higher max_ev_change would be helpful.
The video starts from picture 150 in the log file. I’m using only 1920*1080 pixel from the full sensor size.
http://youtu.be/pzHb8i1EtlE (http://youtu.be/pzHb8i1EtlE)
Update:
I analyzed the log file. It’s the time, which the algorithm needs when he comes from the highest ISO and longest Tv time. With ev_change from 8 it takes time until overexposure is gone. I made a new run this night with Tv max = 1s. This goes much faster. Additional I will increase now max ev_change.
-
Updated scripts attached. This is a minor update that doesn't affect script logic. Changes are:
* Fix the over /under frac logging logging pointed out by c_joerg in http://chdk.setepontos.com/index.php?topic=11081.msg124673#msg124673. (http://chdk.setepontos.com/index.php?topic=11081.msg124673#msg124673.) When I updated the histogram to parts per million in the previous version, I forgot to update the format string to 4 digits, so what should have been 1.0nnn would be logged as 1.nnn. This only affected the logged values, not the calculations.
* Update the rawop ev <=> raw calls for trunk r4241
* Log script version in the log
* Add a wait to allow shooting to complete before restoring settings like image size and raw enable.
* saved scripts in LF instead of CRLF format to reduce size
-
I came just back from a trip and I had a strange error with my G1x. I can’t reproduce it anymore so I am not sure, if it is worse to talk about.
On my G1x I had an older version of rawopint and CHDK. With this combination I had a lot of runs at home without any problems.
On the trip, the same combination makes problems. I started rawopint and after a couple of shoots (between 40 -80 shoots), the camera goes off. I made this 3 times, only the number of shoots differs.
Interesting about this runs was, the rawopint log file include always only the first info line, no shoots where logged. The ROMLOG of all this runs says something about ‘System Panic’.
I formatted the SD card and I copied the same versions of rawopint and CHDK on the SD card again. After this, I had no problems anymore…
-
I came just back from a trip and I had a strange error with my G1x. I can’t reproduce it anymore so I am not sure, if it is worse to talk about.
Good to post anyway, in case someone else runs into it later.
Interesting about this runs was, the rawopint log file include always only the first info line, no shoots where logged.
This isn't too surprising, the script writes and flushes the header at start up. After that, it's buffered by the OS, and presumably only written when the buffer is full or the script ends.
The ROMLOG of all this runs says something about ‘System Panic’.
Was this firmware 101a? I seem to remember you had two different ones, and one had some hardware problems?
The romlog doesn't show anything immediately obvious to me. It seems to point to ff45f958 which the sigfinder calls _divmod_signed_int. Unfortunately the panic doesn't tell us what task was involved, and I'm not sure what the module and panic codes mean (figuring them out would probably be useful).
I formatted the SD card and I copied the same versions of rawopint and CHDK on the SD card again. After this, I had no problems anymore…
Card problems are possible, but it seems a little strange.
-
Was this firmware 101a? I seem to remember you had two different ones, and one had some hardware problems?
Yes I have a 101a and a 100e. I’m using only the 101a right now. The 101a had the hardware problems in the past. But since I reset the 101a one, I didn’t have problems anymore.
Card problems are possible, but it seems a little strange.
For me as well. Unfortunately I deleted also all CHDK and rawopint setting. It might be also a combination of setting, which I have not used before…
I've found another reason for using the overexposure stuff in timelapse. White boots in the ocean on sunrise (or sunsets). When I have processed and investigated my runs, I will post it here.
-
Hello reyalp,
I have just a question for understanding.
Which influence has ‘ui_ev_shift_e’ (Ev shift) in case of a scene which is controlled by overexposure?
I understand what happens on a scene without overexposure. When I set Ev shift to -1/2EV, meter96 goes to -48, which I expect.
But when I have an overexposed scene, may be overexposure is set to 1%, than ‘ui_ev_shift_e’ don’t have an influence or?
After starting of analyzing the Data from my last trip, I’m thinking, I it would be helpful, to have two parameters for ‘ui_max_ev_change_e’. One for up and one for down. So on a overexposed scene, the algorithm could go faster down and slower up.
Greats
-
Which influence has ‘ui_ev_shift_e’ (Ev shift) in case of a scene which is controlled by overexposure?
I understand what happens on a scene without overexposure. When I set Ev shift to -1/2EV, meter96 goes to -48, which I expect.
But when I have an overexposed scene, may be overexposure is set to 1%, than ‘ui_ev_shift_e’ don’t have an influence or?
The ev shift shouldn't change the behavior, it just changes the target meter96 value.
The meter high and meter low thresh/limit values are relative to the shifted value, so if you have those enabled, ev shift will affect at what exposure level the meter and over/under balance.
After starting of analyzing the Data from my last trip, I’m thinking, I it would be helpful, to have two parameters for ‘ui_max_ev_change_e’. One for up and one for down. So on a overexposed scene, the algorithm could go faster down and slower up.
This would affect both meter and over exposure related changes, but meter would only be affected when scene change is > the max value, so maybe that's not a big deal. It might be better to just set the limits for over, under and meter separately.
-
Hello reyalp,
Thanks for explanation.
There is another interesting stuff on the data on my last trip from G1x.
Most runs with G1x I made with 3s Interval and JPG + Canon RAW. On most runs, this is not a problem along the exposure time is short enough. I never had problems before. As you see on the run with timing OK, I had still more than 1s time.
But I have also two runs, where I had massive problems. Sometimes sleep is greater -4000ms and the Interval go near 8s. Only a few shoots have 3s Interval. Max exposure time is less than 0.1s.
All runs where made with same SD Card with 64GByte and Class 10.
I don’t have logs off combinations, so where it runs for a while OK and then it made problems.
Yesterday I tested G1x with this SD Card again. No problems. Sleep always greater 1s. Write test in the Camera with CHDK says always > 14Mbytes/s. I checked Card on a PC for 30 Minutes. Writing was always greater > 14Mbytes/s.
I removed the Card after each run and copied data to PC. Can it be a connection problem with the SD on the Cam?
Update:
The following short video shows, how it looks with a non-continuous interval
http://youtu.be/tUShxzAq0tA (http://youtu.be/tUShxzAq0tA)
-
After more than 20000 shoots from my last trip with rawopint,
here are a few suggestions for the adaptation of the default parameters:
#ui_meter_width_pct=90 "Meter width %" [1 100]
#ui_meter_height_pct=90 "Meter height %" [1 100]
I think on most scenes, this will give you a better result as the smaller spot window from 30%, especially if point to the wrong spot.
#ui_log_append=true "Append log"
It is always better, to delete parts in the log as to have no log for the run…
#ui_do_draw=false "Draw debug info"
Yes, I know, we talked about it before… but especially beginners get nervous with it…
Yes, I know, I can save parameter files, but sometime, when I changed too much, I go back to defaults…
Is it possible, to put comments in the parameter set?
Sometimes I copy them on a PC and modify them for different runs.
Like this parameter
ui_max_ev_change_e=3 "Max Ev change" {1/16 1/8 1/4 1/3 1/2 1}
It would be helpful, to see what is possible to set as a comment.
-
Hello reyalp,
it's time to say thank you once :)
I read from your script at the German CHDK forum in which I am also too rarely :-(
Great. I've done some tests, unfortunately, do not understand even half of the parameters but I'm still impressed :-)
Very cool script. thx :xmas
https://www.youtube.com/watch?v=dA3WJxi5EO4 (https://www.youtube.com/watch?v=dA3WJxi5EO4)
h3px
-
I run rawopint with the SX50. Normally I stop the script by pressing the Menu bottom. On all my other cams this works without any problems. On the SX50 I notice the following:
1) After pressing the Menu bottom, the cam shows no reaction for about 10-15s.
2) The cam makes on more picture.
3) Than another 10-15s without reaction after the script finished.
This only happens in continues mode. On Quick mode it works fine.
This is now also on my SX230 in continues mode with R4293.
I don’t notice that on my last trip with R4203.
-
Moved from release thread
I run rawopint with the SX50. Normally I stop the script by pressing the Menu bottom. On all my other cams this works without any problems. On the SX50 I notice the following:
1) After pressing the Menu bottom, the cam shows no reaction for about 10-15s.
2) The cam makes on more picture.
3) Than another 10-15s without reaction after the script finished.
This only happens in continues mode. On Quick mode it works fine.
This is now also on my SX230 in continues mode with R4293.
I don’t notice that on my last trip with R4203.
I suspect this is a bug I introduced in the latest version of the script (0.18). I haven't been able to reproduce it on my cameras, but it wouldn't be surprising the there was some variation due to timing and hook placement.
Please try adding
hook_shoot.set(0)
hook_raw.set(0)
just above "if yield_save_count then" around line 1700.
The camera may still take one additional shot after you press menu, but the delay should go away.
-
hook_shoot.set(0)
hook_raw.set(0)
Yes, thanks, that’s it. It works as you say…
In the time before, I find out that this part of the end of the script made the delay.
When I delete it, the script works as well.
-- allow final shot to end before restore + possible shutdown
repeat sleep(10) until not get_shooting()
sleep(1000)
-
-- allow final shot to end before restore + possible shutdown
repeat sleep(10) until not get_shooting()
sleep(1000)
Right. The reason I added this code is because the restore() at the and will reset some settings like raw enable, image size etc to what they were before the script started. In the old version of the script, this reset could happen before the final shot was finished, and so would use the restored settings.
The extra delay was caused by the hooks still being set, so if the shooting process reached a hook after the menu click was detected (probably timing and port specific) get_shooting() wouldn't go false until the hook timed out.
-
I made more analyzing of the data from my last trip.
I found one more run from G1x with massive timing problems. I checked the SD Card again. I had no problems with the card on an interval from 2s at home (JPG + Canon RAW). On this run now, the interval was set to 3s but sometime it goes close to 6s. This can be really good seen in the not continuous moving of the ship. It might be helpful to have a warning (LED or focus light on or a noise?), when sleep goes to negative values.
Another think of this run is that it has much more flicker as the other runs. I’m really sure that comes from the waves in front of the video. The meter window was set to 90%/90% so the waves are full in. In cases like this, it would be not good to use the full picture for measure.
I’m thinking loud:
I’m not sure if an auto window size could be helpful for most situations.
Instead the meter_width and meter_height only a parameter like:
Spot_Window (30%/30%), Full_Window (90%/9%), Sky, Ground
How does it work to detect the Sky or Ground?
1) Using the pre shoot and dividing the full picture into a number of equal rows (may be 12).
2) Calculating a meter value of each row.
3) Calculating a mean (or median) value from all meter values.
4) Looking, which rows are above or under the mean value.
5) Combining all rows which are above or under the mean value to one widow for the following shoots.
http://youtu.be/RWvEtKy2qY4 (http://youtu.be/RWvEtKy2qY4)
-
I found one more run from G1x with massive timing problems. I checked the SD Card again. I had no problems with the card on an interval from 2s at home (JPG + Canon RAW). On this run now, the interval was set to 3s but sometime it goes close to 6s.
Unfortunately, without being able to reliably reproduce this, I'm not sure there's much I can do. It's still good to post, the data may become valuable later.
There are some general scenarios that come to mind:
1) As discussed before SD cards firmware is optimized for specific usage patterns, which is probably quite different from what happens under the script. Even though a card can do 14mb/s sustained writes, it's possible the actual usage pattern triggers some pathological behavior in the SD card.
2) The whole "shooting hook" principle used by the script relies on blocking the normal firmware execution deep in the shooting process, in ways the firmware designers would not expect. It seems to work, but weird side effects would not be a surprise. I suspect the shutter issues I see on D10 are an example of this, but I'm still not certain.
Some ideas of things that might have some impact
1) The interval. Maybe if the interval is short the Canon or SD card firmware sometimes gets "behind" and can never catch up.
2) Continuous mode on / off.
3) CHDK raw vs Canon raw vs jpeg only.
4) Log file writing. This is tricky, because if you disabled the log completely, it would be difficult to measure the real behavior. The log could be done entirely to RAM and written at the end, but this would require a lot of RAM for long runs. It would be possible to come up with more compact log format that just had the timing information.
5) The raw hook sleep / disable script yield settings.
It might also be possible narrow down where the delays occur, but this would probably require adding code to the CHDK core.
This can be really good seen in the not continuous moving of the ship. It might be helpful to have a warning (LED or focus light on or a noise?), when sleep goes to negative values.
This is a good idea, I'll look into to adding it for the next version.
Another think of this run is that it has much more flicker as the other runs. I’m really sure that comes from the waves in front of the video. The meter window was set to 90%/90% so the waves are full in. In cases like this, it would be not good to use the full picture for measure.
I’m thinking loud:
I’m not sure if an auto window size could be helpful for most situations.
Instead the meter_width and meter_height only a parameter like:
Spot_Window (30%/30%), Full_Window (90%/9%), Sky, Ground
How does it work to detect the Sky or Ground?
1) Using the pre shoot and dividing the full picture into a number of equal rows (may be 12).
2) Calculating a meter value of each row.
3) Calculating a mean (or median) value from all meter values.
4) Looking, which rows are above or under the mean value.
5) Combining all rows which are above or under the mean value to one widow for the following shoots.
I have intended to make the meter position adjustable, and possible add the ability to have multiple meter areas with different priorities. However, this makes both the configuration and final behavior even more complex. One idea I've toyed with is having a scripted GUI for configuration, but this would be a lot of work.
Now that 1.4 is released, I hope to spend some more time on the script.
-
Since the raw hook features are in the stable release now, I've created separate threads for rawopint (http://chdk.setepontos.com/index.php?topic=12697.0) and contae (http://chdk.setepontos.com/index.php?topic=12696.0). Further discussion of the scripts should go in their respective threads.