rawopint.lua: Fast, accurate intervalometer with raw exposure metering - page 18 - Completed and Working Scripts - CHDK Forum
« previous next »
Pages: 1 ... 13 14 15 16 17 [18] 19   Go Down

rawopint.lua: Fast, accurate intervalometer with raw exposure metering

  • 182 Replies
  • 51413 Views
*

Offline reyalp

  • ******
  • 13625
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #170 on: 10 / July / 2021, 14:36:33 »
Advertisements
Thanks for posting this, it seems like a good idea. I'll take a closer look later, but just one comment for now:
Quote from: dolomiti_timelapse on 10 / July / 2021, 14:14:51
After I wrote this lines of code I asked my self why I did not used a float or double. I actually was not sure if there is a performance or memory issue using doubles since I could only find Integers in the code.
CHDK Lua numbers are integers. To use real numbers, you need imath fixed point (https://chdk.fandom.com/wiki/Lua/Lua_Reference#Mathematical_functions_.28imath_library.29 ) or fmath  (https://chdk.setepontos.com/index.php?topic=14305.msg145763#msg145763 added in CHDK 1.6 )

Performance for mathematical operations with regular integers or either of the above shouldn't be a concern unless you're doing hundreds or thousands of calculations in tight loop (see https://chdk.setepontos.com/index.php?topic=14305.30 and post linked from there)
Logged
Don't forget what the H stands for.
*

Offline dolomiti_timelapse

  • **
  • 92
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #171 on: 11 / July / 2021, 18:30:32 »
The filter parameter is now an input parameter so I can control it with CHDKPTP remotely.
Also I added the raw d_ev and the filtered d_ev to the log file.
The Plot is attached. (with ND filter kicking in at around 450. frame, it works flowlessly! before I checkt the plot I was not able to find the moment where it drops in, very pleasant :D)
I think the filtering effect is quite visible, but maybe not necessary, since the camera changes the exposure only every 1/3 stop...(?)
I will try the next days to increase the value from 0 to 9 and then upload the plots here.
« Last Edit: 11 / July / 2021, 18:33:32 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline reyalp

  • ******
  • 13625
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #172 on: 11 / July / 2021, 19:10:08 »
Quote from: dolomiti_timelapse on 10 / July / 2021, 13:07:47
I want to share my experience and some code changes I did to rawopint.lua in this and the next post.
Thanks for doing this and sharing it.

Quote
For a better understanding of the settings/parameters ui_bv_ev_shift_pct, ui_bv_ev_shift_base_e, ui_ev_shift_e I did a bit of reverse engineering by gradually changing one parameter and then recording a sunset and so forth.
As I mentioned earlier ui_ev_shift_e is a bit broken. This could be a confounding factor trying to understand the other values.
Quote
I attached a pdf with the plots of the ui_bv_ev_shift_pct changes from 2% to 80%, which I think could be also interesting to others for the understanding of this functionality.
For my kind of timelapse a value of about 16-24% is best depending on the ui_bv_ev_shift_base_e and ui_ev_shift_e parameters. At the beginning it was quite strange that high values would overexposed the image in daylight. But in the end it makes sense since it lowers the exposure when it's dark and rises the exposure when it is higher than the "neutral" exposure. The ui_bv_ev_shift_base_e did basically the same as the ui_bv_ev_shift_pct.
bv_ev_shift_base and bv_ev_shift_pct shouldn't be equivalent in the general case. base should define the Bv value (derived from meter and exposure) where the effect of bv_ev_shift is 0, regardless of the bv_ev_shift_pct value. Bv is an absolute measure of the amount of light, where a scene in full daylight would typically be around 10 APEX.

Looking at your plots, I guess bv_ev_shift_base_bv is APEX 4? I would generally set it around the brightest you expect the scene, since you don't want to overexpose much, but meter and overexposure limits should stop it from going too far.

In your examples, bv_ev_shift crosses the origin right around where bv96 crosses 400. When Bv is higher, it pushes exposure brighter, and when it's lower, it pushes it darker, subject to the meter thresh/limit values (and I think a bit confused by the ui_ev_shift values, but it's hard to say without the actual log).

Quote
An other topic was the under_expo_weight, over_exp_weight. As far as I understand for my type of sunset it is best to switch this two parameters off and only relay on the meter exposure, because I think that if a bright cloud comes into the frame the hole frame should not darken.
I would set the "thresh_frac" values to 0 if you want to completely disable under or over protection. This should completely disable all the related calculations.

For sunrise/sunset, I might disable under (or set very large % and low ev limit), but I would probably not disable over. If the scene is half bight sky and half shadowed land, you probably want the sky not to be totally blown out, even when the land is dark.

Quote
What does actually mean a rise of the meter weight from 100 to say 200? I attached a plot where exactly that happens but I could not see any changes in the timelapse video itself.. ???
The weights are arbitrary numbers that determine the relative influence of the meter and over / under exposure, subject to various limits. The key line is
Code: [Select]
ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)
ev_change is the difference between the meter value and the target value, clamped to +/-ev_change_max.
over_weight and under_weight are calculated from the percentage of over and under exposure, such that they reach 100 when "thresh" is reached, and quickly ramp up to 200 after that. You can see if over and under are 0, then ev_change is unaffected. Meter weight starts at 100, and ramps up to 200 between from thresh to limit.

The intent is that the three factors will tend balance in the range defined by the thresh and meter limit values, without any sudden transitions triggered by if/else logic. For example, if overexposure is at the "thresh" value, it will balance with meter the thresh value in the opposite direction, and if that is exceeded, push meter to the limit value without allowing much more overexposure.

The actual change in any given frame is limited to ev_change_max.

I haven't tested much with the weights at non-default values. I'd generally use the thresh / limit and over / under fractions instead.

Quote
The last part I'm not sure is, if whether the ui_meter_width_pct and ui_meter_height_pct are dependent on the selected image format. E.g. if I choose 1:1 or 16:9 image format stays the area that rawopint uses to calculate the exposure always the same?  ???
Anything related to size/position in rawopint refers to the full raw buffer. So if you use a different jpeg aspect ratio than the native sensor resolution, you'll have to account for that yourself.
Logged
Don't forget what the H stands for.
*

Offline dolomiti_timelapse

  • **
  • 92
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #173 on: 11 / July / 2021, 20:04:41 »
Thank you very much for all the information and clarifications of my misunderstandings. It really helps a lot!
Quote from: reyalp on 11 / July / 2021, 19:10:08
bv_ev_shift_base and bv_ev_shift_pct shouldn't be equivalent in the general case. base should define the Bv value (derived from meter and exposure) where the effect of bv_ev_shift is 0, regardless of the bv_ev_shift_pct value. Bv is an absolute measure of the amount of light, where a scene in full daylight would typically be around 10 APEX.

Looking at your plots, I guess bv_ev_shift_base_bv is APEX 4? I would generally set it around the brightest you expect the scene, since you don't want to overexpose much, but meter and overexposure limits should stop it from going too far.

In your examples, bv_ev_shift crosses the origin right around where bv96 crosses 400. When Bv is higher, it pushes exposure brighter, and when it's lower, it pushes it darker, subject to the meter thresh/limit values (and I think a bit confused by the ui_ev_shift values, but it's hard to say without the actual log).

thank you very much! I think this is a very good explanation of how bv works and how it should be setup! You should really add it to the wiki of the rawopint script! The explanation there is not so comprehensively explained.

I actually tried to use the bv value to try to influence the bv_ev_shift in a way that the timelapse get darker later in a sunset (by lowering the bv value as the amount of light decreases over time and so should a lower value activate the ev_shift later. I don't like it, when it gets dark too early in the timelapse.
I have done also a series where I change only the bv_base but the effect is not so prominent as I would like.
I attached the pdf with the bv_base change series.
I think this plot series shows quite well how the bv_ev_shift_base_bv parameter influences the algorithm.
The point where the curve of the ev_shift crosses the zero line shifts depending on how high bv_base is set to be.

Quote
I would set the "thresh_frac" values to 0 if you want to completely disable under or over protection. This should completely disable all the related calculations.
That seems like a good idea!
Quote
The weights are arbitrary numbers that determine the relative influence of the meter and over / under exposure, subject to various limits. The key line is
Code: [Select]
ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)
ev_change is the difference between the meter value and the target value, clamped to +/-ev_change_max.
over_weight and under_weight are calculated from the percentage of over and under exposure, such that they reach 100 when "thresh" is reached, and quickly ramp up to 200 after that. You can see if over and under are 0, then ev_change is unaffected. Meter weight starts at 100, and ramps up to 200 between from thresh to limit.

The intent is that the three factors will tend balance in the range defined by the thresh and meter limit values, without any sudden transitions triggered by if/else logic. For example, if overexposure is at the "thresh" value, it will balance with meter the thresh value in the opposite direction, and if that is exceeded, push meter to the limit value without allowing much more overexposure.

The actual change in any given frame is limited to ev_change_max.

I haven't tested much with the weights at non-default values. I'd generally use the thresh / limit and over / under fractions instead.
This makes sense to me. Thanks.
Quote
Anything related to size/position in rawopint refers to the full raw buffer. So if you use a different jpeg aspect ratio than the native sensor resolution, you'll have to account for that yourself.
Thanks, that is great to know! With this I can adjust the ui_meter_width_pct and ui_meter_height_pct according to the crop of the final video without any doubt ;)
« Last Edit: 11 / July / 2021, 20:08:24 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline dolomiti_timelapse

  • **
  • 92
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #174 on: 24 / July / 2021, 08:57:15 »
Here are the results of the test run I did with rawopint.lua + G1X + exponential smoothing code mod (moving average filter), I presented earlier. All settings can be found in the pdf.

Attached are 2 pdf with some plots. One with 4 runs from 0 (no filter) to 9 (max filter) in steps of 3.
shoot all 4 in the same afternoon.
In the other pdf you can see the sunrise plots with filter value set from 0 to 9.

2 Observations:
Without smoothing the ev change value some flickering in the video appears, most noticeable when light condition are stable and don't change at all.
I would suggest a minimum value of 3-4 to get rid of the "flickering".
Also notice that I used max ev change of 1/8 which kind of prevent flickering form its own.
At 0 the flickering is noticeable but not a problem. At 3 it is not more or less gone.

With max filtering some transient response of the filtered ev change value can be observed, especially in the beginning as rawopint has to lower the overall ev a bit (as I underexpose the image a bit).
But the timelapse after the settle in is perfectly smooth.

I think, I will keep the filtering functionality and choose for my application a value of 4 to 7.

Bottom line:
It works as it should. Edge values to be avoided imho.
« Last Edit: 24 / July / 2021, 09:01:39 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline c_joerg

  • *****
  • 1179
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #175 on: 08 / August / 2021, 11:58:41 »
Yesterday I did an M3 time lapse with rawopint. When heavy rain set in, I unfortunately switched off the camera via Power Off. In the logfile there are now about 130 entries of 8s each missing. I wondered why so many lines were lost.
Logged
M100 100a, M3 101a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline reyalp

  • ******
  • 13625
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #176 on: 08 / August / 2021, 15:12:47 »
Quote from: c_joerg on 08 / August / 2021, 11:58:41
Yesterday I did an M3 time lapse with rawopint. When heavy rain set in, I unfortunately switched off the camera via Power Off. In the logfile there are now about 130 entries of 8s each missing. I wondered why so many lines were lost.
The file is only closed when the script ends, so if the camera is powered off, anything in the OS buffer is lost. There may be additional layers where things can be lost if the file isn't closed.

The log module can write every line if you want (commented line buffer_mode = 'sync'), but that may make the shooting interval less consistent.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 13625
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #177 on: 08 / August / 2021, 15:59:46 »
Quote from: dolomiti_timelapse on 24 / July / 2021, 08:57:15
2 Observations:
Without smoothing the ev change value some flickering in the video appears, most noticeable when light condition are stable and don't change at all.
I would suggest a minimum value of 3-4 to get rid of the "flickering".
So I went to add exponential smoothing, and realized I already implemented it without knowing ;)

The original smoothing algorithm was (ev_change + last_ev_change)/2, or effectively the same as yours with a factor of 0.5.

In the old code, this is followed by additional logic:
* Avoid "overshoot", where the calculated ev_change would land exactly on the target exposure. That is, if meter == target, then ev_change is 0, and including the previous value would make the next value different from the target. In this case, ev_change is left at 0, so the next shot should end up exactly at the target value. However, this isn't necessarily optimal if there's a trend...

* Avoid smoothing pushing the exposure in the wrong direction. That is, if the sign of the calculated ev_change and the smoothed ev_change are opposite, the change is set to 0.

* If the sign of ev_change changes (after smoothing), reduce the change by half.

So my new plan is to make the factor an option and add some control for the additional logic.

This all begs for some kind of controlled, reproducible testing, but I haven't figured out a manageable way to do it. Maybe a video playing on a monitor?
Logged
Don't forget what the H stands for.
*

Offline c_joerg

  • *****
  • 1179
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #178 on: 09 / August / 2021, 01:34:53 »
Quote from: reyalp on 08 / August / 2021, 15:12:47
The file is only closed when the script ends, so if the camera is powered off, anything in the OS buffer is lost.

Do you know the size of the buffer?

Quote from: reyalp on 08 / August / 2021, 15:12:47
The log module can write every line if you want (commented line buffer_mode = 'sync'), but that may make the shooting interval less consistent.

Thanks for the hint.

Quote from: reyalp on 08 / August / 2021, 15:59:46
This all begs for some kind of controlled, reproducible testing, but I haven't figured out a manageable way to do it. Maybe a video playing on a monitor?

At least that's how I do it for simple cases. It works very well with that. But I can't test the oscillation with it.



Logged
M100 100a, M3 101a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline reyalp

  • ******
  • 13625
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #179 on: 09 / August / 2021, 01:50:21 »
Quote from: c_joerg on 09 / August / 2021, 01:34:53
Do you know the size of the buffer?
I think it's usually 32K.

Quote
At least that's how I do it for simple cases. It works very well with that. But I can't test the oscillation with it.
Yeah, could be tricky. What kind of exposure range do you get from that video?
Logged
Don't forget what the H stands for.
Pages: 1 ... 13 14 15 16 17 [18] 19   Go Up
« previous next »
 

Related Topics