rawopint.lua: Fast, accurate intervalometer with raw exposure metering - page 17 - Completed and Working Scripts - CHDK Forum supplierdeeply
« previous next »
Pages: 1 ... 12 13 14 15 16 [17] 18 19 20 21   Go Down

rawopint.lua: Fast, accurate intervalometer with raw exposure metering

  • 207 Replies
  • 73681 Views
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #160 on: 26 / May / 2021, 04:23:22 »
Advertisements
Quote from: reyalp on 25 / May / 2021, 18:56:21
Just to confirm, it works *without* the ui_raw_hook_sleep modification?
Yes, new firmware (g1x-100e-1.6.0-5910-full) and new rawopint v0.25 script copied to the Canon G1X.
And the script works just fine out of the box, without the little plastic cover plate.

Quote
Honestly, seeing people do cool things with it is the best reward for working on this stuff.

Nice to hear that. And nice to see that this little time lapse project also finds favor here.

Quote
Very nice. Have you thought about posting these outside IG?
I actually don't like the facebook company... I don't use Whatsapp or Facebook nor have a private IG account but I found a cool python Instagram Private API (https://github.com/adw0rd/instagrapi) and so I started with IG.
What other platforms would you suggest?

Quote
The moon + clouds definitely gets bright at the end. May be able to offer suggestions if you post the log file.

Yes I will post the log. I think the "Moon flare" comes from the optically bad window the camera has to photograph trough... yeah.. still room for improvements, such as removing the window :lol

Some background info:
The sunrise is in M mode. this is because in that way the script is able to start whit tv ~ 6 seconds
The sunset is in AV mode. this is because the amount of daylight can be very different. The script always start at 0 EV and than goes to the -1.1/4 i have set in the glue script (ui_ev_shift_e=5) which si not much of an Issue. I could just skip the first 5 images, but in the videos its not quite visible.



Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #161 on: 26 / May / 2021, 04:29:00 »
For better understanding, attached the rawopint_rs.lua file and the chdkptp scripts.

sunset:
Code: [Select]
clock -sync
luar set_lcd_display(0) --switch of display during taking photos
luar set_mf(1) --set focus to manual
luar set_focus(1495) --set focus distance
luar set_capture_mode(4) --set to AV mode
luar set_user_av96(320) --set AV value
#AV modes: 3.2  3.5 4.0  4.5  5.0  5.6  6.3 7.1 8.0
#                320 350 380 410 440 470 500 530 560
luar set_ev(-96) --set EV value
#EV modes: -192  -160  -128  -96  -64  -32  0  32  64  96  128  160  192
#                  -2    -12/3 -11/3 -1   -2/3 -1/3 0  1/3 2/3 1   11/3 12/3 2
luar set_iso_mode(80) --set ISO
luar set_raw_nr(0) --auto dark frame
rs "D:/sunset/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=600 -int=15
luar sleep(2000)
d A/rawopint.csv D:/sunset/
sunrise:
Code: [Select]
clock -sync
luar set_lcd_display(0) --switch of display during taking photos
luar set_mf(1) --set focus to manual
luar set_focus(1495) --set focus distance
luar set_capture_mode(5) --set to manual mode
luar set_user_av96(320) --set AV value
#AV modes: 3.2  3.5 4.0  4.5  5.0  5.6  6.3 7.1 8.0
#                320 350 380 410 440 470 500 530 560
luar set_tv96_direct(-256) --set TV value to 6.3s
luar set_iso_mode(200) --set ISO
luar set_raw_nr(0) --auto dark frame
rs "D:/sunrise/input/" -script=D:/sourcecode/rawopint_rs.lua -shots=600 -int=15
luar sleep(2000)
d A/rawopint.csv D:/sunrise/
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #162 on: 26 / May / 2021, 04:35:57 »
Quote from: c_joerg on 26 / May / 2021, 03:59:46
I have two G1x and no cap. But I bought them used.
The original cap of my M3 has a notch so that the switch is not switched.
I don't think your cap originally belongs to the G1x. It makes no sense for me to removing the cap when using the internal flash.

You might be right. At least what you say makes more sense to me.
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline reyalp

  • ******
  • 14080
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #163 on: 27 / May / 2021, 01:25:57 »
Quote from: dolomiti_timelapse on 26 / May / 2021, 04:23:22
I actually don't like the facebook company... I don't use Whatsapp or Facebook nor have a private IG account but I found a cool python Instagram Private API (https://github.com/adw0rd/instagrapi) and so I started with IG.
What other platforms would you suggest?
Hmm, youtube would let people see full resolution, without the square letter box. Even 4k once you get the g1x set up. But probably not something people would regularly watch unless it was embedded somewhere else.

2x daily short videos could work well as a twitter bot, and code for those should be readily available, but the video quality is pretty bad.
Quote
Yes I will post the log.
I took a look... and found a bunch of bugs in my code  :-[

You may have noticed the log is spammed with "bv ev thresh:0" and the bv_ev_shift column is -12, even though you have bv_ev_pct 0, which should disable it. This is related to using the initial ev shift (ui_ev_shift_e=5, = -1.25 stops) being larger magnitude than the meter high threshold (ui_meter_high_thresh_e=2, = 1 stop)

I *think* the net effect in this case is the target exposure will be reduced by a further 1/8th stop, but it's been a long time since I wrote that code.

With that out of the way, a few other observations on the 2021_05_25_sunset log:

Despite that big blown-out looking area around the moon at the end, the over_weight column is all zero, so overexposure as measured by the script never got to the point where it influenced exposure. The highest over exposure was at the very end, around 0.17%. If I've done the math right that would be about ~120 pixels square, or roughly 3x the size of the moon at the reported 43.8 EFL.

If you want the moon to trigger overexposure protection, you'd need a much lower over_thresh_frac, 0.1% or lower. Beware that setting this very low can trigger oscillations and run into precision issues. And of course, if you did correctly expose the moon at night, the rest would get very dark.

If you want more of the bright area to be considered overexposed, you could increase over_margin_ev

Exposure was mostly driven by underexposure, with under_frac ranging from 8% to 40%, and under_weight never below 65. Given that the bottom ~half of your scene is going to be very dark a lot of the time, you probably want to keep the influence of underexposure fairly low. You could turn it off completely by setting under_thresh_frac to 0, or some combination of increased under_margin_ev and under_thresh_frac.

For a scene like this, it might be nice to have the meter not measure the bottom, to bias the exposure for the sky and snowy peaks, but the script currently only allows setting size, not position.

Updated notebook and htmlized output for the sunset log attached.
Logged
Don't forget what the H stands for.
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #164 on: 28 / May / 2021, 18:34:18 »
Quote from: reyalp on 27 / May / 2021, 01:25:57
Hmm, youtube would let people see full resolution, without the square letter box. Even 4k once you get the g1x set up. But probably not something people would regularly watch unless it was embedded somewhere else.
Yes, an upload to YouTube will be a future project too. I believe hardly anyone will be intersted in this timelapse but I want to use it as cloud backup ;) . Actually a cloud backup is also on my list. The problem here is, that the internet is at the moment way too slow for uploading ~100MB of videos. So I think I will have to delay this too. (sometimes the time lapse computer has to deal with 0,2-0,4MBit/s of upload speed, also remote access is a mess :-[ there is a copper wiring issue... and the street will not get fiber before 2023...)

Quote
Despite that big blown-out looking area around the moon at the end, the over_weight column is all zero, so overexposure as measured by the script never got to the point where it influenced exposure. The highest over exposure was at the very end, around 0.17%. If I've done the math right that would be about ~120 pixels square, or roughly 3x the size of the moon at the reported 43.8 EFL.

If you want the moon to trigger overexposure protection, you'd need a much lower over_thresh_frac, 0.1% or lower. Beware that setting this very low can trigger oscillations and run into precision issues. And of course, if you did correctly expose the moon at night, the rest would get very dark.

I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment... 
Since three exposure times (inkl dark frame) would get too long for HDR, I think it is good enough for the moment. Also HDR would be quite difficult with the flare of the window glass. The G1X should make a difference though. The real problem I see is (since the over_weight parameter always remains the same.) in daylight condition the overexposure could trigger too easily with some bright clouds or snow reflection.
Maybe the highlight clipping in clouds and snow are a better way to define over_thresh_frac and over_margin_ev.

Quote
If you want more of the bright area to be considered overexposed, you could increase over_margin_ev

Exposure was mostly driven by underexposure, with under_frac ranging from 8% to 40%, and under_weight never below 65. Given that the bottom ~half of your scene is going to be very dark a lot of the time, you probably want to keep the influence of underexposure fairly low. You could turn it off completely by setting under_thresh_frac to 0, or some combination of increased under_margin_ev and under_thresh_frac.

For a scene like this, it might be nice to have the meter not measure the bottom, to bias the exposure for the sky and snowy peaks, but the script currently only allows setting size, not position.

hmm.. I already did that: the time lapse in the final videos includes only the upper 2/3 of the frame. (I cut the village out of the picture.) And because the missing part did influence too much the metering I excluded it.

line 703ff
Code: [Select]
-- meter rectangle, centered in active area
self.meter_left = rawop.get_active_left() + rawop.get_active_width()/2 - self.meter_width/2
self.meter_x_count = self.meter_width/self.meter_step
self.meter_top = rawop.get_active_top() + rawop.get_active_height()/2 - self.meter_height/2
self.meter_y_count = self.meter_height/self.meter_step/3 --only upper 2/3 counts
Quote
Updated notebook and htmlized output for the sunset log attached.

Thank you very much. I haven't had time to really get into it yet, but would like to create some jupyter notebook as well in the near future.

« Last Edit: 28 / May / 2021, 18:36:18 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline reyalp

  • ******
  • 14080
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #165 on: 28 / May / 2021, 21:22:45 »
Quote from: dolomiti_timelapse on 28 / May / 2021, 18:34:18
I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment... 
Yes, it's always a trade.
Quote
Since three exposure times (inkl dark frame) would get too long for HDR, I think it is good enough for the moment.
FWIW, without darkframe might look fine with the kind of exposures you're using. I'd certainly expect it to be OK on G1x. You can also use pre-made darkframes and subtract them on the PC, but that works better with raw and takes a lot of processing.

Quote
The real problem I see is (since the over_weight parameter always remains the same.) in daylight condition the overexposure could trigger too easily with some bright clouds or snow reflection.
Maybe the highlight clipping in clouds and snow are a better way to define over_thresh_frac and over_margin_ev.
I'm not sure I follow. over_weight was 0 in the sunset run because of the particular conditions. It was triggered quite a bit in the sunrise (https://www.instagram.com/p/CPUtoF8HpuA/ i think), but underexposure still pushed the exposure quite high.

Quote
hmm.. I already did that: the time lapse in the final videos includes only the upper 2/3 of the frame. (I cut the village out of the picture.) And because the missing part did influence too much the metering I excluded it.

line 703ff
Code: [Select]
-- meter rectangle, centered in active area
self.meter_left = rawop.get_active_left() + rawop.get_active_width()/2 - self.meter_width/2
self.meter_x_count = self.meter_width/self.meter_step
self.meter_top = rawop.get_active_top() + rawop.get_active_height()/2 - self.meter_height/2
self.meter_y_count = self.meter_height/self.meter_step/3 --only upper 2/3 counts
That makes sense. It should probably be a script option, but it already has so many  :-[

You could also adjust the area measured by the histogram, if you haven't already.
Logged
Don't forget what the H stands for.
*

Offline c_joerg

  • *****
  • 1248
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #166 on: 29 / May / 2021, 02:37:06 »
Quote from: dolomiti_timelapse on 28 / May / 2021, 18:34:18
I don't think it is a good idea to trigger overexposure in this situation as the rest would get very dark. I think this is the limit of my setup at the moment... 
That's what I usually do. I try to expose the moon as brightly as possible but so that it isn’t overexposed. In post-production, I then pull the shadows up and the lights down. RAW is of course a must. The problem is the dynamic range of the old G1x. Then there is not so much in it. A reason for me to switch to the M100.
Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #167 on: 29 / May / 2021, 15:09:24 »
Quote from: reyalp on 28 / May / 2021, 21:22:45

I'm not sure I follow. over_weight was 0 in the sunset run because of the particular conditions. It was triggered quite a bit in the sunrise (https://www.instagram.com/p/CPUtoF8HpuA/ i think), but underexposure still pushed the exposure quite high.

OK thanks. It will take a few weeks and several changes to the settings and analysis of the result until it fits.

I will wait now and see how it performs with the G1X and match the parameters for the G1X.
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #168 on: 10 / July / 2021, 13:07:47 »
Hello reyalp,

I want to share my experience and some code changes I did to rawopint.lua in this and the next post.

For a better understanding of the settings/parameters ui_bv_ev_shift_pct, ui_bv_ev_shift_base_e, ui_ev_shift_e I did a bit of reverse engineering by gradually changing one parameter and then recording a sunset and so forth.
I attached a pdf with the plots of the ui_bv_ev_shift_pct changes from 2% to 80%, which I think could be also interesting to others for the understanding of this functionality.
For my kind of timelapse a value of about 16-24% is best depending on the ui_bv_ev_shift_base_e and ui_ev_shift_e parameters. At the beginning it was quite strange that high values would overexposed the image in daylight. But in the end it makes sense since it lowers the exposure when it's dark and rises the exposure when it is higher than the "neutral" exposure. The ui_bv_ev_shift_base_e did basically the same as the ui_bv_ev_shift_pct. It should the shift starts later for a sunset or earlier for a sunrise but the result is minimal and very similar to the ui_bv_ev_shift_pct changes even when you use the extreme values.

An other topic was the under_expo_weight, over_exp_weight. As far as I understand for my type of sunset it is best to switch this two parameters off and only relay on the meter exposure, because I think that if a bright cloud comes into the frame the hole frame should not darken. Because the trade off would be the video to flicker if the clouds are changing too fast. Also in the night the under expo tried to raise the exposure up but if it is dark outside it is ok if parts of for me if in the frame are completely dark areas. On the other hand if the clouds are too overexposed I would just underexpose the whole image and if the image in the night is too dark in general lower the ui_bv_ev_shift_pct value a bit.

What does actually mean a rise of the meter weight from 100 to say 200? I attached a plot where exactly that happens but I could not see any changes in the timelapse video itself.. ???

The last part I'm not sure is, if whether the ui_meter_width_pct and ui_meter_height_pct are dependent on the selected image format. E.g. if I choose 1:1 or 16:9 image format stays the area that rawopint uses to calculate the exposure always the same?  ???


« Last Edit: 10 / July / 2021, 13:11:24 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
*

Offline dolomiti_timelapse

  • **
  • 96
Re: rawopint.lua: Fast, accurate intervalometer with raw exposure metering
« Reply #169 on: 10 / July / 2021, 14:14:51 »
I found this line in the rawpint code:
Code: [Select]
1148 -- smooth out rapid changes
1149 -- TODO smoothed fraction should be configurable

What I did was to implement a simple exponential moving average.
https://en.wikipedia.org/wiki/Exponential_smoothing

Disclaimer:
I actually don't know nothing about lua and I just took vscode to implement the changes. I also didn't format the code because it could be that the auto formatting of vscode would apply an other format that you did.
Also I have no experience which code performance in lua and I could not test the script property with some proper test but just loaded it to the camera and compared the d_ev value before and after the change  :lol

But it actually works:
All changes are done in the function exp:calc_ev_change() function.
I always included the line before so it is easier to find, since the line number could vary.
Code: [Select]
930    local last_ev_change
931    local last_ev_change_decimal
932    local median_filter_intensity = 5 -- this is a magic number!! it has to become an input parameter
933    -- allowed values: 0: no filter, 9: max filter (>10 and <0 not allowed here) expample:
934    -- 1: ev_change = 10% * last_ev_change + 90% * ev_change -> low moving average filter intensity
935    -- 8: ev_change = 80% * last_ev_change + 20% * ev_change -> high moving average filter intenisty
Code: [Select]
940        last_ev_change = self.ev_change
941        last_ev_change_decimal = self.ev_change_decimal
Code: [Select]
949        last_ev_change = 0
950        last_ev_change_decimal = 0
Code: [Select]
1126    -- smooth out rapid changes exponential moving average implemented
1127    if self.smooth then
1128        local ev_change_smooth = (last_ev_change * 10 + last_ev_change_decimal) * median_filter_intensity +
                                        (10 - median_filter_intensity) * ev_change * 10
1129        ev_change = ev_change_smooth / 100
1130        last_ev_change_decimal = ev_change_smooth % 100 / 10 -- not sure if modulo works this way in lua
1131    end
1132    self.ev_change = ev_change
1133    self.ev_change_decimal = last_ev_change_decimal

After I wrote this lines of code I asked my self why I did not used a float or double. I actually was not sure if there is a performance or memory issue using doubles since I could only find Integers in the code.
In the end I simulated the first decimal number after the decimal point with an Int which should be ok.

the magic is simply this line of code:
ev_change = ev_change * ( 1 - moving_average_fiter_intensity) + last_ev_change * moving_average_fiter_intensity
the value for the moving_average_fiter_intensity goes from 0 to 1

since I used only int for this calculation I multiplied the new ev_change by 10 and also the moving_average_fiter_intensity goes form 0 to 9 (10 would mean no change at all)
after the calculation the result has to be divided by 100 and to save also the first decimal number line 1130 comes into place.

The two pdfs shoes the d_ev value with the moving average filter (set to 9) and without the moving average filter.

What is still missing: the moving_average_fiter_intensity is a magic number at the moment and has to be declared as parameter at the beginning.
I also deleted the smooth overshoot and smooth sign switch as the moving average does a better job.

If the filter is set to high it will get quite sluggish.

EDIT:
What I forgot to mention: the d_ev was scaled by a factor of 10 in the plot. (because it is very small compared to the other values)
« Last Edit: 10 / July / 2021, 15:09:44 by dolomiti_timelapse »
Logged
If you want to see a sunset or sunrise of Dolomiti Val Gardena shot with CHDK visit
Instagram: dolomiti_timelapse
YouTube: https://www.youtube.com/channel/UCEJHg--ujxLkjMrevJXh-Gw
Pages: 1 ... 12 13 14 15 16 [17] 18 19 20 21   Go Up
« previous next »
 

Related Topics