proposal - script shooting hooks - page 26 - General Discussion and Assistance

c_joerg

1240

Re: proposal - script shooting hooks

« Reply #250 on: 29 / August / 2015, 19:17:46 »

Advertisements

Quote

For the sx230 runs, I'm not clear which section of the log matches each run, but the meter weight setting suggestion still applies.

I updated the log file for SX230, which includes now only the 4 runs from video.

I had some funny thinks, when I manual focus. On 2 runs without changing script parameters, the moon was totally overexposed. I don’t have the log for this runs anymore.

I see also on the Display
SD: 65486
Adj:Off
But only on manual focus. On auto focus not.

« Last Edit: 29 / August / 2015, 19:28:00 by c_joerg »

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

reyalp

13878

Re: proposal - script shooting hooks

« Reply #251 on: 30 / August / 2015, 01:34:51 »

Quote from: c_joerg on 29 / August / 2015, 19:17:46

I see also on the Display
SD: 65486
Adj:Off
But only on manual focus. On auto focus not.

This is normal CHDK behavior.

Here's a moon rise taken with sx160

Settings:
Zoom 100mm (35 mm equivalent)
Focus manual at infinity
Interval 3 sec
Meter width and height 90%
Meter step 13
Bv Ev shift % 30
Bv Ev shift base Bv 9
Max Tv 1000 (1 second)
Target ISO 100
ISO adj TV 250 (1/4 sec)
Max ISO 400
Meter low thresh -Ev 5
Meter low limit -Ev 6
Meter low max weight 100 (no meter weight increase for low, but thresh and limit are still used for Bv Ev shift)
Overexp thresh 5
Overexp Ev range 1/2
Oeverexp prio 10
Underexp thresh 0 (disabled)
Histogram step 9

I think this worked pretty well technically, but even with the wisps of cloud it's pretty boring when the moon is visible. In the log it looks like there is some oscillation, but it's not really visible in the video.

It stays dark and doesn't quite hit the ISO limit after the moon leaves the frame due to Bv Ev shift. I should have either set the base value lower, set the meter low limits higher, or not used it at all.

rawopint.csv (363.4 kB - downloaded 26 times.)

test-rawmeter-2015-08-29-exp.png (88.75 kB, 1096x678 - viewed 93 times.)

Logged

Don't forget what the H stands for.

c_joerg

1240

Re: proposal - script shooting hooks

« Reply #252 on: 30 / August / 2015, 04:21:05 »

Quote

I think this worked pretty well technically

Yes, I can only agree. My first two races are also very good.

Quote

but even with the wisps of cloud it's pretty boring when the moon is visible.

Yes of course. But as you can see, this is quite well so already quite complex.
I wanted to make just a couple of similar runs and only change exp_over_thresh_frac to see the effects.
Yes of course, the more interesting races are already, if clouds are in play or the moon goes through trees or buildings.

Quote

n the log it looks like there is some oscillation, but it's not really visible in the video.

The moon in your video is very small .May be you would see it when the cutting would be greater.

Quote

It stays dark and doesn't quite hit the ISO limit after the moon leaves the frame

The same in my first run, but very close...

Quote

I should have either set the base value lower, set the meter low limits higher, or not used it at all.

But it can be seen but the complexity of the whole.
Changes or one forgets only one parameter, the result is quite different.
Especially if you come from one day run into a night run, a lot of parameters have to be changed.
Also I'm not sure whether the change of a particular parameter always conceded the desired effect .
This really is very complex ...

Do you have any idea why run 3 and 4 have not properly focused ?
The log file is in all 4 races :

sd:-1 af_ok:true

So I have the runs focused :

The start I have the moon set in the center of the image.
After the start of the script, I then slightly shifted the camera.
The problem is, that can be seen only at the end of the run, if the moon was really sharp.

With manual focus I have the impression, that infinite is not really infinity at this focal length from 400mm.

Quote

(the menu value is in parts per 100k, but the actual value is per million)

That makes understanding not easier...

Update:
I 'm not sure if the overexposure must be weighted. Why not just stop or reduce the exposure, when overexposure exceed?

What would happen in the following example with 1% overexposure?

Putting a hysteresis window around overexposure maybe 20%. So the window is 0.8% to 1.2%

Increasing exposure (depending on meter window) until limits or as long overexposure < 0.8%
Between 0.8% - 1.2% no changes in exposure at all to avoid oscillation.
If overexposure > 1.2% only decreasing exposure.

Maybe that's too easy. But you can estimate fairly accurately what happened.
So I did it in the past with my script, but with the old histogram with 1 % resolution, it did not work really well ...

I have already tried to integrate the new histogram in my old script , but that was a bit too complex for me .

Why I play at all with my old script ?
My script is very simple. I change only the exposure time. The exposure time I calculate out from camera values by smoothing. I use Canon ISO settings. I look like that’s this works on G1x with higher ISO values than 320 (But fixed ISO

« Last Edit: 30 / August / 2015, 05:34:04 by c_joerg »

Logged

reyalp

13878

Re: proposal - script shooting hooks

« Reply #253 on: 30 / August / 2015, 14:16:40 »

Quote from: c_joerg on 30 / August / 2015, 04:21:05

But it can be seen but the complexity of the whole.
Changes or one forgets only one parameter, the result is quite different.
Especially if you come from one day run into a night run, a lot of parameters have to be changed.
Also I'm not sure whether the change of a particular parameter always conceded the desired effect .
This really is very complex ...

Yes, the script has too many options. Eventually I'd like to simplify it, but I don't know in advance what will give the best results in different situations, so I have lots of options to try things out. As I've said before, the script was mostly written to prove the new features in the C code.

In the CHDK script menu, you can save parameter sets, so you could save one for moon rises, another for daytime, and so on.

Quote

Do you have any idea why run 3 and 4 have not properly focused ?

On a camera that has manual focus, if you set it using the Canon UI before running the script, it should stay at whatever you set.

Quote

So I have the runs focused :

The start I have the moon set in the center of the image.
After the start of the script, I then slightly shifted the camera.

Are you using Canon AF? If so, I suggest you don't do that. Use MF, take a test shot and look at it in review mode if you need to. On my cameras, I find just using the "focus at infinity" mode or selecting MF and setting the maximum distance works fine.

edit:
I should note that the Canon "focus at infinity" (mountain icon on the cameras that have it) mode is only treated as a suggestion by the Canon firmware. I haven't had trouble with it for timelapse, but in other situations I've seen it fail.

Quote

I 'm not sure if the overexposure must be weighted. Why not just stop or reduce the exposure, when overexposure exceed?

There are many possible algorithms, but I don't have time to code them all. The current algorithm was intended to try to keep the overall scene reasonable, rather than strictly limiting over exposure.

The rawopint script is modular, so it should be relatively easy to replace the exposure calculation. You should only need to replace exp:calc_ev_change() and set the desired exposure change in self.ev_change. You may have to set a few other values to keep the debug drawing and logging code happy (self.ev_target_base, self.ev_target...) but they can just be dummy values.

« Last Edit: 30 / August / 2015, 14:24:10 by reyalp »

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: proposal - script shooting hooks

« Reply #254 on: 30 / August / 2015, 14:44:50 »

Quote from: reyalp on 30 / August / 2015, 14:16:40

Quote
Do you have any idea why run 3 and 4 have not properly focused ?
No
On a camera that has manual focus, if you set it using the Canon UI before running the script, it should stay at whatever you set.

There's a firmware feature that can alter focus in MF mode: "Safety MF". It should be set to "off".

Logged

c_joerg

1240

Re: proposal - script shooting hooks

« Reply #255 on: 30 / August / 2015, 15:00:21 »

Quote

As I've said before, the script was mostly written to prove the new features in the C code.

Yes, of course and I’m really happy to use it….

Quote

In the CHDK script menu, you can save parameter sets, so you could save one for moon rises, another for daytime, and so on.

Yes, I did this before. Unfortunately, the parameter sets have only number and no name as sunset , moon rise and so on.
Where I find information, what happens when a parameter name changes or when a new parameter comes in?

Quote

Are you using Canon AF?

Yes, all 4 runs one the SX230 I made with Canon AF. That is why I was wondering…

Quote

I suggest you don't do that

After the runs I tried to play with manual focus. That brought then also unsuccessful. I ended up being quite as tired. I want to try it with MF again ...

Quote

but I don't have time to code them all

Yes of course, I understand this.

Quote

You should only need to replace exp:calc_ev_change()

Well, I will try it….

Quote

Don't forget what the H stands for.

In rawopint not for hacking… I have reviewed a lot off code in my live and rawopint is definitely no hacking (may be the C code behind

)

@srsa_4c

Quote

"Safety MF". It should be set to "off".

It is always off on all my cams….

Logged

reyalp

13878

Re: proposal - script shooting hooks

« Reply #256 on: 31 / August / 2015, 02:26:15 »

Another moon test run

Camera elph130
Zoom 224mm (35 mm equivalent)
Focus Focus at infinity mode.
Interval 4 sec
Meter width and height 90%
Meter step 13
Bv Ev shift: off
Max Tv 2500 (2.5 second)
Target ISO 100
ISO adj TV 250 (1/4 sec)
Max ISO 400
Meter low max weight 100 (no meter weight increase for low)
Overexp thresh 5
Overexp Ev range 1/2
Oeverexp prio 10
Underexp thresh 0 (disabled)
Histogram step 7

This is similar to the last run, except Bv Ev is disabled and the zoom is higher. Due to some mistakes on my part, it doesn't include the moon entering or leaving the scene, so the bv ev doesn't actually have any impact. Some oscillation is visible.

edit:

Quote from: c_joerg on 29 / August / 2015, 18:10:17

OK, script starts with ISO800.

This is probably a quirk in CHDK, which you can see in my log too.

The first line in the log (after the column headers) is from half press, before the actual shot. When CHDK overrides ISO (from script or the menu) it actually sets the Canon firmware to auto ISO. It doesn't use Canon auto ISO, it just needs to be in this mode to make the override work. However, once you stop using the override, the Canon firmware is still in auto ISO mode, even though it might look like it is set to something else in the UI. You can reset this by changing the ISO in the Canon UI.

This means if you run rawopint more than once, without rebooting or touching the Canon ISO settings, the runs after the first one will start with the Canon auto ISO values. This shouldn't normally be a problem, because the values will set before the actual shot, but it might cause problems on g1x where high ISO has other impacts.

rawopint-2015-08-30_1.csv (76.09 kB - downloaded 26 times.)

test-rawmeter-2015-08-30_1-exp.png (85.68 kB, 1188x661 - viewed 66 times.)

« Last Edit: 31 / August / 2015, 02:42:44 by reyalp »

Logged

Don't forget what the H stands for.

c_joerg

1240

Re: proposal - script shooting hooks

« Reply #257 on: 31 / August / 2015, 04:04:09 »

Quote

This means if you run rawopint more than once, without rebooting or touching the Canon ISO settings, the runs after the first one will start with the Canon auto ISO values. This shouldn't normally be a problem, because the values will set before the actual shot, but it might cause problems on g1x where high ISO has other impacts.

Interesting discovery. This can be very well seen in the log file of G1x. I had 2 test starts. The first starts at ISO100. The other with ISO1600. So when I use the G1x, I have to remember ...

Update:

Just for understanding, if I have understood the code correct:
I made it very simple, just for understanding. The hysteresis window is not implemented…

What I want to do:
1) Ignore underexposure at all
2) On overexposure take only care about overexposure
3) No changes in exposure when meter96 is close to 0 and no overexposure.

Your code

Code: [Select]

 ev_change = (ev_change*meter_weight - self.ev_change_max*over_weight + self.ev_change_max*under_weight)/(meter_weight + over_weight + under_weight)

 -- clamp to ui max
 -- TODO everything above should already be in limits
 ev_change = self:clamp(ev_change,self.ev_change_max)

 -- smooth out rapid changes
 -- TODO smoothed fraction should be configurable
 if self.smooth then
  local ev_change_smooth = ev_change + last_ev_change
  ev_change_smooth = ev_change_smooth/2 + ev_change_smooth%2
  -- but don't allow overshoot in the wrong direction
  if (ev_change == 0 and ev_change_smooth ~= 0) 
   or (ev_change_smooth > 0 and ev_change < 0) 
   or (ev_change_smooth < 0 and ev_change > 0) then
   logdesc('smooth overshoot:%d',ev_change_smooth)
   ev_change = 0
  else
   ev_change = ev_change_smooth
  end
  -- TODO hacky hardcoded
  -- sign changed, reduce by half (and trunc to 0 if 1)
  if ev_change * last_ev_change < 0 then
   logdesc('smooth sign switch')
   ev_change = ev_change/2
  end
 end

 self.ev_change = ev_change

My changes:

Code: [Select]

    if self.over_frac > self.over_thresh_frac then
  
         -- on overexposure take only care about the overexposure
         ev_change = ( - self.ev_change_max)

    else

         -- avoid oscillation around meter96 = 0
         if (self.mval96 > -8 and self.mval96 < 8)
             -- no changes when meter96 is close to 0 and no overexposure
             ev_change = 0  
         end
 
    end
 
 -- clamp to ui max
 -- TODO everything above should already be in limits
 ev_change = self:clamp(ev_change,self.ev_change_max)
 
    self.ev_change = ev_change

« Last Edit: 31 / August / 2015, 06:51:05 by c_joerg »

Logged

reyalp

13878

Re: proposal - script shooting hooks

« Reply #258 on: 31 / August / 2015, 15:59:32 »

Quote from: c_joerg on 31 / August / 2015, 04:04:09

Interesting discovery.

Sadly not a discovery, but remembering a very old CHDK bug

Quote

Just for understanding, if I have understood the code correct:
I made it very simple, just for understanding. The hysteresis window is not implemented…

What I want to do:
1) Ignore underexposure at all
2) On overexposure take only care about overexposure
3) No changes in exposure when meter96 is close to 0 and no overexposure.

Without actually testing, you code seems like it would do what you described.

A couple of notes

Code: [Select]

 ev_change = ( - self.ev_change_max)

the current code scales the weight (and so, the eventual effect on ev) depending how much over exposure there is. As discussed before (http://chdk.setepontos.com/index.php?topic=11081.msg123350#msg123350), you can't know how much change is required, but if you are just slightly over thresh, you probably don't want a full ev_change_max. Maybe your hysteresis will take care of this.

My plan to deal with this is to make the algorithm know something about how much the previous change affected overexposure, but I'm not sure when I will get to it, and obviously it will make the algorithm even more complex.

Code: [Select]

         if (self.mval96 > -8 and self.mval96 < 8)
             -- no changes when meter96 is close to 0 and no overexposure
             ev_change = 0  
         end

For meter, the existing code calculates the exact change require to get correct exposure (so if mval96 == 9, it would give a change of -9 before smoothing) If you keep that logic, there will be a sudden jump when mval96 leaves the window. Also some settings (ev shift, bv ev) change the target exposure, so they won't work if you use mval96 directly like this.

FWIW, I originally started with lots of ifs like this. The reason I switched to the current system with weights ramping smoothly up and down is that I always had corner cases where there would be jumps or oscillation switching from one state to another. I'm not saying it can't be done, I just personally found it kept getting more complicated and harder to test.

Logged

Don't forget what the H stands for.

c_joerg

1240

Re: proposal - script shooting hooks

« Reply #259 on: 01 / September / 2015, 09:01:42 »

Quote

the current code scales the weight (and so, the eventual effect on ev) depending how much over exposure there is.

Yes, of course. Over exposure changes has to be weighted. This was only to simply the example for understanding.

Quote

For meter, the existing code calculates the exact change require to get correct exposure (so if mval96 == 9, it would give a change of -9 before smoothing) If you keep that logic, there will be a sudden jump when mval96 leaves the window.

This was also just to simplify. I want to try a dumping curve (see plot) for ev_changes . So maybe I have not check mval96 I can just but ev_changes in translation.

Quote

settings (ev shift, bv ev) change the target exposure, so they won't work if you use mval96 directly like this.

Yes, of course. On the first run, I want only to try how my stuff works.
So an –1EV shift gives you mval96 == -96?

Quote

I originally started with lots of ifs like this.

May be it get even worse, but then I have learned a little bit more about your code…

Update:
This is the code, which i want to try:

Code: [Select]

    
    ---------------------------------------------------------------------------------------------------
    -- Window is 5% of over_thresh_frac
    local win_over_thresh_frac = self.over_thresh_frac / 20 
    if win_over_thresh_frac < 1 then
        win_over_thresh_frac = 1
    end

    
    if self.over_frac > self.over_thresh_frac then
     
        -- on over exposure take only care about the overexposure
        -- for every 5% over exposure one step
        local ev_change_factor = (self.over_frac - self.over_thresh_frac) / win_over_thresh_frac
        if ev_change_factor < 1 then
            ev_change_factor = 1
        end
        
        ev_change = ( - ev_change_factor)
        logdesc("Overexposure ev_change",ev_change)
         
    elseif self.over_frac  > (self.over_thresh_frac - 2 * win_over_thresh_frac) then    

         -- only negative ev_change when close to overexposure
        if ev_change > 0 then
            ev_change = 0
        end
        logdesc("Only negative ev_change",ev_change)
             
    else

         --if (self.mval96 >= -10 and self.mval96 <= 10) then
         -- no changes when meter96 is close to 0 and no overexposure

         -- dumping ev_changes to avoid oscillation ev_change = (ev_change^3)/100
         local ev_change_mod = ev_change

         if (ev_change >= -10 and ev_change <= 10) then         
             ev_change_mod = (ev_change * ev_change * ev_change)/100
         end
         logdesc("Change ev ",ev_change," to ",ev_change_mod)
         ev_change = ev_change_mod
    
    end
    
    -- clamp to ui max
    -- TODO everything above should already be in limits
    ev_change = self:clamp(ev_change,self.ev_change_max)
    
    self.ev_change = ev_change

ev_change.png (163.34 kB, 3359x2330 - viewed 58 times.)

« Last Edit: 01 / September / 2015, 10:32:07 by c_joerg »

Logged