supplierdeeply

G1X + others ISO override issues (was Re: rawopint.lua)

  • 65 Replies
  • 14668 Views
G1X + others ISO override issues (was Re: rawopint.lua)
« on: 01 / July / 2017, 03:18:28 »
Advertisements
I've attached a copy of rawopint.lua with set_sv96(sv) replaced (at line 1055) by the code I've been using for years without problems. Will you see if it fixes the ISO problem on the G1X if you have time? Run the script in continuous mode.
Code: [Select]
-- *****lapser sv test
--   set_sv96(self.sv96)
  set_prop(props.DELTA_SV,self.sv96-get_prop(props.SV)+get_prop(props.DELTA_SV))
  set_prop(props.SV,self.sv96)
-- *****lapser sv test
I tried the changes with G1x against this test video (only to the middle).
 
/>
I run twice. With and without @lapser changes. What should I say it works great!
Thank you lapser for that. Attached a plot without and with his changes.
But why? Looks like that’s the same what we did in isoinc. And we tested it in isoinc in continuous mode.
Code: [Select]
-- *****lapser sv test
 set_prop(props.DELTA_SV,self.sv96-get_prop(props.SV)+get_prop(props.DELTA_SV))
set_prop(props.SV,self.sv96)
-- from isoinc
-- from http://chdk.setepontos.com/index.php?topic=11081.100
local dsv = v - get_prop(props.SV) + get_prop(props.DELTA_SV)
set_prop(props.DELTA_SV,dsv)
set_prop(props.SV,v)
 
So what can be the different? I run only RAW and not JPG + RAW?
If it's really only specific to the g1x I wouldn't want to add a bunch of features to script just to work around problems with one port.
Not in this case 😉
We have seen this problem also on S110 and SX50. So I will test it on my S110 again.


*

Offline lapser

  • *****
  • 1093
G1X + others ISO override issues (was Re: rawopint.lua)
« Reply #1 on: 01 / July / 2017, 13:05:06 »
I'm glad to see that it worked. Saving raw+jpg doesn't really change anything, so it will work the same. I'm not sure how isoinc does this, but the timing is critical when the script can do this and have it work. It has to be in continuous mode while hook_raw is ready and before hook_raw.continue().

This method works for all my cameras (haven't test the M3 yet). Even in continuous mode, you can't use this method before the first shot. The camera has to set the first shot (sets svbase).

The camera sets ISO by adding svbase+svdelta. The sum, sv, is stored in props.SV for EXIF purposes. In continuous mode, the camera doesn't modify these parameters. So I assume this equation is true:

sv=svbase+svdelta

The problem with set_iso() in CHDK is that it sometimes changes svbase. In continuous mode, you only want to set svbase on the first shot. After the first shot in continuous mode, set_iso() should switch to my method. That should work for all cameras.

Let's say svnew is the new sv you want to set. You don't want to change svbase, only svdelta. So you add the change you want, (svnew-sv), to svdelta

svdelta=svdelta +(svnew-sv)
then you finish with
sv=svnew
« Last Edit: 01 / July / 2017, 18:50:57 by lapser »
EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

*

Offline philmoz

  • *****
  • 3070
    • Photos

 
This method works for all my cameras (haven't test the M3 yet). Even in continuous mode, you can't use this method before the first shot. The camera has to set the first shot (sets svbase).


The camera sets ISO by adding svbase+svdelta. The sum, sv, is stored in props.SV for EXIF purposes. In continuous mode, the camera doesn't modify these parameters. So I assume this equation is true:


sv=svbase+svdelta


The problem with set_iso() in CHDK is that it sometimes changes svbase. In continuous mode, you only want to set svbase on the first shot. After the first shot in continuous mode, set_iso() should switch to my method. That should work for all cameras.


Let's say svnew is the new sv you want to set. You don't want to change svbase, only svdelta. So you add the change you want, (svnew-sv), to svdelta


svdelta=svdelta +(svnew-sv)
then you finish with
sv=svnew


Hi Lapser, love the timelapse videos, hope to get time to try this one day.


Let me try and shed a bit more light on this.
First, sv = svbase + svdelta is correct; but the 'svbase' value is a constant, CHDK never changes this once it has been initially set. Neither does the Canon firmware as far as I can tell.


Under normal conditions, on most cameras, the CHDK set_sv96 function and your script code to set sv96 actually do the very same thing.
To prove this consider the following.
Terminology:
- cam_sv, cam_svdelta - property values for sv and svdelta currently set in the camera
- sv - target sv value we want to override to for the next shot


In both cases the first sv0 override is done using set_sv96 (CHDK code).
The result is
    cam_sv0 = sv0
    cam_svdelta0 = sv0 - svbase


For the next override, sv1, the CHDK set_sv96 code will set:
    cam_sv1 = sv1
    cam_svdelta1 = sv1 - svbase
The script version does this
    cam_sv1 = sv1
    cam_svdelta1 = sv1 - cam_sv0 + cam_svdelta0
Substituting values we get
    cam_svdelta1 = sv1 - sv0 + sv0 - svbase
or
    cam_svdelta1 = sv1 - svbase


The above works provided the value of cam_svdelta remains intact.


Herein lies the problem for the G1X.
For Tv < 1.3 seconds the above holds and both methods should produce the same result.
When Tv >= 1.3 seconds, then at ISO 400, the camera does strange things.
It should result in:
    svbase = 576 (constant)
    cam_sv = 603
    cam_svdelta = 96
It actually end up with
    svbase = 576 (constant)
    cam_sv = 603
    cam_svdelta = 0


The camera shoots at 1 stop lower ISO; and then digitally boosts the exposure of the CR2 & JPG images (while recording the original requested ISO in the EXIF).
The raw_hook metering sees the underexposed image so the metering calculates a 1 stop lower value.


This is where the two methods now diverge.
For ISO 403 (the next increment):
- CHDK set_sv96 sets cam_sv = 604 and cam_svdelta = 97
- Script set sv96 sets cam_sv = 604; but cam_svdelta = 1


For the CHDK version, this time firmware does not alter cam_svdelta so it remains at 97 and shoots at ISO 403; but it still digitally boosts the CR2 & JPG images. So the raw_hook meter is correct; but the images are overexposed by 1 stop.


For the script version, the camera is now shooting at 1 stop lower and still boosting the images. The raw_hook meter is off by 1 stop; but the images are the correct brightness.


In summary, for the G1X with Tv >= 1.3s and ISO >= 400:
CHDK set_sv96 - saved CR2/JPG overexposed by 1 stop, metering should be correct (except at ISO 400)
Script set sv96 - saved CR2/JPG correct exposure, metering underexposed by 1 stop


 
Phil.

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)

Lapsers changes also fix the problem at the S110. At S110, the problem with ISO800 occurs, so the peak is to the right. I will do another test with EOS M3.



This method works for all my cameras (haven't test the M3 yet). Even in continuous mode, you can't use this method before the first shot. The camera has to set the first shot (sets svbase).

I made a run with EOS M3.  There are no differences between original version and lapsers changes.

But, from both runs, the EXIF ISO is always ISO100. The run was made with (RAW+JPG). All files have EXIF ISO100 (RAW+JPG).

Update:
EXIF ISO100
« Last Edit: 02 / July / 2017, 08:35:50 by c_joerg »

*

Offline lapser

  • *****
  • 1093
I made a run with EOS M3.  There are no differences between original version and lapsers changes.

But, from both runs, the EXIF ISO is always ISO100. The run was made with (RAW+JPG). All files have EXIF ISO100 (RAW+JPG).
I discovered a new propcase on the EOS M3 that sets the exif ISO

  set_prop(pDELTA_SV,sv-get_prop(pSV)+get_prop(pDELTA_SV)) -- sets actual exposure
  set_prop(pSV,sv) -- sets EXIF data
  if eosm3 then set_prop(354,sv) end -- EXIF data on EOSM3

I tried in a test script last night and it worked!
EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

Another test with the SX50. Again, the problem is solved.
I have now changed all my local scripts versions with this fix.

How would a general solution look like?
1) replace set_sv96 (self.sv96).
2) replace set_sv96 (self.sv96) with additional script parameter.
3) replace set_sv96 (self.sv96) only in continuous mode.

@reyalp
I still want a log for the battery temperature ;)

I discovered a new propcase on the EOS M3 that sets the exif ISO
  set_prop(pDELTA_SV,sv-get_prop(pSV)+get_prop(pDELTA_SV)) -- sets actual exposure
  set_prop(pSV,sv) -- sets EXIF data
  if eosm3 then set_prop(354,sv) end -- EXIF data on EOSM3
I tried in a test script last night and it worked!
I haven’t checked this …

*

Offline reyalp

  • ******
  • 11335
Re: G1X + others ISO override issues (was Re: rawopint.lua)
« Reply #7 on: 03 / July / 2017, 15:37:59 »
Split from the rawopint thread, since it really doesn't have anything to do with that specific script. There was previous related discussion in the isoinc tread: https://chdk.setepontos.com/index.php?topic=12165.10

edit:
Also in the raw hook development thread: https://chdk.setepontos.com/index.php?topic=11081.msg121660#msg121660
« Last Edit: 03 / July / 2017, 16:36:44 by reyalp »
Don't forget what the H stands for.


*

Offline reyalp

  • ******
  • 11335
In summary, for the G1X with Tv >= 1.3s and ISO >= 400:
CHDK set_sv96 - saved CR2/JPG overexposed by 1 stop, metering should be correct (except at ISO 400)
Script set sv96 - saved CR2/JPG correct exposure, metering underexposed by 1 stop
 
Phil.
In the previous discussion, there was a difference between continuous mode and not https://chdk.setepontos.com/index.php?topic=12165.msg122574#msg122574 which may explain why the fix "works" in rawopint
Don't forget what the H stands for.

*

Offline philmoz

  • *****
  • 3070
    • Photos
Re: G1X + others ISO override issues (was Re: rawopint.lua)
« Reply #9 on: 04 / July / 2017, 17:48:34 »
Finally I can report some progress with the G1X ISO overrides.


I was able to find the task and location where the svdelta value is being overwritten, and by patching this code I can now get all the different variations to produce the same results in the isoinc.lua test script (Cont/CHDK, Cont/Script, Single/CHDK & Quick/CHDK).


I can now also determine for which images the svdelta is being overwritten and by how much - this means I can theoretically pass a compensation value to the rawop meter calculation function to adjust the metering.


The camera still shoots at 1/2 the requested ISO (and sometimes with a different black point), and then cooks the JPG & CR2 data before saving.
If you use CHDK DNG files they will still be 1/2 brightness and record 1/2 the ISO in the EXIF data.


I need to run some more tests to make sure there are no other side effects, and I need to figure out the metering adjustment; but I hope to be able to post test builds over the next few days if anyone wants to try it out.


Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)