Measuring exposure performance (was Re: proposal - script shooting hooks)

c_joerg

@reyalp:
Made just a fast run with isoinc on the S110 with 1s exposure time and ISO from 960 to 1040 (just 12 pictures)

You mean to set the initial Tv? You should be able to use M mode. It should always stay the same between shots, unless you selected "Keep exposure constant"

Yes. I set manual 1s in M Mode, const_exp=true and start_ev_shift=0. But I did not get the 1s. So I fixed it in the code as you descript before.

I made 2 runs. One in continue and one in quick mode. As you can see, I can reproduce it and the result is different from the two runs. I canâ€™t believe it why I have it not seen beforeâ€¦.

From my last post here I had 2 open question. May be you find some time and give me a short answer. It will may be help me to understand a little bit more about the stuff..

isoinc_S110_1s.csv

3.23KB

reyalp

c_joerg
Yes. I set manual 1s in M Mode, const_exp=false and start_ev_shift=0. But I did not get the 1s. So I fixed it in the code as you descript before.

That's what I get for going from memory rather than looking at the code. This is expected behavior, if the cameras current ISO doesn't match the lowest test value.

From my last post here I had 2 open question. May be you find some time and give me a short answer. It will may be help me to understand a little bit more about the stuff..

Sorry, sometimes even a short answer takes lot of time 🙁

The first thing which I notice was that I got sometimes from the pre shot ISO200. Yes, we have seen it before and it looks like this is some stupid stuff from G1x. So I fixed that in the code from isoinc.

I would guess this is just the camera being in auto-ISO at the start. Remember, this happens if CHDK ISO overrides have been used, even if the camera already had a specific ISO set. To get back to true manual ISO, you need to set something in the menu.

Why the RAW values are not going under 512? Yes I know a little bit about the black level but not really much.

Some previous discussion https://chdk.setepontos.com/d/7069

The G1x has a 14bit AD Converter. The pictures with +3EV are totally overexposed. Why I canâ€™t see 16384 in the RAW Data? Yes I know, it is an average of R G B Pixels.

I'm not sure what raw data you are looking at. isoinc meter value?

Is this the answer that only on color goes to full scale?

Typically, the greens clip a lot before the red on a neutral subject.

Is it possible just to meter one color?

Yes. the drtest.lua script does this, for example. In fact, I have a drtest csv file you sent me from G1x at some point. You can see the greens are clipping at +2 stops (I assume 16382 vs 16385 is probably due to some dead or weak pixels)

drtest-c_joerg-g1x.csv

2.64KB

reyalp

[unknown]
I am happy to run the test, are there any specifications or parameters you need? Or is just pointing the camera outside fine?

The subject should have constant, uniform lighting and not contain anything that moves. An LCD monitor showing a blank page or a blank wall under artificial light should be good.

Set the "ISO input type" to ISO, and enter the range of ISOs you want to test. For your test, you could just do something like 980 to 1020.

If you are only testing a small range like, you can set "Initial ev shift" to 0.

The other settings can be left default.

It's possible the "Shoot mode" setting will affect ISO override behavior.

Setting it to "cont" and making sure continuous mode is enabled in the canon firmware should be most similar to your rawopint run that showed the funny ISO jump. If "quick" doesn't show the same behavior, you may be able to avoid the problem by turning off cont mode in rawopint.

edit:
FWIW, ISO overrides behaving oddly at high values is pretty common. As mentioned in the first post, the last stop on some cameras is done entirely in JPEG processing, with no change in the raw data.

c_joerg

[left]

This is expected behavior, if the cameras current ISO doesn't match the lowest test value.

[/left]

[left]Understood. This is the d_sv96 part. It might be easier, to set this value to 0, when the camera is in M Mode.[/left]

[left] [/left]
[left]

Sorry, sometimes even a short answer takes lot of time

[/left]

[left] [/left]
[left]Itâ€™s OK. I know this. Maybe the time comes, where I can really support you.[/left]
[left]Thanks about giving me the answers.[/left]

[left] [/left]
[left]

I would guess this is just the camera being in auto-ISO at the start.

[/left]

[left] [/left]
[left]Iâ€™m really sure that the Camera was set to ISO100. On auto-ISO I could understand itâ€¦[/left]

[left] [/left]
[left]

I'm not sure what raw data you are looking at. isoinc meter value?

[/left]

[left] [/left]
[left]Yes the meter values. But I understand now. [/left]

[left] [/left]
[left]

If "quick" doesn't show the same behavior, you may be able to avoid the problem by turning off cont mode in rawopint.

[/left]

[left] [/left]
[left]But when I understand my measure correct, "quick" made it worseâ€¦[/left]

[left] [/left]
[left]

As mentioned in the first post, the last stop on some cameras is done entirely in JPEG processing, with no change in the raw data.

[/left]

[left] [/left]
[left]Yes I understand this.[/left]
[left]But everything which can be seen in the meter (or meter96) is not just a JPEG processing or?[/left]
[left]In case, where is no change in the raw data, and the raw data is not overexposed, this can be corrected when you saving the RAW files and doing the JPEG processing on a PC or?[/left]

reyalp

c_joerg Iâ€™m really sure that the Camera was set to ISO100. On auto-ISO I could understand itâ€¦

Just to be clear: using any CHDK ISO override (script or menu) puts the camera in auto-ISO, even if the Canon UI looks like it is set to some specific ISO. So if you ran insoinc twice, without rebooting or manually changing the Canon UI ISO setting between runs, the second run would effectively start in auto ISO.

But everything which can be seen in the meter (or meter96) is not just a JPEG processing or?

[left]Right, what I mean by "in jpeg" is that we don't see it in the raw data. But this isn't the the issue we see on s110 or g1x, it's just a general point that "ISO" in the Canon firmware is not only amplifier gain (as it would be in a classic, generic description of a digital imager), it's a combination of amplifier gain, software processing in the raw->jpeg conversion, and maybe other stuff that varies depending on the specific camera model. CHDK overrides do not always control all of these elements successfully.
[/left]

[left]The point of this point 😳 is that it shouldn't be a surprise when high ISO values are wonky. Specific cases should still be reported and fixed if possible, but in the end CHDK is hack and this is one of the hacky bits. [/left]
[left]

In case, where is no change in the raw data, and the raw data is not overexposed, this can be corrected when you saving the RAW files and doing the JPEG processing on a PC or?

[/left]
In cases like the D10, where the last stop is in JPEG, you might be able to recover images that were overexposed in the "ISO 1600" jpeg from raw. I'm not sure off the top of my head how this would work with rawopint. In general, I'd recommend keeping the ISO limits in the range that works normally.

MarkB

reyalp

[unknown]
I am happy to run the test, are there any specifications or parameters you need? Or is just pointing the camera outside fine?

The subject should have constant, uniform lighting and not contain anything that moves. An LCD monitor showing a blank page or a blank wall under artificial light should be good.

Set the "ISO input type" to ISO, and enter the range of ISOs you want to test. For your test, you could just do something like 980 to 1020.

If you are only testing a small range like, you can set "Initial ev shift" to 0.

The other settings can be left default.

It's possible the "Shoot mode" setting will affect ISO override behavior.

Setting it to "cont" and making sure continuous mode is enabled in the canon firmware should be most similar to your rawopint run that showed the funny ISO jump. If "quick" doesn't show the same behavior, you may be able to avoid the problem by turning off cont mode in rawopint.

edit:
FWIW, ISO overrides behaving oddly at high values is pretty common. As mentioned in the first post, the last stop on some cameras is done entirely in JPEG processing, with no change in the raw data.

I did two tests on the s110 , in both the Canon firmware was set to Continuous Mode. One test was set to "cont" in the script the second to "quick"

s110 cont isoinc.csv

1.55KB

s110 quick isoinc.csv

1.09KB

c_joerg

[left]

MarkB
I did two tests on the s110 , in both the Canon firmware was set to Continuous Mode. One test was set to "cont" in the script the second to "quick"

[/left]
[left]The results looking comparable to my camera. Both of your runs have not the same exposure time so there are not full comparable. But I looks like that the effect is independent from the exposure time, because I use 1s and you 0.05s. [/left]
[left] [/left]
[left]It looks like; there is already an offset between â€˜Contâ€™ and â€˜Quickâ€™ before ISO1000.[/left]

c_joerg

I made another 4 runs with isoinc, just for interests. Every run gives a different resultâ€¦
Combinations: Cont CHDK, Cont Script, Quick CHDK, Quick Script.
The script method of ISO settings avoids the JPG boost, but meter values going down.
For me it looks like, that there is still a real amplifier working, because meter increases in all cases.
If I understand the Dxomark ISO sensitive plot correct, then the real amplifier works until ISO6400 for the S110. But that might be a misinterpreting from my side
http://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-PowerShot-SX50-HS-versus-Canon-Powershot-S110-versus-Canon-PowerShot-G1X___848_838_769

ISOINC_S110_1s_2.csv

14.17KB

reyalp

c_joerg
For me it looks like, that there is still a real amplifier working, because meter increases in all cases.
If I understand the Dxomark ISO sensitive plot correct, then the real amplifier works until ISO6400 for the S110.

I would interpret the "script" runs as the camera turning the real amplifier gain down by 1 ev, setting the jpeg process to add a constant +1 ev, and then running the amp through the range again. IIRC this is similar to one of the things we see on G1x, but G1x did it at much lower ISO.

The "chdk" ones look like the same thing happens initially, but then the -1 on the amp is lost. The ~half stop difference between cont and quick is just weird though ???

c_joerg

I would interpret the "script" runs as the camera turning the real amplifier gain down by 1 ev, setting the jpeg process to add a constant +1 ev, and then running the amp through the range again.

May beâ€¦
But why does the cam do this so complicate? He could just go to the max real amplifier gain and add smaller steps to jpeg...

reyalp

From the g7x thread

[unknown]
For me itâ€™s important the dynamic range of a camera. According DxO the dynamic range of G7x is 2EV better than the G1x.
It would be nice if you could start drtest.lua on the camera, and post the results.

Here's a drtest from g7x, along with one from d10 with the the same script version for comparison.

One notable differences is that for this particular scene (defocused on an LCD monitor showing a blank page), cameras idea of "correct" exposure is ~2/3 of a stop lower than the "neutral" value used in CHDK. On most other cameras I've tested, the m96 value is closer to 0. The d10 is almost exactly 0, but that's because it's the camera I derived the value from.

Another oddity of this camera is that the blacklevel seems to be about 511, not 127 as seen on other 12 bit cams. This is both from testing and what is appears in DNGs converted with Adobe DNG converter.

I'll do some isoinc runs later.

drtest-g7x.csv

2.91KB

drtest-d10.csv

2.87KB

reyalp

Here are isoinc runs on g7x from ~116-424 and 394-872

I'm not sure if the glitch at the start is a bug in port or the script, but aside from that it looks very linear.

Interestingly it seems to work even in the range below the 125 minimum.

These runs were all done in cont mode.

I also did a run at from ~800-6400, but the initial exposure value was bad (likely related to the first shot glitch seen in the others) and it ended up being over exposed. I'll try to run it again later, but it looked OK up to around 4000.

isoinc-g7x-sv500-680.csv

21.03KB

isoinc-g7x-sv670-780.csv

12.97KB

c_joerg

[left]Thanks for sharing this result with us.
[/left]
[left]I notice, that drtest logs m96 now in float and not in integer anymoreâ€¦.[/left]
[left]If I understand the results correct, the G7x has 1 step more dynamic as the D10?[/left]

[left]

Another oddity of this camera is that the blacklevel seems to be about 511, not 127 as seen on other 12 bit cams.[\quote][/left]

[left]Thatâ€™s interesting for me that the cam made a higher dynamic range (12.7 EV) with a lower value range (511-4095). The G1x has (10,😎 with a value range from 511 â€" 16383. Yes, thatâ€™s not just the bits from AD converterâ€¦[/left]

[left]Iâ€™m not sure, how I can trust the following link:[/left]

[left]http://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-PowerShot-G7-X-versus-Canon-PowerShot-SX50-HS-versus-Canon-PowerShot-G1X___978_848_769[/left]

[left]
Here are isoinc runs on g7x from ~116-424 and 394-872 [\quote][/left]

[left]Have you seen any jumps in JPG results like on G1x?[/left]

[left]Would be interesting to know, if the cam make also blacklevel jumps on exposure time > 1.3s. [/left]
[left]May be you could try it with some values Tv >1.3s and higher ISO values.[/left]

[left]
One notable differences is that for this particular scene (defocused on an LCD monitor showing a blank page), cameras idea of "correct" exposure is ~2/3 of a stop lower than the "neutral" value used in CHDK. On most other cameras I've tested, the m96 value is closer to 0. The d10 is almost exactly 0, but that's because it's the camera I derived the value from.[\quote][/left]

[left]Interesting. How the JPG does look in that case (D10 to G7x)? [/left]
[left]If I have m96 = 0 than the average on JPG is close to 128. Does the cam correct the 2/3 lower stop on the JPG processing?[/left]

[left]One reason to do this is could be, that this protect the cam more for overexposure. The range from "neutral" to overexposure is larger. If you do this with another camera by setting it to -2/3EV and pushing it up in RAW processing, this gives you more noise. But maybe not on this camâ€¦[/left]

reyalp

I fixed a bug in the isoinc script that caused the initial ISO and exposure to be wrong in some cases. This probably happened in some of your runs too. Updated script attached.

I re-ran the isoinc tests on g7x. ISO seems to start at 115, and stay linear all the way up to 12800, with no jumps in the jpeg or funny stuff with the black level. All my runs have been with relatively short exposure times (all < 1 sec, and > 1/1000th for the highest ISO)

c_joerg
If I understand the results correct, the G7x has 1 step more dynamic as the D10?

It looks one more step of usable range to me, but the offset in neutral value makes it harder to compare directly.

Iâ€™m not sure, how I can trust the following link:[left]http://www.dxomark.com/Cameras/Compare/Side-by-side/Canon-PowerShot-G7-X-versus-Canon-PowerShot-SX50-HS-versus-Canon-PowerShot-G1X___978_848_769[/left]
[left][/l]

[left]I don't know, my impression is their benchmarks are well regarded, but I don't see how g7x could have > 12 stops, since my test only covers 11 and has clipping at both ends.[/left]
[left] It is interesting that their dynamic range chart for g1x DR is flat until around 400, then starts drop. The same point we see the blacklevel weirdness etc.[/left]

Interesting. How the JPG does look in that case (D10 to G7x)?

That's a good question. Just pointing them at the same blank monitor and letting them auto-expose, the g7x is a bit darker, average ~138 while d10 is ~147, but this wasn't a very scientific test.

isoinc.lua

8.78KB

c_joerg

I was just playing with my IXUS160â€¦
I made 3 isoinc runs (sv96 -> 400-600, 600-800 and 800 -1000) and drtest. Result attached in the zip file.
I thought the IXUS has only 10Bits but the results show me 12bitsâ€¦.

It looks like, that ISO works already <ISO100 but it stops exactly at ISO1600.
EXIF data is OK and works even with ISO6400.

But I see a small jump from 5 bits in the JPG Data around ISO560. Itâ€™s not much and you will not really see this when you only watch the JPGâ€™s.

reyalp
with no jumps in the jpeg or funny stuff with the black level. All my runs have

Are you sure the JPGâ€™s are OK on the G7x? How do you investigate this?

IXUS160_isoinc.zip

16.67KB

reyalp

c_joerg
Are you sure the JPGâ€™s are OK on the G7x? How do you investigate this?

Not completely sure, I just scrolled through the images, so I certainly could have missed smaller effects. I'm confident it doesn't jump a full stop or anything like that though.

I thought the IXUS has only 10Bits but the results show me 12bitsâ€¦.

Most cameras from 2009 or so have 12 bit raw.

c_joerg

This is a full isoinc run with my G1x from ISO80 to ISO10000 (in 3 steps). As you can see, as long the exposure time is less than 1s the G1x make also no funny stuff.

There only two little jumps from '3' m96 values at ISO1600 and '6' m96 values at ISO3200. But this can be fixed at post processing.

Would be really interested what the G7x will do at longer exposure timesâ€¦

I plan also for completeness this full runs with my SX230, S110 und SX50 in the next timeâ€¦

G1x_isoinc_Full.csv

79.19KB

c_joerg

This is a full isoinc run of my SX50.

Every of my cams show other funny things. This is the first time where I see a plateau in the JPG data. Itâ€™s happen around ISO160, ISO750 and ISO 3600. But this will probably no trouble for time lapse.

M96 looks Ok until ISO 3600 where a jump from 1EV step is.

SX50_isoinc_full.csv

74.79KB

c_joerg

reyalp
One notable differences is that for this particular scene (defocused on an LCD monitor showing a blank page), cameras idea of "correct" exposure is ~2/3 of a stop lower than the "neutral" value used in CHDK. On most other cameras I've tested, the m96 value is closer to 0. The d10 is almost exactly 0, but that's because it's the camera I derived the value from.

Iâ€™m not sure, if this is the same what I notice here:

I have quite a few Canon PowerShot models (G1x, S110, SX230, SX50) and a EOS1200D. And now for testing a g7x as well.

I hold the PowerShot models (without G7x) on a white wall with identical settings (Av mode with the same aperture, ISO, equivalent focal length ...), then I get on all models the same exposure time. Images are then all identical brightness and the histogram peak is in the middle. This works even with the IXUS500, when I then use the values from the EXIF for PowerShot.

In the DSLR that is different. Since exactly get half the exposure time. Even when G7x I currently this is so. Even with her you get half the exposure time (May be the 2/3 of a stop). Ultimately have these images have an identical brightness compared to the other power shots. Again, the histogram peak is located in the middle.

What insights can eventually win it? Is the EOS1200D and G7x around 1EV sensitive than the other power shots? So is ISO100 with the DSLR or G7x ultimately ISO200 on Powershot?

I again repeated the measurements yesterday, and must revise my statements above part.

Basically it is so that the light meter of EOS1200D and G1x tend (about 2/3 EV) to expose a little below. Probably about to have a little more dynamic upward.

On same manually settings, the pictures nearly the same.

c_joerg

The results look funny from M10.

How did you run?

You should definitely do it in M Mode. The light should not change during the measurement.

The easiest way: Display a white area on a monitor and place the camera directly in front of it. The focus is not important.

A run from ISO 100 ... to ISO 400 would be interesting.

isoinc.csv

5.07KB

« Previous Page Next Page »