Improving CHDK ND filter support

reyalp

12688

Improving CHDK ND filter support

« on: 29 / August / 2017, 16:54:07 »

Advertisements

Trying to implement ND support in rawopint raised some issues. (background: https://chdk.setepontos.com/index.php?topic=12697.msg134341#msg134341

https://chdk.setepontos.com/index.php?topic=12766.msg134375#msg134375)
Edit:
Background on ND_FILTER_STATE propcase
https://chdk.setepontos.com/index.php?topic=7889.msg95186#msg95186

Issues
1) There's no reliable way to determine the ND value on cameras with both an ND and iris
2) On cameras with a "hidden" ND, there's no known way to determine the ND state set by the Canon firmware.
3) There's no way for script to distinguish between hidden and non-hidden ND cameras

Some general notes:

For cameras with only an ND filter, the ND state can be determined by comparing the AV and MIN_AV propcases.
If the ND was put in by the Canon firmware, the value can be determined by the difference between AV and MIN_AV. The values determined from this seem to be quite accurate, and vary between camera models. There is no obvious way to determine the value from script if the ND was not put in (and CHDK ND override does not set the appropriate propcase) but in many cases the values are available in the shooting.c AV table.

On cameras with manual ND control (mostly CAM_HAS_NATIVE_ND_FILTER but also on some ND-only cameras like SD990) there is a propcase which reflects the manual ND state (ND_FILTER_STATE). This is not set if the ND is controlled automatically (even on SD990, which also allows manual control of the ND)

Cameras with an ND and iris do not put the ND value in aperture exif, while ND-only cameras do.

On G7X, I thought I found propcases that reflected the ND value, however MarkB and c_joerg reported different values for the same camera model, so this may be incorrect.

There are 3 propcases that all have the same behavior. On g7x (propset 7), they are 85,93,94. On S110 (propset 5) they are 80,87,88. The lowest number is identified with the string "\r\nError : GetDeltaNdResult" in response to a propcase call. The second one similarly associated with "\r\nError : PCH_DELTAND_MEASURE"

The behavior of these propcases is: If the ND filter is selected (ND_FILTER_STATE = 1 before half press), they have a ~3 stop value after half press, otherwise 0.

Scripted ND control on manual ND cameras does not cause these propcases to update, unless ND_FILTER_STATE set *before* halfpress. The current ND override code sets ND_FILTER_STATE in half press for CAM_HAS_NATIVE_ND_FILTER

Some initial things to do:
1) Create a script like isoinc.lua to test ND value (here https://app.assembla.com/spaces/chdk/subversion/source/HEAD/trunk/CHDK/SCRIPTS/TEST/ndtest.lua)
1.1) determine if the "delta ND" propcases accurately reflects ND value. (Pretty close, maybe?)
2) Create a spreadsheet of of cameras with ND details (ND only, CAM_HAS_NATIVE_ND_FILTER, ...)
3) Investigate how the ND is used on "hidden ND" cameras, and whether it affects any propcases when the Canon firmware uses it.

I'll try to get to #1 later today.

« Last Edit: 03 / September / 2017, 21:06:58 by reyalp »

Logged

Don't forget what the H stands for.

c_joerg

Re: Improving CHDK ND filter support

« Reply #1 on: 30 / August / 2017, 03:08:26 »

On my first G1x I became on (80,87,88) the value 284 and on the second one 285 So not much differences.
On 83 of my first G1x I became always 0. On my second G1x I have a value like 65440. But I have different Firmware on my G1x.

Quote from: reyalp on 29 / August / 2017, 23:39:03

I would be surprised if there was that much variation in production of a given model, but who knows. Anyway,

From external ND filter I've heard several times, that there are the same manufacturer larger deviations.
In the G7x I had four different models. All were different from the imaging performance at 24mm. One model was the left, then the right corner better. I believe that there is a large series of deviation.

I've heard of someone who has changed the lens unit a P&S. But that was not enough. The service had to carry out a calibration. Perhaps the ND filter will be calibrated at this time.

May be there are some function of calibration in the firmware?

Quote from: reyalp on 29 / August / 2017, 16:54:07

1) There's no reliable way to determine the ND value on cameras with both an ND and iris

But when the propcases includes a calibrated ND value, why the script can’t use it?

Quote from: reyalp on 29 / August / 2017, 16:54:07

1) Create a script like isoinc.lua to test ND value
1.1) determine if the "delta ND" propcases accurately reflects ND value.

I thought about it. My idea was:
1) Make 10 shoots with ND Filter and 10 without ND Filter with isoinc under constant light condition.
2) Calculate average meter values with and without ND Filter to see the differences
3) Additional I would calculate the standard deviation to see how accurate the values are.
But the accuracy starts with the light source…
https://chdk.setepontos.com/index.php?topic=12165.msg124556#msg124556

« Last Edit: 30 / August / 2017, 03:11:18 by c_joerg »

Logged

M100 100a, M3 101a, 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

reyalp

12688

Re: Improving CHDK ND filter support

« Reply #2 on: 30 / August / 2017, 13:14:35 »

Quote from: c_joerg on 30 / August / 2017, 03:08:26

On 83 of my first G1x I became always 0. On my second G1x I have a value like 65440. But I have different Firmware on my G1x.

On g1x the values should be in 80, 87 and 88.

65440 can't be an ND value.

Quote

I've heard of someone who has changed the lens unit a P&S. But that was not enough. The service had to carry out a calibration. Perhaps the ND filter will be calibrated at this time.

May be there are some function of calibration in the firmware?

Yes, we know there is camera specific calibration, I'm just surprised the ND is included, if that is in fact the explanation.

Quote

Quote from: reyalp on 29 / August / 2017, 16:54:07
1) There's no reliable way to determine the ND value on cameras with both an ND and iris

But when the propcases includes a calibrated ND value, why the script can’t use it?

If that's what it this value, and we can find the propcase on all the different propsets, it would be a solution. What I mean was that there is no known reliable way to use in script now.
One thing we don't know is if this value is present in "hidden" ND cameras like SX260.

edit: Another thing that makes this inconvenient is that the value is only available after enabling the ND in the UI (or setting ND_FILTER_STATE) and half pressing.

Quote

I thought about it. My idea was:
1) Make 10 shoots with ND Filter and 10 without ND Filter with isoinc under constant light condition.
2) Calculate average meter values with and without ND Filter to see the differences

This is pretty much what the script I started does. There are still some issues to sort out before I post, but initial result on elph130 is the ND value from meter is within ~3 of the value calculated from the "AV" with ND and MIN_AV. I always assumed these "AV" values didn't vary between individual cameras of the same model, but I guess I don't know for sure.

« Last Edit: 30 / August / 2017, 13:31:53 by reyalp »

Logged

Don't forget what the H stands for.

c_joerg

Re: Improving CHDK ND filter support

« Reply #3 on: 31 / August / 2017, 01:46:55 »

Quote from: reyalp on 30 / August / 2017, 13:14:35

One thing we don't know is if this value is present in "hidden" ND cameras like SX260

Looks like I have two cams with "hidden" ND (SX50, SX230).

What we know is that the prop value goes from 0 to something between 100 and 300.
Is it helpful to make a script which

Code: [Select]

1) switch ND off (set_nd_filter(1)), half press
2) read all propset values, note all values with 0
3) switch ND on (set_nd_filter(2)), half press
4) read all propset values,  note all values which changes from 0 to something between 100 and 300.
5) repeat step 4 to verify

Or just write every prop value in a file to search on a PC….

Just to understand a little bit more about the stuff. What is behind a prop value? Just a specific memory or a register?

Logged

reyalp

12688

Re: Improving CHDK ND filter support

« Reply #4 on: 31 / August / 2017, 03:04:08 »

Quote from: c_joerg on 31 / August / 2017, 01:46:55

What we know is that the prop value goes from 0 to something between 100 and 300.
Is it helpful to make a script which
Code: [Select]
1) switch ND off (set_nd_filter(1)), half press 2) read all propset values, note all values with 0 3) switch ND on (set_nd_filter(2)), half press 4) read all propset values, note all values which changes from 0 to something between 100 and 300. 5) repeat step 4 to verify

Unfortunately, this doesn't help, because the canon firmware function we use in set_nd_filter doesn't set the propcases. (there is a case where CHDK sets a propcase when set_nd_filter is called, but that only applies to ones we already know about)

So first, you have to figure out what makes the canon firmware use it. I would guess the Canon firmware uses it in Tv or Av mode where the other limit is reached. So for example, if you set Tv = 1/15 in a with light so bright that even the highest aperture would be over-exposed, the canon firmware probably puts in the ND. Same if you set Av to the widest value with enough light to make the shutter go over the Canon max.

Once you've figured out how to trigger it in the canon firmware, then you can compare the propcases between the ND being active and not.

SX50 is propset 5 (like S110) and SX230 is propset 4 (like G1x) so if it works like a manual ND, the value should be 80, 87 and 88.

You should also check PROPCASE_ND_FILTER_STATE (196 for both propset 4 and 5)

Quote

Or just write every prop value in a file to search on a PC….

This can be done with a script, or the built in propcase comparison tool (http://chdk.wikia.com/wiki/PropertyCase#Discovering_PropertyCase_meanings_and_values).

If you have chdkptp set up, the devutil module makes this very easy. See https://chdk.setepontos.com/index.php?topic=12759.msg129296#msg129296

Quote

Just to understand a little bit more about the stuff. What is behind a prop value? Just a specific memory or a register?

It's some kind of value accessed by the "PropertyCase" API. The underlying source and behavior depends on the value and isn't obvious.

Logged

Don't forget what the H stands for.

c_joerg

Re: Improving CHDK ND filter support

« Reply #5 on: 31 / August / 2017, 13:31:02 »

Quote from: reyalp on 31 / August / 2017, 03:04:08

You should also check PROPCASE_ND_FILTER_STATE (196 for both propset 4 and 5)

Checked on S110 ND off 196:0 ND on 196:1

Next Test SX50:

Quote from: reyalp on 31 / August / 2017, 03:04:08

So first, you have to figure out what makes the canon firmware use it. I would guess the Canon firmware uses it in Tv or Av mode where the other limit is reached. So for example, if you set Tv = 1/15 in a with light so bright that even the highest aperture would be over-exposed, the canon firmware probably puts in the ND. Same if you set Av to the widest value with enough light to make the shutter go over the Canon max.

Correct. Canon Firmware switches ND on and off but 80,87 and 88 always 0. So, no changes in prop values.

Next test.

I went to the CHDK menu and made 3 dumps and 3 pictures (In M mode) with
1)   ND Off
2)   ND In
3)   ND Out
Interesting think
Picture ND off is correct exposed.
Picture ND In is under exposed.
Picture ND out is over exposed.
EXIF of all three pictures are the same (pictures attached).
Comparing of dumps didn’t show something about ND or?
While I do this, the cam crashes several times (romlog attached).

I put the ‘half press’ in the beginning of the script. Hope this was correct. Script attached.

SX50.zip (797.02 kB - downloaded 16 times.)

Logged

reyalp

12688

Re: Improving CHDK ND filter support

« Reply #6 on: 31 / August / 2017, 17:34:59 »

Quote from: c_joerg on 31 / August / 2017, 13:31:02

Quote from: reyalp on 31 / August / 2017, 03:04:08
You should also check PROPCASE_ND_FILTER_STATE (196 for both propset 4 and 5)
Checked on S110 ND off 196:0 ND on 196:1

Right, this one is observed to be the Canon UI manual setting of the ND, for cameras that have it.

Quote

Correct. Canon Firmware switches ND on and off but 80,87 and 88 always 0. So, no changes in prop values.

That's unfortunate

It really seems like the ND value should appear somewhere, since it's necessary for the Canon exposure calculation. You could check to see if there are propcases elsewhere using the prop compare feature or a script to dump the full range.

IIRC people did look when we first realized these "hidden" ND existed and didn't find anything.

Quote

I went to the CHDK menu and made 3 dumps and 3 pictures (In M mode) with

How did you control the ND? Using CHDK overrides (from menu or script) would probably not affect propcases normally set by the canon firmware. I would expect Canon firmware never uses the ND in M mode.

Logged

Don't forget what the H stands for.

c_joerg

Re: Improving CHDK ND filter support

« Reply #7 on: 01 / September / 2017, 01:21:28 »

Quote from: reyalp on 31 / August / 2017, 17:34:59

You could check to see if there are propcases elsewhere using the prop compare feature or a script to dump the full range.

What is the full range? 400? I thought default of the script where full range…

Quote from: reyalp on 31 / August / 2017, 17:34:59

IIRC people did look when we first realized these "hidden" ND existed and didn't find anything.

Quote from: reyalp on 31 / August / 2017, 17:34:59

How did you control the ND? Using CHDK overrides (from menu or script) would probably not affect propcases normally set by the canon firmware.

Ok, I did the in the CHDK menu…

But I did a dump before with your method in AV and TV and I couldn’t see differences. Unfortunate this dumps where overwritten. But I can do this dumps again with full range.

Quote from: reyalp on 31 / August / 2017, 17:34:59

I would expect Canon firmware never uses the ND in M mode.

How do you explain the three different exposed pictures? I understand off and in but not out…

Logged

reyalp

12688

Re: Improving CHDK ND filter support

« Reply #8 on: 01 / September / 2017, 02:04:09 »

Quote from: c_joerg on 01 / September / 2017, 01:21:28

What is the full range? 400? I thought default of the script where full range…

I wrote before I realized there were prop dumps in the zip, meant to delete that before posting. Sorry about that.

However, there are known propcase values for propset 5 up to 377, so a larger range might be worthwhile. For g7x, I dumped 1000 in various cases and the highest with a non-zero value was 617.

Quote

Quote from: reyalp on 31 / August / 2017, 17:34:59
I would expect Canon firmware never uses the ND in M mode.
How do you explain the three different exposed pictures? I understand off and in but not out…

I don't have an explanation for that

But I think M mode should use the Av and Tv you set, and that should be that, unless you have auto ISO enabled (if the camera supports it in M, which I think most powershots don't) Maybe the function we use for ND has some side effect...

Logged

Don't forget what the H stands for.

reyalp

12688

Re: Improving CHDK ND filter support

« Reply #9 on: 01 / September / 2017, 02:23:20 »

edit: script is now in svn, latest version: https://app.assembla.com/spaces/chdk/subversion/source/HEAD/trunk/CHDK/SCRIPTS/TEST/ndtest.lua
Here's the test ND test script I wrote, along with some results.

To use it, put the camera on a stable surface and aim it an a uniform, unchanging scene like a blank wall. The default options should be OK. Using "quick" or "cont" will likely give slightly more stable results, but might fail on some cameras.

The "Shots per state" option controls how many shots to take in each ND position.

The "ND shot comp EV96" specifies how much to adjust the shutter speed when putting in the ND filter. This defaults to 3 stops just to keep exposures reasonably similar. If it's set exactly equal to the real ND value, then the meter values shouldn't change with the ND state.

The last line of the log gives the average meter96 values with and without the ND, and calculates the ND value using these and the "ND shot comp" value.

Running on my cameras, I get the following
d10: 276-279 (value from AV - MIN_AV = 283)
elph130: 274 (value from AV - MIN_AV = 276)
g7x: 292 (value from DeltaND propcase = 293)

Tests were done under artificial (mix of LED and incandescent) lighting. The meter96 value generally didn't change by more than 1 over a sequence of shots in a given state.

Because the non-ND shots are taken first, there is potential for sensor temperature effects.

On both elph130 and d10, I noticed that the zoom level seems to change very slightly with/without ND.

edit:
here are images showing the with/without ND zoom effect on D10. These were taken in cont mode, so there was no refocusing etc between shots.
No ND: https://app.box.com/s/0jwn5wtiiblsflcppetb3tox6c9nnuxm
ND: https://app.box.com/s/173ggsrx85570q10h168iecrsam4fr4l

They were taken with ev shift 278, and the levels are virtually identical. The shots before and after do not have the zoom change.

ndtest.lua (8.94 kB - downloaded 17 times.)

ndtest-results-reyalp.zip (3.37 kB - downloaded 16 times.)

« Last Edit: 03 / September / 2017, 21:05:45 by reyalp »

Logged

Don't forget what the H stands for.