BlackLevel and WhiteLevel anomalies: IXUS 1000, SX110, SX120

koshy

1096

« on: 20 / August / 2014, 10:13:47 »

Advertisements

Out of all 112 camera models I worked with three show an anomaly in BlackLevel.

camera.h's

Code: [Select]

    #define CAM_WHITE_LEVEL                 ((1<<CAM_SENSOR_BITS_PER_PIXEL)-1)      // 10bpp = 1023 ((1<<10)-1), 12bpp = 4095 ((1<<12)-1)
    #define CAM_BLACK_LEVEL                 ((1<<(CAM_SENSOR_BITS_PER_PIXEL-5))-1)  // 10bpp = 31 ((1<<5)-1),    12bpp = 127 ((1<<7)-1)

should lead to either 31 or 127 depending on sensor bits per pixel.

For IXUS 1000 I get

Code: [Select]

  | | 16) BlackLevel = 110
  | |     - Tag 0xc61a (4 bytes, int32u[1])
  | | 17) WhiteLevel = 4080
  | |     - Tag 0xc61d (4 bytes, int32u[1])

platform_camera.h doesn't define CAM_BLACK_LEVEL so that's a puzzle.

SX110 and SX120 have something along the lines of

Code: [Select]

    #define CAM_SENSOR_BITS_PER_PIXEL       12
    #undef  CAM_BLACK_LEVEL
    #define CAM_BLACK_LEVEL                 128     // black level set to 128 per reports on the forum

in platform_camera.h It doesn't seem to make much sense to replace the default 127 with 127. Especially the "per reports on the forum" is funny. I'd like to meet the person who can see a 1/4096 difference in it.

The IXUS 1000 offset seems of a magniture that could be seen,
the SX110 / SX120 inconsistency I list just for the sake of completeness.

I have a general question on this. Does CHDK ever alter sensor pixels that go into RAW files to actually comply with the CAM_BLACK_LEVEL or are the sensor pixels already in that form. Put differently: On the IXUS 1000 if this were changed would that affect meta data only or would the actual pixel data in two hypothetical RAW files produced under idealized identical conditions differ between the CHDK version with BlackLevel = 110 and the one with BlackLevel = 127?

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

reyalp

14118

Re: BlackLevel and WhiteLevel anomalies: IXUS 1000, SX110, SX120

« Reply #1 on: 20 / August / 2014, 22:58:39 »

Quote from: koshy on 20 / August / 2014, 10:13:47

platform_camera.h doesn't define CAM_BLACK_LEVEL so that's a puzzle.

I think you are looking at trunk code, but running a 1.2 build. The 110 is defined in the 1.2 tree, for reasons unknown. This port had a long, troubled development. I expect the 1.3 version has been cleaned up by nafraf.

The 1.2 tree can be found at http://trac.assembla.com/chdk/browser/branches/release-1_2

Quote

SX110 and SX120 have something along the lines of
Code: [Select]
#define CAM_SENSOR_BITS_PER_PIXEL 12 #undef CAM_BLACK_LEVEL #define CAM_BLACK_LEVEL 128 // black level set to 128 per reports on the forumin platform_camera.h It doesn't seem to make much sense to replace the default 127 with 127. Especially the "per reports on the forum" is funny. I'd like to meet the person who can see a 1/4096 difference in it.

It's not quite that simple, because the black level setting affects how the raw processor interprets the data. If the real blacklevel is greater than the setting, then the raw processor could treat pixels that are just noise as having data, instead of being black.

Quote

I have a general question on this. Does CHDK ever alter sensor pixels that go into RAW files to actually comply with the CAM_BLACK_LEVEL or are the sensor pixels already in that form.

No, blacklevel defines the value to put in the DNG, it does not change the values of data stored when shooting. It does affect some raw operations like subtract.

A better way to get the actual black level is to use the masked areas of the sensor. The DNG spec actually has provisions for this, but we haven't implemented it in CHDK, and I don't know how many raw processors actually support it.

Logged

Don't forget what the H stands for.

koshy

1096

Re: BlackLevel and WhiteLevel anomalies: IXUS 1000, SX110, SX120

« Reply #2 on: 21 / August / 2014, 18:58:52 »

Quote from: reyalp on 20 / August / 2014, 22:58:39

Quote from: koshy on 20 / August / 2014, 10:13:47
platform_camera.h doesn't define CAM_BLACK_LEVEL so that's a puzzle.
I think you are looking at trunk code, but running a 1.2 build.

Not quite but I think something like it. It was a 1.3 auto build but it was more than a month ago that I took the photo I inspected.

Code: [Select]

  | 15) Software = CHDK ver. 1.3.0
  |     - Tag 0x0131 (16 bytes, string[16])
  | 16) ModifyDate = 2014:07:06 16:31:02
  |     - Tag 0x0132 (20 bytes, string[20])

Since the 110 is no longer in the trunk it's fixed obviously.

Quote from: reyalp on 20 / August / 2014, 22:58:39

Quote
SX110 and SX120 have something along the lines of
Code: [Select]
#define CAM_SENSOR_BITS_PER_PIXEL 12 #undef CAM_BLACK_LEVEL #define CAM_BLACK_LEVEL 128 // black level set to 128 per reports on the forumin platform_camera.h It doesn't seem to make much sense to replace the default 127 with 128. Especially the "per reports on the forum" is funny. I'd like to meet the person who can see a 1/4096 difference in it.
It's not quite that simple, because the black level setting affects how the raw processor interprets the data. If the real blacklevel is greater than the setting, then the raw processor could treat pixels that are just noise as having data, instead of being black.

Yes I am familliar with the details of the process.

The difference between 127/4096 and 128/4096 as the value to subtract remains arbitrary. Well, why not look at it for the SX110. I'll encode the sampe, image data to a Gamma of 2.2 so that the value range gets more evn in a histogram I'm going to show (opposed to linear sensor data). So we're interested in the following intensities:

127/4095=0.03101343101343101343101343101343
128/4095=0.03125763125763125763125763125763

Let's gamma encode them and put them in the 0 to 255 value range of an 8-bit histogram

(127/4095)^(1/2.2)*255=52.587
(128/4095)^(1/2.2)*255=52.775

Both round to histogram bin 53 in this form.
OK so let's get at some black data from the covered sensor bit.
exiftool -ActiveArea= -BlackLevel= image.dng will do to make it acessible and remove any BlackLevel subtraction. The data is RAW but I'll gamma encode it. I'll attach a screenshot for the SX100 and a Tiff of the crop.

Quote from: koshy on 20 / August / 2014, 10:13:47

Especially the "per reports on the forum" is funny. I'd like to meet the person who can see a 1/4096 difference in it.

Maybe it's clearer now why that was funny. The spread of the darkest pixels on the covered sensor (not afflicted by noise) to the brightest ones (afflicted by max noise there is at this ISO setting) is wider than the difference between the two values. I don't think a 128 exception makes sense to have.

Canon PowerShot SX110 IS.jpg (42.49 kB, 409x374 - viewed 49 times.)

Canon PowerShot SX110 IS.zip (20.25 kB - downloaded 23 times.)

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)

koshy

1096

Re: BlackLevel and WhiteLevel anomalies: IXUS 1000, SX110, SX120

« Reply #3 on: 21 / August / 2014, 19:24:21 »

Quote from: reyalp on 20 / August / 2014, 22:58:39

No, blacklevel defines the value to put in the DNG, it does not change the values of data stored when shooting. It does affect some raw operations like subtract.

Thanks I was uneasy if there was any pre computation of values in the CHDK DNG files. Some camera makers do things like that.

Quote from: reyalp on 20 / August / 2014, 22:58:39

A better way to get the actual black level is to use the masked areas of the sensor. The DNG spec actually has provisions for this, but we haven't implemented it in CHDK, and I don't know how many raw processors actually support it.

This struck my interest enough to find out.

Again I prepared an image (SD850 dng) to see the covered bit of the sensor

Code: [Select]

exiftool -ActiveArea="2 4 2478 3332" image.dng
I found out where a black rectangle was to use for the DNG tag MaskedAreas you mentioned and which was added in DNG version 1.1 so it's been around for close to a decade. I set the meta data up accordingly:

Code: [Select]

exiftool -MaskedAreas="8 3308 2474 3332" image.dng
exiftool -BlackLevel="0" image.dng

With this I hoped to see if the Adobe products would use the MaskedAreas. Long story short: They don't. Adobe ignores this completely. They take their own spec literally "Note that DNG writers are still required to include estimate and store the black encoding level using the black level DNG tags. Support for the MaskedAreas tag is not required of DNG readers." So they use BlackLevel and ignore MaskedAreas. No matter if I set BlackLevel higher (511) than what the masked area would yield or stripped the BlackLevel tag alltogether current Adobe products (PS / LR / DNG converter) wouldn't use the MaskedAreas tag.

After that I tried dcraw. It uses the MaskedAreas tag. So many programs based on dcraw will, too. Interestingly dcraw cannot deal with the simple BlackLevel itself at all right now. That was broken somewhere after dcraw Revision: 1.456 I found. I don't intend to fix it but did write to the author.

So it seems that it would be not too difficult to implement the MaskedAreas tag in CHDK. I could even supply the information of where actual fully covered areas are for all 112 cameras on auto build but I'm not at all sure how much good this would do. At low ISOs it seems potentially useful. At high ISOs the signal amplification makes the noise spread so wide that it's hard to work with. I'll attach a histogram screen shot of a photo from the SD850 at ISO 1600. Same processing as on the previous post. Histogram bin 53 is right in the middle of that spread. It seems Canon does fashion the data (tweak A/D conversion and sensor output) so that this BlackLevel of 127/4096 || 31/1024 all cameras share is a generally working thing.

sd850_iso_1600.jpg (16.99 kB, 269x377 - viewed 45 times.)

Logged

Koshy had a little ELPH which wasn't white as snow but everywhere that Koshy went the ELPH was sure to go. (actually an SD, but that detail ruins the rhyme...)