Black level importance in RAW - page 2 - RAW Shooting and Processing - CHDK Forum
« previous next »
Pages: 1 [2]   Go Down

Black level importance in RAW

  • 18 Replies
  • 16566 Views
*

Offline Lebeau

  • ***
  • 187
Re: Black level importance in RAW
« Reply #10 on: 10 / November / 2011, 15:51:09 »
Advertisements
By definition, black noise is present when photon is absent. At the opposite, non-black noise is present when photons are present. These definitions are superset of detailed definitions found in your links. It's just to expose black level meaning.

I see the differences too. In black area, it reduce noise. In white area, the high pixel value reduce the apperance of noise. In mid-range value, the pre-demosaicing denoiser seams to have diffixulties to reduce noise, most of these would be photonic noise.

I never said that removing the 5 lsb would remove noise. I only said that more the black level is high, more the black noise would be high, reducing the effective bits-length of the analog-digital converter.

I repeat, the precision of a 5 lsb black-level for 10b system is similar to a 7 lsb black-level for 12b system, leaving most of the weighting to 5 msb.

Concerning big company investment, it's not my concern. If they don't want to publish this kind of simple information, I don't mind. They probably already know but rush for megapixels, speed, compactness, weight, user interface, .. all these marketing stuffs, they are not so concerned about black-level.

Note, I am not talking against scientific research, I like that. I am only saying that black-level value is set upon black-level noise of the sensor once it's is converted and depend of the bits-length of the converter. Therefore, big company already know about that and set the black-level value in consequence.

I already read your links and many more before writing this.
Logged
*

Offline Lebeau

  • ***
  • 187
Re: Black level importance in RAW
« Reply #11 on: 10 / November / 2011, 16:07:31 »
Simple question for philmoz,

If you had to choose between
 a) 10b system with black-level value at 31
 b) 12b system with black-level value at 127
 c) 12b system with black-level value at 31

Which one would you choose ?

I would choose "c". Yes, "b" is better than "a" but "c" is soo much better than "b". On my knowledge, black-level noise (from many sources, upon scientific readings) is the most disturbing noise, spreaded over the full range of data level but downsized when considering signal-to-noise ratio.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Black level importance in RAW
« Reply #12 on: 10 / November / 2011, 16:40:48 »
I'm sorry; but I give up - I will never convince you it seems so I will leave you to your misconceptions.

To summarise, in your original post you claimed:
"The level of a white on 12 bits is (2 ^ 12-1). The black level for a Canon CCD is 7 bits (127 = 2 ^ 7-1). So 12 bits stored in the raw, 7 bits are random. There are only 5 significant bits, as for the 10-bit CCD."

You can make up all the rationalisations and definitions you want; but this is just complete and utter nonsense.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline Lebeau

  • ***
  • 187
Re: Black level importance in RAW
« Reply #13 on: 10 / November / 2011, 19:04:41 »
"7 bits are random" when no photons are present, that's, by definition, black level. This black-level noise propagate through all values. Other noises add and substract to this noise, generating complex noise pattern.

Don't give up and answer my simple question, please !
Logged
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: Black level importance in RAW
« Reply #14 on: 10 / November / 2011, 20:03:52 »
Quote from: Lebeau on 10 / November / 2011, 19:04:41
"7 bits are random" when no photons are present, that's, by definition, black level. This black-level noise propagate through all values. Other noises add and substract to this noise, generating complex noise pattern.
There seems to be some confusion here about what the 7 bits of black means.  Philmoz has it right - it does not mean 7 bits of noise across the whole measurement range of the sensor.   It just means that the charge level in each pixel never goes right to zero.  So as photons strike a pixel,  the energy level increases above this threshold value and every bit of the resolution is significant from there on up.   

So the idea that the bottom bits are insignificant is incorrect.  A seven bit black level does not mean seven bits of noise.  A ten bit sensor will give you 1024 discrete levels of measurement although the black level cuts off the first 128 or so levels of the range.

Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline reyalp

  • ******
  • 14128
Re: Black level importance in RAW
« Reply #15 on: 11 / November / 2011, 01:51:45 »
Quote from: Lebeau on 10 / November / 2011, 19:04:41
"7 bits are random" when no photons are present, that's, by definition, black level.
You keep saying this, but you haven't shown it, and your own sample image appears to contradict it. As philmoz points out, if your version were correct, you'd expect this to be mentioned in CCD literature somewhere (astrophotographers would certainly be onto it!), but I've certainly never seen it.

I'm not imaging expert, but my understanding is that black level is an offset, not a random value from 0-BLACKLEVEL.

If your idea is correct, a histogram of a short, low iso dark frame should be a nice broad bell curve peaking around BLACKLEVEL/2

If my version is correct, there should be a sharp peak around BLACKLEVEL and very little below (with a spike a zero for the pixels canon marked bad).

Attached is histogram from a dark frame (using the dfshoot.lua from chdkde) taken on my d10, converted to 8 bit, discarding the the upper bits (we are only interested in the lower 8 anyway)
rawconvert -12to8 -w=4104 -h=3048 -noshift -pgm CRW_0677.CRW CRW_0677_8.PGM
/ImageMagick-6.7.3-Q16/convert.exe CRW_0677_8.PGM histogram:histogram.png

CHDK black level on this camera is set at 127. The actual values (aside from the expected small spike at zero) range from about  100-150, with a very strong peak at about 130. There is essentially nothing between 0 and 100, so your claim that there is 7 bits of "dark noise" is clearly incorrect.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Black level importance in RAW
« Reply #16 on: 11 / November / 2011, 01:59:28 »
Quote from: Lebeau on 10 / November / 2011, 19:04:41
"7 bits are random" when no photons are present, that's, by definition, black level. This black-level noise propagate through all values. Other noises add and substract to this noise, generating complex noise pattern.

Don't give up and answer my simple question, please !

If you mean this question
Quote
If you had to choose between
 a) 10b system with black-level value at 31
 b) 12b system with black-level value at 127
 c) 12b system with black-level value at 31

Then c) is may be a better choice since it may have a higher dynamic range. It could also be a completely awful sensor that just happens to have lower black point. It is also irrelevant to this discussion.

So lets talk about what happens in a black image - perhaps this will help.

Attached is a graph from a completely black image taken on my G12.
To generate this I did the following:
- set ISO to the lowest in camera value
- covered the lens for the G12
- set the aperture to widest and shutter speed to 1/250
- take an image captured as RAW using CHDK
- count the number of pixels for each pixel value present in the image.

On the graph the X axis is RAW pixel value and the Y value is % of pixels in the image with this RAW pixel value.

In an ideal sensor with a black level of 127 then every pixel in this image would have a value of 127. Noise alters the value read from the sensor up or down so we get a range of values instead of a single value.

Look carefully at this graph - 99.94% of the pixels in the image fall between 123 and 132.

So the 'black noise' (to use your terminology) in this image ranges from -4 to + 5 from the 127 black level set in CHDK.

What this graph also tell me is that black level set in CHDK for the G12 should be 128 not 127.

Noise in the image is not something that uniformly affects the same bits in each pixel value as you seem to be claiming. The image sensor is an analog device - the noise comes from variations in the charge captured in each cell, leakage between cells, heat, and variations in the analog to digital conversion. The end result is a +/- deviation on each pixel by a certain amount (-4 to +5 in the above example). If you took an image of a uniform white object you would get a similiar variation across the pixels in the image.

On a dark image the deviation as a percentage of the average pixel value is higher (~9% for the above test). In a bright image where the average pixel value is 4000 say then a -4 to +5 deviation is only 0.25%. This is a simplification of signal to noise ratio (SNR).

I hope this helps clarify why your initial claim is wrong.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline Lebeau

  • ***
  • 187
Re: Black level importance in RAW
« Reply #17 on: 14 / November / 2011, 08:08:13 »
The following is based on Roger N. Clark articles

Canon's black level in raw data corresponds to the lowest digital number (DN) value of the photo sensor. In 10b system, this value is set to 31. In 12b system, this value is set to 127, about (2^(12-10) * 31).

For same kinds of photo sensors, this DN value is set to (2^(b-5) - 1), where "b" represent bit-range of the analog/digital (A/D) converter, leaving an even possible numbers of DN values (10b: 992, 12b: 3968) to be converted during de-mosaicing and color mapping processes.

Why the black level is set to this DN value instead of been set to DN zero, as per Nikon (see M. Clark' articles), leaving full DN range to count photons ?

First, and most, to have the possibility to evaluate floor noise, composed of sensor read noise, A/D conversion limitations and amplifier noise, relevant to empty cell's well, in opposition to full cell's well, ideally set to full range DN value. Also, in fact, with Canon sensors, a non-exposed sensor area exist on the left side to statistically estimate black level and its uncertainty.


From http://www.barrypearson.co.uk/articles/dng/specification.htm#areas

The following schemas represent G9 (10b) and G12 (12b) statistical results. Assuming Gaussian distribution, these black levels are respectively 32 +/- 10 DN and 128 +/- 40 DN. On similar A/D base, DN values and uncertainties are identical. Other cameras could have different values and uncertainties.
(see the first schema)
(see philmoz attachement)

Second, under some conditions, as long exposure, ambiant temperature, ... thermal noise could be modelised in conjonction with floor noise to contribute to total noise estimation. Upon M. Clarks's model, total noise is expressed as square root of the sum of photons in the cell's well (Pw), the squared floor noise (r) and the squared of thermal noise (t). [ Total = sqtr( Pw + r^2 + t^2 ) ] Photons in the cell's well are square root of the incident photons, due to Poisson distribution. Floor noise could be measured using the black level uncertainty of the black Canon's area. Thermal noise could be deduced from black level value, at different temperatures, IMHO.

Third, and therefore, black value and noises (P, r, t) estimations could be used to remove undesired randomness in DN values during de-mosaicing and color mapping convertion under some raw converters. This task is more hard with zeroed black level DN value and longer without sensor's black area (i.e.: subsequent black frame exposure), as per Nikon.

Non-zero black level DN value effectively cost, on G9 and G12, about 3% of the DN dynamic range [( black level + 1 ) / ( white level + 1 )]. In terms of photographic stop, where zeroed full range is log2 (white level +1), Canon's full range is log2 ( (WL+1)-(BL+1) ), and finally cost respectively, for G9 and G12, 0,46% over 10 stops and 0,38% over 12 stops, at the benefit of immediatly available noise information.

Concerning differences of black levels and uncertainties between colors, these could be consequent to the cell's filter transparencies and cell's sensitivity to these wavelengths (cell threshold sensitivity is in infrared wavelength), necessiting different amplifications for blue, green and red, to give similar linear response upto full well capacity.

To conclude, constant DN black level value could be used as is, but estimation of colored black level DN values could be considered by some raw converters. But, also, estimation of colored black level uncertainties could be used, before de-mosiacing, to reduce noises over the dynamic range, giving more realistic pictures at the price of about 0,4% photographic stops.

P.S.: I think it is better like that.
My error was Canon's full range is log2 ( (WL+1) ) - log2 ( (BL+1) ) instead of Canon's full range is log2 ( (WL+1) - (BL+1) ). Sorry and thank you.
« Last Edit: 18 / November / 2011, 11:49:29 by Lebeau »
Logged
*

Offline Lebeau

  • ***
  • 187
Re: Black level importance in RAW
« Reply #18 on: 18 / November / 2011, 11:53:58 »
Now, from my understanding, black level is 31 for 10b and 127 for 12b systems and becomes the zero of the dynamic range.

Read noise is the randomness about this black level and is approximatly proportional to sqrt(iso) where intensity noise is proportional to sqrt of incident photons. Right !
Logged
Pages: 1 [2]   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal