Dark Noise Analysis (or - optimizing noise for low light/stars) - RAW Shooting and Processing

jmac698

Dark Noise Analysis (or - optimizing noise for low light/stars)

« on: 14 / July / 2016, 19:16:53 »

Advertisements

Hello,
I've been doing some deep analysis of dark noise statistics on my cameras, and made some surprising discoveries.

Introduction
It's typical in astrophotography (specifically, taking long exposures for nebulas or stars), to prepare calibration files like this:

flat frames
a picture of something grey, as a calibration for the astronomy program to remove vignetting (vignetting is the darkness at the corners of a raw pic, that you don't usually see in a calibrated program, but it's there, as a kind of shadow from the lens)

bias frames
black pictures at the shortest shutter speed. This is supposed to have no dark current noise, but does show any slight variation in the brightness of each pixel.

dark frames
black pictures to find the level of noise. affected by temperature, exposure time, and ISO.

Why take these pics of grey card and some black ones?
The astronomy program can remove 'hot pixels' and reduce noise in the most optimal way. It really works; you can see the difference in the pics even on a high-end camera, and both the bias and dark frames make a difference. The nebula comes out less clouded and more contrast, and less noisy of course.

The Work I've Done
I've taken black, bias, and white (not flat) frames to check the ranges of noise and levels in raw, at different ISO

My Results

-The bias frames, which are supposed to have the least noise, actually have more noise than a dark frame. The only vague theory I have to account for this is that the camera switches to a faster clock rate for the ADC in order to support the quicker shutter speed, and the ADC gets noisier.

A Note About the black level area first

There are some areas which are covered by metal on the chip, so are always in darkness. These are analysed everytime to find the average value to be considered as 'black'. For example, if the average is 128 at ISO80, then every value in the imaging/active area will have 128 subracted, thus leading to 0, for a black picture. This is needed because the black level changes with each picture (by temperature, iso and exposure).

I found that there are some hot pixels (pixels stuck at always the same value) in the black area, so this ruins the black level calibration by some small amount.

-The noise can become so large, that it swings to less than 0. For example if the (true) average is 128, and the noise varies by +-500, then some pixels become less than zero. This affects the average! You are now trying to read an average with some values clipped, which gives you a value higher than it should be! I can actually see this, with averages coming out 4 higher than they should be, ruining the deepest shadows of your pic.

-In the highest ISOs, the colours have different stats! This is strange, as pixels are supposed to be all the same with only come colour filters over them. My theory for this, is that there is an analog gain amplifier somehow connected to each colour, for the purpose of daylight balance.

-Sometimes the picture is too bright! By a level of about 2. My theory for this is that sometimes the shutter speed is inaccurate. The impact of this, is that creating your calibration frames will be wrong by some amount. The bright frames should be thrown out.

The Work
These are stats for an SX50HS with 1/50s, with the lens covered.

Code: [Select]

    white    black                bias
iso    max    min    max    avg    std    min    max    avg    std
80        97    185    127.5    1.60    101    181    127.6    1.71
200    4092    74    191    127.3    3.61    
400    4091    43    240    127.3    6.53    
800    4089    21    261    127.7    11.9    24    263    127.6    13.0
1600    4084    0    338    127.3    23.1
3200    4074    0    648    127.4    43.8    0    643    128.5    47.3
6400    4095    0    1336    126.7    86.9

So what does this table mean? For each ISO, I took 16 pics, then found the highest and lowest pixels values of the active/imaging area in all the pics. I also came up with general averages over all pixels (this actually varies by colour) and the worse standard deviation I found (the higher, the more noise).

What can be learned from this table

-Noise increases linearly with ISO. If you double the ISO, you double the noise. Except not as much as you expect; it's like a higher ISO is more 'efficient'. I can't think of a way this helps yet.

-Starting at ISO1600, black frame statistics can go wrong, due to low values being clipped.

-Bias frames are about 9% noisier

-White pixels don't always saturate the ADC. The least dynamic range is at ISO3200 (this may not really be true overall, I have to think about it more). I would explain this by the steps in gain settings not being accurate?

-I didn't show it in the table, but at lower ISO's there's some white pixels somewhere that are dark, I didn't manage to see it myself yet though. At higher ISO, every single pixel is exactly white. The problem could be I didn't make the light bright enough to account for the vignetting, though I didn't see any vignetting.

The difference by colours

Code: [Select]

iso    colour    min    max    avg    std
3200    B    0    430    128.1    32.7
    G    0    502    129.6    43.7
    G2    0    538    129.8    43.7
    R    0    406    130.1    41.6

6400    B    0    768    127.1    85.5
    G    0    1336    127.0    86.9
    G2    0    1064    129.5    86.9
    R    0    736    130.9    86.8

-There is a huge difference in the max values, with green pixels being twice as large as the others.
-Green pixels always have similar noise
-Blue is least noisy, maybe it's boosted less? I would have to look at the daylight coefficients to see if my daylight balance idea is true.

Logged

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #1 on: 14 / July / 2016, 19:17:30 »

Part 2

Further Directions

-To look for a pattern in noise through different shutter values, and maybe explain why shorter shutters are noisier

-To explain the difference in noise by colour by experimenting with white balance settings

-To profile dark noise by ISO, shutter time, and temperature. I already have some dark frames and logs of the sensor temperature, for 0-25 degrees, by leaving the camera in the freezer (yes it still works!

-To write a script with RAWOPS etc. to do the entire calibration automatically, in-camera, and even fix your photos with no external program! I'd love this for perfect night photos and no external programs needed. This is huge actually, I'll tell you why.

If you import your RAW into DXO Pro program, for example, then you have to export each correct (vignetting, lens distortions, sharpness) pic, then average them for noise, that's slow. Instead, you can export a NOISELESS RAW direct from you camera, then apply all the benefits of the RAW editor with one pic.

The other way of doing it, stack in some astronomy program, works for noise fine but then you can't -export- as RAW again to the other program (DXO etc) to apply the fancy corrections to it... so basically it's a choice now, either noiseless, or corrected photos (even the corrected photos, then stacked, you lost precision there too)...

Kinda hard to explain but this will give you the best possible pictures, both noiseless and corrected.

What can be accomplished by this research?

-highly optimized noise and defect management beyond what any program currently does, as I doubt anyone has worked in so much detail. I don't know if the difference will be noticable yet.

(A Note About Taking Dark Frames)

I want to explain that it is usual to have to take dark frames at the same time as your star pics, to be in the same conditions of temperature and settings. This doubles the time of your pics, and can also ruin your star trails etc. because of the interruption in between.

-with an exact formula for the dark current level at all iso, shutter speed, and temp, you never have to take a dark frame again! Such analysis have been done for, for example the Juno camera aboard a satellite around Jupiter right now, had such a calibration done. This is NASA level calibration I'm doing here.

If you are one of those people who go 'why?' and 'the differences aren't worth it', or 'too much trouble' I say, you just don't know until you do the research, there can be surprises! Also I do it because of the 'INTP' or Scientist personality type, which is why few people bother with such things. It may seem like a waste of time, but I can enjoy it and that's normal but rare personality type

The science itself is interesting.

Logged

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #2 on: 15 / July / 2016, 04:58:30 »

Part 3

To recall, I advertised this continuing research:

Further Directions

-To look for a pattern in noise through different shutter values, and maybe explain why shorter shutters are noisier

-To explain the difference in noise by colour by experimenting with white balance settings

-To profile dark noise by ISO, shutter time, and temperature. I already have some dark frames and logs of the sensor temperature, for 0-25 degrees, by leaving the camera in the freezer (yes it still works!

I now have quick answers to those questions.

ISO3200 std G/G2

Code: [Select]

Tv    22deg        0deg    
1/1600    47.3        44.4
1/800    44.8        42.4
1/400    43.4        41.1
1/200    43.4        41.2
1/100    43.3        41.3
1/50    43.3
1/25    43.3        41.3
1/13    43.2
1/6    43.1        41.4
1/3    43.1
1    43.1        41.5

What This Table Means
This is the noise by shutter time, at two temperatures. The warm test was done from long to short shutter time, the cold test from short to long shutter time. This was done to account for the self-heating/warming up effects on the noise.

What You Can Learn From This Table
-Self-heating has a measureable effect on noise, even over a minute.
-Both bias and dark frames are affected by temperature. The assumption that bias frames are little effect by temperature are false! This means they should be taken at the same time as dark frames, for best calibration.
-This is very important: the standard deviation of noise changes at .145/deg C. This important value is what's essential to avoiding taking dark frames ever again! I don't expect this value isn't linear, I need to do more careful tests.
-1/1600 and 1/800 are very noisy for some reason! I think bias frames should be taken at 1/400
-Dark current wasn't really noticeable up to 1"
-You can reduce noise by up to 3DN by putting you camera in the freezer first! The value 3 is in Data Number, or the actual raw pixel value.

Not Shown
-The average value can be as low and 125 and high as 131 or so.

The Test of WhiteBalance Affecting Noise

-my theory was correct, WB setting affects RAW noise:

Code: [Select]

Tungsten
iso    colour    min    max    avg    std
3200    B    0    449    126.0    40.3
    G    0    525    127.5    42.7
    G2    0    493    127.8    42.7
    R    0    401    128.2    40.6

What This Table Means
The stats for the Tungsten WB setting, which is
WB RGGB Levels Tungsten : 1135 782 782 2421

Compare to:

Code: [Select]

Daylight
iso    colour    min    max    avg    std
3200    B    0    430    128.1    32.7
    G    0    502    129.6    43.7
    G2    0    538    129.8    43.7
    R    0    406    130.1    41.6

*WB RGGB Levels As Shot : 1636 922 922 1636

What Can Be Learned From These Tables
-Now that we know temperature affects standard deviation dramatically, these runs can't be compared directly for their noise statistics
-If you look at the relative noise of Blue, in each case it is the least, but dramatically lower in the Daylight setting.
-The WB setting for Blue is (2421/782)/(1636/922)=1.74 more, and the noise is:

40.3/42.7=.93
Noise of blue/Noise of Green, for Daylight
32.7/43.7=.75
Noise of blue/Noise of Green, for Tungsten

So the noise is .93/.75=1.26 less. It's hard to say what this means, I will have to do it again in a temperature controlled way, also there could be a base (read noise) noise to account for the difference.

« Last Edit: 15 / July / 2016, 05:03:57 by jmac698 »

Logged

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #3 on: 30 / August / 2016, 18:23:02 »

So, been doing some more research. The dependency of dark current on temp is exponential, and about doubles every 5 degrees. Fortunately, I found a way to avoid calibrating by temperature, which saves a whole lot of trouble. There's an algorithm to find the best scaling of a dark frame to a particular temperature automatically.

Add on top of that, that the scaling is linear to exposure time, and probably ISO. I just want the true calibration to prove that these estimates are reliable.

In the end, all that's really needed is one master dark frame, bias frame, at a stable temperature, and I calculate about 200 of them to be noise-free.

As for temperature, it follows an exponential curve, and the camera self-heats as it's taking frames. In some cases, it's stable for only 2 minutes, but as it reaches equilibrium, I had up to 17 minutes to take frames. This is not nearly enough for a large set of 10 minute darks. I had the idea of opportunistic frames, for example just take frames continuuosly and cycle the temperature several times in the fridge, then sort by temperature later until enough at each temp is collected. Probably what I will do is just use two temps, equilibrium at room temp and fridge temp.

I've also been learning imagemagick, and could prototype all this into a script. I love IM for experimenting. I could probably use a bit of Matlab for some stuff as well.

I'm also looking into kappa-sigma clipping and Artificial Skepticism, these are like median but better.

I also found that Pixinsight is an amazing, highly developed program with all the best ideas, highly recommend it for Astrophotography (I would buy it), but still I have a few ideas better than what they do, they are about 95% there.

I also found out how to calibrate well capacity and sensitivity in electron volts. For this you need some flat frames. You can't avoid vignetting (with my point and shoot), but I had an idea, just make a pattern on the monitor that's the inverse of vignetting to get a pretty good flat frame

This won't help you calibrate your vignetting, it's only for flat frames for the purpose of linearity and electron volts sensitivity of the sensor.

I say this, because even though you can put an inverse vignette on your monitor, there's no telling if the monitor itself has even brightness, in fact it doesn't and that's proven in monitor reviews, where up to 10% brightness differential is in the corners. I found suggestions for, a) the horizon at dawn b) rest the lens against an EL light (I've seen square nightlights like this), or c) out of focus against a white wall or piece of paper. My own idea was d) out of focus pointing at a high brightness card stock on a sunny day with the sun at meridian. You have a few minutes where the sun hardly moves and the light should be very even, if you look straight down at the paper.

Meanwhile, I've started on my other camera. This one is the model before, and wow the sensor is so much worse. There's 9 stuck pixels, 7 stuck to black and 2 stuck to white, and a complete range of random values. It's so bad I wonder how it can even make a decent picture. I'm trying various techniques to detect bad and noisy pixels. For example, I looked at 3 frames in a row to see how many pixels were at the same value, and there's a lot, more than just the stuck pixels. I'm working on some very robust outlier detection, and proving which pixels are actually responding to light etc. I'm working on the theory that some pixels are just random values all the time, so that's another possible class of bad pixels.

« Last Edit: 30 / August / 2016, 18:33:51 by jmac698 »

Logged

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #4 on: 18 / September / 2016, 13:45:42 »

Just an update on my research.

I've given up on imagemagick. It takes a long time to figure out the command lines, but the worst problem is it buffers an entire image sequence before doing anything to it, even writing stats of each image. I made my own program instead, now processing can complete and is faster.

I noticed something I didn't explain above, I listed a bunch of numbers after white balance, these are multipliers to the raw pixel values. Another update, it is true that large telescope calibrate their flats against the sky at twilight, almost straight up, but this isn't applicable to cameras because the brightness varies a few percent per degree, it works for big telescopes because of their zoom level. Even then, that's not accurate enough for their needs, and they do some fancy stuff with reference stars. (photometric calibration). And, the reason for flat frames isn't just for vignetting, it can be used to find pixel response non uniformity (which means each pixels is a slightly different size or has different chemical makeup making it more or less sensitive to light), and can also be used to calibrate well capacity and sensitive to electrons.

I gave up using the fridge for getting a stable temp, as it turns on and off, the temp varies quite a bit. It's better to let the camera stabilize at a steady ambient, such as indoors or outdoors. I saw some small dips outdoors, I think even the wind can change the temp, so shield from that also. The standard deviation is extremely sensitive to temperature, I could see it changing perhaps every 1/10 degree while the sensor temp readout was steady. I could probably read temp directly from sensor stats to within 1/3 degree with little noise. Also the sensor temp readout and sensor stats implied temp don't match, there seems a bit of themal load between them. By this I mean, that the stats reacts to changes in ambient quicker than the sensor. It's so sensitive, I might even be able to make a thermal image from different regions. What a unique idea - thermal imaging from noise!

Also found out that stuck pixels aren't always stuck! If it's cold enough, they unstick, perhaps they will respond to light now?

So far, I only wrote part 1 of my program, which let me screen some pics to get very steady temp/stats, but now I have GB and hundreds of pics I have to average, median, and get more stats, and I have to write some more software. It won't be long now until I can finally make some pics with all my work!

« Last Edit: 18 / September / 2016, 14:09:35 by jmac698 »

Logged

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #5 on: 29 / September / 2016, 11:55:21 »

Anyone else find crop circles in their master darks?

https://s4.postimg.org/nmmbmj8kd/circles.jpg

I can't explain why there'd be perfect circles on my sensor. The circles extend to the shielded black area as well. There's also column noise, which is to be expected, this is from differences in the parallel adc's.

I've seen such a pattern come up in polishing something to be flat though.

Logged

reyalp

11546

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #6 on: 29 / September / 2016, 16:46:31 »

Quote from: jmac698 on 29 / September / 2016, 11:55:21

Anyone else find crop circles in their master darks?

https://s4.postimg.org/nmmbmj8kd/circles.jpg

I can't explain why there'd be perfect circles on my sensor. The circles extend to the shielded black area as well.

Yes, this is quite pronounced on g7x. My theory (based on some comments by srsa_4c) is that this is part of some very coarse vignette correction done in readout, essentially adjusting the amplifier through a preprogrammed profile as the sensor is read out. If this theory is correct, the circles should probably change depending on the zoom stetting.

edit:
https://chdk.setepontos.com/index.php?topic=6725.msg118032#msg118032

« Last Edit: 29 / September / 2016, 16:53:53 by reyalp »

Logged

Don't forget what the H stands for.

jmac698

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #7 on: 29 / September / 2016, 17:54:45 »

Very interesting! I'd thought it might have something to do with vignetting, but I wasn't sure how. I noticed it was too perfect to be anything natural. I will have to experiment with aperture and zoom now.

This problem of generating darks is much more complicated than it seems at first. I'm having to solve simultaneous equations to separate various effects. Also the amp glow responds differently to temp than the dark noise, so that has to be separated too. I can't even measure the magnitude of noise without separating out other things first.

But it makes it an interesting challenge

Here's standard deviation vs temperature. It shows a sequence of 1000 pictures with a few hundred at a steady temperature, however the noise is extremely sensitive and loosely tracks the temperature, it always responds more quickly than the temperature reading. For example over the course of 30deg, you can see the s.d. slowly march up by the sub degree. The relation is exponential which is why it's so sensitive.

https://s12.postimg.org/iq83mxg7h/std_vs_temp.jpg

and here is some strong amp glow from several minutes exposure:
https://s18.postimg.org/nwyd1dgsp/amp_glow.jpg

I expected amp glow to react basically the same as light, but I don't know why it wouldn't affect the shield pixels area.. hmm.. maybe the metal shields the electric field, so it's both a light and EM shield.

Also I wonder if sensors can have some kind of burn-in? I've read that a dark frame calibration can be valid for several months, I'd like to track to see how mine change. I actually have already, since I posted some stats here from one of my cameras years ago, and it did change a little bit. I'm so glad I tend to write complete and specific info in my posts, mostly for myself I think!

Another thing that is going on, I've noticed that the stats are not repeatable, with the same settings and temp, they vary a lot more when warmer, then follow temp trends as usual, that is it might start out a lot higher than normal for example. Haven't looked further, I can't think of why though. Things warming up at different rates would eventually reach a steady state...

« Last Edit: 29 / September / 2016, 18:11:38 by jmac698 »

Logged

reyalp

11546

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #8 on: 29 / September / 2016, 23:00:44 »

Quote

Another thing that is going on, I've noticed that the stats are not repeatable, with the same settings and temp, they vary a lot more when warmer, then follow temp trends as usual, that is it might start out a lot higher than normal for example.

FWIW, I'm pretty sure I once saw an analysis by an astronomer showing that some Canon DSLRs exhibited different noise depending on battery state of charge, but I neglected to save/bookmark it at the time, and wasn't able to find it again later.

Logged

Don't forget what the H stands for.

a1ex

671
ML dev

Re: Dark Noise Analysis (or - optimizing noise for low light/stars)

« Reply #9 on: 30 / September / 2016, 09:20:51 »

Quote from: jmac698 on 29 / September / 2016, 17:54:45

the noise is extremely sensitive and loosely tracks the temperature, it always responds more quickly than the temperature reading.

You might be able to use the stdev from optical black areas as an estimation of temperature.

Beware of hot pixels, as they can throw off your stdev. You cannot use MAD here, because it will only return small integer values, but trimmean should do the trick.

Here's another useful trick regarding hot pixels: http://www.photonics.com/Article.aspx?AID=44298

(they can be measured and provide some estimation of the dark current)

Regarding vignette correction, you might be able to disable them from engio registers.

If these gains are really applied during readout, it may be interesting to see their effect on noise, (they might have some potential for improving dynamic range).

Logged