How to subtract a dark frame without taking a dark frame - General Chat - CHDK Forum  
« previous next »
Pages: [1]   Go Down

How to subtract a dark frame without taking a dark frame

  • 9 Replies
  • 14289 Views
*

Offline LjL

  • ****
  • 266
  • A720IS
How to subtract a dark frame without taking a dark frame
« on: 30 / July / 2008, 08:53:40 »
Advertisements
Our cameras automatically take a dark frame after an exposure, with matching shutter speed, and subtract it from the raw data before processing it, when exposure is longer than about one second.

This is a crucial operation to calibrate any CCD sensor.

However, it may sometimes be impractical or undesired to "waste" a long time doing exposures on nothing, especially when taking multiple exposure with the intent of later blending them together.

Additionally, keeping the CCD sensor on for a longer time results in higher temperature increases, with paradoxically increases the image noise (I say "paradoxically" because you're doing this to reduce noise).

Taking one dark frame at the beginning or end of the photographing session is also often not an acceptable solution, both because the dark frame taken must match the total exposure time (or you risk exacerbating noise, by adding in a source with more randomness than the blend of your images) and the temperature (which definitely won't be the same before, during and after a session).


I'll present a method that should result in a decent compromise among the needs.


The data (the noise) contained in a dark frame results mainly from two sources: the CCD and the amplifier.

CCD noise increases with exposure time and temperature.
Amplifier noise is independent of exposure time, and only marginally or not dependent on temperature (article in Italian).

We can isolate CCD noise from amplifier noise by taking a bias frame. To achieve this, one can take a dark frame at an extremely fast shutter speed (the shortest exposure time allowed by the camera).
This will ideally only contain amplifier noise.

Now subtract the bias frame from the dark frame, and you have a representation of CCD noise (thermal noise).

Calculate the average value of all the pixels in the frame, and write it down. If you now take another dark frame, at the same temperature (let the camera cool down), with twice the exposure time, you should find that the average value has doubled. If you take one with half the exposure time, it should halve.

The relationship between temperature and CCD noise is less obvious, but should generally be linear.
So, take a dark frame at a significantly different temperature from the previous (but the same exposure time), and again calculate the average.
If the temperature change was N degrees, then the ratio of the second average to the first approximates the factor you have to multiply each pixel in a thermal frame for to obtain the equivalent frame when the temperature increases by N degrees.

It is possible on most cameras to obtain a temperature reading for the CCD sensor.

Eventually, you will have to add the bias frame back to the rescaled thermal frame to reproduce an artificial dark frame for any given exposure time and temperature.

You can then subtract this from your images, although to avoid introducing random noise, enough dark frames and bias frames must still have been taken to match or exceed the total exposure time.

The advantage of this process, however, is that the dark and bias frames may have been taken an arbitrarily long time before (or after) the actual exposures.

Note, however, that a bias frame is only valid for a given ISO setting, as it represents amplifier noise. It should be easy, however, to determine a linear relationship between ISO settings and bias frame values, if taking a bias frame for each different ISO setting is not desirable.



To summarize briefly; when preparing for calibration:

- take N dark frames (shutter closed or lens capped, medium exposure time) at temperature T1
- take N bias frames (shutter closed or lens capped, minimum exposure time S)
- average all dark frames together
- average all bias frames together
- subtract the resulting bias from the resulting dark
- take some (ideally N) dark frames at temperature T2, average them
- take the average pixel value of the dark at T1 (call it v1) and the dark at T2 (call it v2)
- write down the value of f=(v2/v1)*(T1/T2); this is the factor for a positive 1-degree temperature change

When processing an image (composed of at most N averaged exposures):

- call the shutter speed of each exposure Si
- write down the average CCD temperature at which the exposures were taken, call it Ti
- Multiply the dark frame taken at T1 for f*(Ti-T1)*(Si-S), clip overflowing values
- Add the bias frame to the resulting frame
- Subtract the resulting frame from your image
« Last Edit: 30 / July / 2008, 11:39:51 by LjL »
Logged
*

Offline reyalp

  • ******
  • 14111
Re: How to subtract a dark frame without taking a dark frame
« Reply #1 on: 30 / July / 2008, 17:03:51 »
Nice post.

In my testing, there is significant variation from frame to frame despite the camera reporting identical CCD temperatures. Even canons automatic darkframe (immediately after the exposure) lets some hot pixels get through on a 30 sec exposure, and the difference between 3 30 second exposures in a row is quite significant.

One thing I plan but haven't got to yet is taking a long series so the camera has a chance to reach equilibrium. The difference between frames should drop off if the difference in noise is caused by temperature changes.

Note that on my particular camera (a540) the temperature reported is fairly close to ambient (in the mid 30s when the battery temp reports high 20s) and is the same value as reported for "optical temperature", which suggests the actual sensor isn't on the CCD itself. Some other cameras seem to report much higher and faster changing values for CCD temp, which might indicate a sensor closer to the CCD.
Logged
Don't forget what the H stands for.
*

Offline LjL

  • ****
  • 266
  • A720IS
Re: How to subtract a dark frame without taking a dark frame
« Reply #2 on: 31 / July / 2008, 08:27:02 »
That's possible... or you could just have a cooler CCD with fewer components nearby, although that sounds a bit unlikely given the A540, if I'm not mistaken, is the same size as the A720 and an earlier generation.

As for the hot pixels, couldn't they be a transitory phenomenon? As in, getting stuck during the shift/readout process in a random fashion.

But what is certain is that, if your camera's CCD behavior is really similar to that of mine, and it's just the thermometer that under-reports, then in the spam of 3 30 seconds exposures things do change quite a bit.

The following is a graph of temperatures for my camera starting up at ambient temperature (28 Celsius), or maybe slightly warmer since I had turned it on briefly a couple of minutes before.

You can see that in the span of 90 seconds, a lot actually changes...

When the temperature starts dropping again, that's because the LCD (and thus the CCD) was turned off by power saving.

So when I say it might be better to spend those additional 15 seconds, or whatever, cooling off the CCD, I'm not kidding. Although of course, if the temperature reading is unreliable, that's another story...
« Last Edit: 31 / July / 2008, 09:18:52 by LjL »
Logged
*

Offline vk2him

  • *
  • 22
  • Canon S5IS
Re: How to subtract a dark frame without taking a dark frame
« Reply #3 on: 31 / July / 2008, 19:17:08 »
Quote from: LjL on 31 / July / 2008, 08:27:02

The following is a graph of temperatures for my camera starting up at ambient temperature (28 Celsius), or maybe slightly warmer since I had turned it on briefly a couple of minutes before.


Can you explain how you obtained these temperatures please? Is there a sensor you can read in the camera and if so, how do you save it to a log file?  ???

Cheers
Logged
*

Offline LjL

  • ****
  • 266
  • A720IS
Re: How to subtract a dark frame without taking a dark frame
« Reply #4 on: 31 / July / 2008, 20:58:54 »
It's explained in the link I gave in the first posting. I think you will have to wait some time, however, before the functions to read the sensors are included in any build, although I suspect that it might happen relatively fast in the collaborative build.
Logged
*

Offline reyalp

  • ******
  • 14111
Re: How to subtract a dark frame without taking a dark frame
« Reply #5 on: 31 / July / 2008, 22:37:10 »
Quote from: LjL on 31 / July / 2008, 08:27:02
That's possible... or you could just have a cooler CCD with fewer components nearby, although that sounds a bit unlikely given the A540, if I'm not mistaken, is the same size as the A720 and an earlier generation.
Agreed. The fact that optical and CCD return the same value is pretty suggestive too.
Quote
As for the hot pixels, couldn't they be a transitory phenomenon? As in, getting stuck during the shift/readout process in a random fashion.
They definitely increase with exposure time. Some random processes could do this (radiation comes to mind, although I really don't think my house is that hot ;)) but see below...
Quote
But what is certain is that, if your camera's CCD behavior is really similar to that of mine, and it's just the thermometer that under-reports, then in the spam of 3 30 seconds exposures things do change quite a bit.
Agreed, and further testing appears to confirms this.
Using a sequence of twenty thrity second exposures (10 shots with canon dark frame enabled, saving before and after)
The CCD temp (recorded just before each raw) went like this:
Code: [Select]
1-3:   36
4-9:   37
10-19: 38
20:    39
Clearly still climbing, but trailing off. The slow response again suggests the sensor is off chip.

Some notes on the actual test:
I subtracted the canon NR images from the raw images to make dark frames (call these D1 through 10). Just looking at them, there's a clear increase in noise as the sequence progresses.

To compare the noise levels, I subtracted the first, fifth and last dark frames from all the others, giving me three sets of nine difference frames each.

The temperature v.s. noise relation is definitely real (not that it was in doubt): Looking at the sequence of D2-D1, D3-D1 ... D10-D1, there's a very clear increase in lit pixels. However, that isn't the whole story.

My subtract function outputs zero if the result would be negative, so if the noise were increasing with temperature, subtracting a later image from an earlier one (e.g. D1-D10) should give a completely black frame. This isn't the case.

If the additional noise was purely random, you'd expect all of the later v.s. earlier ones to all have a similar level of noise. That is D1-D10 should have roughly the same number of lit pixels as D5-D10 or D9-D10. They don't.  D1-D10 results in many more lit pixels than D9-D10. It's not nearly as dramatic as the reverse, but it's pretty noticeable.

Jeff Medkeffs posts in this thread that you linked subtracting dark frame - Photo.net Digital Darkroom Forum mentions that on-sensor tricks complicate the process. That thread is from 2004, and it seems a fair guess that the consumer camera manufacturers have been forced to use a lot of tricks to keep up in the megapixel war.

Have you made dark and bias frames for your camera ? If so, how are the results ?
Logged
Don't forget what the H stands for.
*

Offline LjL

  • ****
  • 266
  • A720IS
Re: How to subtract a dark frame without taking a dark frame
« Reply #6 on: 01 / August / 2008, 11:23:02 »
I've done a slightly different test.

I've taken bias and dark frames using this script, which should hopefully be self-explanatory:

Code: [Select]
@title Dark Frame

rem Log console
print_screen 1
rem Enable raw saving
set_raw 1
rem Disable dark frame subtraction
set_raw_nr 1
rem Use low ISO
set_iso 1
rem Use a very fast shutter speed
rem (could be faster but I'm not sure how)
set_tv96_direct 1053
rem The lens must be capped anyway, but it won't
rem hurt to close the iris as much as possible
set_av96_direct 576

s=get_day_seconds
print "Starting at sec "s

rem Shoot (only) four bias frames for now
rem (We want the CCD to still be relatively cool
rem when starting to take the dark frames)
for i=1 to 4
 set_tv96_direct 1053
 set_av96_direct 576
 t=get_temperature 1
 print "Bias #"i" at "t"C"
 shoot
next i

s=get_day_seconds-s
print s" seconds passed"

rem Use four-second speed for darks
rem (I'd have used one second but "0" acts weird)
set_tv96_direct -192

rem Shoot sixteen dark frames
for i=1 to 16
 set_tv96_direct -192
 set_av96_direct 576
 t=get_temperature 1
 print "Dark #"i" at "t"C"
 shoot
next i

s=get_day_seconds-s
print s" seconds passed"

rem Go back to fast shutter speed
set_tv96_direct 1053

rem Shoot another eight bias frames
for i=5 to 16
 set_tv96_direct 1053
 set_av96_direct 576
 t=get_temperature 1
 print "Bias #"i" at "t"C"
 shoot
next i

s=get_day_seconds-s
print s" seconds passed"

end

(The time calculations are obviously wrong, but hey, when I realized I had already saved the whole thing, so I just left it all there).

For each of the resulting raw frames, I have used the following command line:
dcraw -D -W -4 -c $CRWFile | identify -verbose - | grep "Min:\|Max:\|Mean:\|deviation:"
to measure pixel values in the images.

The "-D" and "-W" options of DCRaw should hopefully guarantee no automatic post-processing, which seems to be confirmed by the values I get.

What follows is the script's output, joined to the output of the above command line:

Code: [Select]
Starting at sec 58798
Bias #1 at 31C, Min: 27 (0.000411) Max: 158 (0.002410) Mean: 31.8489 (0.000485) Stddev: 0.874163 (1.33389e-05)
Bias #2 at 32C, Min: 27 (0.000411) Max: 176 (0.002685) Mean: 31.8468 (0.000485) Stddev: 0.876435 (1.33735e-05)
Bias #3 at 34C, Min: 28 (0.000427) Max: 175 (0.002670) Mean: 31.8523 (0.000486) Stddev: 0.877269 (1.33863e-05)
Bias #4 at 35C, Min: 27 (0.000411) Max: 183 (0.002792) Mean: 31.8613 (0.000486) Stddev: 0.878778 (1.34093e-05)
26 seconds passed
Dark #1 at 36C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9422 (0.000487) Stddev: 0.992175 (1.51396e-05)
Dark #2 at 37C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9422 (0.000487) Stddev: 0.997732 (1.52244e-05)
Dark #3 at 38C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9433 (0.000487) Stddev: 1.00203 (1.529e-05)
Dark #4 at 39C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9376 (0.000487) Stddev: 1.00163 (1.52839e-05)
Dark #5 at 40C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9473 (0.000487) Stddev: 1.0072 (1.53688e-05)
Dark #6 at 41C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9443 (0.000487) Stddev: 1.01537 (1.54935e-05)
Dark #7 at 42C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9435 (0.000487) Stddev: 1.01805 (1.55345e-05)
Dark #8 at 42C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9558 (0.000487) Stddev: 1.0221 (1.55962e-05)
Dark #9 at 43C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.952 (0.000487) Stddev: 1.02697 (1.56706e-05)
Dark #10 at 43C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9467 (0.000487) Stddev: 1.03139 (1.5738e-05)
Dark #11 at 44C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9585 (0.000487) Stddev: 1.03898 (1.58539e-05)
Dark #12 at 44C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9553 (0.000487) Stddev: 1.04383 (1.59279e-05)
Dark #13 at 44C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9533 (0.000487) Stddev: 1.05103 (1.60377e-05)
Dark #14 at 45C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9517 (0.000487) Stddev: 1.05318 (1.60705e-05)
Dark #15 at 45C, Min: 27 (0.000411) Max: 1023 (0.01561) Mean: 31.9556 (0.000487) Stddev: 1.06043 (1.61812e-05)
Dark #16 at 45C, Min: 28 (0.000427) Max: 1023 (0.01561) Mean: 31.9559 (0.000487) Stddev: 1.06446 (1.62425e-05)
58968 seconds passed
Bias #5 at 46C, Min: 27 (0.000411) Max: 250 (0.003814) Mean: 31.8481 (0.000485) Stddev: 0.919376 (1.40288e-05)
Bias #6 at 46C, Min: 27 (0.000411) Max: 250 (0.003814) Mean: 31.8401 (0.000485) Stddev: 0.920084 (1.40396e-05)
Bias #7 at 46C, Min: 27 (0.000411) Max: 255 (0.003891) Mean: 31.8497 (0.000485) Stddev: 0.921461 (1.40606e-05)
Bias #8 at 46C, Min: 27 (0.000411) Max: 323 (0.004928) Mean: 31.8458 (0.000485) Stddev: 0.926271 (1.4134e-05)
Bias #9 at 46C, Min: 27 (0.000411) Max: 331 (0.005050) Mean: 31.8448 (0.000485) Stddev: 0.928759 (1.4172e-05)
Bias #10 at 46C, Min: 27 (0.000411) Max: 280 (0.004272) Mean: 31.8522 (0.000486) Stddev: 0.928595 (1.41695e-05)
Bias #11 at 46C, Min: 27 (0.000411) Max: 299 (0.004562) Mean: 31.8393 (0.000485) Stddev: 0.928798 (1.41725e-05)
Bias #12 at 47C, Min: 27 (0.000411) Max: 286 (0.004364) Mean: 31.851 (0.000486) Stddev: 0.932557 (1.42299e-05)
Bias #13 at 47C, Min: 27 (0.000411) Max: 322 (0.004913) Mean: 31.8352 (0.000485) Stddev: 0.934008 (1.42521e-05)
Bias #14 at 47C, Min: 27 (0.000411) Max: 339 (0.005172) Mean: 31.8446 (0.000485) Stddev: 0.936335 (1.42876e-05)
Bias #15 at 47C, Min: 27 (0.000411) Max: 357 (0.005447) Mean: 31.8399 (0.000485) Stddev: 0.939636 (1.43379e-05)
Bias #16 at 47C, Min: 27 (0.000411) Max: 334 (0.005096) Mean: 31.8376 (0.000485) Stddev: 0.939593 (1.43373e-05)
106 seconds passed
*** FINISHED ***

My off-hands thoughts are that there is remarkably little difference between frames as temperature (allegedly) increases. Mean and standard deviations remain about the same, although a slight increasing trend of standard deviations is apparent.

What does seem to change is the value of the few really hot pixels. This is not visible in the dark frames, as they're all clipped to the maximum value (it would probably have been visible with shorter exposures), but it can clearly be observed by comparing the first with the second row of bias frames.

This, I believe, shows either that the bias frames were not taken with a short enough exposure (so aren't really convincing as actual bias frames), or that my assumption that bias frames would be nearly independent of temperature was incorrect.

On the other hand, I'd have expected an overall increasing trend for the mean value of dark frames, which doesn't seem to be apparent.


When you say that your exposures at lower temperatures result in more lit pixels than higher temperature ones, my immediate thought is that there is some sort of noise reduction immediately applied by the camera software to the raw data, the amount of which depends on temperature.
This possibility is reinforced by the fact that we know that the software plays with the raw data somewhat, as it removes [not all of the] hot pixels.

This wouldn't explain why the mean value doesn't change, however; one could conjecture that the camera also subtract an amount from pixel values, which, in turn, depends on temperature, but this seems pointless, as the camera also normally subtracts a proper dark frame - it's CHDK that prevents that.
« Last Edit: 01 / August / 2008, 11:27:14 by LjL »
Logged
*

Offline reyalp

  • ******
  • 14111
Re: How to subtract a dark frame without taking a dark frame
« Reply #7 on: 01 / August / 2008, 19:12:21 »
Quote from: LjL on 01 / August / 2008, 11:23:02
When you say that your exposures at lower temperatures result in more lit pixels than higher temperature ones, my immediate thought is that there is some sort of noise reduction immediately applied by the camera software to the raw data, the amount of which depends on temperature.
I must not have been clear. The lower temperature ones have less noise total. What I was trying to say is there more difference between exposures taken far apart than ones closer together, even when the difference calculated is cooler-hotter.

It's also worth noting (as we did on IRC) that my method of subtracting canons noise reduced images may is not the same as taking straight dark frames, and may introduce more complications.

More in a bit :)

edit to add:
Here's some charts showing what I reported earlier. The data comes from the same dcraw/imagemagick script that LjL used. First is the darkframes, followed by subtracting one darkframe from the other. I haven't included the min value since it's zero. I've also omitted the temp value from all but the first chart. (I should probably have omitted max too, since it doesn't seem to do much meaningful.)

The first (darkframes, created by subtracting noise reduced from original) shows what we'd expect: There's more noise in the later images.
The second also shows what we'd expect: Subtracting the first darkframe from the later ones results in progressively more noisy images.
The third (fifth darkframe subtracted from earlier and later) shows the same patter for the later images. For later subtracted from earlier, we see something else: The standard deviation gets larger as we go farther back, while the mean gets smaller. This pattern continues in the final chart (tenth darkframe subtracted from earlier)

Next up, take real darkframes instead of my before and after hack.
« Last Edit: 01 / August / 2008, 23:51:06 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline LjL

  • ****
  • 266
  • A720IS
Re: How to subtract a dark frame without taking a dark frame
« Reply #8 on: 02 / August / 2008, 09:09:48 »
You might perhaps want to use 'pnmpsnr' (from netPBM) to compare different images, rather than subtracting.

Anyway, I don't know what it is, but there's something terribly wrong with this. You can't have standard deviations one order of magnitude higher than the ones I get (your sound more plausible to me than mine - just one level of standard deviation?!).

I took a second set of readings, this time with 20 32-seconds exposures, again with the lens capped and dark frame subtraction turned off.

The results are puzzling. The standard deviations are higher than in the previous set, but the means only marginally so. This time, a positive trend is apparent in both mean and standard deviation, but it's still unimportant and on a completely different level from your data.

The first exposure is missing because, for some reason, that raw wasn't saved.

Code: [Select]
Starting at sec 406
Dark #1 at 34C
Dark #2 at 40C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.4908 (0.000495778) Stddev: 1.56375 (2.38613e-05)
Dark #3 at 43C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.5333 (0.000496427) Stddev: 1.63857 (2.5003e-05)
Dark #4 at 45C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.5974 (0.000497405) Stddev: 1.69305 (2.58343e-05)
Dark #5 at 46C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 32.6571 (0.000498315) Stddev: 1.74886 (2.66859e-05)
Dark #6 at 47C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 32.7125 (0.00049916) Stddev: 1.79908 (2.74522e-05)
Dark #7 at 47C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.7777 (0.000500155) Stddev: 1.84263 (2.81168e-05)
Dark #8 at 48C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.8437 (0.000501163) Stddev: 1.87952 (2.86796e-05)
Dark #9 at 48C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 32.8968 (0.000501972) Stddev: 1.91737 (2.92572e-05)
Dark #10 at 49C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 32.9457 (0.000502719) Stddev: 1.94558 (2.96877e-05)
Dark #11 at 49C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.0121 (0.000503733) Stddev: 1.97701 (3.01672e-05)
Dark #12 at 50C, Min: 28 (0.000427253) Max: 1023 (0.01561) Mean: 33.0634 (0.000504515) Stddev: 2.00948 (3.06626e-05)
Dark #13 at 50C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.112 (0.000505257) Stddev: 2.03267 (3.10166e-05)
Dark #14 at 50C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.1609 (0.000506003) Stddev: 2.05815 (3.14053e-05)
Dark #15 at 51C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.2107 (0.000506763) Stddev: 2.08483 (3.18124e-05)
Dark #16 at 51C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.2573 (0.000507475) Stddev: 2.10349 (3.20971e-05)
Dark #17 at 52C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.311 (0.000508293) Stddev: 2.13119 (3.25198e-05)
Dark #18 at 52C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.3618 (0.000509068) Stddev: 2.14659 (3.27548e-05)
Dark #19 at 52C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.4006 (0.000509661) Stddev: 2.16988 (3.31102e-05)
Dark #20 at 52C, Min: 27 (0.000411994) Max: 1023 (0.01561) Mean: 33.4437 (0.000510319) Stddev: 2.19378 (3.3475e-05)
782 seconds passed
*** FINISHED ***

At this time, I won't even attempt to conjecture an explanation for the behavior you report on subtraction of former from later images.
Logged
*

Offline reyalp

  • ******
  • 14111
Re: How to subtract a dark frame without taking a dark frame
« Reply #9 on: 02 / August / 2008, 16:29:14 »
I'm actually thinking there's a bug in my subtract code (although it clearly works for most of the pixels), so take all my posts above (except for the plain darkframes which should be legit) with a grain of salt  >:(

Thanks for the tip on pnmpsnr, I'll look into it.

What do your dark frames look like if you examine them in something like ufraw ? Mine look like this (it's been re-sized and jpeged, but you get the idea)

I typically force the the R/G/B channel multipliers to 1, and set the exposure slider to 1 or 3 depending what I'm looking for. Shouldn't really matter as long as you use the same settings between images.

edit:
Turns out my problem was in how dcraw (and related programs like ufraw) deal with raw files that have zeros in the image data. For reasons not entirely clear to me, they will fill the zero values with values from neighboring pixels (this may depend on which camera dcraw decides you are using). In my subtracted frames, there would typically be a few lit pixels and a lot of zeros, resulting in massive smears of color that didn't exist in the raw data. Because it works on neighboring values, the smear can propagate itself down and to the right of a lit pixel as long as there are zeros to fill in.

edit again:
With the dcraw issue fixed, the numbers make much more sense. Charts attached below show real dark frames (not the subtract hack).
The first shows us noise increasing with temperature.
The dn-d1 shows that us pretty much the same thing as before, that the later dark frames contain more noise than the earlier ones.
dn-d20 shows that much more what we expect. The later dark frames are mostly a superset of the earlier ones. There is more difference (increasing mean and SD) between the images that have more noise overall.
« Last Edit: 04 / August / 2008, 22:46:56 by reyalp »
Logged
Don't forget what the H stands for.
Pages: [1]   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal