Bad Pixels: Unusual Behaviour

SkepticaLee

50

Re: Bad Pixels: Unusual Behaviour

« Reply #20 on: 17 / October / 2013, 12:38:17 »

Advertisements

@rkomar
Looking at the images in greater detail, UFRaw does not even remove the bad pixels, but it doesn't turn them into huge blobs as DCRaw does.

@ Everyone Else

I have tried out rawtherapee (RT), and it has succeeded in removing the raw pixels. In the documentation (which is still fairly poor) there are a couple of links to programs that will detect and list hot pixels.

There are a couple of tricks that you need to get RT to do what you want as a batch program. The most important is to modify the file default.pp3, which RT installs in the subfolder "Profiles". You can edit this file by hand, but probably a safer way to do this is to load a raw image, then load the profile "default" and modify it from there, so that you can see what the effects will be on your images. To be on the really safe side, make a copy of the file "default.pp3".

Once the profile is loaded, make some changes that you will want to perform on all the images you run through your batch, e.g.
1) Go to "Raw", "Preprocessing" and check the box "Hot/Dead Pixel Filter";
2) Go to "Detail", "Noise Reduction" and check "Enabled". Also the Luminance value has to be increased to 20-30 for some modest but effective denoising.
I also enabled "Defringe" in the same menu.

Save the profile as "default.pp3" in the original folder, overwriting the original.

Now run a batch like this:

Code: [Select]

"C:\Program Files\rawtherapee\rawtherapee.exe" -d -c *.DNGand you should have all your DNGs converted and tidied up as JPGs.

Now on to the remaining open questions of where the dead and hot pixels come from and how they are influenced by shooting parameters. Stay tuned!

Logged

rkomar

98

Re: Bad Pixels: Unusual Behaviour

« Reply #21 on: 17 / October / 2013, 13:04:23 »

Quote from: SkepticaLee on 17 / October / 2013, 12:38:17

@rkomar
Looking at the images in greater detail, UFRaw does not even remove the bad pixels, but it doesn't turn them into huge blobs as DCRaw does.

Are you using the patched version of ufraw on DNG 1.3 files?

Logged

reyalp

14080

Re: Bad Pixels: Unusual Behaviour

« Reply #22 on: 17 / October / 2013, 16:21:38 »

Quote from: SkepticaLee on 17 / October / 2013, 03:35:06

Quote from: reyalp on 16 / October / 2013, 16:12:19
I would guess this is "preview" sub-ifd and the full resolution image is still in there somewhere. What size are the files?
@reyalp:
After "converting" DNGs to DNGs in Adobe DNG Converter what you see is the JPG "thumbnail" @ 768 * 1024 pixels. By setting the switch "-p2" on the command line the thumbnail is then full size, but it is still the JPG image, and not the raw image.

Sorry I wasn't clear, I was asking about the byte size of the file. TIFF (and DNG since it's a TIFF extension) allow embedding multiple images in a single file, in a number of different ways that can get quite complicated. Many programs will only recognize or display a subset of these. It is very unlikely that Adobe's converter actually re-sizes the raw data, that would completely defeat the purpose of using a raw format like DNG.

Logged

Don't forget what the H stands for.

SkepticaLee

50

Re: Bad Pixels: Unusual Behaviour

« Reply #23 on: 17 / October / 2013, 17:12:48 »

@rkomar

I have tried the latest version of UFRaw on both DNG 1.1 and 1.3 and inspection of both types of converted images shows that the hot pixels are reduced in size/intensity, but not removed.

@reyalp

Yes, I sort of wondered what the point was in converting DNG to DNG, if all you got out of it was a smaller image. In any case I think the Adobe DNG Converter has outlived its usefulness.

@ those interested in developing CHDK further...

I would like to propose the following:

1) The process for identifying dead pixels could be renamed "Dead pixels" instead of "Bad pixels", in order to separate two issues: one where pixels consistently give a zero value and two where pixels give inaccurate values (hot pixels). Could the file currently named badpixel.bin be renamed to dead_pix.bin or something similar?

2) In addition to the binary file, it would be useful if a text file of dead pixels could be generated (with the format "x-value space y-value EOL") with a current datestamp, so that this file could be retrieved and used together with rawtherapee to eliminate the dead pixels for shots taken between various dates. E.g. today's file could be called dp131017.txt and saved to a dedicated subdirectory.

If these suggestions could be effected then I think that the dead pixel issue would be closed. As it stands this is probably the lesser of the two issues that I have raised, and if it could be closed then I think we can move on to the other issue.

Thanks to all contributors so far and stay tuned.

Logged

reyalp

14080

Re: Bad Pixels: Unusual Behaviour

« Reply #24 on: 17 / October / 2013, 22:47:42 »

Quote from: SkepticaLee on 17 / October / 2013, 17:12:48

1) The process for identifying dead pixels could be renamed "Dead pixels" instead of "Bad pixels", in order to separate two issues: one where pixels consistently give a zero value and two where pixels give inaccurate values (hot pixels). Could the file currently named badpixel.bin be renamed to dead_pix.bin or something similar?

Technically it could be, but it's not something I would spend my time on. The effort would be better spent improving the documentation IMO.

The badpixel (no .bin) file exists to deal with hot pixels. Despite the earlier confusion, this should work.

Quote

2) In addition to the binary file, it would be useful if a text file of dead pixels could be generated (with the format "x-value space y-value EOL") with a current datestamp, so that this file could be retrieved and used together with rawtherapee to eliminate the dead pixels for shots taken between various dates. E.g. today's file could be called dp131017.txt and saved to a dedicated subdirectory.

Are you thinking this would be by a menu item like chdk badpixel.bin?

Some way of generating dcraw and rawtherapee compatible badpixel lists could be useful. However, this might be better done with an external tool rather than inside CHDK. I suppose one could also write a lua script that reads badpixel.bin and spits out text.

Logged

Don't forget what the H stands for.

SkepticaLee

50

Re: Bad Pixels: Unusual Behaviour

« Reply #25 on: 18 / October / 2013, 09:59:09 »

Quote from: reyalp on 17 / October / 2013, 22:47:42

The badpixel (no .bin) file exists to deal with hot pixels. Despite the earlier confusion, this should work.

@reyalp
But is this a file that you have to write yourself? Can you point to the documentation of the format of this file?

« Last Edit: 18 / October / 2013, 11:38:15 by SkepticaLee »

Logged

lapser

1093

Re: Bad Pixels: Unusual Behaviour

« Reply #26 on: 18 / October / 2013, 10:38:00 »

Quote from: SkepticaLee on 18 / October / 2013, 09:59:09

Quote from: reyalp on 17 / October / 2013, 22:47:42
The badpixel (no .bin) file exists to deal with hot pixels. Despite the earlier confusion, this should work.
@reyalp
Where is this file?

Looking at the CHDK code, you would create the text file:

"A/CHDK/badpixel"
or
"A/CHDK/badpixel.txt"

CHDK reads this text file at startup and converts it to binary data to use for patching raw pixels.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

reyalp

14080

Re: Bad Pixels: Unusual Behaviour

« Reply #27 on: 18 / October / 2013, 16:08:40 »

Quote from: SkepticaLee on 18 / October / 2013, 09:59:09

Quote from: reyalp on 17 / October / 2013, 22:47:42
The badpixel (no .bin) file exists to deal with hot pixels. Despite the earlier confusion, this should work.

@reyalp
But is this a file that you have to write yourself? Can you point to the documentation of the format of this file?

The format is just text x y coordinates, one per line.

Some documentation can be found at
http://chdk.wikia.com/wiki/CHDK_User_Manual#Manual_bad_pixel_removal
http://chdk.wikia.com/wiki/Badpixel_removal
http://chdk.wikia.com/wiki/CHDK_firmware_usage/AllBest#Hot-Pixel_Removal_.28Build_100-16_and_later.29

There are some tools for creating linked from there, though both the tools and documentation are pretty out of date.

As I mentioned earlier, it's my intention to add the capability to generate this file to chdkptp.

Logged

Don't forget what the H stands for.

reyalp

14080

Re: Bad Pixels: Unusual Behaviour

« Reply #28 on: 19 / October / 2013, 23:20:28 »

I did some more digging in the badpixel / bapixel.txt system

* The actual format is text x,y with one coordinate per line. Spaces are allowed, and stuff following the y is ignored. Both DOS and unix text should work.

* The pixels do get patched

* The input file is arbitrarily limited to 8kb, although I think if you have both badpixel and badpixel.txt they will effectively be combined.

* The text file is parsed into a linked list, which is not very memory efficient. You wouldn't want to use the data from badpixel.bin in this format

* The file(s) are loaded at startup. It would be better to only load them when the option was enabled.

* The show_bad tool suggested in the docs is hard coded to for a few framebuffer formats. It will not work for most cameras without modification.

* show_bad outputs in the format x,y=value. This is not a problem since everything from the = to the end of line is ignored, but it counts against the 8kb limit.

Logged

Don't forget what the H stands for.

SkepticaLee

50

Re: Bad Pixels: Unusual Behaviour

« Reply #29 on: 22 / October / 2013, 04:57:42 »

Thanks to reyalp for clarification on the badpixel/badpixel.txt file.

I have put up a php script that will translate badpixel.bin to badpixel.txt at http://www.skeptic.de/CHDK/Bad%20Pixels/index.htm#Index_3.1 (the page has been extensively revised, so it might be worth your while to have a look at the whole document). However interesting that may be, it still won't solve the problem of fixing hot pixels.

Even if you could get the position of the hot pixels (but not via CHDK), then the 8kB limit on badpixel/badpixel.txt, and the format of these files, means that you could only deal with around 1,600 hot pixels. I would also question the wisdom of trying to do this on the camera. When I am out in the field taking photos (and carrying around sufficient batteries for the task), I am not going to waste battery power trying to fix a small proportion of the hot pixels, something which could be done more comprehensively on a real computer.

Since the JPGs that the camera produces do not seem to have any of the hot pixels that are quite visible on the DNGs, my question now is: Where does the camera store this information, and can it be retrieved? That might be a task worthy of CHDK. Of course, you could experiment with removing hot pixels with rawtherapee, but you would only be relying on whatever algorithm they have. The camera seems to deal with hot pixels accurately enough. I want to know how.

Logged