Bad Pixels: Unusual Behaviour - page 2 - General Help and Assistance on using CHDK stable releases - CHDK Forum
« previous next »
Pages: 1 [2] 3 4   Go Down

Bad Pixels: Unusual Behaviour

  • 33 Replies
  • 11062 Views
*

Offline SkepticaLee

  • **
  • 50
Re: Bad Pixels: Unusual Behaviour
« Reply #10 on: 16 / October / 2013, 04:30:23 »
Advertisements
I'll try to take one topic at a time.

Fixing hot pixels with Adobe

Software: Adobe Digital Negative Converter 7.2.0.46, copyright 2012
Irfanview, DCRAW, UFraw: current versions

All DNG files (both 1.1 and 1.3) produced by CHDK can be converted by Adobe to new DNG files.

Version 1.1. When viewing these files in Irfanview most, but by no means all, of the hot pixels appear to be compensated for.
Version 1.3. In Irfanview almost all hot pixels have been corrected.
From Irfanview these files can be saved (e.g. as PNG) with the hot pixels not reappearing.

However, when these converted files are opened by DCRAW or UFraw most of the hot pixels appear again.

Since I am using an automated system (via PHP) to process very large numbers of files, I need a program that either runs directly in PHP (will try ImageMagick later today and report the results - but I already know that it will not open the original DNG files without the bad pixels appearing, since it uses dcraw as a delegate to do the conversion) or can be addressed via the command line.

Any tips?
« Last Edit: 16 / October / 2013, 04:37:09 by SkepticaLee »
Logged
*

Offline SkepticaLee

  • **
  • 50
Re: Bad Pixels: Unusual Behaviour
« Reply #11 on: 16 / October / 2013, 13:32:44 »
I know this is not a CHDK problem, but when running Adobe DNG converter on the command line the images are always reduced to 768 * 1024. Running the GUI only results in images of original size when under "Preferences" the option "Custom" is chosen and the "Backward Version" is set to "DNG 1.1" (or "DNG 1.3") regardless of whether the images were shot as 1.1 or 1.3. If "Backward Version" is set to "1.4" the files can be saved in the lossy format, but not otherwise. And this only for the legacy version of Adobe 7.2, not the current version (8.2).

The sort of command line (in a .bat file) I'm using is:
Code: [Select]
md Originals
move *.DNG Originals
set OLDDIR=%CD%
chdir Originals
rem for %%i in (*.DNG ) do "C:\Program Files\Adobe\Adobe DNG Converter.exe" -dng1.3 -d "%OLDDIR%" "%%i"
cd "%OLDDIR%"
But I can't seem to find any set of command line options that preserves the size of the image, and the documentation is miserable.

Again, any suggestions?
« Last Edit: 16 / October / 2013, 13:44:51 by SkepticaLee »
Logged
*

Offline reyalp

  • ******
  • 14080
Re: Bad Pixels: Unusual Behaviour
« Reply #12 on: 16 / October / 2013, 16:12:19 »
Quote from: SkepticaLee on 16 / October / 2013, 13:32:44
I know this is not a CHDK problem, but when running Adobe DNG converter on the command line the images are always reduced to 768 * 1024.
I would guess this is "preview" sub-ifd and the full resolution image is still in there somewhere. What size are the files?
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 14080
Re: Bad Pixels: Unusual Behaviour
« Reply #13 on: 16 / October / 2013, 16:46:23 »
Quote from: SkepticaLee on 16 / October / 2013, 04:30:23
All DNG files (both 1.1 and 1.3) produced by CHDK can be converted by Adobe to new DNG files.
Converting 1.1 would be pointless, as far as bad pixel patching goes, since pre-1.3 the DNG spec requires that bad pixels be patched in the original data.

Converting 1.3 to 1.1 would only be useful if the tool applies the badpixel opcodes. I'm not sure if Adobes tools can do that. I do know that if you use their dng_validate tool to convert to tiff, the pixels are fixed.

Quote
Version 1.3. In Irfanview almost all hot pixels have been corrected.
We need to keep terminology straight here. CHDK does not do anything with hot pixels (pixels that are stuck on, or accumulate spurious counts.)

So if you are actually trying to deal with hot pixels, you have to do it with something completely outside of CHDK. dcraw has options to handle arbitrary bad pixels, and I'm sure many other image processing programs do to. See the -P option. You should  (in theory) be able to handle all your hot/bad/dead pixels this way, independent of CHDK.

(There's supposed to be a way to do this in CHDK, but apparently it got broken and no one told me... It should be fixed at some point)

CHDK badpixel.bin and the badpixel opcodes only deals with low value (dead or marked bad by the canon firmware) pixels. As I explained before, Canon cameras make pixels as bad based on criteria that aren't fully understood. It is likely that they depend on exposure time, sensor temperature and possibly ISO. If you want to use CHDK DNG 1.1, you should generate your bad pixel file with long exposure and probably take some shots before hand to make sure the sensor is warmed up. You may also want to use a higher ISO value (though very high ISOs may behave differently...). On some cameras, this will result in a badpixel file which is so large it causes your camera to run out of memory or fail to load the file. In that case, you need to use DNG 1.3 and deal with the bad pixels elsewhere.

Note that the bad pixels is usually only one (red green or blue) bayer element, these pixels can still show up as a saturated pixel. Converting your DNG to greyscale without debayering (dcraw -d, -D or -E) can help tell the difference.

Regarding DNG 1.3, someone on the forum created a patch to dcraw to handle DNG 1.3 badpixel opcodes, see http://chdk.setepontos.com/index.php?topic=9443.msg98243#msg98243

I'm not sure what the status is, but you might be able to find it or get a copy from them.

FWIW, you can also use my chdkptp tool https://www.assembla.com/spaces/chdkptp/wiki to manipulate CHDK DNGs (it probably won't work on DNGs that have been created or processed by other programs.)

See this post http://chdk.setepontos.com/index.php?topic=6231.msg104529#msg104529 and USAGE.TXT for some information about the DNG commands.

I intend to add the ability generate bad pixel and hot pixel lists at some point, but as usual there is no ETA.
Logged
Don't forget what the H stands for.
*

Offline lapser

  • *****
  • 1093
Re: Bad Pixels: Unusual Behaviour
« Reply #14 on: 16 / October / 2013, 17:45:26 »
Quote from: reyalp on 15 / October / 2013, 23:25:39
For DNG 1.3, CHDK simply generates instructions for the DNG processor which say values below a certain value are "bad".
Thank you for the explanation of DNG 1.1 and 1.3. You're an excellent teacher, and explain things very clearly. I really appreciate it.

From your explanation, the camera sets some raw pixels to 0, based on its own algorithm that varies with exposure. This could explain the "Bad Pixels: Unusual Behaviour" with DNG 1.1 that we're talking about.

DNG 1.3 converters replace pixels below a certain value with pixels derived from surrounding pixels. CHDK specifies that value in the DNG 1.3 file. The results should be just as good or better than 1.1, without requiring "badpixel.bin" generation, or the overhead of interpolating pixels in CHDK when saving raw files. So using 1.3 should solve the current problem being discussed.

Now the problem has become a DNG 1.3 file processing issue, and not a CHDK issue.

Maybe it's time to remove the code in CHDK for DNG 1.1?
Logged
EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Bad Pixels: Unusual Behaviour
« Reply #15 on: 16 / October / 2013, 18:26:46 »
Quote from: reyalp on 16 / October / 2013, 00:07:00
Quote from: philmoz on 15 / October / 2013, 23:59:19
In the current code, the manual bad pixel removal only works where the sensor value is 0 when the removal mode is set to 'Average' - this is probably a bug introduced in a previous code cleanup.
That means it doesn't work at all. News to me...

My mistake - it is still working as expected.
Sorry for the confusion.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline rkomar

  • **
  • 98
Re: Bad Pixels: Unusual Behaviour
« Reply #16 on: 16 / October / 2013, 18:39:25 »
Quote from: reyalp on 16 / October / 2013, 16:46:23
Regarding DNG 1.3, someone on the forum created a patch to dcraw to handle DNG 1.3 badpixel opcodes, see http://chdk.setepontos.com/index.php?topic=9443.msg98243#msg98243

I'm not sure what the status is, but you might be able to find it or get a copy from them.

I never heard back from either the ufraw or dcraw maintainers, so the patch went nowhere.  The patch for ufraw-0.19.2 is available inside the zip file available from this post: http://chdk.setepontos.com/index.php?topic=9443.msg98900#msg98900.  A patched version of ufraw automatically corrects the bad pixels in DNG 1.3 images.

If you don't want to patch and build ufraw, then you can use rawtherapee to interpolate over the bad pixels.  CHDK sets them to 0, so enabling the RAW/Preprocessing/Hot/Dead_pixel_filter option will fix them, even though it doesn't know about the badpixel opcode.  In the same way, hacking dcraw.c and setting the "zero_is_bad" variable to 1 somewhere would achieve the same result, but it would apply to DNG 1.1 and 1.3 images alike.
« Last Edit: 16 / October / 2013, 18:41:36 by rkomar »
Logged
*

Offline reyalp

  • ******
  • 14080
Re: Bad Pixels: Unusual Behaviour
« Reply #17 on: 17 / October / 2013, 00:07:40 »
Quote from: lapser on 16 / October / 2013, 17:45:26
DNG 1.3 converters replace pixels below a certain value with pixels derived from surrounding pixels. CHDK specifies that value in the DNG 1.3 file. The results should be just as good or better than 1.1, without requiring "badpixel.bin" generation, or the overhead of interpolating pixels in CHDK when saving raw files. So using 1.3 should solve the current problem being discussed.
Not exactly, for two reasons:
1) It's not clear if the problem is the above mentioned bad pixels, or hot pixels that are unknown to the canon firmware.
2) It only solves it if you have software that can correctly process DNG 1.3
Quote
Now the problem has become a DNG 1.3 file processing issue, and not a CHDK issue.

Maybe it's time to remove the code in CHDK for DNG 1.1?
I would say no, because in reality support for DNG 1.3 opcodes is still very rare outside of adobe products. If you can generate a badpixel.bin that covers the "worst" canon badpixel map, you have a much better chance of getting a DNG that "just works".

@rkomar
Thanks for posting that link.
Logged
Don't forget what the H stands for.
*

Offline SkepticaLee

  • **
  • 50
Re: Bad Pixels: Unusual Behaviour
« Reply #18 on: 17 / October / 2013, 03:35:06 »
Quote from: reyalp on 16 / October / 2013, 16:12:19
I would guess this is "preview" sub-ifd and the full resolution image is still in there somewhere. What size are the files?
@reyalp:
After "converting" DNGs to DNGs in Adobe DNG Converter what you see is the JPG "thumbnail" @ 768 * 1024 pixels. By setting the switch "-p2" on the command line the thumbnail is then full size, but it is still the JPG image, and not the raw image.

Irfanview on its own used to extract just the jpg image and save it in the target format. I say "used to", because it worked until yesterday morning, and hasn't worked since. No idea what is going on here. However, it is not a solution.

UFRaw-batch 0.19 will now work to extract raw images of both DNG 1.1 and 1.3, but the quality of images with a large number of hot pixels is still poor, although better than what can be achieved with DCRaw. That is to say, not all hot pixels are removed. Perhaps fiddling with the wavelet-denoise parameter may improve the quality, I'll give it a try.

I will also see what rawtherapee does, as it appears to have batch processing capability.
Logged
*

Offline rkomar

  • **
  • 98
Re: Bad Pixels: Unusual Behaviour
« Reply #19 on: 17 / October / 2013, 12:19:28 »
Quote from: SkepticaLee on 17 / October / 2013, 03:35:06
UFRaw-batch 0.19 will now work to extract raw images of both DNG 1.1 and 1.3, but the quality of images with a large number of hot pixels is still poor, although better than what can be achieved with DCRaw. That is to say, not all hot pixels are removed. Perhaps fiddling with the wavelet-denoise parameter may improve the quality, I'll give it a try.
.

I use ufraw-batch with the "--wb=auto --interpolation=vng --color-smoothing" options to improve the output over the defaults.
Logged
Pages: 1 [2] 3 4   Go Up
« previous next »
 

Related Topics