badpixel file...."file is unknown" :(

  • 12 Replies
  • 1947 Views
*

Offline gcgc

  • *
  • 14
badpixel file...."file is unknown" :(
« on: 06 / June / 2014, 01:09:38 »
Advertisements
Hi,
I've some troubles in generating badpixel file (no problem with badpixel.bin.....since the autogenerating way to do it). I've read many discussion trying to find my way but I must do something wrong because when I open the file with text file reader function in the camera it says "file is unknown".
What I do is shooting a crw (i also tried with dng without success) full black photo in the same folder as show_bad files unzipped, and wrote:
C:\Users\Rut\Desktop\showbad>showbad raw_file.CRW 30 > badpixel (I used 30 because I thought my camera, a2500 has a10 bit sensor). Then I take the generated badpixel file and put it and the black crw photo inside the CHDK directory (I also tried to put them in the root directory of the SD card without success).
Please help me.
Thanks for spending your time reading me.



*

Offline reyalp

  • ******
  • 11487
Re: badpixel file...."file is unknown" :(
« Reply #1 on: 06 / June / 2014, 01:55:00 »
C:\Users\Rut\Desktop\showbad>showbad raw_file.CRW 30 > badpixel (I used 30 because I thought my camera, a2500 has a10 bit sensor). Then I take the generated badpixel file and put it and the black crw photo inside the CHDK directory (I also tried to put them in the root directory of the SD card without success).
Please help me.
Thanks for spending your time reading me.
showbad is very old and only understands raws from a few old cameras.

You can use chdkptp to generate a CHDK badpixel list from a DNG, see https://www.assembla.com/spaces/chdkptp/wiki/DNG_Processing

You don't need to install the usb driver to do this, just unzip the files and run the exe.
Don't forget what the H stands for.

*

Offline gcgc

  • *
  • 14
Re: badpixel file...."file is unknown" :(
« Reply #2 on: 06 / June / 2014, 03:50:57 »

showbad is very old and only understands raws from a few old cameras.

You can use chdkptp to generate a CHDK badpixel list from a DNG, see https://www.assembla.com/spaces/chdkptp/wiki/DNG_Processing

You don't need to install the usb driver to do this, just unzip the files and run the exe.
[/quote]

Thanks, very useful!!
I've been reading making photos in all the possible ways and reading chkd manuals and many topics about badpixel but I've never found something as interesting as your answer.
I wish I had asked my question before!
When at home (at work now)  I'll try this way and I'll tell you if it works.
Thank you again!



*

Offline gcgc

  • *
  • 14
Re: badpixel file...."file is unknown" :(
« Reply #3 on: 06 / June / 2014, 16:30:14 »
Ok,
I 've just tried and lastly I had the file in the attachment  :)
Sorry for the many silly questions but..........to make this file working right how should I use it?
Should I put it inside the chdk directory?
Shoud I rename it?
Will it works for all raw format (CR2, CRW) or only for DNG?
If I choose dng 1.1 the camera will use badpixel.bin instead of the file generated with chdkptp?
Thank you for your help and for chdkptp  :)


*

Offline reyalp

  • ******
  • 11487
Re: badpixel file...."file is unknown" :(
« Reply #4 on: 06 / June / 2014, 17:22:37 »
Ok,
I 've just tried and lastly I had the file in the attachment  :)
1100 seems like a lot of hot pixels, how was the shot taken and what commands did you use in chkdptp?

Generally, to fix hot pixels you would want to
1) Take a "dark frame" with the longest exposure and highest ISO you'd expect to use.
You can make the dark frame either by covering the lens, or using script to keep the shutter closed during the exposure. You could also just use a scene that's very under-exposed, but you will want to use a very conservative limit for what you call "hot" in that case.

Note that if you use an exposure over a few seconds, the Canon firmware will automatically shoot and subtract a dark frame. The camera will show "busy" and the shot will take something like 2x the selected exposure time. The dark frame takes care of hot pixels, so you wouldn't generally want to use an exposure this long to generate a badpixel list. You can disable the cameras automatic dark frame the CHDK raw settings, but in this case you'd usually want to apply your own dark frames later.

2) Load the resulting DNG into chdkptp with dngload file.dng and generate a badpixel.txt containing pixels with value above some limit.
The exact value is matter of taste, but it should be something substantially above the black level, because there will be some amp glow and dark current. If you covered the lens, there will probably be some stray light too. You don't want to try to "fix" all those pixels with the badpixel list, you only want the ones that are really "hot" or stuck on.

You can use the dnghist command to get an idea of where the hot pixels are. Here's an example using an 4 second, ISO800 dark frame from my D10 (12 bit sensor, like your a2500)
Code: [Select]
$ chdkptp -i
___> dngload CRW_1060.DNG
loaded CRW_1060.DNG
___> dnghist -bin=512
0-511 12338145
512-1023 3
1024-1535 1
1536-2047 3
2048-2559 5
2560-3071 0
3072-3583 0
3584-4095 3
Here we see virtually all the pixels fall below 512. It's probably safe to call the remainder "hot", but we can take a close look at the low end
Code: [Select]
___> dnghist -bin=32 -max=512
0-31 144385
32-63 607806
64-95 1663674
96-127 2718160
128-159 3720824
160-191 2302111
192-223 857943
224-255 250329
256-287 55785
288-319 12795
320-351 3233
352-383 832
384-415 208
416-447 43
448-479 14
480-511 3
512-543 0
The black level on 12 bit cameras is around 128, so we expect to see the values clustered around there. 4 seconds is quite a long exposure, so we expect some dark current etc too. It's clear that values over 512 are exceptional, so we can use that as the maximum:
Code: [Select]
___> dnglistpixels -min=512 -fmt=chdk -out=badpixel.txt
15 matching pixels
If the "dark frame" wasn't shot with the shutter closed, there might be a lot of real values around 512, so you would use a higher limit.
Quote
Sorry for the many silly questions but..........to make this file working right how should I use it?
Should I put it inside the chdk directory?
Shoud I rename it?
Copy it to the CHDK directory as BADPIXEL or BADPIXEL.TXT
Quote
Will it works for all raw format (CR2, CRW) or only for DNG?
If the "Manual bad pxiel removal" setting is not OFF, it will be applied to all images. How it actually works depends on which of the settings you use, see http://chdk.wikia.com/wiki/CHDK_User_Manual#Manual_bad_pixel_removal
For DNG 1.1, the RawConv option will leave you with 0 value pixels in the data, which will look like unfixed badpixels.
Quote
If I choose dng 1.1 the camera will use badpixel.bin instead of the file generated with chdkptp?
No, DNG badpixel.bin does a different thing. See http://chdk.wikia.com/wiki/Badpixel_removal

badpixel lets you specify pixels as bad. This is generally used for "hot" pixels that are not known to the canon firmware.

badpixel.bin fixes patches pixels the camera sets 0 (ones the canon firmware already knows are bad) in order to make a valid 1.1 DNG.
Don't forget what the H stands for.

*

Offline gcgc

  • *
  • 14
Re: badpixel file...."file is unknown" :(
« Reply #5 on: 07 / June / 2014, 02:00:01 »
I was wrong because before  I used  "dnglistpixels -max=0 -out=bad.txt -fmt=rt -coords=rel" that I found in https://www.assembla.com/spaces/chdkptp/wiki/DNG_Processing without knowing what I was doing.
And I overrided iso value of the camera (I think iso around 10) and Tv values 1/4 sec following "Try several RAW files to obtain a stable result (approximate shooting conditions: minimal ISO, exposure time ~1/5s-0,8s)" found here http://chdk.wikia.com/wiki/Badpixel_removal.
As you can see I'm a newbie and need help because I take information from here and there without exactly knowing which is what :(

The commands now:
dngload crw_0060.dng
dnglistpixels -min=512 -fmt=chdk -out=badpixel.txt

Thanks....now is 736 matching pixel....maybe still too much and I don't know why.....I capped the lens and I was in a darkroom to be sure: I'll try other ways to cap the lens.


Used iso 800 (never in my life I used iso 800......could I use something lower (eg iso 600) if I never go to these values?)

Please could you tell me, just to know, where I can see information about bit sensor of a canon camera (same days ago i wasn't able to find that info for my a2500.....luckly you said me it's a 12 bit camera :) ).

*

Offline reyalp

  • ******
  • 11487
Re: badpixel file...."file is unknown" :(
« Reply #6 on: 07 / June / 2014, 16:55:27 »
Thanks....now is 736 matching pixel....maybe still too much and I don't know why.....I capped the lens and I was in a darkroom to be sure: I'll try other ways to cap the lens.
What are you actually trying to do?

If you use a higher iso or longer exposure than real shooting conditions, then you might end up "fixing" a few pixels that weren't actually bad in that shot, but ~700 pixels out of 16 million is probably a small enough number you don't need to care much.  Even most of them are "good" interpolating over them won't affect the overall image quality much. So you could just stop there.

Assuming you are trying to get rid of hot pixels, looking at the image should make it fairly obvious what is happening. Be warned that many simple image viewers (like irfanview, picasa etc) do an automatic contrast stretch on raw images, which will make it harder to interpret. Something like raw therapee will work better.

Here's a screenshot from raw therapee of one of my d10 dark frames (4 sec, iso80) https://app.box.com/s/aka3td6p0r1celsbn4ia

I set the exposure to "neutral" and then turned up the exposure compensation up a bit so you can see the noise and amp glow. You can see handful of bright red, blue and green dots: Those are the things you could fix with badpixel. The ~10 or so that are really bright would be annoying in a real image. There's maybe a few dozen more that are well above the noise, but probably wouldn't be obvious in most shots.

As I mentioned earlier, you can also use dnghist to for this. dnghist generates a histogram in text format, meaning it counts how many pixels are in each value range. The -bin option sets the range of values that are counted together. So bin=1 (the default) would give you one a count for each of the 4096 possible values. That's hard to read, something like -bin=256 will give you a bitter idea what's going on.

Using this on the same DNG used for the screenshot above:
Code: [Select]
___> dnghist -bin=256
0-255 12338087
256-511 42
512-767 15
768-1023 3
1024-1279 3
1280-1535 1
1536-1791 0
1792-2047 1
2048-2303 2
2304-2559 1
2560-2815 0
2816-3071 0
3072-3327 1
3328-3583 0
3584-3839 1
3840-4095 3
This shows the same story as the screenshot. The vast majority are in the first bin, 0-255. The ~13 from 1024 to 4095 are the ones that stand out as hot pixels. There are some in between, but they are less than 1/4 of max value, so in a real image they aren't going to stand out much.
Quote
Please could you tell me, just to know, where I can see information about bit sensor of a canon camera (same days ago i wasn't able to find that info for my a2500.....luckly you said me it's a 12 bit camera :) ).
http://chdk.wikia.com/wiki/CameraFeatureTable has bit depth and some other useful information. Pretty much all canon P&S after 2009 or so are 12 bit (except G1x which is 14 bit)
Don't forget what the H stands for.

*

Offline gcgc

  • *
  • 14
Re: badpixel file...."file is unknown" :(
« Reply #7 on: 08 / June / 2014, 01:01:54 »
Thanks, you were very illuminating  :)
I'll make some "experiments" (not today because I've to go out the whole day with my wife) and.....I'm sure I'll have something more to ask you: please be patient.


*

Offline gcgc

  • *
  • 14
Re: badpixel file...."file is unknown" :(
« Reply #8 on: 10 / June / 2014, 00:53:55 »
Hi,
I had some "experiments" and these are the dnglistpixels and dnghist for  black images taken at different settings.

This is iso 80 and 4 sec

> dngload crw_0077.dng
loaded crw_0077.dng
> dnglistpixels -min=512 -fmt=chdk -out=badpixel.txt
7 matching pixels
> dnghist -bin=256
0-255 15999033
256-511 0
512-767 0
768-1023 7
1024-1279 0
1280-1535 0
1536-1791 0
1792-2047 0
2048-2303 0
2304-2559 0
2560-2815 0
2816-3071 0
3072-3327 0
3328-3583 0
3584-3839 0
3840-4095 0

This is 400 iso 4 sec

> dngload crw_0078.dng
loaded crw_0078.dng
> dnglistpixels -min=512 -fmt=chdk -out=badpixel.txt
10 matching pixels
> dnghist -bin=256
0-255 15976148
256-511 22882
512-767 3
768-1023 0
1024-1279 0
1280-1535 0
1536-1791 0
1792-2047 0
2048-2303 0
2304-2559 0
2560-2815 1
2816-3071 6
3072-3327 0
3328-3583 0
3584-3839 0
3840-4095 0

The following is iso 800 and 4 sec.

> dngload crw_0060.dng
loaded crw_0060.dng
> dnglistpixels -min=512 -fmt=chdk -out=badpixel.txt
736 matching pixels
> dnghist -bin=256
0-255 15124011
256-511 874293
512-767 710
768-1023 19
1024-1279 0
1280-1535 0
1536-1791 0
1792-2047 0
2048-2303 0
2304-2559 0
2560-2815 0
2816-3071 0
3072-3327 0
3328-3583 0
3584-3839 0
3840-4095 7



passing from iso 80 to 400 there's only a small increase in badpixel (only 3 more and in the range  512-767) while passing from iso 400 a 800 there are 710 more in the range 512-767 and 19 more in the range 768-1023.
I was just wondering which one of badpixel file (400 vs 800) it would be better to use if  I never use more than 400 iso.
I know that 710 + pixel of iso 800 are not "lost" (that is why badpixel file stand for) and that are not too much compared with the 16mpix of the camera....but, having the possibility to ask you, I was just wondering if the better choise is to stay with iso 400 badpixel file or use the iso 800 file and make the hotpixel be averadged via software.

PS
chdk ptp is really a usefull tool!! :)

*

Offline reyalp

  • ******
  • 11487
Re: badpixel file...."file is unknown" :(
« Reply #9 on: 10 / June / 2014, 16:41:08 »
I know that 710 + pixel of iso 800 are not "lost" (that is why badpixel file stand for) and that are not too much compared with the 16mpix of the camera....but, having the possibility to ask you, I was just wondering if the better choise is to stay with iso 400 badpixel file or use the iso 800 file and make the hotpixel be averadged via software.
I would guess the ~710 you get in the 512-768 bin are the tail of the noise, not hot pixels.  I probably wouldn't include them. It looks to me like your camera has 7 hot pixels, or maybe they should be called "warm pixels" since they aren't at max value until you go to ISO 800.

Note if you are testing with a 4 sec exposure, you should make sure the dark frame subtraction is disabled, otherwise it may confuse the results.

With shorter exposures, the hot pixels will be less obvious, so if your are mostly doing hand held shooting below ISO 800, those pixels may not matter at all.
Don't forget what the H stands for.

 

Related Topics