badpixel.bin failed - RAW Shooting and Processing - CHDK Forum supplierdeeply
« previous next »
Pages: [1] 2   Go Down

badpixel.bin failed

  • 14 Replies
  • 15738 Views
*

Offline 2frugal

  • *
  • 12
badpixel.bin failed
« on: 01 / May / 2018, 06:29:39 »
Advertisements
Hello!

First, thanks for this wonderful work - CHDK!! :) I'm amazed by what some open source communities do!

I get an error "badpixel.bin failed". Two shots are made, then this error is shown for a second, then it disappears.

Ixus 115 hs = Elph 100 hs (purchased used and cheap). Exif from jpeg showed rev 3.0, i.e. C, so I grabbed the "101c". However, the build info shows firmware "101b". I use the svn revision 5012 build. Otherwise everything works fine (so far).

I'm new, and can't debug (can only "report", like this). Will it be helpful if I create the 4 files as instructed by philmoz? If somebody helps me - would be very sweet! Please!

(I know, there's a workaround, to create an external badpixels list, for PC raw software. I'm trying it...)

Thanks
« Last Edit: 01 / May / 2018, 06:32:16 by 2frugal »
Logged
*

Offline 2frugal

  • *
  • 12
Re: badpixel.bin failed
« Reply #1 on: 01 / May / 2018, 12:02:24 »
And a question, if possible - do I understand it right? - that the firmware can supply to CHDK a factory-made list of bad pixels? And it is this list that CHDK puts in the opcodes in the v1.3 DNG when there's no badpixel.bin?
Logged
*

Offline reyalp

  • ******
  • 14080
Re: badpixel.bin failed
« Reply #2 on: 01 / May / 2018, 13:40:47 »
Quote from: 2frugal on 01 / May / 2018, 12:02:24
And a question, if possible - do I understand it right? - that the firmware can supply to CHDK a factory-made list of bad pixels? And it is this list that CHDK puts in the opcodes in the v1.3 DNG when there's no badpixel.bin?
Not quite.

badpixel.bin gets the list from the firmware indirectly. The canon firmware sets known bad pixels to 0 (or possibly some other small value on some cameras) so we extract the list by searching for 0 valued pixels. The reason we use badpixel.bin is that searching the whole raw buffer is very slow.

In 1.3, we use opcodes to tell the raw software that pixels with 0 value are bad, so we don't need the list.

A lot of software doesn't support DNG opcodes, but some can be told to interpolate over 0 valued pixels anyway. For example, in raw therapee, you can check "Dead pixel filter" in the raw tab.

Quote
the build info shows firmware "101b". I use the svn revision 5012 build.
This is expected, 101b and 101c firmwares use the same build.

Quote
I'm new, and can't debug (can only "report", like this). Will it be helpful if I create the 4 files as instructed by philmoz? If somebody helps me - would be very sweet! Please!
It wouldn't hurt.

If you haven't already, you could also just try running the badpixel process a few times. The badpixel list can be temperature dependent and the process will fail if changes between shots.
Logged
Don't forget what the H stands for.
*

Offline 2frugal

  • *
  • 12
Re: badpixel.bin failed
« Reply #3 on: 01 / May / 2018, 16:15:46 »
Quote from: reyalp on 01 / May / 2018, 13:40:47
badpixel.bin gets the list from the firmware indirectly. The canon firmware sets known bad pixels to 0 (or possibly some other small value on some cameras) so we extract the list by searching for 0 valued pixels.
Thank you! I'd say the manual (and the chdkptp wiki) misses these explanations. Now makes perfect sense, what and why.

Quote
This is expected, 101b and 101c firmwares use the same build.
Thanks.

I tried right after starting (while sensor is cold... idea: put the camera in the fridge for an hour?), and it worked right away! (previously I tried many times, but it was warm)

I also managed the badpixels list obtained with chdkptp dnglistpixels to get recognized by ufraw (the workaround; my mistake was raw cropping="full", now I use "active"). 16000 bad pixels reported.

Now images are smooth using both ways :)

Quote
It wouldn't hurt.
Since it works, no need I guess.

Thank you so much!
« Last Edit: 01 / May / 2018, 16:37:21 by 2frugal »
Logged
*

Offline reyalp

  • ******
  • 14080
Re: badpixel.bin failed
« Reply #4 on: 01 / May / 2018, 16:42:38 »
Quote from: 2frugal on 01 / May / 2018, 16:15:46
Since it works, no need I guess.
It's fine if you don't want to but if you can upload examples taken cold and warm (with DNG 1.3, so nothing is patched on camera), it might provide some insight into the problem.

badpixel creation not working for some people is a long standing problem that we've never really fully figured out.

If you post what the CHDK menu miscellaneous stuff->Show memory info reports while in rec mode after taking at least one picture, that might also be useful. There's generally more bad pixels when the sensor is hot, so it's possible the process runs out of memory.

FWIW, if you are using chdkptp, you can also use dngbatch to interpolate over bad pixels in the dng file. I generally use something like:
Code: [Select]
dngbatch path/to/images { mod -patch ; save -over -keepmtime }
Logged
Don't forget what the H stands for.
*

Offline 2frugal

  • *
  • 12
Re: badpixel.bin failed
« Reply #5 on: 02 / May / 2018, 12:51:41 »
(... took some time) Roughly what I did:
for temerature=10,22,35 (°C)
for i=1 to 5
set iso=100, try badpixel.bin
set iso=3200, try badpixel.bin
end
check free memory
end

Temperatures very approx., and they were changing during tests. Second loop - appr. 5, several times. Always, without a single exception: iso=100 succeeded, iso=3200 failed. So (for me), no dependence on temperature. Same amount of free memory, around 552000 (when cold, was a bit more, 585000). Also, same memory after simply taking 2 raws.

It's not very easy to get proper DNGs: they are always both over- and under-exposed. At iso=100 I could get it almost proper (histogram narrow). At iso=3200 it's impossible (histogram wide). Two DNGs (at these two ISOs) at box.com. May be this increase in dynamic range at high ISOs somehow causes the problem?

If you have any other ideas, and my camera looks interesting - I'll be happy to assist.

Quote from: reyalp on 01 / May / 2018, 16:42:38
you can also use dngbatch...
Thanks, also works. I'm curious though, why CHDK's badpixel.bin needs two shots, and dngbatch patches fine a single image?
Logged
*

Offline reyalp

  • ******
  • 14080
Re: badpixel.bin failed
« Reply #6 on: 02 / May / 2018, 13:35:33 »
Quote from: 2frugal on 02 / May / 2018, 12:51:41
Thanks, also works. I'm curious though, why CHDK's badpixel.bin needs two shots, and dngbatch patches fine a single image?
chdkptp just interpolates over the 0 value pixels in that particular image, like a program that understands DNG opcodes would.

The badpixel.bin creation process is trying to create a list that can be used for future shots, so it checks that it got the same number both shots. Setting a fixed list of known bad pixels to 0 is just observed behavior, so we don't know if it actually applies to every camera.

In reality, the cameras seem to have multiple lists which may depend on things like exposure time, ISO, temperature. The CHDK logic still mostly works because the lists under worse conditions (longer exposure, higher ISO, higher temp) include the pixels  that would be bad in better conditions. CHDK checks if a pixels is 0 before patching, so using a bigger list just costs memory and performance.

Thanks for uploading the DNGs, and narrowing it down to ISO. I'll try to take a look at them later.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 14080
Re: badpixel.bin failed
« Reply #7 on: 03 / May / 2018, 02:46:58 »
I looked at the DNGs briefly.

In the ISO 100 shot, there are 16193 zero value pixels, and no other values lower than 115
In the ISO 3200 short, there are 53543 zero value pixels, and some pixels with every value between that and blacklevel (127)

It would be interesting to see two ISO 3200 shots taken one after the other.

The port is set up so values <= 16 will be treated as badpixels. If the number of pixels in that range varies, that would explain the process failing.
Logged
Don't forget what the H stands for.
*

Offline 2frugal

  • *
  • 12
Re: badpixel.bin failed
« Reply #8 on: 03 / May / 2018, 12:45:45 »
Ok, let's see, 3 new DNGs. The CRW_1594.DNG is a bit underexposed, but hopefully still can be compared with the CRW_1593.DNG.
Logged
*

Offline 2frugal

  • *
  • 12
Re: badpixel.bin failed
« Reply #9 on: 03 / May / 2018, 15:02:06 »
Is there (the proper) source-code documentation?

I tried to manually read the DNGs, myself, in octave. However, the zero-pixel count I get is much lower (1541 vs. 16193). I tried to look at sources, core/raw.c: get_raw_pixel() looks relevant. But I don't see any documentation on how RGBs are packed in the memory (rawadr[]). So, the documentation?

Here's what I did. In case it's easy, may be you can correct me?
dcraw -j -t 0 -4 330___05/CRW_1591.DNG

a1=imread('/mnt/win1/330___05/CRW_1591.ppm');
[iq,jq]=find(a1(:,:,1)==0 & a1(:,:,2)==0 & a1(:,:,3)==0);
size(iq)

(yes... I've got too much time... :) )
Logged
Pages: [1] 2   Go Up
« previous next »
 

Related Topics