How to interpolate or remove black pixels in DNG files

sbeaupre

Re: How to interpolate or remove black pixels in DNG files

« Reply #20 on: 02 / July / 2012, 13:48:54 »

Advertisements

solved: forgot about 'xattr' when using a mac:
xattr -d com.apple.quarantine DISKBOOT.BIN PS.FIR

Quote from: sbeaupre on 02 / July / 2012, 02:00:39

I have the exact same problem with the exact same camera and tried to install the attached files, but somehow they are not recognized. When I go to the menu to select 'Firmware update', I don't see the entry in the menu anymore. When I go back to the old version, I do see it again and can execute the update again.

Any ideas?

Quote from: philmoz on 27 / March / 2012, 03:53:40
Quote from: hari-bhari on 26 / March / 2012, 18:41:41
That's annoying, sorry about that.

http://www.box.com/s/678e641c1c95f9320064
http://www.box.com/s/c8d16625ea4d9bbe3f82
http://www.box.com/s/a943d9b255ffafc5a31c

It could be the same problem that was found on the ixus220.

Try the attached version and see if it makes any difference.

Phil.

Logged

galenus

Re: How to interpolate or remove black pixels in DNG files

« Reply #21 on: 03 / July / 2012, 19:41:20 »

Hello,

I'm trying to figure out the same (it seems) bad pixels issue on my sx230. I, too, get a bad pixels count around 60k while creating the badpixels.bin at f3.1, 1/30 sec., wide. Resulting DNG opened in the Rawtherapee, shows a huge amount of bad pixels. Resulting JPEG contains them as well. A matching native JPEG from camera does not have this problem.

So I wonder:
1) Is there some fix for my camera model?
2) Can I somehow rescue the RAW already taken?

Thank you in advance!

P.S.: attachment contains a screenshot of sample picture in Rawthereapee.

RawTherapee 4.0.9.png (392.09 kB, 564x324 - viewed 169 times.)

Logged

waterwingz

11755

Re: How to interpolate or remove black pixels in DNG files

« Reply #22 on: 03 / July / 2012, 20:28:51 »

Quote from: galenus on 03 / July / 2012, 19:41:20

I'm trying to figure out the same (it seems) bad pixels issue on my sx230. I, too, get a bad pixels count around 60k while creating the badpixels.bin at f3.1, 1/30 sec., wide. Resulting DNG opened in the Rawtherapee, shows a huge amount of bad pixels.

Okay .. your camera sensor has a few bad pixels.

Quote

A matching native JPEG from camera does not have this problem.

Hmmm....... how do you suppose that happened ? The pixels fixed themselves or your camera's firmware corrected for them ? ( aka covered them up)

Quote

So I wonder:
1) Is there some fix for my camera model?

Quote

2) Can I somehow rescue the RAW already taken?

Rescue? Is the RAW image unusable as is with the 60K pixels removed?

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3071

Re: How to interpolate or remove black pixels in DNG files

« Reply #23 on: 03 / July / 2012, 20:31:16 »

Quote from: galenus on 03 / July / 2012, 19:41:20

Hello,

I'm trying to figure out the same (it seems) bad pixels issue on my sx230. I, too, get a bad pixels count around 60k while creating the badpixels.bin at f3.1, 1/30 sec., wide. Resulting DNG opened in the Rawtherapee, shows a huge amount of bad pixels. Resulting JPEG contains them as well. A matching native JPEG from camera does not have this problem.

So I wonder:
1) Is there some fix for my camera model?
2) Can I somehow rescue the RAW already taken?

Thank you in advance!

P.S.: attachment contains a screenshot of sample picture in Rawthereapee.

What version of CHDK are you using?

I suspect you are using the unstable development build and have the DNG version set to 1.3 (the default) so this is normal.

DNG 1.3 uses opcodes to tell the processing software how to map out bad pixels; but it seems that only Adobe software bothers to do this. RawTherapee ignores the DNG 1.3 extensions and so you see the bad pixels - the DNG 1.3 spec was released three years ago so it's surprising that it does not have better support.

You can change the DNG version to 1.1 in the camera and then CHDK will do the bad pixel mapping (this will slow down the DNG saving a bit with that many bad pixels).

For existing images you will need to process them with Lightroom, Photoshop or Elements.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)

galenus

Re: How to interpolate or remove black pixels in DNG files

« Reply #24 on: 04 / July / 2012, 02:46:40 »

Quote from: waterwingz on 03 / July / 2012, 20:28:51

Quote from: galenus on 03 / July / 2012, 19:41:20
I'm trying to figure out the same (it seems) bad pixels issue on my sx230. I, too, get a bad pixels count around 60k while creating the badpixels.bin at f3.1, 1/30 sec., wide. Resulting DNG opened in the Rawtherapee, shows a huge amount of bad pixels.
Okay .. your camera sensor has a few bad pixels.

Quote
A matching native JPEG from camera does not have this problem.
Hmmm....... how do you suppose that happened ? The pixels fixed themselves or your camera's firmware corrected for them ? ( aka covered them up)

Well, I guess that's camera firmware

Quote

Quote
So I wonder:
1) Is there some fix for my camera model?
No

It appears there is... (answer by philmoz)

Quote

Quote
2) Can I somehow rescue the RAW already taken?
Rescue? Is the RAW image unusable as is with the 60K pixels removed?

Generally, I do appreciate your sense of humor very much

Logged

galenus

Re: How to interpolate or remove black pixels in DNG files

« Reply #25 on: 04 / July / 2012, 02:54:04 »

Quote from: philmoz on 03 / July / 2012, 20:31:16

Quote from: galenus on 03 / July / 2012, 19:41:20
Hello,

I'm trying to figure out the same (it seems) bad pixels issue on my sx230. I, too, get a bad pixels count around 60k while creating the badpixels.bin at f3.1, 1/30 sec., wide. Resulting DNG opened in the Rawtherapee, shows a huge amount of bad pixels. Resulting JPEG contains them as well. A matching native JPEG from camera does not have this problem.

So I wonder:
1) Is there some fix for my camera model?
2) Can I somehow rescue the RAW already taken?

Thank you in advance!

P.S.: attachment contains a screenshot of sample picture in Rawthereapee.

What version of CHDK are you using?

I suspect you are using the unstable development build and have the DNG version set to 1.3 (the default) so this is normal.

DNG 1.3 uses opcodes to tell the processing software how to map out bad pixels; but it seems that only Adobe software bothers to do this. RawTherapee ignores the DNG 1.3 extensions and so you see the bad pixels - the DNG 1.3 spec was released three years ago so it's surprising that it does not have better support.

You can change the DNG version to 1.1 in the camera and then CHDK will do the bad pixel mapping (this will slow down the DNG saving a bit with that many bad pixels).

For existing images you will need to process them with Lightroom, Photoshop or Elements.

Phil.

Thank you very much!

It seems you are right: I'm using the unstable build and the DNG version is set to 1.3. I'll try to follow your advise and report the results here.

Same for the existing images.

Thanks again.

Logged

Mike Lee

Re: How to interpolate or remove black pixels in DNG files

« Reply #26 on: 24 / July / 2012, 01:29:32 »

Mightn't it more prudent, at this time, and for the sake of compatibility, to make DNG v1.1 the default? I notice no difference in file saving time (1.1 vs. 1.3), and knowing that only Adobe products can properly map out bad pixels from 1.3 files is a bit of a (real) concern.

I think the default setting of 1.3 is counterproductive to the promotion of CHDK, as it is bound to cause undue problems (for I'm sure a great many users) when Adobe software is not available for processing, and the user was unaware of CHDK's compatiblity at its default settings..

Logged

reyalp

11707

Re: How to interpolate or remove black pixels in DNG files

« Reply #27 on: 24 / July / 2012, 02:12:27 »

Quote from: Mike Lee on 24 / July / 2012, 01:29:32

Mightn't it more prudent, at this time, and for the sake of compatibility, to make DNG v1.1 the default? I notice no difference in file saving time (1.1 vs. 1.3), and knowing that only Adobe products can properly map out bad pixels from 1.3 files is a bit of a (real) concern.

Saving time isn't the issue. The main reason for preferring 1.3 is that it doesn't require the frequently problematic step of producing badpixel.bin. A secondary reason is that badpixel.bin can eat up quite a lot of memory. Whether this is worse than the issues associated with DNG opcode support, I don't know.

Badpixel opcode support should be quite easy to implement, and properly implemented, it should give better results than the simple interpolation done on the camera. If enough users ask, maybe some of these other tools will start to support it. Alternately, writing a utility that just processes the opcodes should be pretty straightforward.

Logged

Don't forget what the H stands for.

sylwb

Re: How to interpolate or remove black pixels in DNG files

« Reply #28 on: 17 / September / 2012, 17:24:36 »

Hi,

Quote

2) Can I somehow rescue the RAW already taken?

I was struggling with the same thing, I have bunch of DNGs 1.3 taken and I'm using Rawtherapee 4 to develop them.

Here
https://docs.google.com/document/d/1DHLb_6xNQsEInxiuU8pz1-sWNinnj09bpBUA4_Vl8w8/edit?pli=1#heading=h.xg9t7d98krag
I found an information that it's possible to create a simple text file with bad pixels coordinates, and they'll be fixed before processing. So I decided to create small lua script that can convert badpixel.bin into text file acceptable by RT4. I was partially successful, script works (attached) but there're still couple of problems to resolve:

Some versions of RT4 doesn't recognize file (enable verbose mode), I think there'a bug in app, sometimes it helps adding an extra character at the beginning of filename (so: _Canon PowerShot A590 IS.badpixels)
Some pixels have huge X, Y values, beyond size of my camera sensor, I added code to skip them, not sure by what they are caused
I read whole file at once, for my A590 with 8k bad pixels it works great, I can imagine some cameras can suffer because of lack of sufficient memory

Anyway, it works for me, if anybody has ideas for improvements please let me know.

bp4rt.lua (1.42 kB - downloaded 65 times.)

Logged