The case for Adobe Digital Negative Converter

Mike Lee

26

« on: 10 / July / 2012, 12:43:36 »

Advertisements

I regularly use CHDK on my SX130 and S3. The in-camera DNG files that they produce are different. In exactly what way(s) I'm not sure, but various image viewers and RAW editors see the files differently, with DNG files from the S3 typically being less recognized (i.e. compatible). I would suspect that the latest changes to CHDK for the SX130 implement some DNG improvements not incorporated into CHDK for the older S3.

I have found that by using Adobe Digital Negative Converter (I'm using the latest v.7.1) to convert these in-camera DNG files they become universally recognizable, and much more manageable. An added benefit is that file size is typically reduced by approximately 40%. There is only modest, and I feel acceptable, metadata purging when the files are converted. The metadata lost is relatively innocuous, and all critical EXIF info (exposure and time data, etc.) is retained.

Embedding a medium-sized preview image during the conversion process also speeds image browsing. My experience shows a several times increase in browsing speed when this option is chosen in the converter.

All in all I've had only good experiences with this extra step in my DNG workflow, and well worth the little bit of time it takes to open a folder of CHDK DNG files and convert them.

Comments and observations?

Logged

KenO

122

Re: The case for Adobe Digital Negative Converter

« Reply #1 on: 13 / July / 2012, 13:05:00 »

Hi Mike,

"The in-camera DNG files that they produce are different."

Are you using the in-camera DNG 1.2 or DNG 1.3?

Thanks,

Ken

Logged

reyalp

14080

Re: The case for Adobe Digital Negative Converter

« Reply #2 on: 13 / July / 2012, 16:35:42 »

Quote from: Mike Lee on 10 / July / 2012, 12:43:36

In exactly what way(s) I'm not sure, but various image viewers and RAW editors see the files differently, with DNG files from the S3 typically being less recognized (i.e. compatible).

If you can identify programs which refuse to open CHDK DNG but do open Adobe created DNGs, we'd be interested to know what they are. Samples of the offending DNGs would also be helpful.

Quote

I have found that by using Adobe Digital Negative Converter (I'm using the latest v.7.1) to convert these in-camera DNG files they become universally recognizable, and much more manageable. An added benefit is that file size is typically reduced by approximately 40%.

Adobe likely compresses the image data. CHDK doesn't do this, because the camera CPU is quite slow, and CHDK DNG already suffers a performance penalty compared to pure raw.

Quote

I would suspect that the latest changes to CHDK for the SX130 implement some DNG improvements not incorporated into CHDK for the older S3.

This should not be the case, DNG is done in common code. If you can upload examples of each, that would help us identify any errors.

Quote

Comments and observations?

As KenO alluded, the CHDK development builds allow you to use DNG 1.3. This uses DNG "opcodes" to deal with badpixels (rather than interpolating over them in camera, as required for DNG 1.2) Few programs support these, but Adobe products probably do. If you can use DNG converter to convert from 1.3 to 1.2, this would give users of programs that don't support 1.3 a good option for badpixel removal.

Logged

Don't forget what the H stands for.

Rogerio

13

Re: The case for Adobe Digital Negative Converter

« Reply #3 on: 18 / July / 2012, 16:50:46 »

Quote

Adobe likely compresses the image data. CHDK doesn't do this, because the camera CPU is quite slow, and CHDK DNG already suffers a performance penalty compared to pure raw.

Wouldn't the extra CPU time be offset by a shorter writing time? CPU speeds have got much faster at each iteration, not so with SD cards except for very expensive ones.

Logged

philmoz

3450

Re: The case for Adobe Digital Negative Converter

« Reply #4 on: 18 / July / 2012, 18:27:55 »

Quote from: Rogerio on 18 / July / 2012, 16:50:46

Quote
Adobe likely compresses the image data. CHDK doesn't do this, because the camera CPU is quite slow, and CHDK DNG already suffers a performance penalty compared to pure raw.

Wouldn't the extra CPU time be offset by a shorter writing time? CPU speeds have got much faster at each iteration, not so with SD cards except for very expensive ones.

Probably not, the camera's CPU is quite slow. Most of the complex work is done by the Digic processor which CHDK does not have access to (no one has reversed engineered any of it yet).

When saving the DNG file, CHDK has to byte-swap the RAW buffer to conform to the DNG standard - this takes approx 1/3rd second. This is a simple operation and the code is pretty optimized. A complex function like lossless JPEG compression that DNG uses could take a long time.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Rogerio

13

Re: The case for Adobe Digital Negative Converter

« Reply #5 on: 18 / July / 2012, 22:23:37 »

Quote from: philmoz on 18 / July / 2012, 18:27:55

Probably not, the camera's CPU is quite slow. Most of the complex work is done by the Digic processor which CHDK does not have access to (no one has reversed engineered any of it yet).

When saving the DNG file, CHDK has to byte-swap the RAW buffer to conform to the DNG standard - this takes approx 1/3rd second. This is a simple operation and the code is pretty optimized. A complex function like lossless JPEG compression that DNG uses could take a long time.

Phil.

Thank you. I thought that we had access to the Digic processor.

Logged

reyalp

14080

Re: The case for Adobe Digital Negative Converter

« Reply #6 on: 18 / July / 2012, 23:45:45 »

Quote from: Rogerio on 18 / July / 2012, 22:23:37

Thank you. I thought that we had access to the Digic processor.

Technically, Digic includes both a plain vanilla ARM CPU (where CHDK and the high level logic of the Canon firmware run) and a Canon proprietary image processing system. We know very little about the latter. Reverse engineering a completely unknown architecture is much more difficult than a well documented one like the ARM.

Logged

Don't forget what the H stands for.

Mike Lee

26

Re: The case for Adobe Digital Negative Converter

« Reply #7 on: 21 / July / 2012, 18:49:53 »

Thank you gentlemen for your suggestions and observations. I apologize for the delay in responding.

I've uploaded in-camera DNG samples from an SX130 and S3:

Canon S3 - https://www.box.com/s/66fa69ca53f53f2f1538
Canon SX130 - https://www.box.com/s/786f909da63f6509427e

They are definitely not the same (variety of DNG). I am using CHDK builds from March (SX130) and April (S3) and they do not offer an option to choose the version of DNG you wish to save. Curiously, Adobe DNG Converter 7.1 offers three selectable options: 1.1, 1.3 and 1.4, but no 1.2!

The very popular Faststone Image Viewer (which I personally use) displays a 3MP preview for SX130 DNG files, and a 128x96 pixel preview (with magenta color cast) for S3 files. I can't explain that, can you? I have Faststone set to display embedded preview.

I also use Silkypix to process DNG files directly from the SX130. Silkypix will not accept S3 DNG files until they have been converted by Adobe DNG Converter. I have the converter compatibility set to Camera Raw 7.1 and later, so I would presume I am saving as DNG version 1.4.

Click the links for original in-camera DNG samples. Attached here are a few screenshots to demonstrate just a few of the compatibility issues I've encountered. Thank you for any additional suggestions, especially - how does one choose the DNG version in CHDK (as mentioned)?

Faststone Image Viewer.png (60.65 kB, 300x280 - viewed 145 times.)

S3.png (15 kB, 423x385 - viewed 155 times.)

« Last Edit: 21 / July / 2012, 19:10:18 by Mike Lee »

Logged

waterwingz

12138

Re: The case for Adobe Digital Negative Converter

« Reply #8 on: 21 / July / 2012, 18:56:03 »

Quote from: Mike Lee on 21 / July / 2012, 18:49:53

I am using CHDK builds from March (SX130) and April (S3) and they do not offer an option to choose the version of DNG you wish to save.
...
Thank you for any additional suggestions, especially - how does one choose the DNG version in CHDK (as mentioned)?

I'd start with loading the most recent version of CHDK from the dev (soon to be stable) trunk : http://mighty-hoernsche.de/trunk/

Logged

Ported : A1200 SD940 G10 Powershot N G16

Mike Lee

26

Re: The case for Adobe Digital Negative Converter

« Reply #9 on: 21 / July / 2012, 19:42:49 »

Thank you for your suggestion.

I downloaded and installed the latest versions. I created my badpixel.bin files and took a few photos, using both DNG 1.1 and 1.3. Unfortunately I can not retract any of my previous observations. Nothing has changed, and the discrepancies I noted earlier still remain.

I must admit I didn't realize that all CHDK versions (for all cameras) were updated with these new developments (e.g. DNG version selection). This particular issue, where DNG files from one model differ significantly from those from another model using the same DNG version, seems a bit more of a challenge to figure out.

Logged