Converting CHDK RAW files to something DXO PhotoLab 2 will read

icameisaw

Converting CHDK RAW files to something DXO PhotoLab 2 will read

« on: 29 / November / 2018, 16:09:00 »

Advertisements

Having just bought this on the basis of what it could do with RAW files from my Sony DSLRs, I am a bit disappointed to discover that it doesn't seem to like ones generated by CHDK.

It will read: .CR2 files from a Powershot S95 running Canon's firmware

It won't read: .CRW files or .DNG from CHDK from the S95 or Powershot S100

(By the point I got the S100, I was enough of a CHDK convert that I don't think I have any Canon firmware photos from it yet!)

A quick look at what else I have shows that Corel AfterShot Pro 3 also doesn't like either of the CHDK ones, but darktable will read the .DNGs.

Is there any way to convert them into something it will read (.CR2 or

)

I can find plenty of ways to go from the .CR? to .DNG, but not the other way round...

Logged

icameisaw

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read

« Reply #1 on: 29 / November / 2018, 17:15:20 »

Oops, just remembered to try RawTherapee - it will do the .CR2, but not the .CRW and does .DNG badly (perhaps related to the bug mentioned elsewhere here?)

Trying with more of them PhotoLab 2 will do a much better job with the most recent .DNG - early ones have a very strong magenta cast. For some reason it improves greatly between shots taken on the same day earlier this month - two have the magenta cast, one doesn't - and ones taken last weekend are great. I will have updated the CHDK version at some point, but I'm (reasonably) sure it wasn't between those three photos.

Logged

reyalp

12868

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read

« Reply #2 on: 29 / November / 2018, 23:18:07 »

Quote from: icameisaw on 29 / November / 2018, 17:15:20

Oops, just remembered to try RawTherapee - it will do the .CR2, but not the .CRW and does .DNG badly

Generally, you'll find that software supports Canon raw much better out of the box than CHDK files. If your camera has Canon raw and your software handles it well, use it. It generally shoots a bit faster than CHDK raw/DNG too.

CHDK .CRW isn't in CRW format, it's just a frame buffer dump without any metadata. It is strongly recommended that you use DNG instead.

DNG compatible software should load CHDK DNGs, but the default rendering probably won't be as pretty as Canon raw. You should be able to get results equivalent to Canon raw with appropriate settings, lens correction and so on.

Quote

(perhaps related to the bug mentioned elsewhere here?)

It would be helpful if you referenced the specific bug, there a lot to choose from

Quote

Trying with more of them PhotoLab 2 will do a much better job with the most recent .DNG - early ones have a very strong magenta cast. For some reason it improves greatly between shots taken on the same day earlier this month - two have the magenta cast, one doesn't - and ones taken last weekend are great.

IMO, the difference almost certainly has to do with the scene and exposure, not the day. Many programs apply a contrast stretch to raw images, which can go wildly wrong in some cases. This is particularly common when large areas of the image should be dark. Some versions of raw therapee do this by default. Pressing the "reset" or "neutral" button on the exposure tab should get you to default settings. The default settings will probably look very dull compared to canon jpeg colors.

If you upload examples of your good and bad DNGs somewhere (they are too large for the forum, use google drive, dropbox etc), I can take a look.

Quote

I will have updated the CHDK version at some point, but I'm (reasonably) sure it wasn't between those three photos.

There haven't been any changes that should affect the look of raw files in a long time.

Logged

Don't forget what the H stands for.

icameisaw

Correcting CHDK 1.3 DNG files with chdkptp

« Reply #3 on: 25 / May / 2020, 06:23:51 »

https://app.assembla.com/spaces/chdkptp/wiki/DNG_Processing suggests that chdkptp can correct the CHDK 1.3 bug that meant DNG files weren't written correctly.

However, if I try doing

Code: [Select]

dngbatch ../bugDNG/ -odir=../goodDNG/ { mod -patch ; save }the resulting file is identical to the original.

What am I doing wrong?

Code: [Select]

___> dnginfo
../bugDNG/IMG_1105.DNG:
4160x3124x12, RGGB, DNG 1.3.0.0 / 1.1.0.0, CHDK ver. 1.3.0, Canon PowerShot S100

Logged

icameisaw

Re: Correcting CHDK 1.3 DNG files with chdkptp

« Reply #4 on: 25 / May / 2020, 06:27:16 »

Or, alternatively, what does actually correct them so that DxO Photolab will read them properly?

Logged

icameisaw

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read

« Reply #5 on: 25 / May / 2020, 06:38:26 »

Argh, I missed this reply - sorry.

Here's one it won't load:

https://www.dropbox.com/s/1jhf1fwxvk7bki2/IMG_1105.DNG?dl=0

Code: [Select]

___> dnginfo
../bugDNG/IMG_1105.DNG:
4160x3124x12, RGGB, DNG 1.3.0.0 / 1.1.0.0, CHDK ver. 1.3.0, Canon PowerShot S100

Logged

reyalp

12868

Re: Correcting CHDK 1.3 DNG files with chdkptp

« Reply #6 on: 25 / May / 2020, 15:04:11 »

Quote from: icameisaw on 25 / May / 2020, 06:23:51

https://app.assembla.com/spaces/chdkptp/wiki/DNG_Processing suggests that chdkptp can correct the CHDK 1.3 bug that meant DNG files weren't written correctly.

Not really. It can interpolate over bad pixels for programs that don't implement DNG 1.3.

Quote

However, if I try doing
Code: [Select]
dngbatch ../bugDNG/ -odir=../goodDNG/ { mod -patch ; save }the resulting file is identical to the original.

Do you mean byte identical (with md5 sum or something like that), or they look identical?
chdkptp print say something like
patched <some number> pixels
if it found pixels below the threshold.

Quote

4160x3124x12, RGGB, DNG 1.3.0.0 / 1.1.0.0, CHDK ver. 1.3.0, Canon PowerShot S100

patch doesn't change the version, it just interpolates over pixels at or below a certain value (0 if you don't specify)

Quote

Or, alternatively, what does actually correct them so that DxO Photolab will read them properly?

What specific problem do you see when you open them in DxO?

Logged

Don't forget what the H stands for.

reyalp

12868

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read

« Reply #7 on: 25 / May / 2020, 15:35:55 »

Raw therapee 5.8 loads that file fine for me.

With the default profile, the colors are quite flat, which is normal. There is also some CA, which you can correct in the raw tab.

dng_validate complains about the active area and forward matrix.

Code: [Select]

*** Warning: Too little padding on left edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on top edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on right edge of CFA image (possible interpolation artifacts) ***
*** Warning: Too little padding on bottom edge of CFA image (possible interpolation artifacts) ***
*** Error: ForwardMatrix2 does not map equal camera values to XYZ D50 ***
*** Error: File format is invalid ***

The active area can be controlled in CHDK settings, and should only hurt the very edges of the images at worst.

The forward matrix issue appears to be a bug in the port, which should have been fixed in 2014, build 3497 and later https://chdk.setepontos.com/index.php?topic=7887.msg114303#msg114303

It is probably possible to correct these tags using chdkptp, if they are causing problems.

Logged

Don't forget what the H stands for.

reyalp

12868

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read

« Reply #8 on: 25 / May / 2020, 15:38:19 »

I merged these two topics, as they seem to be about the same issue.

Logged

Don't forget what the H stands for.

icameisaw

Re: Correcting CHDK 1.3 DNG files with chdkptp

« Reply #9 on: 25 / May / 2020, 17:05:48 »

Thanks!

Quote from: reyalp on 25 / May / 2020, 15:04:11

Not really. It can interpolate over bad pixels for programs that don't implement DNG 1.3.

Ah, ok.

Quote

Do you mean byte identical (with md5 sum or something like that)

Byte identical, same SHA256 hash.

Quote

patch doesn't change the version, it just interpolates over pixels at or below a certain value (0 if you don't specify)

I wasn't expecting the version to change, that was me trying to be helpful by showing the versions and camera involved

Quote

What specific problem do you see when you open them in DxO?

They don't open! DxO PhotoLab says 'Cannot load image data'. Per another comment, the preview is heavily tinted magenta. Skylum's programs (Luminar, AuroraHDR, Photolemur) don't open it either.

Serif's Affinity Photo will open them, so clearly it's not as fussy about correct format.. but it's also nowhere near as good at developing RAW files as PhotoLab is (or as neat as the Skylum ones are).

If I look at one that was done after I upgraded CHDK past 1.3.0, I see

Code: [Select]

___>___> dnginfo
../nobugDNG/IMG_2559.DNG:
4160x3124x12, GBRG, DNG 1.3.0.0 / 1.1.0.0, CHDK ver. 1.5.0, Canon PowerShot S100
___>

.. so the other explanation I've seen is that the Bayer colour matrix is wrong in the buggy ones. (It would certainly explain the magenta preview image issue.)

But the program that's elsewhere here (https://chdk.setepontos.com/index.php?topic=13060.msg131602#msg131602) makes some change..

Code: [Select]

./fixdng bugDNG/IMG_1105.DNG
FixBadPixelOpcode BayerPattern is wrong. Has 01000000. Should be 00000000. CFA 02010100

.. and the hash indeed changes, but the result still doesn't load in PhotoLab and has the same result if I look at dnginfo:

Code: [Select]

___> dnginfo
../bugDNG/IMG_1105.DNG:
4160x3124x12, RGGB, DNG 1.3.0.0 / 1.1.0.0, CHDK ver. 1.3.0, Canon PowerShot S100

So something else needs changing. If necessary, I can write a 'load image assuming it's buggy, write it so it's not' program that looks at the whole thing (step one, learn the file format!) but it must be easier than that, possibly just changing a handful of bytes.

Logged