Converting CHDK RAW files to something DXO PhotoLab 2 will read - page 2 - RAW Shooting and Processing - CHDK Forum supplierdeeply

Converting CHDK RAW files to something DXO PhotoLab 2 will read

  • 26 Replies
  • 4816 Views
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #10 on: 25 / May / 2020, 17:12:41 »
Advertisements
Just to see what would happen, I tried the fixdng program from the other thread on the 'good' DNG file:

Code: [Select]
./fixdng nobugDNG/IMG_2559.DNG
FixBadPixelOpcode BayerPattern is wrong. Has 03000000. Should be 02000000. CFA 01000201

Is the CFA* really as simple as

00 = blue
01 = green
02 = red

and would just changing it from 02010100 on the problematic ones to the 01000201 of the non-problematic ones (given they're the same camera) work?

* As a long-time Forth user, I think of this as 'code field address'!

*

Offline reyalp

  • ******
  • 13393
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #11 on: 25 / May / 2020, 17:57:58 »
The bayer patter is clearly correct in the DNG you uploaded.

I'm guessing DxO doesn't like the buggy forward matrix.

I've attached a lua file which can be used to patch the affected DNG files. You could also use something like exiftool.

To use the lua file, put it in the lua/extras directory of your chdkptp install
From chdkptp, load with
Code: [Select]
!require'extras/fixolds100fwm'
To fix a single file, use something like (adjusting paths as you want)
Code: [Select]
!fixfwm('IMG_1105.DNG','IMG_1105-fix.DNG')
To fix all the files in the current directory, outputting to a different directory, you could use something like
Code: [Select]
!for f in lfs.dir('.') do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end

Quote
Per another comment, the preview is heavily tinted magenta.
The thumbnail created by CHDK is always terrible. It's a very low res, primitive conversion of the raw data to keep it fast.
Don't forget what the H stands for.

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #12 on: 25 / May / 2020, 19:05:05 »
I've attached a lua file which can be used to patch the affected DNG files. You could also use something like exiftool.

If that's Phil Harvey's Perl program, I'm not spotting any obvious way to do it with that, but..

Quote
To fix a single file, use something like (adjusting paths as you want)
Code: [Select]
!fixfwm('IMG_1105.DNG','IMG_1105-fix.DNG')

.. works! Thank you!

My Lua is just about up to thinking that running this on files that are ok won't do anything (tests, yes, that's right) and to altering it if the S95 examples I also have need a different fix to their matrix.

Quote
The thumbnail created by CHDK is always terrible. It's a very low res, primitive conversion of the raw data to keep it fast.

Yes, in the 'good' files, it's a dark "muddy" very low res thing. But it's not magenta :) and that provides an easy test for whether one needs this fix or not.

Thank you again.

*

Offline reyalp

  • ******
  • 13393
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #13 on: 25 / May / 2020, 19:58:16 »
My Lua is just about up to thinking that running this on files that are ok won't do anything (tests, yes, that's right) and to altering it if the S95 examples I also have need a different fix to their matrix.
Oh, sorry. I missed that you had a different camera with another problem.

There's another file in the extras directory "hackcfa.lua" which has an example of changing the CFA.

The CFA pattern is specified as you described. If you use the Lua script, you'd use like 0x01000201. Note the value is little endian, so 0x01000201 value is G B R G (edit: but, without an example, I have no idea if the CFA is wrong. It hasn't changed for S95 since 2012 at least)

This thread also has an example of changing CFA with exiftool https://chdk.setepontos.com/index.php?topic=10520.10

S95 does not have the same forward matrix problem S100 did, in fact it doesn't define ForwardMatrix at all.
« Last Edit: 25 / May / 2020, 20:01:11 by reyalp »
Don't forget what the H stands for.


Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #14 on: 26 / May / 2020, 10:49:31 »
Oh, sorry. I missed that you had a different camera with another problem.
..
S95 does not have the same forward matrix problem S100 did, in fact it doesn't define ForwardMatrix at all.

My memory was, in part failing. Looking, I started using the S100 in Sep 2013, but went back to the S95 with SHDK 1.3.0 from Feb 2015 to Sep 2015 for reasons I don't remember (probably not this, but the DNGs from then are indeed fine) before returning to the S100 again.

The problematic S95 ones are in CRW format from Dec 2012 and before.

Thank you again!

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #15 on: 26 / May / 2020, 10:56:07 »
Interestingly, looking at the thumbnails for the S100 DNG files that were done with the later CHDK version, there are about ten that have a magenta tint that is reflected in the actual image.

Would you like to have a look to see if they are curable?

Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #16 on: 26 / May / 2020, 13:15:52 »
To fix all the files in the current directory, outputting to a different directory, you could use something like
Code: [Select]
!for f in lfs.dir('.') do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end

All done! The only issue was failing in one directory:

Code: [Select]
___> !require'extras/fixolds100fwm'                 
< do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end               
patched IMG_1060.DNG -> ../fixed/IMG_1060.DNG
patched IMG_1082.DNG -> ../fixed/IMG_1082.DNG
patched IMG_1070.DNG -> ../fixed/IMG_1070.DNG
patched IMG_1067.DNG -> ../fixed/IMG_1067.DNG
patched IMG_1096.DNG -> ../fixed/IMG_1096.DNG
patched IMG_1071.DNG -> ../fixed/IMG_1071.DNG
patched IMG_1069.DNG -> ../fixed/IMG_1069.DNG
patched IMG_1094.DNG -> ../fixed/IMG_1094.DNG
patched IMG_1087.DNG -> ../fixed/IMG_1087.DNG
patched IMG_1093.DNG -> ../fixed/IMG_1093.DNG
patched IMG_1075.DNG -> ../fixed/IMG_1075.DNG
patched IMG_1084.DNG -> ../fixed/IMG_1084.DNG
ERROR: call failed:/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:798: size larger than data
stack traceback:
/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:798: in function 'set_data'
/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:880: in function 'load'
...Documents/CHDK/chdkptp-r921/lua/extras/fixolds100fwm.lua:7: in function 'fixfwm'
[string "for f in lfs.dir('.') do if f:match('.DNG$') ..."]:1: in main chunk
[C]: in function 'xpcall'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:748: in function </home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:739>
(...tail calls...)
[C]: in function 'xpcall'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:285: in function 'execute'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:400: in function </home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:391>
(...tail calls...)
/home/me/Documents/CHDK/chdkptp-r921/lua/main.lua:301: in main chunk
[C]: in function 'require'
[string "require('main')"]:1: in main chunk

___> quit

Because it's looking at the files in a semi-random order, it was not immediately apparent which one is causing this, but having worked out which two it was via moving files in and out of the directory, it should have been obvious: they were smaller than they should have been :)




*

Offline reyalp

  • ******
  • 13393
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #17 on: 26 / May / 2020, 13:20:08 »
Would you like to have a look to see if they are curable?
I'll take a look if you upload an example image somewhere.

Quote
The problematic S95 ones are in CRW format from Dec 2012 and before.
As I mentioned earlier, CHDK non-DNG raw (usually saved with a CRW extension) is not really a format. It's a straight dump of the sensor frame buffer, with no metadata. Some programs support it for some cams, but they can only guess based on the file size, and default rendering will generally be bad because there's no white balance etc.
If you have the corresponding jpegs for these images, it may be possible to use dng4ps2 https://chdk.fandom.com/wiki/DNG4PS-2 but in general I wouldn't bother unless they are really important to you.

Quote
but having worked out which two it was via moving files in and out of the directory, it should have been obvious: they were smaller than they should have been
CHDK DNG files for a given camera should always be exactly the same size, so if these are CHDK files, they are likely unrecoverable broken.



« Last Edit: 26 / May / 2020, 13:35:02 by reyalp »
Don't forget what the H stands for.


Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #18 on: 26 / May / 2020, 19:16:41 »
I don't have the JPEGs on this PC, although they may still be on a memory card somewhere.

Here's one from the garden of the president of Italy. I'm unlikely to be able to go around that again :)

https://www.dropbox.com/s/epxtgo3kk41jap6/IMG_1858.DNG

*

Offline reyalp

  • ******
  • 13393
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #19 on: 26 / May / 2020, 21:51:36 »
I don't have the JPEGs on this PC, although they may still be on a memory card somewhere.

Here's one from the garden of the president of Italy. I'm unlikely to be able to go around that again :)

https://www.dropbox.com/s/epxtgo3kk41jap6/IMG_1858.DNG
This one is weird. It looks like a bad CFA at first glance, but on closer inspection I pretty sure it's not: There are small areas of fairly solid green in the foliage. White balancing on the lighter tiles in Raw Therapee gives me the attached, which looks like it could be roughly correct, except for the sky which is massively over-exposed. Of course, I don't know what the colors on the mosaic are supposed to be, but they look Italian :haha

This maxes out the green/purple tint slider in RT, so something is clearly not right, but I don't know what it is.

The CFA related DNG values (active area, CFA pattern, default crop) are the same as the other image (IMG_1105.DNG) we looked at earlier. The camera WB values are different, but not wildly so, and swapping in the IMG_1105 white balance values doesn't have any significant effect.
Don't forget what the H stands for.

 

Related Topics