Converting CHDK RAW files to something DXO PhotoLab 2 will read - page 2 - RAW Shooting and Processing - CHDK Forum supplierdeeply
« previous next »
Pages: 1 [2] 3   Go Down

Converting CHDK RAW files to something DXO PhotoLab 2 will read

  • 26 Replies
  • 4655 Views
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #10 on: 25 / May / 2020, 17:12:41 »
Advertisements
Just to see what would happen, I tried the fixdng program from the other thread on the 'good' DNG file:

Code: [Select]
./fixdng nobugDNG/IMG_2559.DNG
FixBadPixelOpcode BayerPattern is wrong. Has 03000000. Should be 02000000. CFA 01000201

Is the CFA* really as simple as

00 = blue
01 = green
02 = red

and would just changing it from 02010100 on the problematic ones to the 01000201 of the non-problematic ones (given they're the same camera) work?

* As a long-time Forth user, I think of this as 'code field address'!
Logged
*

Offline reyalp

  • ******
  • 13361
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #11 on: 25 / May / 2020, 17:57:58 »
The bayer patter is clearly correct in the DNG you uploaded.

I'm guessing DxO doesn't like the buggy forward matrix.

I've attached a lua file which can be used to patch the affected DNG files. You could also use something like exiftool.

To use the lua file, put it in the lua/extras directory of your chdkptp install
From chdkptp, load with
Code: [Select]
!require'extras/fixolds100fwm'
To fix a single file, use something like (adjusting paths as you want)
Code: [Select]
!fixfwm('IMG_1105.DNG','IMG_1105-fix.DNG')
To fix all the files in the current directory, outputting to a different directory, you could use something like
Code: [Select]
!for f in lfs.dir('.') do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end

Quote
Per another comment, the preview is heavily tinted magenta.
The thumbnail created by CHDK is always terrible. It's a very low res, primitive conversion of the raw data to keep it fast.
Logged
Don't forget what the H stands for.
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #12 on: 25 / May / 2020, 19:05:05 »
Quote from: reyalp on 25 / May / 2020, 17:57:58
I've attached a lua file which can be used to patch the affected DNG files. You could also use something like exiftool.

If that's Phil Harvey's Perl program, I'm not spotting any obvious way to do it with that, but..

Quote
To fix a single file, use something like (adjusting paths as you want)
Code: [Select]
!fixfwm('IMG_1105.DNG','IMG_1105-fix.DNG')

.. works! Thank you!

My Lua is just about up to thinking that running this on files that are ok won't do anything (tests, yes, that's right) and to altering it if the S95 examples I also have need a different fix to their matrix.

Quote
The thumbnail created by CHDK is always terrible. It's a very low res, primitive conversion of the raw data to keep it fast.

Yes, in the 'good' files, it's a dark "muddy" very low res thing. But it's not magenta :) and that provides an easy test for whether one needs this fix or not.

Thank you again.
Logged
*

Offline reyalp

  • ******
  • 13361
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #13 on: 25 / May / 2020, 19:58:16 »
Quote from: icameisaw on 25 / May / 2020, 19:05:05
My Lua is just about up to thinking that running this on files that are ok won't do anything (tests, yes, that's right) and to altering it if the S95 examples I also have need a different fix to their matrix.
Oh, sorry. I missed that you had a different camera with another problem.

There's another file in the extras directory "hackcfa.lua" which has an example of changing the CFA.

The CFA pattern is specified as you described. If you use the Lua script, you'd use like 0x01000201. Note the value is little endian, so 0x01000201 value is G B R G (edit: but, without an example, I have no idea if the CFA is wrong. It hasn't changed for S95 since 2012 at least)

This thread also has an example of changing CFA with exiftool https://chdk.setepontos.com/index.php?topic=10520.10

S95 does not have the same forward matrix problem S100 did, in fact it doesn't define ForwardMatrix at all.
« Last Edit: 25 / May / 2020, 20:01:11 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #14 on: 26 / May / 2020, 10:49:31 »
Quote from: reyalp on 25 / May / 2020, 19:58:16
Oh, sorry. I missed that you had a different camera with another problem.
..
S95 does not have the same forward matrix problem S100 did, in fact it doesn't define ForwardMatrix at all.

My memory was, in part failing. Looking, I started using the S100 in Sep 2013, but went back to the S95 with SHDK 1.3.0 from Feb 2015 to Sep 2015 for reasons I don't remember (probably not this, but the DNGs from then are indeed fine) before returning to the S100 again.

The problematic S95 ones are in CRW format from Dec 2012 and before.

Thank you again!
Logged
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #15 on: 26 / May / 2020, 10:56:07 »
Interestingly, looking at the thumbnails for the S100 DNG files that were done with the later CHDK version, there are about ten that have a magenta tint that is reflected in the actual image.

Would you like to have a look to see if they are curable?
Logged
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #16 on: 26 / May / 2020, 13:15:52 »
Quote from: reyalp on 25 / May / 2020, 17:57:58
To fix all the files in the current directory, outputting to a different directory, you could use something like
Code: [Select]
!for f in lfs.dir('.') do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end

All done! The only issue was failing in one directory:

Code: [Select]
___> !require'extras/fixolds100fwm'                 
< do if f:match('.DNG$') then fixfwm(f,'../fixed/'..f) end end               
patched IMG_1060.DNG -> ../fixed/IMG_1060.DNG
patched IMG_1082.DNG -> ../fixed/IMG_1082.DNG
patched IMG_1070.DNG -> ../fixed/IMG_1070.DNG
patched IMG_1067.DNG -> ../fixed/IMG_1067.DNG
patched IMG_1096.DNG -> ../fixed/IMG_1096.DNG
patched IMG_1071.DNG -> ../fixed/IMG_1071.DNG
patched IMG_1069.DNG -> ../fixed/IMG_1069.DNG
patched IMG_1094.DNG -> ../fixed/IMG_1094.DNG
patched IMG_1087.DNG -> ../fixed/IMG_1087.DNG
patched IMG_1093.DNG -> ../fixed/IMG_1093.DNG
patched IMG_1075.DNG -> ../fixed/IMG_1075.DNG
patched IMG_1084.DNG -> ../fixed/IMG_1084.DNG
ERROR: call failed:/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:798: size larger than data
stack traceback:
/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:798: in function 'set_data'
/home/me/Documents/CHDK/chdkptp-r921/lua/dng.lua:880: in function 'load'
...Documents/CHDK/chdkptp-r921/lua/extras/fixolds100fwm.lua:7: in function 'fixfwm'
[string "for f in lfs.dir('.') do if f:match('.DNG$') ..."]:1: in main chunk
[C]: in function 'xpcall'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:748: in function </home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:739>
(...tail calls...)
[C]: in function 'xpcall'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:285: in function 'execute'
/home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:400: in function </home/me/Documents/CHDK/chdkptp-r921/lua/cli.lua:391>
(...tail calls...)
/home/me/Documents/CHDK/chdkptp-r921/lua/main.lua:301: in main chunk
[C]: in function 'require'
[string "require('main')"]:1: in main chunk

___> quit

Because it's looking at the files in a semi-random order, it was not immediately apparent which one is causing this, but having worked out which two it was via moving files in and out of the directory, it should have been obvious: they were smaller than they should have been :)



Logged
*

Offline reyalp

  • ******
  • 13361
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #17 on: 26 / May / 2020, 13:20:08 »
Quote from: icameisaw on 26 / May / 2020, 10:56:07
Would you like to have a look to see if they are curable?
I'll take a look if you upload an example image somewhere.

Quote
The problematic S95 ones are in CRW format from Dec 2012 and before.
As I mentioned earlier, CHDK non-DNG raw (usually saved with a CRW extension) is not really a format. It's a straight dump of the sensor frame buffer, with no metadata. Some programs support it for some cams, but they can only guess based on the file size, and default rendering will generally be bad because there's no white balance etc.
If you have the corresponding jpegs for these images, it may be possible to use dng4ps2 https://chdk.fandom.com/wiki/DNG4PS-2 but in general I wouldn't bother unless they are really important to you.

Quote
but having worked out which two it was via moving files in and out of the directory, it should have been obvious: they were smaller than they should have been
CHDK DNG files for a given camera should always be exactly the same size, so if these are CHDK files, they are likely unrecoverable broken.



« Last Edit: 26 / May / 2020, 13:35:02 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline icameisaw

  • *
  • 21
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #18 on: 26 / May / 2020, 19:16:41 »
I don't have the JPEGs on this PC, although they may still be on a memory card somewhere.

Here's one from the garden of the president of Italy. I'm unlikely to be able to go around that again :)

https://www.dropbox.com/s/epxtgo3kk41jap6/IMG_1858.DNG
Logged
*

Offline reyalp

  • ******
  • 13361
Re: Converting CHDK RAW files to something DXO PhotoLab 2 will read
« Reply #19 on: 26 / May / 2020, 21:51:36 »
Quote from: icameisaw on 26 / May / 2020, 19:16:41
I don't have the JPEGs on this PC, although they may still be on a memory card somewhere.

Here's one from the garden of the president of Italy. I'm unlikely to be able to go around that again :)

https://www.dropbox.com/s/epxtgo3kk41jap6/IMG_1858.DNG
This one is weird. It looks like a bad CFA at first glance, but on closer inspection I pretty sure it's not: There are small areas of fairly solid green in the foliage. White balancing on the lighter tiles in Raw Therapee gives me the attached, which looks like it could be roughly correct, except for the sky which is massively over-exposed. Of course, I don't know what the colors on the mosaic are supposed to be, but they look Italian :haha

This maxes out the green/purple tint slider in RT, so something is clearly not right, but I don't know what it is.

The CFA related DNG values (active area, CFA pattern, default crop) are the same as the other image (IMG_1105.DNG) we looked at earlier. The camera WB values are different, but not wildly so, and swapping in the IMG_1105 white balance values doesn't have any significant effect.
Logged
Don't forget what the H stands for.
Pages: 1 [2] 3   Go Up
« previous next »
 

Related Topics