DNG calibration, again (was Re: pixls.us wants your raw samples) - RAW Shooting and Processing - CHDK Forum supplierdeeply

DNG calibration, again (was Re: pixls.us wants your raw samples)

  • 23 Replies
  • 23276 Views
Advertisements
Finally got around to looking at this again as I was cleaning up the ixus300 code.  Which got me to the CAM_COLORMATRIX1 values - the color calibration part of a DNG image.  And that got me to wondering just where those values come?  My tentative conclusion is that in many cases, they may just be a C&P from another port.

If so, I probably need to validate (or replace) the ones in the cameras I've ported before posting to pixls.us .  And I'm not sure that I understand how that happens.  Doing a detailed forum search produces the usual array of somewhat relevant posts but nothing really definitive that I could find.

But I did find this : DNG4PS-2/Calibration  and this Adobe DNG Profile Editor Tutorial. Before I go down this route, are these the right approach?

Also, both docs require a target image. The link in the first doc ( pallete.PNG ) is dead but I did find some samples online like (Color Checker Color Rendition Chart) but it's not clear to me how useful they are? I have access to a $1000 calibrated IPS UHD monitor but if you don't have some sort of calibration value for each of the 18 colors how does this work?  (edit : Color Checker Colors)

Or do I need to buy one of those $69  X-Rite ColorChecker Classic Cards to really do this properly?

edit : Found this New project: camera color profile calculation : "Gamma value and monitor calibration no matter because the program calculates color matrix by comparing RAW and Jpeg image, it doesn't use only RAW"

Interesting!  But then I also found this :
CHDK DNG do contain a color matrix, but the quality varies between ports and there no simple, documented process to make a good calibration (dng4ps2 can be used for this, but in practice it is difficult or impossible to get it to work on current cameras). If you come up with a better calibration, we can include it in the source.
« Last Edit: 13 / July / 2017, 21:51:42 by waterwingz »
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 14079
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #1 on: 13 / July / 2017, 21:13:45 »
Finally got around to looking at this again as I was cleaning up the ixus300 code.  Which got me to the CAM_COLORMATRIX1 values - the color calibration part of a DNG image.  And that got me to wondering just where those values come?  My tentative conclusion is that in many cases, they may just be a C&P from another port.
Yup.

Quote
If so, I probably need to validate (or replace) the ones in the cameras I've ported before posting to pixls.us.
If you have the capability, it would be good to fix them, but the reality is most CHDK ports aren't properly calibrated.

Quote
edit : Found this New project: camera color profile calculation : "Gamma value and monitor calibration no matter because the program calculates color matrix by comparing RAW and Jpeg image, it doesn't use only RAW"

Interesting!  But then I also found this :
CHDK DNG do contain a color matrix, but the quality varies between ports and there no simple, documented process to make a good calibration (dng4ps2 can be used for this, but in practice it is difficult or impossible to get it to work on current cameras). If you come up with a better calibration, we can include it in the source.
If you can make DNG4PS2 work, that should be good enough. If you can't, I don't know the solution. The elph130 and sx160 both have C&P, and they seem fairly OK (meaning "neutral" settings in raw therapee have reasonable colors).

For cameras with native canon raw, where you can get the color matrix directly from that or from converting with Adobe DNG converter.

If someone could make a standalone utility like the DNG4PS2 process that actually worked with modern cams, that would be great ;)
Don't forget what the H stands for.

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #2 on: 13 / July / 2017, 22:24:26 »
The elph130 and sx160 both have C&P, and they seem fairly OK (meaning "neutral" settings in raw therapee have reasonable colors).
Interesting. 

Getting back to the original subject matter of this thread,  I guess the pixls.us project mostly cares about the row/column values needed to interpret different RAW files - DNG is easier due to that data being in the header.  Inaccurate color calibration probably does not matter to the project?
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 14079
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #3 on: 13 / July / 2017, 22:57:07 »
Getting back to the original subject matter of this thread,  I guess the pixls.us project mostly cares about the row/column values needed to interpret different RAW files - DNG is easier due to that data being in the header.  Inaccurate color calibration probably does not matter to the project?
I think it's mostly about having examples of what a RAW program might encounter in the wild. It does says "if you can provide a more useful photo from a camera model we already support (e.g. a photo of a color target)..."
Don't forget what the H stands for.


Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #4 on: 13 / July / 2017, 23:52:58 »
It does says "if you can provide a more useful photo from a camera model we already support (e.g. a photo of a color target)..."
I noticed that - and do not remember that from when I originally looked at it. 

Do you interpret that to mean a color target photo of something like the ColorChecker Classic Card (or maybe a shot of it from a calibrated monitor) instead of a "normal landscape mode" photo - or in addition to?
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #5 on: 14 / July / 2017, 01:34:28 »
It does says "if you can provide a more useful photo from a camera model we already support (e.g. a photo of a color target)..."
I noticed that - and do not remember that from when I originally looked at it. 

Do you interpret that to mean a color target photo of something like the ColorChecker Classic Card (or maybe a shot of it from a calibrated monitor) instead of a "normal landscape mode" photo - or in addition to?


http://www.ludd.ltu.se/~torger/photography/camera-profiling.html


Recommended shooting a good quality colour checker card, not sure you would get good results from an LCD no matter how well calibrated. You can't daylight and tungsten illuminate an LCD since it's already a light source.


Phil.

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #6 on: 14 / July / 2017, 02:17:45 »
Quote from: philmoz link=topic=13076.msg133924#msg133924 date=
http://www.ludd.ltu.se/~torger/photography/camera-profiling.html
Wow. This takes pixel peeping to a new level.  I tend to look for subject matter first, composition next, followed by lighting and everything else.  I really had to strain to see the differences in the test images - and even the author called them subtle.

Quote
Recommended shooting a good quality colour checker card, not sure you would get good results from an LCD no matter how well calibrated. You can't daylight and tungsten illuminate an LCD since it's already a light source.
Thanks!  Good advice.

I've already spent enough time on this to have paid for a $70 card from B&H.  But looking at all the different CHDK ports, I guess I was also looking for something that would be "good enough" and better than the C&P matrix values too many of the ports have used.
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 14079
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #7 on: 14 / July / 2017, 16:17:22 »
Do you interpret that to mean a color target photo of something like the ColorChecker Classic Card (or maybe a shot of it from a calibrated monitor) instead of a "normal landscape mode" photo - or in addition to?
I would personally stay away from using a monitor, no matter how well calibrated. (dng4ps2 is a special case because it's only trying to match the camera jpeg, but they don't collect camera jpegs)

The person I talked to about the original request was LebedevRI in the #pixls.us on freenode, it would probably be better to talk to them directly if you want to do more than just a basic sample. My impression was they are pretty happy with just a sample raw that has a recognizable image (as they put it: Image in focus and properly exposed, Landscape orientation)


IMO, the best general solution for CHDK calibration would be a DNG4PS2 style "match the jpeg" system. In theory, it should be possible to make this almost totally automatic. Unless we can extract something convertible to a DNG color matrix from the firmware...
Don't forget what the H stands for.


*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #8 on: 14 / July / 2017, 18:01:48 »
Unless we can extract something convertible to a DNG color matrix from the firmware...


I spent quite a while trying to find something like this for the G1X; but I wasn't able to find anything useful.


I also came to the conclusion that Canon don't embed any sort of usable colour profile in their CR2 raw file format - the matrices extracted by the DNG convertor trick are actually from the Adobe Camera Raw standard camera profiles.


Even dcraw doesn't know how to extract this from CR2 files - it also has copies of the ACR matrices in the code for supported cameras.


Phil.


(This reminds me I need to finish creating the matrices for the IXUS 310 and SX40 someday.)

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #9 on: 14 / July / 2017, 18:27:56 »
IMO, the best general solution for CHDK calibration would be a DNG4PS2 style "match the jpeg" system.


Best for keeping the general populace happy; but probably the worst solution photographically.


This is essentially what Adobe try to do with their standard camera profiles (and what you will get close to by embedding the ACR matrices in the DNG).


But the ACR profiles have a very aggressive tone curve built in that crushes the blacks and kills highlights in order to get a punchy, contrasty default look that is close the camera JPEG. Even the so called 'camera neutral' profile in ACR does this.


To extract the most data out of the image you need to start from a linear profile; but the problem with this is the images look like [admin: avoid swearing please] when you first load them (extremely flat and insipid). Adobe knows that this would create a negative impression (especially for Lightroom) so they don't do it - unfortunately they don't even provide the option for people who want to use linear profiles.


Phil.

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)