DNG calibration, again (was Re: pixls.us wants your raw samples) - page 2 - RAW Shooting and Processing - CHDK Forum

DNG calibration, again (was Re: pixls.us wants your raw samples)

  • 23 Replies
  • 20447 Views
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #10 on: 14 / July / 2017, 18:37:57 »
Advertisements
To extract the most data out of the image you need to start from a linear profile; but the problem with this is the images look like [admin: avoid swearing please] when you first load them (extremely flat and insipid). Adobe knows that this would create a negative impression (especially for Lightroom) so they don't do it - unfortunately they don't even provide the option for people who want to use linear profiles.
I asked about embedding a linear profile on CHDK IRC last night - mostly because I was looking for a neutral starting point for exploring calibration.  But now the possibility of embedding one in the DNG gets interesting based on your comments. 

If I understand correctly, that would give you the most useful image for extracting data as an input to something like Lightroom, as Lightroom won't let you override the DNG's built-in calibration matrix with a neutral one?

What if we add an option to the CHDK DNG menu for calibration?

DNG Calibration [ Normal ]   or   DNG Calibration [ Flat      ]

Default to normal so the punters don't get confused but make the other choice an option?

Or is creating a linear profile tricky too?

Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline philmoz

  • *****
  • 3332
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #11 on: 14 / July / 2017, 19:31:34 »
 
I asked about embedding a linear profile on CHDK IRC last night - mostly because I was looking for a neutral starting point for exploring calibration.  But now the possibility of embedding one in the DNG gets interesting based on your comments. 


If I understand correctly, that would give you the most useful image for extracting data as an input to something like Lightroom, as Lightroom won't let you override the DNG's built-in calibration matrix with a neutral one?


If you have alternate profiles installed you can select them in LR (you can even set the default import profile). So it's easy to use a linear profile if you have one.


Quote
What if we add an option to the CHDK DNG menu for calibration?


DNG Calibration [ Normal ]   or   DNG Calibration [ Flat      ]


Default to normal so the punters don't get confused but make the other choice an option?


This should be possible; but I'm not sure what the results would be like - it's probably worth experimenting with.


The colour matrices we embed into DNG files aren't a complete profile; just the basic RAW conversion to something more closely resembling the colours we see. The ACR camera profiles have a lot more stuff in them (not all of which I understand).


Quote
Or is creating a linear profile tricky too?


I know of two options for creating a linear profile that will work with ACR / LR:
- creating a true custom camera profile using a colour checker and software like dcamprof, and saving with a linear tone curve. This is not all that hard; but requires a colour card and extra software (after my previous post I found that the authors of dcamprof have released a new program called Lumariver Profile Designer - it's not free; but it make the process a lot easier).
- create a copy of an existing ACR profile with a linear tone curve - see https://www.dpreview.com/forums/post/58785763 for instructions on doing this.


Phil
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #12 on: 14 / July / 2017, 19:39:23 »
This is not all that hard; but requires a colour card and extra software
I waited my mandatory 24 hours before ordering something I think that I want (avoids a lot of really bad purchases that way) and ordered a color card. Shipwright's disease again.
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline philmoz

  • *****
  • 3332
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #13 on: 14 / July / 2017, 20:06:08 »
It seems I was not entirely correct in my previous posts (not the first time  :) ).


The colour matrices we embed in the DNG files don't have any tone curve - obvious when you thing about it.
There is another tag in the DNG file (Tone Curve - doh), that can be set to control the rendering tone curve.
If this is not set then LR automatically applies it's standard tone curve. Setting this to (0,0 - 1,1) gives a linear tone curve in LR.


This might be an option worth to add to the DNG settings in CHDK.


Linear profiles are still useful in LR (at least to me); but not so much for CHDK - getting good matrices is key.


Phil.

CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)


Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #14 on: 14 / July / 2017, 20:16:29 »
This might be an option worth to add to the DNG settings in CHDK.
Easy enough to do and pretty much no risk. 

Quote
Linear profiles are still useful in LR (at least to me); but not so much for CHDK - getting good matrices is key.
So an option for linear makes some sense?

This is pretty much where I came into this discusssion.  If most of the CHDK ports are just using C&P matrices from cameras that used CRT projections of color cards (lots of posts about that - it's where I got the bad idea) then we are off to a bad start. Or do "pixel peepers" using full Photoshop just ignore the embedded DNG info and do their own corrections?  And for everyone else a shared matrix is somehow close enough?
Ported :   A1200    SD940   G10    Powershot N    G16

*

Offline reyalp

  • ******
  • 13446
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #15 on: 14 / July / 2017, 21:46:56 »
Best for keeping the general populace happy; but probably the worst solution photographically.
I meant "best" as an achievable lowest common denominator that would be better than copy/pasting from a random existing port. Realistically any solution that requires specific hardware or complicated processes isn't going to be done for the vast majority of ports.
Don't forget what the H stands for.

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #16 on: 16 / July / 2017, 10:38:36 »
@reyalp :   I was scrolling through the pixls.us user forum and found this comment from six months ago when this all started:

Rawsamples.ch and hosting raw sample files

Quote
damonlynchDec '16
I've analyzed every file in the raw sample DB. Last I checked those CHDK files are impossible to get metadata out of using mainstream free software tools. Maybe that's not important to you as someone interested primarily in rawspeed? But it is important to myself, and maybe to others too. Personally I find them useful as samples of garbage input that my program should handle gracefully. So I think it would be very useful for the DB to differentiate between categories of RAW.

Were you able to explain to them what a CHDK RAW contains (i.e. image data only)? 
Ported :   A1200    SD940   G10    Powershot N    G16

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #17 on: 16 / July / 2017, 10:39:02 »
This might be an option worth to add to the DNG settings in CHDK. Linear profiles are still useful in LR (at least to me); but not so much for CHDK - getting good matrices is key.

tl;dr -> most popular RAW processing software will usually ignore the embedded exif Color Matrix calibration data in a DNG image if it can.

Did a bunch of experimenting yesterday with the CAM_COLORMATRIX1 using my G10 and thought I would record my findings here.

I had observed that the Color Matrix 1 tag values were all incorrectly scaled to be 100x too small but that it did not seem to make any difference when I fixed that (looking at before and after images in Darkroom,  RawTherapee, UFRaw). 

After much experimentation, here's what I think  I learned (mostly about RAW software) :
  • The embedded calibration matrix is usually ignored if the software has a valid profile for the camera model.
  • The software gets the camera model info from the DNG header. If I used hexedit to change G10 to G13 in the image exif header, then the camera became "unknown" and the software's G10 profile calibration was no longer used.  (Yes - I know dcraw determines cameras for RAW from the file size but I was only working with DNG's)
  • RawTherapee's  Color Management -> Input Profile will let you select between No ProfileCamera Standard , and Auto-matched camera profile.   The Camera Standard setting will sometimes use the embedded DNG exif Color Matrix 1 values (see below).
  • I used hexedit to purposely create a wildly wrong Color Matrix 1 in one DNG image (I rescaled every second value by setting the two high order bytes to zero). With RawTherapee set to Camera Standard and the camera model edited to G13 (a non-existant camera) the colors of the resulting image were truly bizarre.
  • If I used the exact same settings and wildly wrong Color Matrix 1 values with a valid camera model (G10) in the exif info, the exif Color Matrix 1 values were ignored!
  • The original G10 Color Matrix 1 values that we all off by 100x worked okay with the camera info set to G13 and Camera Standard  selected because apparently scaling them all by the same amount cancels out in the matrix math used to apply them?

It might be time to move the DNG Color Matrix part of the discussion to its own thread?  I google searched for something existing on DNG calibration but the existing threads have been inactive for over seven years.
Ported :   A1200    SD940   G10    Powershot N    G16


*

Offline philmoz

  • *****
  • 3332
    • Photos
Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #18 on: 16 / July / 2017, 18:23:54 »
This might be an option worth to add to the DNG settings in CHDK. Linear profiles are still useful in LR (at least to me); but not so much for CHDK - getting good matrices is key.

tl;dr -> most popular RAW processing software will usually ignore the embedded exif Color Matrix calibration data in a DNG image if it can.

  • The embedded calibration matrix is usually ignored if the software has a valid profile for the camera model.


Yep - that's why we can extract the matrices from the Adobe profiles and use them. Profiles contain their own matrices so if the software finds a profile it ignores the embedded ones.


Overall finding matrices for cameras with native RAW support is probably not all that important, since most software will probably have built in profiles (or use the Adobe ones), and whatever we put in will be ignored.


For cameras without native raw, profiles won't exist so it would be good to get decent matrices if possible.


I spent some time with the DNG4PS2 code over the weekend, and I was not able to get it to calculate a matrix for the IXUS 310 no matter what I tried. Looking at the code I can't see how it was ever able to work. Essentially it takes the JPEG image and converts it to XYZ colour space with the standard sRGB matrix, then tries to find a matrix that will convert the DNG raw data to the same (or close) XYZ values. Since the JPEG was created (in camera) with unknown hue/saturation maps and an unknown tone curve, I can't see how this could possibly work.
The code applies an increasing gamma correction which might be trying to compensate for the tone curve on the JPEG. Maybe the raw data from the old CCD sensors was closer to the final JPEG than our current CMOS sensors, and just applying a gamma correction worked.


If anyone ever got DNG4PS2 to work creating a colour matrix, I'd be interested in hearing how.

I have been able to create good profiles using a colour checker card - these work well in Lightroom so not having the right matrices in the CHDK DNG files becomes less of an issue for me (one reason I sort of forgot about the IXUS 310 and SX40).
One option might be to try and create a collection of proper colour profiles for cameras without native RAW and host them somewhere.

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)

Re: DNG calibration, again (was Re: pixls.us wants your raw samples)
« Reply #19 on: 16 / July / 2017, 18:37:50 »
One option might be to try and create a collection of proper colour profiles for cameras without native RAW and host them somewhere.
Should I assume by this you mean that color profiles created with a color card and standard tools are not easy to translate back into exif CAM_COLORMATRIX info for DNG headers?

Frankly, the whole idea of getting DNG  exif calibration data for Powershots that support RAW natively is pretty much a redundant exercise, isn't it?   The real fun of CHDK is getting RAW support for the low end P&S cams that would not have it otherwise.  And that needs the color profiles you mention - or the ability to convert those back into DNG CAM_COLORMATRIX values.

Unless you also think that CHDK DNG gives us something that Canon native CR2 does not?

Failing the ability to get those into the DNG header somewhat accurately  with CHDK,  a repo of color profiles would make sense. Except for the punters having to pay $70+ for a color card ($25 for a cheap ebay knock-off from china) to calibrate a five year old+ Powershot bought off eBay fo $15 ..

To quote Gilda Radner "If it's not one thing, it's another".
Ported :   A1200    SD940   G10    Powershot N    G16