New project: camera color profile calculation

hiker_jon

I have an A720. DNG4PS works great for me, but I would like to also use DCraw directly on my camera raw files. DCraw can read the data, but its colors are way wrong. Does anyone know how to modify the source for DCraw to get the colors right? I find the code in that program particularly hard to understand.

brainwash

Are you using DCRaw as-is without any frontend? You should try UFRaw as frontend. RAW files usually have no color balance set, this is usually and advantage since you could set it manually without inducing banding in the histogram. These front-ends have options for setting color balance (cyan-magenta is the most used in-camera) and also an auto-balance setting.

hiker_jon

brainwash
Are you using DCRaw as-is without any frontend? You should try UFRaw as frontend.

Unfortunately the current version (.13) of UFRaw cannot read my A720 raw files.

I find color matching to be a mystery. I need to start with a good approximation to Canon's jpegs. dng4ps conversion produces good results. Rawtherapee and dcraw give poor results when used directly on the A720 raw file

brainwash

dng4ps is currently outdated, so it seemed for my S2IS. Please use dng4ps2 which might give an accurate color balance (the camera's, that is).
But, as I stated, RAW files as written by the camera do not have any white balance performed on them (someone please correct me if I'm wrong).
Also, DNG4PS2 has profile information for A720IS: DNG4PS-2 but I think you must be using the latest beta version, which is what I'm using.

oversky

Hi, ArtDen

Have you consider to use the test image from imaging-resource? Such as
Digital Cameras, Canon PowerShot A650 IS Digital Camera Test Image

Here are some good reasons:
1. The test image was taken from real color check board. This eliminates the color issue from the current LCD method.
2. The lab has very high qulity lighting setup. According to Digital Camera Test Lighting - HMIs! , it gives true ~5500K daylight, balanced to within 0.1EV across our entire target area. This is definitely much better than the most users on this forum.
3. There are three different color check boards in the test image. These should result in a more accurate matrix.
4. imaging-resource has tested almost all canon cameras.

The only issue I see from here is that we do not know exactly how they setup the camera. For example, according to the exif of the test image, the canon a650is uses custom WB on the test image, while G9 uses auto WB. Following is how they prepare for the test image, Imaging Resource Digital Camera Test Methods. It would be interested to compare it to our current method.

OK, I see the problem. We don't have the raw file.

lukg

[unknown]
lukg,

I've tried to download your S5 calibration photos and 7zip won't open the rar archives you've uploaded, and some of the links you've included result in "file not found" errors.

Can you please try to recompress and upload the calibration photos?

Thanks,
Matt

Ok!!! I really appreciated the work you're doing!

EDIT: I've just downloaded my palette pics from zshare...and winrar opens it with no problem! Should I use some particular settings to make it compatible with 7zip?

EDIT2: 😁...i did a try to calibrate by myself using my palette, but with no success! I took many palette pics with still no success! (best gamut error about 42%). Can I ask you what are the reasons for setting zoom to middle and focus to infinity? Is it just to smooth and have homogeneous colors?
Thanks!

lukg

This the only palette I get some success with! I had to deselect the black and dark grey patches (used 22 patches) to get an acceptable gamut error, and also I had to set im. bright. to 0.5.
But I have not understood how to use this matrix now without compile dng4ps 😁
Hope someone could take a better palette than mines (i took many)

Camera -> XYZ matrix:
3,679790 1,286464 1,690627
1,145118 3,685682 0,174926
0,245843 -1,636507 11,881179
XYZ -> Camera matrix:
0,314732 -0,128898 -0,042887
-0,096843 0,309220 0,009228
-0,019851 0,045259 0,086325

zSHARE - s5palette.rar

EDIT: ok, i found "dngsetmatrix" instructions...with this matrix I get some good results:
If someone wants to try (you need dngsetmatrix.exe in the same directory) I wrote a simple batch file, just to avoid to rewrite the matrix everytime:

@echo off
dngsetmatrix.exe %~nx1 M1 0.340055 -0.138680 -0.051226 -0.129504 0.346033 0.005147 -0.026039 0.033814 0.103860
pause

Just save it as "blabla.bat". To execute just drag n drop DNG file over the bat file.

For what I saw it seems better than DNG4PS matrix (S5 and also a720).

lks

It's a much better color matrix than the default provided by dcraw / dng4ps2. Still looks a tad green to me.

lukg

ok, I'm going to take some better palette pics (i hope) before uploading it again!
My question about the zoom was about the problem of taking pics at middle zoom with a S5 with 12x.
As I remember I got many BAD areas.
About dngsetmatrix batch I forgot to change it, because I was experimenting with it since it's used M2 even in the instructions I followed. (I wrote both fields in my experiment....do you think it's better to write both M1 and M2 with same values?)
I'm going to upload conversion logs for different im. brightness so you can take a look at them.

lukg

I don't know if it's normal, but this are my results changing only the im.brightness:

Brightness 1.00: BEST RESULTS FROM G=1,96000, min(E)=20,4% BAD <----- ONLY 20.4 ????!?
Brightness 0.85: BEST RESULTS FROM G=2,02000, min(E)=38,4% BAD
Brightness 0.75: BEST RESULTS FROM G=2,04000, min(E)=35,4% BAD
Brightness 0.65: BEST RESULTS FROM G=1,84000, min(E)=21,4% BAD
Brightness 0.63: BEST RESULTS FROM G=1,84000, min(E)=21,3% BAD
Brightness 0.62: G=1,40328, range(E)=13,4%->13,4%, aver(E)=10,1% POOR RESULTS
Brightness 0.61: G=1,42006, range(E)=11,3%->11,3%, aver(E)=8,4% POOR RESULTS
Brightness 0.60: G=1,45888, range(E)=7,6%->7,6%, aver(E)=6,8%
Brightness 0.59: G=1,48614, range(E)=7,1%->7,1%, aver(E)=6,2%
Brightness 0.58: G=1,53668, range(E)=5,6%->5,6%, aver(E)=5,2%
Brightness 0.55: G=1,53812, range(E)=5,6%->5,6%, aver(E)=5,2%
Brightness 0.50: G=1,61465, range(E)=4,1%->4,1%, aver(E)=4,0%

Since the first error value (im. bright. 1.00) made me thinking, I tried with im. brightness values over 1.00, but with an error never less 20%.

Here are the logs:
zSHARE - logs.zip

lukg

I did some tests using two different matrix. Indeed, as Dawmatt said, I get better results using the matrix with a little higher error.

Here are some examples:

(the pics are in order:
top-left= A720matrix
top-right= matrix obtained from my palette with 0.60 image brightness
bottom-left= camera jpg
bottom-right= matrix obtained from my palette with 0.50 image brightness
)

http://img92.imageshack.us/img92/861/catturadx5.jpg

http://img92.imageshack.us/img92/3770/catturakb1.jpg

http://img92.imageshack.us/img92/9393/catturalt8.jpg

I think the top-right pictures give, in overall, better results (more fidelity with the jpg camera). Except for the green color that doesn't match at all in all matrix. 😁

These are only basic tests, since I get many bad errors and area no. doesn't match. during calculation.

lukg

This is the palette I've used (zipped). I can't take a good palette anymore, I don't know why, it's like to win a lottery.

zSHARE - palette.zip

lks

I tried calibrating the S5 today. I took a picture of the Gretag-Macbeth color chart in 3pm sunlight. The best result I got was this with Im. Bright = 0.75:

RESULTS
White area: 9
Errors:
Zone 1: 11.1%
Zone 2: 10.8%
Zone 3: 10.7%
Zone 4: 4.2%
Zone 5: 10.3%
Zone 6: 4.9%
Zone 7: 7.8%
Zone 8: 10.6%
Zone 9: 6.0%
### Max error is greater then 10%, RESULTS ARE POOR ###
Camera -> XYZ matrix:
1.558489 0.923604 0.792446
0.260645 2.211214 -0.107758
0.044825 -0.855401 6.696108
XYZ -> Camera matrix:
0.698670 -0.325843 -0.087927
-0.083100 0.493829 0.017781
-0.015293 0.065266 0.152201

Unfortunately it doesn't appear much better than lukg's. I do note that if I use tint +10 in ACR that it looks pretty close, though.

lukg

Thanks lks, if you have a real gretag color chart you can provide better results than everyone else. Keep trying, play with focus and zoom.
I have a question for the author or anyone knows...does this method depends on the palette provide in the first post?? because i tried with different color palette with horrible results (errors of about 100000%).

EDIT: LKS, I tried your matrix, but it doesn't match colors with my 3 test pics. It gives colder colors sometimes and warmer colors in other pics (like the a720matrix). It's a mission impossible, in my opinion. I'm worry that in this way, there are millions of good matrix for the calibration pic, but only few are good for any other picture too. Anyway I hope we'll find some of those.

lks

I agree my last calibration is not that great. I tried again and got the following slightly better results:

RESULTS
White area: 9
Errors:
Zone 1: 10.3%
Zone 2: 3.9%
Zone 3: 8.0%
Zone 4: 9.5%
Zone 5: 9.4%
Zone 6: 10.2%
Zone 7: 9.9%
Zone 8: 5.9%
Zone 9: 4.6%
Zone 10: 1.4%
### Max error is greater then 10%, RESULTS ARE POOR ###
Camera -> XYZ matrix:
1.761105 0.931603 0.791588
0.325948 2.337791 -0.149091
0.065097 -0.923729 6.943318
XYZ -> Camera matrix:
0.622181 -0.278327 -0.076910
-0.087866 0.470720 0.020125
-0.017523 0.065233 0.147422

If in addition I manually set white balance in Adobe Camera Raw and then set the camera calibration tab (based on http://www.21stcenturyshoebox.com/tools/ACRcalibrator.html) to the following settings:

Red Hue, Sat:  -1, -38
Green Hue, Sat: 51, 42
Blue Hue, Sat: 6,18

then the result seemed better on a few test pictures. This is still a bit too much work for my tastes, though.

lukg

You're right! I tried to create a calibration setting in camera raw too but with no absolute solution that could be good for all my pictures (maybe cause i don't have a gretag chart, I don't have the skill and there are too many combinations 😁 ). The only thing I noticed is:
-with my matrix and yours, if you set RGB channels hue to +100 and saturation to -100 I think (i suppose) that you should get a gray scale image. But the image is not gray. If you set Blue or red or green channel sat to -50 you get a gray scale image. This is the point from where I started but I lost my patience soon 😁
I'm really thinking that the camera apply some further strange algorithm from Raw to Jpg instead of a simple color transformation (selective color boost or something else). Maybe i'm wrong, I don't know much about camera working and color theory 😁.
Anyway for what I see your latest matrix is way better than the previous one. I see also that you too got only 10 zones instead of 22 (or 24)...and for what I understood, it is not a good thing. 🙁

lks

I'm thinking part of the problem is that we need to also do a calibration for Tungsten light (Type A). I'm guessing that's why the colors shift a bit with different color light, even after correcting for the white balance.

lukg

I got a better palette pic.
I got a matrix at 0.90, and it seems to get 22/24 areas (my previous matrix got 7/22 areas). Obviously "Max error is greater then 10%, RESULTS ARE POOR" 😁, but it seems to be a lot better than the others we found.

RESULTS
White area: 1
Errors:
Zone 1: 7,3%
Zone 2: 12,4%
Zone 3: 9,3%
Zone 4: 6,7%
Zone 5: 3,3%
Zone 6: 5,4%
Zone 7: 13,4%
Zone 8: 8,8%
Zone 9: 12,3%
Zone 10: 13,5%
Zone 11: 9,1%
Zone 12: 1,8%
Zone 13: 13,3%
Zone 14: 13,5%
Zone 15: 13,4%
Zone 16: 8,9%
Zone 17: 5,1%
Zone 18: 6,9%
Zone 19: 6,8%
Zone 20: 7,8%
Zone 21: 11,1%
Zone 22: 11,1%
### Max error is greater then 10%, RESULTS ARE POOR ###
Camera -> XYZ matrix:
1,637840 0,508453 0,781881
0,147034 1,924260 -0,003592
-0,104238 -0,650416 5,358281
XYZ -> Camera matrix:
0,622294 -0,195168 -0,090936
-0,047538 0,534707 0,007295
0,006335 0,061109 0,185744

Here a simple comparison with my previous matrix (Temperature,hue fixed for both images). On the right the new matrix.
You always need to adjust white balance (and primary colors) to let the colors match better with camera jpg, but this is a better matrix without doubt.
http://img217.imageshack.us/img217/6591/cattura1lj7.jpg
http://img76.imageshack.us/img76/966/cattura2lv7.jpg

lks

I agree that does look better!

Can you share a DNG that has the raw pallette? I can then run the ACR calibration program above.

lukg

Sure, the problem is that I used a "virtual" gretag chart found online, not a real gretag chart (i took a photo of my LCD).
I should have a calibrated monitor to do ACR calibration...but I don't 😁.
If you have a real gretag you're the only that can do an empirical calibration in ACR, using a real DNG of the charts, and hoping to apply a "good" matrix on it. My DNG would be useless cause it depends on the calibration of my monitor.

Here the DNG (with latest matrix) and the camera JPG.

zSHARE - palette.rar

I did a test. Opened both in Photoshop, and set the white balance for both, picking up the color of the "white" square area. It's far from a definitive solution but it doesn't seem so bad.
(camera jpg on the right)

PS. Could you upload a real gretag chart photo taken in daylight by the S5 (jpg+crw)?

« Previous Page Next Page »