CHDK Forum
Using CHDK => RAW Shooting and Processing => Topic started by: vit40 on 02 / October / 2010, 16:22:32
-
As I mentioned about a month ago, I have plans to generate a new type of DNG profiles for CHDK cameras. This is the idea: put a raw containing a kind of color checker in the camera, use RAW develop function to get jpeg, compare input and output and calculate a profile. Sounds simple, but there are problems - output result is only 8 bit compressed image, so it is not an easy task to get smooth result. Anyway, after long experimenting, I finally have beta version profiles for my A620 and A650, that are producing quite good results. Here is the link
http://www.megaupload.com/?d=6Z3NDQON (http://www.megaupload.com/?d=6Z3NDQON)
Included are profiles and sample - test chart as developped by the camera, by ACR using embedded profile (from dng4ps) and by ACR using attached profile for A620, so you can compare. As you would see, this is not a trivial macbeth color checker with 24 patches, but it contains millions of colors. Profiles are made similar way as latest profiles by Adobe for 550D, where color conversion from sensor color space to Photo Pro (working color space in ACR and LR) is done using only a big lookup table, without matrix operation (color matrices are used only for whitebalancing)
As my intention was simply to produce the same color as camera and CHDK functionality is used for this, profiles are named CHDK Standard (according to standard picture style set in the camera). There are arguments for and against using 'Canon colors', some people like them and some don't, but my opinion is that Canon is a company that is dealing with colors for decades, so it is to be supposed that they know something about producing colors
I would like to get some feedback about this project. If enough members are interested, I will try to finish a program, that would be reasonably simple to use for others (it's not in that phase yet, as many parameters are hardcoded etc), so profiles for other cameras can be produced
As you probably know, dng profiles are usable only with ACR and LR. To get the same result as from camera, you have to set the following:
WB as shot
exposure 0.50 (because of baseline exposure tag in dngs, that contains value -0.5)
brightness 50
blacks 0
contrast 25
curve medium contrast
Enjoy
-
So, there were no replies on my post about dng profiles (I like that idea / I don't like it /
I don't care / I think it would be better this way / I tried a profile for A620/650 and ...)
Anyway, I made some minor progress with calibration algorithms in the meantime,
and here are beta profiles for SX110. I made 9 profiles, for different saturation
and contrast (set on the camera in Custom Color mode). As this is relatively new camera,
I hope that someone will try these profiles
http://www.megaupload.com/?d=U0S7IIV0 (http://www.megaupload.com/?d=U0S7IIV0)
Unlike my old A620, where results seem to be quite consistent with jpegs from camera,
SX110 behave differently. On scenes where saturation or contrast is low, camera raises
saturation and/or contrast on the jpeg. A650 also works this way. New DSLR cameras
also behave this way, which was already mentioned by Eric Chan, one of authors of ACR,
on ACR forum. So on real life, it's not possible to get exactly the same photo from raw as from
camera. Color tones would be similar, but with some differences in contrast, saturation
and possibly brightness also, because jpeg processing in the camera depends on the scene
-
@Vit40
I'm now eager to get started, however a couple of queries.
a) Dng "not from dng4ps" do I replace the CHDK colour matrix with the unity matrix before taking a picture?
b) does the picture need to be highly coloured ?
Thanks Aged
-
Hi,
I described the procedure briefly when you contacted me last week (I was in a hurry), so I'll try to be more descriptive this time
First, I need a dng from the camera and corresponding jpeg just to get some exif information (camera name, model, size and position of jpeg, white balance, color matrix), so it's not very important what's in the photo. Here is what you should do:
1) in CHDK menu, select RAW parameters, then option "DNG format" and "DNG file extension"
2) in factory menu of the camera, set jpeg size to large
3) for the start, I suggest that you simply choose ordinary daylight scene, like a shot through the window on ordinary sunny day, avoiding too many shadows, cloudy/rainy/foggy weather, so that camera will measure a color temperature around 5000 K and I can check that white balance of color matrix, embedded in a dng, is aproximately ok. Content of this picture is not important for calibration process
4) switch white balance to manual (last option in WB menu). Set manual white balance by pressing menu or disp key (depending on the camera model). You can use gray card or a sheet of white paper here, but I think it would be ok if you just point a camera to the scene and press menu / disp. Don't use Daylight white balance or something else - must be Manual
5) take a photo using that manual white balance. You must not set different manual white balance (by pressing disp/menu key in manual WB again) until you develop raw image that I will send you, but you can freely use the camera with any WB settings (auto, daylight etc) in the meantime - just don't recalibrate manual WB in the meantime
6) upload a dng and jpeg somewhere where I can download it. Use zip or rar, so that file is smaller. Don't change any information in a dng with some external program like exiftool, just leave it as it is
7) I'll check exif informations in a dng, generate a crw file and send it to you
8 ) You will upload crw to SD card - card reader is needed
9) If camera has custom color mode (like A650, SX110), select custom color, move sharpening to lowest position. You can leave contrast and saturation in the middle position or choose some other position if you are using it, but I suggest leaving it in the middle for the beginning. If camera doesn't have that mode (like A620), use low sharpening mode. If there is no low sharpening mode, just leave on normal / effect off mode
10) select ISO 400 to decrease in-camera sharpening further (to minimise calibration error because of sharpening effects)
11) check that jpeg size is still on large. Also, select superfine jpeg quality this time
12) select manual exposure, 1/500s or something similar. Aperture is not important. Make sure flash is deactivated / closed
13) set WB to manual (if you changed to something else in the meantime), BUT don't press disp or menu again - you must use the same manual WB as first time, because test crw file will be generated according to that white balance; otherwise we will have to repeat the procedure
14) in CHDK menu, select RAW parameters, then RAW develop. Browse to the name of crw file that you uploaded to SD, press set. You will get a message to switch to record mode and take a shot
15) exit chdk menu, press the shutter. On LCD, you will see a test image with many different colors, developed from crw
16) send me that jpeg
17) if everything goes well, I will generate a profile and post it
I hope it's not too complicated
As I said, this profile will be usable only with latest ACR or Lightroom versions, so it will be useless with other programs that don't use dng profiles or very old ACR versions that didn't support dng profiles
Vit
Additional note: this calibration procedure probably won't be possible for some cameras with Digic IV (V, VI ...) processor, because Digic IV corrects vigneting and geometrical distorsions of the lens when developping jpeg. However, it worked on SX 200 IS and SX 130 IS, with these additional settings for the second part of the procedure
- zoom at tele position (no lens correction at that position for these cameras)
- i-Contrast OFF
- ISO set to 200 (because of blur applied by SX130 at ISO 400; this wasn't an issue in SX200)
-
@vit
Really nice work!
I tried the profile for my SX110 with WB set to sunny. In most cases it works well, but sometimes the photo looks a bit pink. I adjusted the color temperature=5000K (originally 5500K) and hue=0 (originally 10) to solve the problem.
The contrast and saturation of my SX110 are set to the middle level. I guess it should be fine using "CHDK Standard 0.345.dcp" for this combination. There are 5 levels for the saturation and contrast. So I just assume that +1/-1 in the file names means max/min value and 0 stands for the medium one. Hope I didn't misunderstand anything.
-
Hi, fotoli
Contrast and saturation +1/-1 mean first position of the slider left/right of the middle. It's written that way in the exif of the photo. Minimum position is Saturation -2 / Contrast -2, while maximum is +2
I tried a bunch of landscape photos with these profiles on SX110. Most of jpegs are slightly more saturated than profile Standard and slightly less than Standard S+1. My explanation is that during calibration, camera slightly lowered saturation of jpeg, to compensate for highly saturated test image, that was used for the calibration. On the other side, profile for A620, which is older camera, is almost perfect match. But I still have to check and tune some parts of the code, so we will see. I sorted highlight part of the profile in the meantime, where there were some hue shifts of blown colors, compared to camera (had to use some tricks with modified test chart, to fool the camera to give me the right result, because workflow in ACR is obviously different than in the camera), and shadows still need to be addressed. But most parts of the profile seem to be right on these beta versions
As about white balance, if you want the same white balance as jpeg, you have to set WB as shot, but you must weather use dng right from the camera (which has WB as shot exif tag), or if you use dng4ps to make dng from crw files, you must copy wb as shot tag from corresponding jpeg with my utility that I posted about a year ago. If you set WB manually, as you noticed, tint close to 0 is usually the right one (don't know why Adobe default is 10), except with flourescent lighting, where you usually need tint 20-30 to kill the green cast
-
I am familar with DNG4PS2 but since I use Adobe Profile Editor, I am satisfy with the integration of Photoshop and Lightroom workflow.
See my comments here:
http://chdk.setepontos.com/index.php?topic=1532.msg56842#msg56842 (http://chdk.setepontos.com/index.php?topic=1532.msg56842#msg56842)
-
Vit40 has been pursuing an interest in CHDK Colour for many years and recently been working on this project. It uses the work done on the Adobe Camera Raw (ACR)to produce
profiles for the Canon CHDK range of cameras. The early Chdk camera Raw profiles owed much to Dave Coffin and ArtDen with their 3 x 3 colour matrices.
Meanwhile Adobe were looking for a way of producing better profiles Thomas Fors, Rags Gardner, Eric Chin and others worked their way towards digital camera profiles (dcp).
These profiles, Standard and a few others come with the ACR plug-in updates (free). Lightroom and CS5 have later versions which each cover more cameras and features. The
basic input has to be a .DNG file. You will need for the cheap version Photoshop Elements 7, ACR 5.6 and Vit40's CHDK dng profile generator to produce good results from
your dng images.
My S3is seems to over expose, as it did when using Rags calculation, ACR can compensates so I now set it to under expose by 1/3 -> 2/3.
Vit40 produced a Standard CHDK profile for me offline. Many Thanks!
Links
DcpTool. Use to check results. Useful to write a one line batch file to input expression (Like examples, save in Binaries/Windows, then click dcptool.exe)
OpenXmlEditor - OXEDIT-1-6.bin.zip Examine output from DcpTool.xml files. Allow some time to do conversion
ACR How to, features - 16 How to Tips ACR Peachpit. online Adobe
Result comparision of 'identical' images
1) 'Difference' blending mode
2) Bruce Fraser Subtract 128 RGB gray is identical
3) from Color Management and Photoshop Windows forums (sorry misplaced link) Shows Marching ants on colour profile
A question for Vit. I have a couple of additional lenses Wide-angle and Tele-converter. Should I have other profiles?
-
Lenses/converters generally add some color tone to the image, but I suppose you can correct that, if needed, with white balance sliders, or by using DNG profile editor. My program will always produce the same profile, no matter which converter/filter you put on the lens, because raw image is always the same - it is software generated and not produced by taking the shot of some test chart. So produced profile is simlpy a kind of copy of the profile in the camera
-
Hi, vit40, This is the most exciting post in this year. Could you briefly describe how to load the profile in lightroom 3? Our lab just bought a Luminance Colorimeter. It can measure color and give L, x, y. Maybe I can measure the color differences between the original jpeg and the dng when my boss is out. Do you have any idea?
-
Hi,
For Photoshop + ACR on Windows XP (my environment), location for profiles is
c:\Documents and settings\all users\Application data\Adobe\Camera Raw\Camera Profiles
profiles can be in this folder or any of its subfolders
On Vista and Windows 7, this folder is different. Also, I'm not sure what's the profile folder for Lightroom. Easyest way is to search a hard disk for files with extension .dcp (after you installed Lightroom) and put profiles in the same folder. Or you can check Adobe's documentation
This will only work if you have some of cameras I made profiles for (A620, A650, SX110). If you don't, I can generate a profile for your camera the same way I did with Aged member (procedure is described above)
To check a difference between jpeg from camera and jpeg from ACR, I used a simple pixel pick program, whick can show RGB value of individual pixel. I opened both images in two different windows, swiched between them and checked how good they match on different parts of extensive calibration chart that was used to generate the profile
-
Does this design for the dng generated by CHDK directly or also work with the old raw-> dng4ps2 process?
-
It doesn't matter whether dng is created in camera or converted from chdk raw format by dng4ps program (if I understood the question correctly). Im using raw->dng4ps workflow
Anyway, advantage of dng from camera is that it has WB as shot info, so you can use it in ACR. Dng from dng4ps doesn't have WB info. I have an utility to copy WB info from jpeg to dng, but at the moment, only A620,A650,SX110 are supported also
-
@vit40,
I noticed that you produced an S3IS profile for Aged.
I have a S3IS camera. Can I have a copy of the profile.
Thanks in advance.
-
Hi. Here is the profile for S3IS
http://www.megaupload.com/?d=63PHU4FK (http://www.megaupload.com/?d=63PHU4FK)
Unfortunately, jpeg that Aged send me to make profile was made in Custom color mode wit contrast set to lowest position, so this profile has tone curve with relatively flat upper part. I asked him to send me also a jpeg with normal contrast, but he didn't. You can try it and see the result. If you find that photos doesn't have enough contrast, there are two options
- move the contrast slider to the right (from default +25 position). Result won't be the same as from camera, as lookup table in the profile is also somewhat different, but maybe you will find it satisfying
- read the instructions above, so we can make another profile. Instructions are relatively long, but procedure is in fact quite simple and straightforward
-
@vit40,
Thanks, I have downloaded the file and saved this page. It will take me time to digest the info.
-
Can you share this utility? I am going to try this profile on the photos I took in Japan and see how it work.
BTW, I noticed that the camera profile your created is larger than the built-in profiles in CameraRaw (239kB vs 110 kB), is that because you calibrate more colors than adobe?
Another idea is that since A650 share the same sensor and lens with G9, is it possible to hack G9 profiles and make them recognized as A650
It doesn't matter whether dng is created in camera or converted from chdk raw format by dng4ps program (if I understood the question correctly). Im using raw->dng4ps workflow
Anyway, advantage of dng from camera is that it has WB as shot info, so you can use it in ACR. Dng from dng4ps doesn't have WB info. I have an utility to copy WB info from jpeg to dng, but at the moment, only A620,A650,SX110 are supported also
-
I had intention to share the utility, as I wrote in the first post, but since nobody was interested, I terminated working with it. Calibrating algorithm is finished, but interface isn't, so usage needs changing some parameters in the code and recompiling (it's written in Delphi)
Anyway, I'm available to make profiles for your cameras - it will take less time for me than finishing the program and will minimize possibility of errors. In case there is suddenly too much requests, which I doubt, then I will finish it, so other members can do it on their own
About profile size, latest Adobe camera profiles are bigger, because they have bigger lookup table, with 90x16x16 points (it was 36x16x16 on older camera profiles). I suppose that the reason is banding in some very saturated red and blue colors because of lookup table interpolation during raw development, that is considerably reduced this way, although still present to small extent. In my profiles, lookup table is 90x15x16 - it has 1 saturation segment less, due to number of patches in calibration chart, but it's not a problem.
It is possible to try Adobe Camera profiles for G9 with A650, or my profiles for A650 with G9. All that is needed is to change an exif tag "Unique camera model" in the profile to match the camera name. This can be done by freeware command line utility exiftool.
And one more detail I didn't mention before - exiftool can be used also for setting Baseline Exposure Tag in dng files to 0; in that case, there is no need for moving exposure slider to 0.5 to match the camera jpeg with my profiles
-
Hi, Vit,
I forgot to mention that my camera is A650. So I think I could use the file you just compiled. Hopefully it won't be too job for you. As for hacking G9 profiles, I was thinking about changing some hex code in the G9 profiles, so we don't need to update EXIF everytime.
-
Hi vit, I've got an A650 (and I already downloaded your profile, thank you very much) but I also own a S5 IS, have you got a profile for this camera too? If not, can you generate a profile if I send you the files?
-
Hi nonno
I don't have a profile for S5IS, but I can make it. Just read the first part of described procedure and send me a jpg and dng pair. If something is not clear, ask. Some steps in the procedure are different than people usually do when taking photos, because whole procedure is very specific for this program, so just do as described, don't do anything different, trying to "improve" the procedure, and everything will be ok
Vit
-
thank you very much for your helpfulness. Right now the weather is really foggy, but as soon as I can I'll take the shots, following exactly your instructions, and I'll send you them!!! (Maybe you should remove your email address from the forum or it will be filled by spam)
-
Can you generate profile for SX130? I hope this picture is sunny enough - it's hard here to get sunny day in this time of year :)
http://www.megaupload.com/?d=8OAYNHPI (http://www.megaupload.com/?d=8OAYNHPI)
Do you think we can use this method with generated CRW and calculations like in DNG4PS to create better 3x3 matrix for camera?
-
Hi, here is test crw
http://www.megaupload.com/?d=OVWEEOHR (http://www.megaupload.com/?d=OVWEEOHR)
I also included a profile for SX200 that I made for eapl, just changed camera name in exif tag to SX 130 IS, so you can try it in the meantime. I tried it with your dng and colors are very similar as in jpeg from the camera, just that jpeg has more contrast, probably because of different processing in the camera for this scene
About white balance, it would probably need some corrections (for SX130 or maybe for SX200 also), as displayed color temperature is too high. If you have some old dngs, made in fine weather, you can try and see how much correction is needed (temperature and tint) to get close to daylight preset (5500K, tint +10), so I can correct it (I must do it manually - can't be calculated with this method)
-
I've uploaded developed jpeg.
http://www.megaupload.com/?d=OEFYZG6Z (http://www.megaupload.com/?d=OEFYZG6Z)
I don't have any old photos with fine weather (new camera, even newer CHDK port :)), so we have to wait until I will be able to take a new one.
-
-
quid,
Could you develop another jpeg, just with ISO set to 100 instead of 400, and all other settings the same. Processing in SX130 is different than in camera I calibrated so far (including SX200) - in ISO 400 there is a kind of strong color bluring (besides less sharpening, like in other cameras). I didn't spot it in the first set of two jpegs that you sent (first was ISO 80 and the second was ISO 400). With this jpeg, I'm affraid that profile won't be accurate enough in the shadow region
Can you also try developing test crw in wide zoom position. You don't have to send jpegs, I'm just interested is resulting picture deformed and with brightened corners (because of built-in correction for the lens) or not
Vit
-
Aepl,
in case you select Embedded profile, ACR will use default "Adobe tone curve", that IS different than tone curve in the camera or in the profile I generated (as you noted in DNG profile editor). It is generic tone curve for Adobe standard profiles for all cameras. Default settings for that profile are exposure = 0 and blacks = 5. That's one of the reasons for the difference in brightness. Second reason is that all CHDK dngs have baseline exposure tag -0.5, which originated from some older Power shot cameras, but every camera model has slightly different mettering, so this value is probably slightly wrong for most new cameras.
Also, depending on the scene, digic can change tone curve or saturation for in-camera processing, resulting in different picture than ACR with this profile. If you have i-Contrast on, it will also probably brighten shadows - you can simulate that using fill light slider and/or raising brightness slider
Bit depth for SX200 and SX130 is 12 bit, while it was 10 bit for older CHDK cameras. You can get various information about dng using command prompt utility exiftool (there are probably other similar tools too)
In case you can make some daylight pictures on sunny weather, or have some older dngs, taken in fine weather, check what is average color temperature and tint for those pictures shown in ACR, when you select this profile and WB as shot (if you had AWB selected in the camera), so we can correct this information in the profile if needed
Vit
-
I've made more pictures and I think best results are on ISO200 with lowest setting on Sharpness slider. There are three files in archive: IMG_1210.JPG - ISO100, lowest Sharpness, IMG_1211.JPG - test CRW on wide, IMG_1212.JPG - ISO 200, lowest Sharpness.
http://www.megaupload.com/?d=NLDAFXX6 (http://www.megaupload.com/?d=NLDAFXX6)
I think there may be slight brightening of corners on test image developed on wide, but it's impossible to say, because of heavy distortion correction - most of gray frame is cropped out. Amount of processing is better visible on this picture (CRW+JPG):
http://www.megaupload.com/?d=NYZH03WJ (http://www.megaupload.com/?d=NYZH03WJ)
-
Hi Vit40.
You said:
"Bit depth for SX200 and SX130 is 12 bit, while it was 10 bit for older CHDK cameras. You can get various information about dng using command prompt utility exiftool (there are probably other similar tools too)"
I am confused. I executed Exiftool and I can't understand the meaning of these values:
---- IFD0 ----
Bits Per Sample : 8 8 8
Samples Per Pixel : 3
---- SubIFD ----
Bits Per Sample : 12
Samples Per Pixel : 1
Thanks in advance.
-
aepl,
- first pair of values is for thumbnail image (width 128, height 96) inside DNG. It has 3 colors per pixel (RGB), 8 bits each
- second pair is for raw image from sensor. It has 1 color per pixel (R, G or B), 12 bits each in this case
-
quid, here is the profile
http://www.megaupload.com/?d=HCCUJDH2 (http://www.megaupload.com/?d=HCCUJDH2)
I used ISO 200 image. Profile made with ISO 400 indeed produces oversaturated shadows, which would boost the color noise. This profile is slightly more saturated than profile for SX200 and there are slight hue differences for some colors, whether that's how this camera renders colors, whether it's slightly different sensor and calibration is different.
For this version, I left color matrix from dng in the profile (like for v1 version for SX200). Using WB as shot would be ok, but WB presets in ACR probably won't work as expected, as temperature and hue calculated by ACR using that matrix are probably both too high. When you make some daylight pictures in fine weather, write what's average color temperature and tint shown in ACR (ACR WB must be set to As shot and camera WB to AWB), and I'll correct color matrix to get it close to daylight preset. You can also make a picture or two with incadescent light and see what is the temperature/tint shown in ACR, so I can also separately adjust color matrix for tungsten illuminant
Vit
-
Thanks, vit40! Currently I don't have too much time to play with it, but first tests of your profile are very promising :)
-
As I mentioned about a month ago, I have plans to generate a new type of DNG profiles for CHDK cameras. This is the idea: put a raw containing a kind of color checker in the camera, use RAW develop function to get jpeg, compare input and output and calculate a profile. Sounds simple, but there are problems - output result is only 8 bit compressed image, so it is not an easy task to get smooth result. Anyway, after long experimenting, I finally have beta version profiles for my A620 and A650, that are producing quite good results. Here is the link
http://www.megaupload.com/?d=6Z3NDQON (http://www.megaupload.com/?d=6Z3NDQON)
Hi vit,
Sorry to revive such an old thread.
Can I copy and try your A620 profile?
The link above is no longer working...
I have the same problem with you to match DNG (generated by CHDK) with JPG.
-
Here it is
http://www.sendspace.com/file/rk0nrm (http://www.sendspace.com/file/rk0nrm)
-
Here it is
http://www.sendspace.com/file/rk0nrm (http://www.sendspace.com/file/rk0nrm)
Thank you Vit!
I'm very excited with this... :D
One thing I'd like to ask:
by using DCP profile like this, will it "replace" the embedded matrix in the DNG file itself? or it will have a "stacking effect" on the image (so it's like map the image color using the embedded matrix, then map the already mapped color using the DCP profile) ?
-
If you select a profile, all information about color rendition are taken from the profile - color matrices and some other metadata from raw file (dng in this case) are ignored
-
Hello vit40,
I have recently bought Canon SX130IS too, and that especially for the raw capability. I noticed someone else has done the lens correction profile already which is great, and now I am only stuck with what appears to be slightly washed out colours, especially skin tones, when developing raw. If I understand it correctly, by using your colour correction profile, colours should pretty much match the jpegs. This would be great, however your link is not working anymore (would you be able to upload again?).
First, after reading some articles I thought I would have to buy some calibration chart (like x-rite), but they are quite expensive. Would you think they are superior when colour correcting though?
Many thanks for your great work.
BR,
D
quid, here is the profile
http://www.megaupload.com/?d=HCCUJDH2 (http://www.megaupload.com/?d=HCCUJDH2)
I used ISO 200 image. Profile made with ISO 400 indeed produces oversaturated shadows, which would boost the color noise. This profile is slightly more saturated than profile for SX200 and there are slight hue differences for some colors, whether that's how this camera renders colors, whether it's slightly different sensor and calibration is different.
For this version, I left color matrix from dng in the profile (like for v1 version for SX200). Using WB as shot would be ok, but WB presets in ACR probably won't work as expected, as temperature and hue calculated by ACR using that matrix are probably both too high. When you make some daylight pictures in fine weather, write what's average color temperature and tint shown in ACR (ACR WB must be set to As shot and camera WB to AWB), and I'll correct color matrix to get it close to daylight preset. You can also make a picture or two with incadescent light and see what is the temperature/tint shown in ACR, so I can also separately adjust color matrix for tungsten illuminant
Vit