CHDK dng profile generator project - RAW Shooting and Processing - CHDK Forum
« previous next »
Pages: [1] 2 3 4   Go Down

CHDK dng profile generator project

  • 37 Replies
  • 38058 Views
*

Offline vit40

  • ****
  • 276
CHDK dng profile generator project
« on: 02 / October / 2010, 16:22:32 »
Advertisements
As I mentioned about a month ago, I have plans to generate a new type of DNG profiles for CHDK cameras. This is the idea: put a raw containing a kind of color checker in the camera, use RAW develop function to get jpeg, compare input and output and calculate a profile. Sounds simple, but there are problems - output result is only 8 bit compressed image, so it is not an easy task to get smooth result. Anyway, after long experimenting, I finally have beta version profiles for my A620 and A650, that are producing quite good results. Here is the link

http://www.megaupload.com/?d=6Z3NDQON

Included are profiles and sample - test chart as developped by the camera, by ACR using embedded profile (from dng4ps) and by ACR using attached profile for A620, so you can compare. As you would see, this is not a trivial macbeth color checker with 24 patches, but it contains millions of colors. Profiles are made similar way as latest profiles by Adobe for 550D, where color conversion from sensor color space to Photo Pro (working color space in ACR and LR) is done using only a big lookup table, without matrix operation (color matrices are used only for whitebalancing)

As my intention was simply to produce the same color as camera and CHDK functionality is used for this, profiles are named CHDK Standard (according to standard picture style set in the camera). There are arguments for and against using 'Canon colors', some people like them and some don't, but my opinion is that Canon is a company that is dealing with colors for decades, so it is to be supposed that they know something about producing colors

I would like to get some feedback about this project. If enough members are interested, I will try to finish a program, that would be reasonably simple to use for others (it's not in that phase yet, as many parameters are hardcoded etc), so profiles for other cameras can be produced

As you probably know, dng profiles are usable only with ACR and LR. To get the same result as from camera, you have to set the following:

WB as shot
exposure 0.50 (because of baseline exposure tag in dngs, that contains value -0.5)
brightness 50
blacks 0
contrast 25
curve medium contrast

Enjoy
« Last Edit: 11 / October / 2010, 05:12:08 by vit40 »
Logged
*

Offline vit40

  • ****
  • 276
CHDK dng profile generator project
« Reply #1 on: 11 / October / 2010, 05:11:22 »
So, there were no replies on my post about dng profiles (I like that idea / I don't like it /
I don't care / I think it would be better this way / I tried a profile for A620/650 and ...)

Anyway, I made some minor progress with calibration algorithms in the meantime,
and here are beta profiles for SX110. I made 9 profiles, for different saturation
and contrast (set on the camera in Custom Color mode). As this is relatively new camera,
I hope that someone will try these profiles

http://www.megaupload.com/?d=U0S7IIV0

Unlike my old A620, where results seem to be quite consistent with jpegs from camera,
SX110 behave differently. On scenes where saturation or contrast is low, camera raises
saturation and/or contrast on the jpeg. A650 also works this way. New DSLR cameras
also behave this way, which was already mentioned by Eric Chan, one of authors of ACR,
on ACR forum. So on real life, it's not possible to get exactly the same photo from raw as from
camera. Color tones would be similar, but with some differences in contrast, saturation
and possibly brightness also, because jpeg processing in the camera depends on the scene

« Last Edit: 11 / October / 2010, 05:13:49 by vit40 »
Logged
*

Offline Aged

  • *
  • 39
Re: CHDK dng profile generator project
« Reply #2 on: 14 / October / 2010, 15:49:36 »
@Vit40
I'm now eager to get started, however a couple of queries.
a) Dng  "not from dng4ps" do I replace the CHDK colour matrix with the unity matrix before taking a picture?
b) does the picture need to be highly coloured ?
Thanks Aged
Logged
*

Offline vit40

  • ****
  • 276
Re: CHDK dng profile generator project
« Reply #3 on: 14 / October / 2010, 17:01:50 »
Hi,

I described the procedure briefly when you contacted me last week (I was in a hurry), so I'll try to be more descriptive this time

First, I need a dng from the camera and corresponding jpeg just to get some exif information (camera name, model, size and position of jpeg, white balance, color matrix), so it's not very important what's in the photo. Here is what you should do:

1) in CHDK menu, select RAW parameters, then option "DNG format" and "DNG file extension"
2) in factory menu of the camera, set jpeg size to large
3) for the start, I suggest that you simply choose ordinary daylight scene, like a shot through the window on ordinary sunny day, avoiding too many shadows, cloudy/rainy/foggy weather, so that camera will measure a color temperature around 5000 K and I can check that white balance of color matrix, embedded in a dng, is aproximately ok. Content of this picture is not important for calibration process
4) switch white balance to manual (last option in WB menu). Set manual white balance by pressing menu or disp key (depending on the camera model). You can use gray card or a sheet of white paper here, but I think it would be ok if you just point a camera to the scene and press menu / disp. Don't use Daylight white balance or something else - must be Manual
5) take a photo using that manual white balance. You must not set different manual white balance (by pressing disp/menu key in manual WB again) until you develop raw image that I will send you, but you can freely use the camera with any WB settings (auto, daylight etc) in the meantime - just don't recalibrate manual WB in the meantime
6) upload a dng and jpeg somewhere where I can download it. Use zip or rar, so that file is smaller. Don't change any information in a dng with some external program like exiftool, just leave it as it is
7) I'll check exif informations in a dng, generate a crw file and send it to you
8 ) You will upload crw to SD card - card reader is needed
9) If camera has custom color mode (like A650, SX110), select custom color, move sharpening to lowest position. You can leave contrast and saturation in the middle position or choose some other position if you are using it, but I suggest leaving it in the middle for the beginning. If camera doesn't have that mode (like A620), use low sharpening mode. If there is no low sharpening mode, just leave on normal / effect off mode
10) select ISO 400 to decrease in-camera sharpening further (to minimise calibration error because of sharpening effects)
11) check that jpeg size is still on large. Also, select superfine jpeg quality this time
12) select manual exposure, 1/500s or something similar. Aperture is not important. Make sure flash is deactivated / closed
13) set WB to manual (if you changed to something else in the meantime), BUT don't press disp or menu again - you must use the same manual WB as first time, because test crw file will be generated according to that white balance; otherwise we will have to repeat the procedure
14) in CHDK menu, select RAW parameters, then RAW develop. Browse to the name of crw file that you uploaded to SD, press set. You will get a message to switch to record mode and take a shot
15) exit chdk menu, press the shutter. On LCD, you will see a test image with many different colors, developed from crw
16) send me that jpeg
17) if everything goes well, I will generate a profile and post it

I hope it's not too complicated

As I said, this profile will be usable only with latest ACR or Lightroom versions, so it will be useless with other programs that don't use dng profiles or very old ACR versions that didn't support dng profiles

Vit

Additional note:  this calibration procedure probably won't be possible for some cameras with Digic IV (V, VI ...) processor, because Digic IV corrects vigneting and geometrical distorsions of the lens when developping jpeg. However, it worked on SX 200 IS and SX 130 IS, with these additional settings for the second part of the procedure

- zoom at tele position (no lens correction at that position for these cameras)
- i-Contrast OFF
- ISO set to 200 (because of blur applied by SX130 at ISO 400; this wasn't an issue in SX200)




« Last Edit: 02 / February / 2011, 05:49:17 by vit40 »
Logged
*

Offline fotoli

  • *
  • 3
Re: CHDK dng profile generator project
« Reply #4 on: 15 / October / 2010, 21:38:22 »
@vit

Really nice work!

I tried the profile for my SX110 with WB set to sunny. In most cases it works well, but sometimes the photo looks a bit pink. I adjusted the color temperature=5000K (originally 5500K) and hue=0 (originally 10) to solve the problem.

The contrast and saturation of my SX110 are set to the middle level. I guess it should be fine using "CHDK Standard 0.345.dcp" for this combination. There are 5 levels for the saturation and contrast. So I just assume that +1/-1 in the file names means max/min value and 0 stands for the medium one. Hope I didn't misunderstand anything.
« Last Edit: 15 / October / 2010, 21:41:16 by fotoli »
Logged
*

Offline vit40

  • ****
  • 276
Re: CHDK dng profile generator project
« Reply #5 on: 16 / October / 2010, 02:39:33 »
Hi, fotoli

Contrast and saturation +1/-1 mean first position of the slider left/right of the middle. It's written that way in the exif of the photo. Minimum position is Saturation -2 / Contrast -2, while maximum is +2

I tried a bunch of landscape photos with these profiles on SX110. Most of jpegs are slightly more saturated than profile Standard and slightly less than Standard S+1. My explanation is that during calibration, camera slightly lowered saturation of jpeg, to compensate for highly saturated test image, that was used for the calibration. On the other side, profile for A620, which is older camera, is almost perfect match. But I still have to check and tune some parts of the code, so we will see. I sorted highlight part of the profile in the meantime, where there were some hue shifts of blown colors, compared to camera (had to use some tricks with modified test chart, to fool the camera to give me the right result, because workflow in ACR is obviously different than in the camera), and shadows still need to be addressed. But most parts of the profile seem to be right on these beta versions

As about white balance, if you want the same white balance as jpeg, you have to set WB as shot, but you must weather use dng right from the camera (which has WB as shot exif tag), or if you use dng4ps to make dng from crw files, you must copy wb as shot tag from corresponding jpeg with my utility that I posted about a year ago. If you set WB manually, as you noticed, tint close to 0 is usually the right one (don't know why Adobe default is 10), except with flourescent lighting, where you usually need tint 20-30 to kill the green cast
« Last Edit: 16 / October / 2010, 02:57:08 by vit40 »
Logged
*

Offline Lebeau

  • ***
  • 187
Re: CHDK dng profile generator project
« Reply #6 on: 10 / November / 2010, 17:06:15 »
I am familar with DNG4PS2 but since I use Adobe Profile Editor, I am satisfy with the integration of Photoshop and Lightroom workflow.

See my comments here:
http://chdk.setepontos.com/index.php?topic=1532.msg56842#msg56842
Logged
*

Offline Aged

  • *
  • 39
Re: CHDK dng profile generator project
« Reply #7 on: 11 / November / 2010, 12:18:23 »
Vit40 has been pursuing an interest in CHDK Colour for many years and recently been working on this project. It uses the work done on the Adobe Camera Raw (ACR)to produce

profiles for the Canon CHDK range of cameras. The early Chdk camera Raw profiles owed much to Dave Coffin and ArtDen with their 3 x 3 colour matrices.

Meanwhile Adobe were looking for a way of producing better profiles Thomas Fors, Rags Gardner, Eric Chin and others worked their way towards digital camera profiles (dcp).
These profiles, Standard and a few others come with the ACR plug-in updates (free). Lightroom and CS5 have later versions which each cover more cameras and features. The

basic input has to be a .DNG file. You will need for the cheap version Photoshop Elements 7,  ACR 5.6 and Vit40's CHDK dng profile generator to produce good results from

your dng images.

My S3is seems to over expose, as it did when using Rags calculation, ACR can compensates so I now set it to under expose by 1/3 -> 2/3.

Vit40 produced a Standard CHDK profile for me offline. Many Thanks!

  Links
DcpTool.    Use to check results. Useful to write a one line batch file to input expression (Like examples, save in Binaries/Windows, then click dcptool.exe)
OpenXmlEditor  -  OXEDIT-1-6.bin.zip   Examine output from DcpTool.xml files. Allow some time to do conversion
ACR  How to, features -  16 How to Tips ACR Peachpit. online Adobe

Result comparision of 'identical' images
1) 'Difference' blending mode
2) Bruce Fraser  Subtract 128  RGB gray is identical
3) from Color Management and Photoshop Windows forums (sorry misplaced link) Shows Marching ants on colour profile

A question for Vit. I have a couple of additional lenses Wide-angle and Tele-converter. Should I have other profiles?
Logged
*

Offline vit40

  • ****
  • 276
Re: CHDK dng profile generator project
« Reply #8 on: 12 / November / 2010, 01:55:56 »
Lenses/converters generally add some color tone to the image, but I suppose you can correct that, if needed, with white balance sliders, or by using DNG profile editor. My program will always produce the same profile, no matter which converter/filter you put on the lens, because raw image is always the same - it is software generated and not produced by taking the shot of some test chart. So produced profile is simlpy a kind of copy of the profile in the camera
Logged
*

Offline oversky

  • *
  • 41
Re: CHDK dng profile generator project
« Reply #9 on: 10 / December / 2010, 12:11:05 »
Hi, vit40,  This is the most exciting post in this year. Could you briefly describe how to load the profile in lightroom 3? Our lab just bought a Luminance Colorimeter. It can measure color and give L, x, y. Maybe I can measure the color differences between the original jpeg and the dng when my boss is out. Do you have any idea?
Logged
Pages: [1] 2 3 4   Go Up
« previous next »
 

Related Topics