CHDK Forum
Using CHDK => RAW Shooting and Processing => Topic started by: MartijnH on 29 / August / 2011, 10:32:27
-
By accident I made two errors: not checking DNG and using low resolution JPG. So, I have 100 diving photo's in S95 CHDK CRW format that I need to convert to DNG. UFRaw, DNG4PS2 and RawTherapee do not read them. I know they are just a dump of the sensor data and I can write software, but without any specification this is impossible. Is there any software solution or specification available to solve this problem?
-
You can create a profile for DNG4PS2 http://chdk.setepontos.com/index.php?topic=6754.0 (http://chdk.setepontos.com/index.php?topic=6754.0)
You can find the details of your cameras sensor layout in the platform_camera.h file for your camera. For s95, this is http://www.assembla.com/code/chdk/subversion/nodes/trunk/platform/s95/platform_camera.h?rev=1308 (http://www.assembla.com/code/chdk/subversion/nodes/trunk/platform/s95/platform_camera.h?rev=1308)
-
Thanks for the information. However, I am not sure this will work. This means DNG4PS2 can be applied to camera's with a Digic-4 processor, like the S95, while the readme_en.txt says "DNG for PowerShot-2 (DNG4PS-2) is free software for converting RAW files from Canon PowerShot cameras with Digic-II processor", so I assumed DNG4PS2 would work for the S95. May be some one can tell me this is no problem before I follow this advice?
-
May be some one can tell me this is no problem before I follow this advice?
If you look at the article linked by reyalp, it references an IXUS100 - a Digic-4 camera.
-
Thanks. It seems to work! I used the following, which did show a picture in photoshop, but I still need to fill in the right Black-white levels and matrix to get the colors ok.
Sensor size: 3744 x 2784
Jpeg size: 3648 x 2736
Active zone x1, y1: 64, 22
Active zone x2, y2: 3728, 2782
Black-white levels: wrong and just a guess
Mosaic type Red,Green Green,Blue
Bits per unit: 12 bit
Illuminant: Standard Light A
Matrix: wrong and just a guess
Multiplicator: 1.0 1.0 1.0
-
Welcome to the world of CHDK RAW files. With a bit of work your images can look just as good as the jpg your camera already produces. With a lot of work you can probably make your image look slightly better. Have fun !
-
Thanks. It seems to work! I used the following, which did show a picture in photoshop, but I still need to fill in the right Black-white levels and matrix to get the colors ok.
Sensor size: 3744 x 2784
Jpeg size: 3648 x 2736
Active zone x1, y1: 64, 22
Active zone x2, y2: 3728, 2782
Black-white levels: wrong and just a guess
Mosaic type Red,Green Green,Blue
Bits per unit: 12 bit
Illuminant: Standard Light A
Matrix: wrong and just a guess
Multiplicator: 1.0 1.0 1.0
Black level = 127
White level = 4095
Color matrix (from DCRAW) = 1.3440 -0.5896 -0.1279 -0.1236 0.9598 0.1931 -0.0180 -0.1001 0.4651
I can't remember if you need commas between the matrix values or not. If matrix doesn't work try it with commas.
Phil.
-
Welcome to the world of CHDK RAW files. With a bit of work your images can look just as good as the jpg your camera already produces. With a lot of work you can probably make your image look slightly better. Have fun !
I can assume you (at least for the A495) that with only a minimal amount of fiddling in e.g. RawTherapee the images transformed from CHDK RAWs (DNGs more precisely) are WAY better than the original JPEGs. The chromatic aberration so ugly in the original JPEGs can be reduced a lot using simply the auto CA correction option, while a less aggressive compression factor preceded by RL or USM sharpening give you much better looking JPEGs with smaller file sizes.
-
Finally, I used this
Sensor size: 3744 x 2784
Jpeg size: 3648 x 2736
Active zone x1, y1: 64, 22
Active zone x2, y2: 3728, 2762
Black-white levels: 127 - 4095
Mosaic type Red,Green Green,Blue
Bits per unit: 12 bit
Illuminant: Daylight
Matrix: 0.13440 -0.05896 -0.01279 -0.01236 0.09598 0.01931 -0.00180 0.01001 0.04651
Multiplicator: 1.0 1.0 1.0
- Black-white levels 127 - 4095, see post of philmoz in this thread;
- Illuminant: Daylight, my guess;
- Matrix: 0.13440 -0.05896 -0.01279 -0.01236 0.09598 0.01931 -0.00180 0.01001 0.04651, see source code dcraw.c http://www.libraw.org/download (http://www.libraw.org/download)
"Canon PowerShot S95", 0, 0,
{ 13440,-5896,-1279,-1236,9598,1931,-180,1001,4651 };
- everything else, see source code chdk Camera - S95 - platform_camera.h http://www.assembla.com/code/chdk/subversion/nodes/trunk/platform/s95/platform_camera.h?rev=1308. (http://www.assembla.com/code/chdk/subversion/nodes/trunk/platform/s95/platform_camera.h?rev=1308.)
I conclude that the chdk source code Camera - S95 - platform_camera.h can be improved by using the right color matrix from dcraw.c. I don't know if this has anything to do with the wrong colors of my JPG files, created with custom white balance using a white card. The colors of the DNG files I generated from CHDK CRW with the above settings are much better.
-
Hm ... is there a reason for using CHDK/dng raw files, as S95 supports native cr2 raw files, which are saved faster in the camera and supported properly by various programs?
Matrix in dcraw is probably taken from Adobe, by simply converting native cr2 file to a dng with Adobe dng converter and reading matrix from that dng file. BTW, dcraw supports only one color matrices, while dng converted from native cr2 has four matrices
-
The remark of Vit40 triggered me to do some research. From the DNG specification http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf (http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf) and the CR2 specification http://lclevy.free.fr/cr2/ (http://lclevy.free.fr/cr2/) I understand that only two matrices for two illumination sources can be specified. I think that Dcraw supports two matrices, but the second one is rarely used. You can specify a second matrix in DNG4PS2, but it is not filled in for all predefined camera types.
As an experiment I took a picture of a Macbeth color checker on my computer screen and created three DNG files:
- with Adobe DNG converter from CR2
- internal with CHDK
- with DNG4PS2 from CHDK CRW
A difference for the DNG files I found out with Raw Therapee and Lightroom is:
- only CR2 DNG contains the Exif data to allow for automatic lens distortion correction and rescaling (with border removement) in for example Lightroom;
- the colors and as shot white balance for internal CHDK DNG were ok, but a slight exposure correction was needed;
- the as shot white balance for CHDK CRW had to be adjusted and a slight exposure correction was needed.
So, if you use CHDK raw (CRW or DNG) you loose automatic lens correction and may need some exposure correction. For CHDK CRW you may also need to adjust white balance.
For dive pictures I have no problem with adjusting white balance and exposure for every picture, but in the future I will use CR2 raw.
-
Old dng specification used 2 color matrices for conversion. However, all adobe profiles in last years are up to dng 1.3, where color conversion is performed using two forward matrices, while two color matrices are used for WB. So there are 4 matrices in total in dng profiles. So it's not the same workflow as in dcraw which is using these matrices a bit incorrectly (whitebalancing is performed differently). However, ACR will work up to old dng specification and use only color matrices if there are no forward matrices in profile or in dng. I studied it in detail, since I needed that to make my dng profile generator program that is still unmatched by adobe, although lately they came quite close with "v3 profiles", as I documented on ACR forum several months ago.
I suppose all chdk dngs have baseline exposure tag = 0.5, while this value is generally different for each camera, hence the exposure difference between cr2 and chdk dng. You can check the value of this tag in a dng converted from cr2, using exiftool utility. With the same utility you can also change it, so results will be the same.
dngs made from crw have no WB info, since dng4ps2 doesn't know how to extract this info from jpeg, because it is in manufacturer info tag, and it's position depends on the camera model. A while ago I made an utility to copy this value from jpg into dng for my compacts. Without that, WB set to "as shot" is the same as WB Auto in ACR
You can use lens profiles with every kind of raw/dng files in ACR, just you have to select it manually - if you have it for your lens. But lens profiles provided by adobe so far seem to ... well ... need some refinement ...
-
You are absolutely right vit40!
With exiftool it is easy to derive the S95 Adobe color matrices from an S95 CR2 image converted with Adobe DNG converter 6.4. They are:
Color Matrix 1
Illuminant: Standard Light A
Matrix: 1.5419 -0.8268 -0.0232 -0.0656 0.8621 0.2437 0.0163 0.0404 0.5172
Color Matrix 2
Illuminant: D65
Matrix: 1.3440 -0.5896 -0.1279 -0.1236 0.9598 0.1931 -0.0180 0.1001 0.4651
Hence, the illuminant should be D65 and not daylight as I assumed.
It appears that it does not work to put both matrices in DNG4PS2 (the latest beta), Exiftool displays the use of illuminant 1 together with matrix 2, so you have to choose one of them.
The baseline exposure of CHDK DNG and DNG4PS2 is -0.5, while it should be 0 and a correction of 0.5 exposure is needed.
-
In case there is only one color matrix in dng or dng profile, it doesn't matter what is the value of illuminant tag. Can be D65, Daylight, Tungsten, whatever. These tags are important only in case there are 2 color matrices, because in that case ACR will interpolate a color matrix from matrix 1 and 2, depending on color temperature set in ACR and values of illuminant tags
If you like canon colors, you can use camera profiles for S95 that are, I suppose, provided by Adobe. In that case it doesn't matter what color matrix is in a dng, because all 4 matrices will be taken from selected profile. Matrix from a dng will be used only in case you select "embedded" profile. After much experimenting, I concluded that Canon standard camera profile is better than simple matrix conversion in most situations for several reasons ... I suppose many would not agree with me and try to prove it with macbeth color checker that has several colors, but real life has many more colors so this checker and whole calibration stuff based on it is actually of limited use ...
-
Thanks, you are again right Vit40. In practice it does not matter what illuminant is mentioned for one matrix, but it is stilll nice to hnow for which illuminant the matrix is measured and optimal.
I nearly never used the embedded colors, because I also liked de Adobe Standard colors in Lightroom. Thanks for explaining the difference. There is a small problem that Lightroom does not show you which camera profile is used. When the dng contains the right Exif tag for the camera (I assume), Lightroom uses automaticly its camera profile and names it Adobe Standard. The camera profiles, including for Canon Powershot S95, are in
C:\Documents and Settings\All Users\Application Data\Adobe\CameraRaw\CameraProfiles
For lensprofles Lightroom shows you which profile is chosen. The lensprofiles, including for Canon Powershot S95, are in
C:\Documents and Settings\All Users\Application Data\Adobe\CameraRaw\LensProfiles
-
There is a small problem that Lightroom does not show you which camera profile is used. When the dng contains the right Exif tag for the camera (I assume), Lightroom uses automaticly its camera profile and names it Adobe Standard.
Really? I don't use LR, but according to this (old) webpage
http://www.thelightsright.com/UsingCameraProfilesWithLightroom2 (http://www.thelightsright.com/UsingCameraProfilesWithLightroom2)
you can select profile on calibration page like in ACR
-
Well this has been changed in Lightroom 3
http://help.adobe.com/en_US/Lightroom/3.0/Using/WS939594D8-4279-41b4-B8E9-B06BC919EC7C.html (http://help.adobe.com/en_US/Lightroom/3.0/Using/WS939594D8-4279-41b4-B8E9-B06BC919EC7C.html)
"Adobe Standard [...] The Profile pop-up menu displays only one Adobe Standard profile for your camera"
-
I know it's been a while since anyone posted this, but when I try this, DNG4PS-2 will see the CRW and JPG files once I set up the camera, but when I try to convert the files to DNG, there is a stream of
Error: Can't find camera profile for this file
Z:\Shared\RAW\IMG_1071.CRW
messages and nothing is output.
Here's the export of the camera profile:
CameraCount=-1
[Camera000]
Id={11111111-11110000-08a420c3-32f90020}
Name=PowerShot S95
ShortName=S95
SensorWidth=3744
SensorHeight=2784
CroppedWidth=3648
CroppedHeight=2736
ActiveOriginX=64
ActiveOriginY=22
ActiveOriginRight=16
ActiveOriginBottom=2
BlackLevel=127
WhiteLevel=4095
Mosaic=1
BitsPerUnit=12
[Camera000/Matrix1]
Illuminant=21
Matrix=1.344000 -0.589600 -0.127900 -0.123600 0.959800 0.193100 -0.018000 -0.100100 0.465100
Mult=1.0 1.0 1.0
[Camera000/Matrix2]
Illuminant=-1
Matrix=0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
Mult=0.0 0.0 0.0
Is it the Id that's wrong? How on earth are you supposed to know what it is?
I've looked through the CHDK source code and can't spot any obvious Id in platform/S95. (I am now even more in awe of the people who worked out what was where in each camera though...) Searching for that string, or sensible bits of it, doesn't find it anywhere in the trunk tree.
If I look at some of the cameras DNG4PS-2 supports natively, and will export the settings for, the ID does vary, slightly:
A430 - Id={11111111-11110000-08a420c3-32f90026}
A450 - Id={11111111-11110000-08a420c3-32f9001E}
.. do I need to try every other couple of last two hex digits?
-
It's been a very long time since I did anything with DNG4PS2, but I don't think that ID is anything you need to set or manipulate.
Under settings -> camera types you may need to set the slot that matches your cameras resolution to the correct camera.
-
Thanks - because the program had the S95 set up and I had JPGs for most of the files, I didn't think I needed to do that.
Once I did, it worked..
.. but only with the files that didn't have associated JPGs. The ones that did had the 'can't find camera profile' error, so something clearly does want setting up differently, because that's supposed to be helpful!
It turns out that the 'Name' has to be "Canon PowerShot S95" - rather than anything else, like "PowerShot S95" - presumably because that's the name in the JPGs.
On the off-chance that it might make a difference (being recognised as being from a known camera when something looks at the DNG rather than it going "'PowerShot S95'? I know about a 'Canon PowerShot S95', but this isn't that..") I've made that change.
Thank you again - I know this is something you typically only need to do once!