CRW vs DNG (was Re: New project: camera color profile calculation)

alex73

« on: 10 / June / 2015, 16:25:45 »

Advertisements

Quote from: reyalp on 10 / June / 2015, 16:17:04

In general, we recommend using CHDK DNG.

Thank you. I used dcraw for RAW->TIFF conversion earlier.
Will camera work slower if it will save DNG instead RAW ? As far as I understand, it should convert each pixel by color profile.

« Last Edit: 10 / June / 2015, 16:34:23 by reyalp »

Logged

reyalp

13393

Re: CRW vs DNG (was Re: New project: camera color profile calculation)

« Reply #1 on: 10 / June / 2015, 16:41:08 »

Split from http://chdk.setepontos.com/index.php?topic=390.msg122873#msg122873

Quote from: alex73 on 10 / June / 2015, 16:25:45

Thank you. I used dcraw for RAW->TIFF conversion earlier.
Will camera work slower if it will save DNG instead RAW ?

It's a little bit slower, something like 100-200ms on most cameras.

Quote

As far as I understand, it should convert each pixel by color profile.

No, DNG is a raw format, so it saves the original data, plus the information needed for raw software to convert it.

I understand your project is using PTP controlled shooting. In this case, if you can use the "remote shoot" function to shoot and download in one step without saving to the SD card. In this case, there should be essentially no performance difference between DNG and CRW.

Logged

Don't forget what the H stands for.

alex73

Re: CRW vs DNG (was Re: New project: camera color profile calculation)

« Reply #2 on: 27 / June / 2015, 04:18:41 »

I tried some software and learned more about DNG, so, lets continue.

I'm trying to choose between DNG and CRW format for use with CHDK. Please correct me if I'm wrong somewhere.

As far as I understand, to have best quality picture, I need to have(whatever CRW/DNG) :
1. RAW image data
2. badpixel info
3. camera color profile

Both formats (CRW or DNG) contains RAW image data.

DNG can contains badpixel info, but RAW cannot. Desktop software can use DNG built-in badpixel info for correct output image. For CRW, desktop software can use non built-in badpixel file. So, result of DNG and CRW conversion can be the same.

Other option - badpixel data can be used on camera side, i.e. CHDK can interpolate RAW image data before writing to DNG/CRW. dcraw's manual said that desktop-side interpolation can be better because desktop software uses more accuracy algorithm than camera since desktop processor is much faster. Additionally, camera-side badpixel processing requires more time. So, it's possible option, but unwanted for me.

Camera color profile is more complex thing. As far as I understand, not DNG nor the RAW contain color profile, i.e. processing depends on desktop software.

The important part of choosing DNG/CRW - what software I need for convert DNG/CRW to well-known image formats like PNG, using badpixel info and color profiles. For my purpose, I need multiplatform software(preferable open-source) with ability to batch conversion. I.e. Photoshop and other Windows/Mac is not usable.

For CRW, we have dcraw, that supports badpixel files, ability to create color profile for specific camera model, and some other good options for color balancing even without color profile.

For DNG, I found only ImageMagick for DNG->PNG conversion, but it doesn't supports badpixels and color profile.

So, looks like DNG could be more preferred format since it's "more standard", but there is no good software for DNG processing.

P.S. I read http://chdk.wikia.com/wiki/Badpixel_removal, but could you confirm: as far as I understand, "badpixel file" used only for camera-side badpixels removing, but "badpixel.bin file" used only for desktop-side camera pixels removing.

Logged

reyalp

13393

Re: CRW vs DNG (was Re: New project: camera color profile calculation)

« Reply #3 on: 27 / June / 2015, 16:43:51 »

Quote from: alex73 on 27 / June / 2015, 04:18:41

I tried some software and learned more about DNG, so, lets continue.

I'm trying to choose between DNG and CRW format for use with CHDK. Please correct me if I'm wrong somewhere.

Just to avoid confusion, "CRW" RAW in CHDK is just a dump of the sensor frame buffer. It is not related to the actual Canon CRW/CR2 which are propriety formats based on TIFF. This means that any software which uses CHDK RAW must be updated for every new sensor.

Quote

DNG can contains badpixel info, but RAW cannot. Desktop software can use DNG built-in badpixel info for correct output image. For CRW, desktop software can use non built-in badpixel file. So, result of DNG and CRW conversion can be the same.

Sort of. There are several complications.
There are two different categories of bad pixels: Pixels known to the firmware to be bad, and those that aren't
The former are set to 0 by the canon firmware, while the latter will be "hot" or "stuck". The canon firm "fixes" the 0 value pixels when it creates a jpeg, but the 0 shows up in the raw data.
Further complicating this, most cameras have multiple Canon bad pixel maps, so which pixels are set 0 might depend on exposure length, ISO and sensor temperature. Additionally, some cameras handle them in a different way, such that no zero value pixels are seen in the raw.

DNG bad pixel handling depends on the DNG version
The DNG 1.1 spec requires that "bad" pixels be interpolated over in the raw data.

In DNG 1.3, the header can just specify DNG opcodes which specify that pixels with a particular value should be interpolated over by the DNG software, but a lot of software doesn't support this.

CHDK DNG
In CHDK, the DNG code only directly addresses the 0 value known bad pixels, but there is a separate system you can use to handle hot pixels for both DNG and jpeg.

If you use DNG 1.1, then you must create badpixel.bin and bad pixels are interpolated over on the camera. badpixel.bin is purely a speed optimization, because we don't know how to directly access the Canon list and searching for 0 value pixels every time is too slow.

If you use DNG 1.3, then you do not need badpixel.bin and 0 value pixels are not modified on the camera. Your raw processor must understand DNG opcodes, or you must have some other way to tell it to interpolate over the affected pixels.

PTP remote shoot is different
Since I gather from other posts you are using PTP remote shoot, this deserves a mention. If you remote shoot with DNG, then the data is always 1.3 format, using opcodes. However, in chdkptp remoteshoot you can use the -badpix=n option to interpolate over pixels with a specific value. (note however that this depends stuff your java/chdkptp port may not implement)

CHDK has different systems for known (0 valued) bad pixels and hot pixels
All of the above stuff about DNG refers to pixels the camera already knows are bad. The "badpixel" file can be used to specify an arbitrary list of pixels that should either be interpolated over or set to 0, depending on the "manual bad pixel removal" setting.

Quote

Camera color profile is more complex thing. As far as I understand, not DNG nor the RAW contain color profile, i.e. processing depends on desktop software.

CHDK DNG do contain a color matrix, but the quality varies between ports and there no simple, documented process to make a good calibration (dng4ps2 can be used for this, but in practice it is difficult or impossible to get it to work on current cameras). If you come up with a better calibration, we can include it in the source.

Quote

The important part of choosing DNG/CRW - what software I need for convert DNG/CRW to well-known image formats like PNG, using badpixel info and color profiles. For my purpose, I need multiplatform software(preferable open-source) with ability to batch conversion. I.e. Photoshop and other Windows/Mac is not usable.

If dcraw + CHDK framebuffer dump RAW works for you, then use it. Most other programs that understand CHDK RAW are based on dcraw.

For DNG processing, I have had good results with raw therapee. You can create settings files to apply standard adjustments to things like color, hot / dead pixels etc, and output 8 or 16 bit png. AFAIK it can be used in to do batch conversions from the command line, but I have not used this.

There are also a lot of open source image processing libraries available (imagej, scipy + scikit-image, opencv...) I don't know off hand which ones might support DNG or bayer data.

chdkptp itself also has some limited DNG processing capability:
https://www.assembla.com/spaces/chdkptp/wiki/DNG_Processing

Quote

P.S. I read http://chdk.wikia.com/wiki/Badpixel_removal, but could you confirm: as far as I understand, "badpixel file" used only for camera-side badpixels removing, but "badpixel.bin file" used only for desktop-side camera pixels removing.

Not really, but hopefully my comments above clear this up.

Logged

Don't forget what the H stands for.

alex73

Re: CRW vs DNG (was Re: New project: camera color profile calculation)

« Reply #4 on: 29 / June / 2015, 15:36:19 »

Hi reyalp. Thank you for detailed description. It's much clearer now.

Quote from: reyalp on 27 / June / 2015, 16:43:51

CHDK DNG do contain a color matrix, but the quality varies between ports and there no simple, documented process to make a good calibration (dng4ps2 can be used for this, but in practice it is difficult or impossible to get it to work on current cameras). If you come up with a better calibration, we can include it in the source.

dcraw has some calibration flow. See description on the https://www.cybercom.net/~dcoffin/dcraw/ under "Why does dcraw output have a green tint and weak color?" Using calibrated color tables with real sunlight looks promising.

I checked raw therapee and other software. For my case, looks like dcraw with CHDK CRW will be best solution because dcraw is very simple solution with very predictable result and with great control for batch processing. I will understand and control which pixels will be removed and how colors will be calibrated.

Logged

Anaglyphic

129
Anaglyphic lives!

Re: CRW vs DNG (was Re: New project: camera color profile calculation)

« Reply #5 on: 18 / June / 2016, 10:27:56 »

Quote from: reyalp on 10 / June / 2015, 16:41:08

Split from http://chdk.setepontos.com/index.php?topic=390.msg122873#msg122873
Quote from: alex73 on 10 / June / 2015, 16:25:45
Thank you. I used dcraw for RAW->TIFF conversion earlier.
Will camera work slower if it will save DNG instead RAW ?
It's a little bit slower, something like 100-200ms on most cameras.
Quote
As far as I understand, it should convert each pixel by color profile.
No, DNG is a raw format, so it saves the original data, plus the information needed for raw software to convert it.

I understand your project is using PTP controlled shooting. In this case, if you can use the "remote shoot" function to shoot and download in one step without saving to the SD card. In this case, there should be essentially no performance difference between DNG and CRW.

In case anyone still cares, I did some testing on a few older cameras, ones that even Class4 cards make no difference, much less a faster UHS type. You can enable RAW writing time under the RAW menu, then open the Console to see the data.

The widest variable I got was: saving time = 1630 (DNG) t0 1470 (CRW) on an SX100IS, running 1.4.1
On a slightly older IXUS80 (SD1100IS) it was negligible; 1330 to 1300.

Please note that darkframe and badpixel and and and all matter, as does WithJPEG, as does in-camera ISO noise reduction as does long exposure... it can seem like the camera is taking 4 seconds to give back control, but it's not the fault of the DNG conversion.

As was said in the Shooting RAW first post however, the ability to process a 10-bit RAW image, be it DNG or CRW, to look better than the in-camera JPEG, is highly elusive.

(note that in spring 2016, the SX100IS still isn't supported in dcraw. I'll send some data to Dave for the other 6 people who still keep 10-year old 8MP cameras running.) :-)

Logged

Since we cannot know all that there is to be known about anything,
we ought to know a little about everything. -- Blaise Pascal