Save RAW files in DNG format instead of RAW

PhyrePhoX

yes, that is the reason, you need the badpixel.bin

PhyrePhoX

ewavr

PhyrePhoX
i gather you also write the exif stuff - do you write the canon values or the chdk (overridden) values?

For Av and Tv - overridden values. For ISO - Canon value, but this can be corrected.

hm, no build for the s3is, damn 😉

No problem, added (also S2, S5, SX100).

Many thanks to ArtDen and cppasm from russian forum.

so i guess adding dng support for cameras that are NOT in dng4ps yet is more difficult, unless we ignore the color profile and people set it themselves in the raw editing program of their choice. this would mean you could add dng support to the svn 😉
what happens if you take a shot without the badpixel.bin?
and what could be negative side-effects of that raw buffer caching? so far i havent seen any downside, unless you call significantly faster raw saving a downside 😉
also maybe you can leave that timecounter that shows how long raw saving takes in the final svn committ, perhaps with an option in the debug menu? this way you can also benchmark your sdcards and the raw vs dng vs caching thing.

again: great work!
oh and ps: your english is fine 😀

pps: can we write the jpg exif as well now?

ewavr

PhyrePhoX
what happens if you take a shot without the badpixel.bin?

Try it yourself - make 4 bytes long badpixel.bin with zero bytes contents (this mean one bad pixel with 0,0 coordinates). My result - in attached image (dcraw, 100% crop).

and what could be negative side-effects of that raw buffer caching? so far i havent seen any downside, unless you call significantly faster raw saving a downside 😉

Maybe on some cameras in some cases this cause image artifacts - more testing is required.

pps: can we write the jpg exif as well now?

At present - no. But some time ago I found 'CreateHeader' task, which writes some EXIF info in JPEGs.

edit: 25 Nov. Dummy badpixel.bin is removed. Use REAL bad pixels list instead it.
edit2: We can increase writing speed still more (saving time ~1.7s in my cam), but JPEG will be corrupted. 😀

PhyrePhoX

well, some people even dont want jpgs to be saved. for real - if we can have the dngs with a useful colorprofile - why use jpg. perhaps make it an option - jpg / jpg+crw / jpg+dng / just crw / just dng + cache/no cache. this would make a lotta people happy 😉 *

the output of the raw without the badpixel.bin looks similar to what i had found on my camera today, even though i DID use the badpixel.bin - i could not reproduce it though.

wow, dng support and MAYBE exif writing later. two of the most wanted features 😀

* edit: in case of the ixus40 100k this feature already exists: the crw is written, the jpg isnt 😁

edit: i GUESS that with dng, we cannot develop the raw in-camera anymore. which is why we should leave the "old crw" option in there. right?

ewavr

PhyrePhoX
the output of the raw without the badpixel.bin looks similar to what i had found on my camera today, even though i DID use the badpixel.bin - i could not reproduce it though.

This depends on RAW converter - Adobe Camera RAW successfilly eliminates single bad pixels (but not bad pixels blocks). Previous screenshot is made by FastStone Image Viewer (dcraw based).

that with dng, we cannot develop the raw in-camera anymore. which is why we should leave the "old crw" option in there. right?

Yes. But DNG can be easily converted to usual CHDK CRW and vice versa. "DNG develop" is also possible.

PhyrePhoX

ha, that was the reason, i was using faststone image viewer AND adobe camera raw, at least i think so.
wow, dng develop and i guess dng merge/average would be possible... very cool if that was available 😀

lrossel

Wow, I tried loaded in card the dummy badpixel.bin and it worked fantastic.

A720IS with 4GB ExtremeIII card.
1.- In camera took about 2 seconds to save the raw. No big difference with the CHDK Raw save.
2.- The Camera Window from canon was able to donwload from camera the new CRW. Perfect.
3.- I opened the Adobe Bridge, and it shows ok the new Raws.
4.- The Photoshop Camera Raw works fine with the new Raws.
5.- The new Raws looks fine in my notebook display.

Congratulations Ewavr!!!

One question to the masters of the forum... ¿what other test you suggest me to do?
In order to help to the release of this big new feature to the trunk.

Hacki

ewavr
edit2: We can increase writing speed still more (saving time ~1.7s in my cam), but JPEG will be corrupted. 😀

How? 😀

ewavr

Hacki
How? 😀

In raw.c:

#if DNG_SUPPORT
            if (conf.dng_raw&&get_dng_header()) {
     //      revert_bytes_order(get_raw_image_addr(), hook_raw_size());
     //      unpatch_bad_pixels_b();
             free_dng_header();
            }
#endif

vit40

Ewavr, checked version for A620, but can't turn DNG format feature on (pressing set on that menu item has no effect). Am I doing something wrong ?

Just an idea, don't know how much programming efforts is needed for this - what about separating a dng header into external file, that will be located somewhere on the SD card, similar to headers in DNG4PS version 1? That way, we can customize it way we want with exiftools, dngsetmatrix and other tools(changing color matrix and other tags), because header is a tiff structure . What do you think ?

zincopal

Very Great work ! now with this mods also IrfanView (good only in daylight or flash) and ACDsee , RawTherapee that shows some EXIF data and of course Dcraw and all related programs open the file without problem and with the same quality of the CHDK.CRW files, I have make a test with two sd card one with the DNG mods and another with #535 build developing with Dcraw and Camera raw the same subject with same parameters, cropping at 200% no significative differrencies in terms of increase of noise or losing of definitions or other artifacts the only very minimal are the tipical of the developing of matrix AHD vs VNG, about the colors, I have seen that actually are much neutrals this is good because give the possibility of simple corrections in post processing.
Regards

lrossel

vit40
Ewavr, checked version for A620, but can't turn DNG format feature on (pressing set on that menu item has no effect). Am I doing something wrong ?

Just an idea, don't know how much programming efforts is needed for this - what about separating a dng header into external file, that will be located somewhere on the SD card, similar to headers in DNG4PS version 1? That way, we can customize it way we want with exiftools, dngsetmatrix and other tools(changing color matrix and other tags), because header is a tiff structure . What do you think ?

I'm sure you are making the same mistake I made.

Be sure to save the badpixel.bin file in the CHDK folder of your card before to try.

You can use for testing the badpixel.bin file attached on the Reply #42.

Good luck

PhyrePhoX

vit40
Ewavr, checked version for A620, but can't turn DNG format feature on (pressing set on that menu item has no effect). Am I doing something wrong ?

read the last few posts, you MUST use the badpixel.bin file, either selfcreated or downloaded from ewavrs attachment.

edit: i was beaten. damn 😁

acetone

So, I ran into something interesting poking about in the DNG spec: compressed DNG files use the older JPEG lossless standard (LJPEG, not JPEG-LS), because it gives better compression than ZIP for >8-bit color data. I'm wondering if there's any way to hook into the existing JPEG-processing hardware on the DIGIC chip, to accelerate the compression?

From what I gather, the differencing/prediction step of compression could be implemented as a 3x3 convolution, and the RLE and Huffman coding that follows is just like normal JPEG. The DIGIC has to do the RLE & Huffman anyway to encode JPEGs, and the prediction/differencing shouldn't require more than a few operations per pixel. I strongly suspect the DIGIC has some facility that could accelerate the latter, assuming the JPEG compression isn't done as a 1-step pure-hardware process.

If there are any appropriate software hooks in the firmware's assembly code, then there's no reason the DNG couldn't be compressed almost as fast as the JPEG is generated.

vit40

lrossel, PhyrePhoX,

thanx ... obviously I didn't follow the rule "read the f... manual" 😳

silas

ewavr
edit: For any camera, supported by DNG4PS2, DNG support can be added.

Can you add a550?
Is supported by dng4ps 0.2.3-beta16.

Hacki

I'm experiencing some problems here. Programs which try to open the resulting files (to either show them in full size, or create thumbnails) tend to get killed:

Nautilus (creating thumbnails): +++ killed by SIGSEGV +++
Eog (Gnome imageviewer):+++ killed by SIGSEGV +++
Rawstudio: +++ killed by SIGSEGV +++
Rawtherapee: works fine...
Ufraw: works fine too..
ImageMagick: +++ killed by SIGSEGV +++
Gimp's tiff plugin: fatal error: Segmentation fault

None of them have no problems processing the original chdk "RAW-RAW" sensor dumps, or the DNG's created by DNG4PS2.
Either they do their job showing the files/creating the thumbnails file, or they just say "Format not supported", but not crash.

Please, dont tell me "Its working fine with adobe lightroom/this /that", i see it myself that some software is capable of handling those files fine, but most has some kind of problem, at least on my machine / Operating system. (Mandriva Linux 2009.0, Gnome 2.24)

So. Are those bugs in the programs named above, or is something "wrong" with the chdk created DNG's ?

vit40

I checked a dng from A620 with dng_validate, that is included in a dng sdk from Adobe. It didn't complain about file, except warning that there is no WB info in the file, just like in files produced by DNG4PS1 and DNG4PS2

Since Adobe written a dng specification, I suppose we should believe to Adobe's validation program

ewavr

vit40
Just an idea, don't know how much programming efforts is needed for this - what about separating a dng header into external file, that will be located somewhere on the SD card, similar to headers in DNG4PS version 1?

Why not? Of course, it is possible (it's easy without EXIF tags: data/time, aperture etc.) But maybe is better to read color matrix (or someting else?) from special text file in CHDK folder.

vit40
I checked a dng from A620 with dng_validate, that is included in a dng sdk from Adobe.

I too 😀

Hacki
Are those bugs in the programs named above, or is something "wrong" with the chdk created DNG's ?

I don't know exactly. Some ideas:
- (stupid question) are you renamed files from CRW to DNG?
- CHDK DNG do not contains thumbnail (at present)
- CHDK DNG is 10 bit, DNG4PS produces 16-bit DNGs
- CHDK DNG contains some errors (but dng_validate tool from Adobe shows no errors, and one warning (like DNGs from DNG4PS2)).

Well, I have Ubuntu Linux (in virtualbox) on my home computer, but can test something on monday only.

ewavr

silas
Can you add a550?
Is supported by dng4ps 0.2.3-beta16.

OMG, I'm using old version DNG4PS2 without A550 support. This camera (and some others) will be added in nearest future.

edit: Added A550 (also G7 and TX1). IXUS (SD series) on next week.

« Previous Page Next Page »