Is DNG saving in "Big-endian" 20 360 times faster than "Little-endian" ???

Lebeau

« on: 21 / December / 2010, 21:35:06 »

Advertisements

May I suggest to create a DNG file using Big-Endian "MM" (Motorola) instead of "II" (Intel) ?

Quote

JHOVE TIFF-hul Module
Header with 0x4D4D002A (if big-endian) or 0x49492A00 (if little-endian) at offset 0

Then in dng.c,

Code: [Select]

 // TIFF file header
 var = 0x4D4D; // big-endian endian "MM" instead of 0x4949 "II"

In raw.c,
- generating reverse-ordered thumbnail data (

) [really not sure!]

Code: [Select]

 * buf++ = b;
 * buf++ = g;
 * buf++ = r;

- reversing byte-order of the DNG header
- and saving straight Canon Raw data

Code: [Select]

 reverse_bytes_order ( get_dng_header ( ), get_dng_header_size ( ) );  //including var = 0x2A !
 write ( fd, get_dng_header ( ), get_dng_header_size ( ) );
 write ( fd, thumbnail_buf, DNG_TH_WIDTH * DNG_TH_HEIGHT * 3 );
 write ( fd, get_raw_image_addr ( ), hook_raw_size ( ) );

Ending with ONE dng_header_size (1536 bytes) instead of TWO hook_raw_size byte-reversing (31 272 480 bytes) !!! 20 360 faster !!!

Am I wrong ? Did I forget something?
I am ready to try and recompile !

P.S.: We could possibly save thumbnail data in YUV !!! PhotometricInterpretation = 6 !

« Last Edit: 21 / December / 2010, 21:38:10 by Lebeau »

Logged

Galerie Photobucket, Canal YouTube, Galerie Panoramio

philmoz

3070

Re: Is DNG saving in "Big-endian" 20 360 times faster than "Little-endian" ???

« Reply #1 on: 21 / December / 2010, 22:04:06 »

Unfortunately you can't do this.

The DNG format supports either big or little endian for the file container.

However the raw image data must be stored in big endian format if the bits per pixel is not 8, 16 or 32. (See the BitsPerSample restriction in section 3 of the DNG spec document.)

Canon P&S cameras have either 10 or 12 bits per pixel so you're stuck, you have to save the raw image data in big endian format.

Regards,
Phil.

« Last Edit: 22 / December / 2010, 19:18:51 by philmoz »

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)

philmoz

3070

Re: Is DNG saving in "Big-endian" 20 360 times faster than "Little-endian" ???

« Reply #2 on: 21 / December / 2010, 22:10:42 »

Quote from: Lebeau on 21 / December / 2010, 21:35:06

Ending with ONE dng_header_size (1536 bytes) instead of TWO hook_raw_size byte-reversing (31 272 480 bytes) !!! 20 360 faster !!!

With some optimisations I have reduced the DNG saving overhead to 1/2 second on the SX30 camera (14MP = 21Mbyte file size).
RAW save time is ~1 sec on a class 10 SD card.
DNG save time is ~1.5 sec.

I've posted some other topics that discuss the things I have done.

Regards,
Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)

Lebeau

Re: Is DNG saving in "Big-endian" 20 360 times faster than "Little-endian" ???

« Reply #3 on: 25 / December / 2010, 09:40:57 »

Merci beaucoup Philmoz !

Concerning Chapter 3,

Code: [Select]

BitsPerSample
Supported values are from 8 to 32 bits/sample. The depth must be the same for each sample if SamplesPerPixel is not equal to 1. If BitsPerSample is not equal to 8 or 16 or 32, then [b]the bits must be packed into bytes [/b] using the TIFF default FillOrder of 1 (big-endian), even if the TIFF file itself uses little-endian byte order.

This make restriction on bits within byte, not byte ordering, IMHO.

Code: [Select]

TIFF Tag FillOrder
Default  1 (lower column values in higher order bits) 
The [b]logical order of bits within a byte[/b]. 
The specification defines these values: 
1 = pixels with lower column values are stored in the higher-order bits of the byte.
In practice, the use of FillOrder=2 is very uncommon, and is not recommended.

And this correspond to LSB/MSB where B means bit, nor byte, IMHO.

But for now, I will concentrate on YUV video capture and may be come back on DNG generation later.

I am still convinced that a byte swap related to dir_entry's tag, type and/or count would permit reversing DNG header content instead of RAW data.

More to come some day ...

Logged

Galerie Photobucket, Canal YouTube, Galerie Panoramio

reyalp

11583

Re: Is DNG saving in "Big-endian" 20 360 times faster than "Little-endian" ???

« Reply #4 on: 25 / December / 2010, 13:33:00 »

Quote from: Lebeau on 25 / December / 2010, 09:40:57

Concerning Chapter 3,
Code: [Select]
BitsPerSample Supported values are from 8 to 32 bits/sample. The depth must be the same for each sample if SamplesPerPixel is not equal to 1. If BitsPerSample is not equal to 8 or 16 or 32, then [b]the bits must be packed into bytes [/b] using the TIFF default FillOrder of 1 (big-endian), even if the TIFF file itself uses little-endian byte order.
This make restriction on bits within byte, not byte ordering, IMHO.

I'm pretty sure your interpretation is wrong, but you can always try constructing a DNG with your suggested settings and see if any tool understands it correctly.

Logged

Don't forget what the H stands for.