Save RAW files in DNG format instead of RAW - page 9 - Feature Requests - CHDK Forum  
« previous next »
Pages: 1 ... 4 5 6 7 8 [9] 10 11 12 13 14 ... 40   Go Down

Save RAW files in DNG format instead of RAW

  • 392 Replies
  • 185855 Views
*

Offline MartinBudden

  • *
  • 16
Re: Save RAW files in DNG format instead of RAW
« Reply #80 on: 17 / November / 2008, 04:15:56 »
Advertisements

Quote from: ewavr on 16 / November / 2008, 17:46:54
Most time waste happens in reverting bytes order in RAW buffer, but I don't know how avoid this procedure or decrease its time (now 2.5 CPU instructions/byte).

Looking at the code I thought that reversing the bytes might be a bottleneck. (I hope you don't mind me correcting your English, reverting has a similar meaning to reverse, but it's not the correct term to use here).  I spent quite some time reading through the DNG specification to see if it would be possible to avoid the byte-reversal. Unfortunately the spec is quite clear that for packed bytes they must be in big-endian order, even if the TIFF header specifies the file is in little-endian order.

I had two thoughts on this:

a) Writing the routine in assembly language will produce some quite good savings, especially since the ARM instruction set is good at byte manipulation. There is an instruction that specifically reverses byte order (REV16), see:

ARM Information Center

Using the REV16 instruction could get it down to 4 CPU instructions per 4 bytes (LDR, REV16, STR, branch)

b) It would be useful to have an option not to reverse the bytes. That way people who wanted to could post-process the DNG file after downloading. But this is complicated by the need to somehow store the byte-order in the file (naturally there isn't a tag available for this). Probably something to do later on.

Martin
Logged
*

Offline vit40

  • ****
  • 276
Re: Save RAW files in DNG format instead of RAW
« Reply #81 on: 17 / November / 2008, 04:24:26 »
Yes, there isn't a tag for byte order, obviously because byte order is specified by the dng specification
I think we could "mark" a dng with reversed byte order simply by giving some of existing tags a special value. I have 3 ideas:

- in CameraModel tag, replace spaces with underscores
- in BaselineExposure tag, change a value n/100 to n/-100 for example (checked with dng_validate - it didn't complain; even value n/0 was fine :D  and ACR didn't crash on such file, it opened the file normaly) or n/101 or something like that ... or use some other less important tag (BaselineNoise, BaselineSharpness etc)
- change DNG version from 1.1.0.0  to something like 1.1.0.128
« Last Edit: 17 / November / 2008, 04:54:09 by vit40 »
Logged
*

Offline reyalp

  • ******
  • 14079
Re: Save RAW files in DNG format instead of RAW
« Reply #82 on: 17 / November / 2008, 04:58:38 »
Quote from: ewavr on 16 / November / 2008, 17:46:54
- interpolation of 7000 bad pixels: 110
Not much, but might be sped up using cached memory (malloc and load with fopen/fread)
Quote
- reverting bytes order in RAW buffer: 650

Most time waste happens in reverting bytes order in RAW buffer, but I don't know how avoid this procedure or decrease its time (now 2.5 CPU instructions/byte).
How about 1.1 instruction/byte ?  Reduced from 550ms -> 290ms on my 6MP camera (a540):
Code: [Select]
void revert_bytes_order(char* start, int count){
// note, we will go to count rounded up to the nearest 32
asm volatile(
"ldr r2, =0xFF00FF\n" // r2 = mask
"add r1, r0, r1\n" // r1 = start + count
"LOOP:\n"
"ldm r0, {R4-R11}\n" // load 8 words

// out = ((in>>8) & 0xFF00FF) | ((in&0xFF00FF) << 8);
"mov r3, r4, lsr #8\n" // r3 = in >> 8
"and r3, r3, r2\n" // r3 &= 0xFF00FF
"and r4, r4, r2\n" // r4 = in & 0xFF00FF
"orr r4, r3, r4, asl #8\n" // out = r3 | (r4 << 8)

"mov r3, r5, lsr #8\n"
"and r3, r3, r2\n"
"and r5, r5, r2\n"
"orr r5, r3, r5, asl #8\n"

"mov r3, r6, lsr #8\n"
"and r3, r3, r2\n"
"and r6, r6, r2\n"
"orr r6, r3, r6, asl #8\n"

"mov r3, r7, lsr #8\n"
"and r3, r3, r2\n"
"and r7, r7, r2\n"
"orr r7, r3, r7, asl #8\n"

"mov r3, r8, lsr #8\n"
"and r3, r3, r2\n"
"and r8, r8, r2\n"
"orr r8, r3, r8, asl #8\n"

"mov r3, r9, lsr #8\n"
"and r3, r3, r2\n"
"and r9, r9, r2\n"
"orr r9, r3, r9, asl #8\n"

"mov r3, r10, lsr #8\n"
"and r3, r3, r2\n"
"and r10, r10, r2\n"
"orr r10, r3, r10, asl #8\n"

"mov r3, r11, lsr #8\n"
"and r3, r3, r2\n"
"and r11, r11, r2\n"
"orr r11, r3, r11, asl #8\n"

"stmia r0!, {r4-r11}\n" // store and increment

"cmp r0, r1\n"
"bcc LOOP\n"
:::"r0","r1","r2","r3","r4","r5","r6","r7","r8","r9","r10","r11"
);
}
I tried interleaving the memory accesses more, but it nothing seemed to get me below 280-290ms. It seems like you should be able to gain more by starting to get one batch of words into cache while you are processing the next, but either it is already doing it, or I didn't get it right.

Note this will go a bit past the end of the raw buffer if it isn't a multiple of 8 words. Seems harmless on mine, but you could actually just do a few less, since all the cameras have some border pixels that are zero and don't need to be swapped. In fact, you might be able to do a few thousand bytes less if you take the border into account.

@MartinBudden:
We are armv5te, rev is armv6 and above :(
Logged
Don't forget what the H stands for.
*

Offline ewavr

  • ****
  • 1057
  • A710IS
Re: Save RAW files in DNG format instead of RAW
« Reply #83 on: 17 / November / 2008, 05:59:50 »
Quote from: reyalp on 17 / November / 2008, 04:58:38
How about 1.1 instruction/byte ?  Reduced from 550ms -> 290ms on my 6MP camera (a540):
Great! Now total time recuded from 2.4s to 2s!
Quote
@MartinBudden:
We are armv5te, rev is armv6 and above :(
Yes, camera crashes on such instruction.
Logged
*

Offline lrossel

  • **
  • 53
  • CHDK - Take it underwater
    • My Photos
Re: Save RAW files in DNG format instead of RAW
« Reply #84 on: 17 / November / 2008, 07:48:27 »
Ewavr,

All this changes you guys are talking about, are there available somewhere to be compiled or tested?.

I would like to be useful like a beta tester, well, if you need it.
Logged
If you saw me on land, It was just the surface interval
G10 without CHDK
*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Save RAW files in DNG format instead of RAW
« Reply #85 on: 17 / November / 2008, 13:27:01 »
Quote
- reverting bytes order in RAW buffer: 650
- reverting bytes order in RAW buffer back (if we want to see correct jpeg): 650

I guess there's a good reason for it, but without knowing how RAW saving works in CHDK, it strikes me odd to first reverse everything in the RAW buffer, then write DNG and then reverse them back instead of reversing single words in a register or smaller blocks of words in RAM while writing DNG to the flash card (without modifying RAW buffer contents at all, that is).
Logged
*

Offline ewavr

  • ****
  • 1057
  • A710IS
Re: Save RAW files in DNG format instead of RAW
« Reply #86 on: 17 / November / 2008, 20:09:43 »
@ Hacki:
Well, I tested this DNGs on Ubuntu 7.10
Quote
Eog (Gnome imageviewer):+++ killed by SIGSEGV +++
If DNG contains thumbnail, it shows correctly (thumbnail only)
Quote
Rawstudio: +++ killed by SIGSEGV +++
Works fine
Quote
ImageMagick: +++ killed by SIGSEGV +++
If DNG contains thumbnail, it shows correctly (thumbnail only)

Conclusion: we need thumbnail in DNG.

Logged
*

Offline ewavr

  • ****
  • 1057
  • A710IS
Re: Save RAW files in DNG format instead of RAW
« Reply #87 on: 17 / November / 2008, 20:21:40 »
Quote from: lrossel on 17 / November / 2008, 07:48:27
All this changes you guys are talking about, are there available somewhere to be compiled or tested?.
I would like to be useful like a beta tester, well, if you need it.
Test files are updated. No significant changes, except reyalp's fast reversing buffer subroutine. Sorry, I'm slightly busy now  :(

@all
Which EXIF tags would you like to see?
Logged
*

Offline vit40

  • ****
  • 276
Re: Save RAW files in DNG format instead of RAW
« Reply #88 on: 18 / November / 2008, 05:06:52 »
Do you mean which tags we want to see in "config file" (to be able to configure) or in a saved dng file?
About saved file, I think it contains all needed tags, except maybe a possibility to have 2 color matrices. Having 2 matrices is totaly unimportant for me though, because I'm using profiles now and don't need a matrix in a dng at all, as profile is attached in ACR.

Did you consider to have a possibility to save dng in reverse byte order, to make save time comparable with CDHK raw, and reverse bytes on the PC ?
Logged
*

Offline PhyrePhoX

  • *****
  • 2254
  • make RAW not WAR
    • PhyreWorX
Re: Save RAW files in DNG format instead of RAW
« Reply #89 on: 18 / November / 2008, 05:17:36 »
if you want to do/have additional work done on the pc you can also take the raw sensor dump imo O_o

as to the tags: would be nice having the chdk settings in there, together with the used version etc. oh and the temperatures would be nice to have as well :D but i guess all this isnt specified to be in the dngs as exif tags, eh. isnt there some "free" tag you can fill with all the stuff you like? hm.
Logged
Pages: 1 ... 4 5 6 7 8 [9] 10 11 12 13 14 ... 40   Go Up
« previous next »
 

Related Topics