Writing DNG in parallel with reversing - page 3 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 2 [3] 4 5 6 7 8 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 33506 Views
*

Offline mastodon

  • **
  • 93
Re: Writing DNG in parallel with reversing
« Reply #20 on: 23 / May / 2013, 11:21:04 »
Advertisements
Thank you. CHDK loads and works, but it can not load any script. I got the message, atfer selecting "Load script from file...":
Fail to load fselect.flt.
bad magicnum
After that I reailsed, that I have not made badpixel.bin.  I got the message:
Fail to load _dng.flt: ba
d magicnum
Logged
(A 410), SX 200, (SX 280), SX 700
*

Offline srsa_4c

  • ******
  • 4451
Re: Writing DNG in parallel with reversing
« Reply #21 on: 23 / May / 2013, 12:29:21 »
Quote from: mastodon on 23 / May / 2013, 11:21:04
CHDK loads and works, but it can not load any script. I got the message, atfer selecting "Load script from file...":
Fail to load fselect.flt.
bad magicnum
After that I reailsed, that I have not made badpixel.bin.  I got the message:
Fail to load _dng.flt: ba
d magicnum
Please copy the whole archive content to the card. If you're using a partitioned card, copy DISKBOOT.BIN to both partitions, and the CHDK directory to the big one.
Logged
*

Offline mastodon

  • **
  • 93
Re: Writing DNG in parallel with reversing
« Reply #22 on: 23 / May / 2013, 13:07:04 »
I have a 16 Gb card, with two partitions (small for DISKBOOT.BIN, big for CHDK and pictures).
Same error messages after I have copied the DISKBOOT.BIN to the big partition (wth CHDK), too.
Logged
(A 410), SX 200, (SX 280), SX 700
*

Offline msl

  • *****
  • 1280
  • A720 IS, SX220 HS 1.01a
    • CHDK-DE links
Re: Writing DNG in parallel with reversing
« Reply #23 on: 23 / May / 2013, 13:14:48 »
Copy DISKBOOT.BIN to the small partition and the rest of the archive (CHDK/MODULES) to big partition. The error says, you need module files from the subdir CHDK/MODULES.

msl
Logged
CHDK-DE:  CHDK-DE links
*

Offline outslider

  • *****
  • 762
Re: Writing DNG in parallel with reversing
« Reply #24 on: 23 / May / 2013, 16:31:46 »
I could test this on s2is if wanted. Could you compile this branch for s2is firmware G? :)
Logged
if (2*b || !2*b) {
    cout<<question
}

Compile error: poor Yorick
*

Offline mastodon

  • **
  • 93
Re: Writing DNG in parallel with reversing
« Reply #25 on: 23 / May / 2013, 17:35:54 »
I had removed the old config files, and it works! Thanks!
It seems to me much faster.
« Last Edit: 23 / May / 2013, 17:54:46 by mastodon »
Logged
(A 410), SX 200, (SX 280), SX 700
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #26 on: 25 / May / 2013, 21:12:59 »
Very happy that you are interested by optimizing the dng generation process.

From 1054 release, I worked hard, with my limited knowledge, to optimize this process, well before modularity.

I would suggest something that could interest you

You are creating a spooler-like task to chunk write dng file infos. That a good idea.
Pros: optimize reversal bytes and writing concurrency
Cons: starting the task after creating the thumbnail info

Here is my suggestion, a dng init buffer task and a byte reversal task.
-------------------------
1) at startup, if dng is enabled (upon config), or when user activate dng, launch DNG buffer init task which fill dng buffer with static data
2) when user disabled dng, free dng buffer
3) on "write_dng",
 ... patch bad pixels (if ...)
 ... launch byte reversal task
 ... fill dng ifds with dynamic data (about 25 tags)
 ... write dng ifds to file (chunk write?)
 ... fill thumbnail section with YCbCr viewport data !!! not raw data :)
 ... write thumbnail section to file (chunk write?)
 ... ... write already reversed raw data in chunk
 ... ... if necessary, dereverse written data
 ... until all raw data written

-------------------------------------------------
Here is my DNG file structure is: (at this time)
 - IFH : information file header
 - IFD0 : thumbnail IFD's tags
 - IFD0 data : more-than-4-bytes IFD0 data
 - IFD1 : full-res IFD's tags
 - IFD1 data : more-than-4-bytes IFD1 data
 - ExifIFD : exif IFD's tags
 - ExifIFD data : more-than-4-bytes exif IFD data
 - Thumbnail Data
 - Raw Image Data

I attached my 1054 DNG.c version FYI and for inspirations.

I hope this would help you. If you need more info, I would be pleased :)
Logged
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #27 on: 25 / May / 2013, 21:46:26 »
Hi Lebeau, good to see you here again.
Quote from: Lebeau on 25 / May / 2013, 21:12:59
... launch byte reversal task
Byte reversing can't start before thumbnail is done, or at least done with the part we want to reverse because...
Quote
... fill thumbnail section with YCbCr viewport data !!! not raw data :)
...viewport occupies same memory as raw on many cameras, once you are in the raw hook, the viewport buffer is destroyed. You could theoretically capture the thumbnail before it's destroyed, but this would be more complicated.

For me, the important thing is to focus on where a big improvement is possible. On many cameras, reversing is ~600ms. On a fast card, this can be a significant portion of the save time. See mastodon sx200 data, reverse time (both ways) is ~660 ms, time to write the raw data is 880.

By writing in parallel we can reduce the reverse overhead to almost nothing (~100ms or less). Outside of reversing (thumbnail, exif etc), the difference between "CHDK raw" and DNG is quite small, 100-200ms or so, mostly dominated by the time to actually write the extra data.

One improvement I do plan to do (once I get the vxworks hang figured out) is put the header and thumb in the same buffer. Saving a write() call should be worth something, and it shouldn't make the code much more complicated.
Logged
Don't forget what the H stands for.
*

Offline outslider

  • *****
  • 762
Re: Writing DNG in parallel with reversing
« Reply #28 on: 26 / May / 2013, 04:34:50 »
I have tested on s2is. No problems as far. DNG saving ~1s faster than from CHDK trunk 1.2.
Logged
if (2*b || !2*b) {
    cout<<question
}

Compile error: poor Yorick
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #29 on: 26 / May / 2013, 09:15:27 »
Thanks reyalp for the re-welcome.

My 1054 dng version save viewport-thumbnail correctly on my a650, that's why I suggest to launch reversing before thumbnailing. Don't know for other cams.

I just remember. You are right, reyalp, reversion impact viewport data :(

Which one is faster ?
 - Creating a buffer spool task that write in 512 bytes chunk ?
Sending (id, bufptr, bufsize) to write task, and waiting back for (id, chunks#) to acknowledge actual writing.

Or writing a big chunk (header & thumbnail) with concurrent reversal and chunking raw with concurrent de-reversal purpose (only usefull for raw = alt) ?

In fact, which one is faster with concurrent task, writing chunky or bulky buffer ? You are suggesting bulky but I am intrigated by the reason.

My reviewed suggestion (?):
1) at startup, if dng is enabled (upon config), or when user activate dng, launch DNG buffer init task which fill dng buffer with static data (about 50% tags) [my dng_header_init ( void ), line 395 .. 578]
2) when user disabled dng, free dng buffer
3) on "write_dng",
 ... patch bad pixels (if ...)
 ... fill thumbnail section with already YCbCr viewport data !!! not raw data
 ... launch byte reversal task
 ... fill dng ifds with dynamic data (about 50% tags) [my create_dng_header, line 583 .. 695]
 ... write dng ifds to file (chunk write?)
 ... write thumbnail section to file (chunk write?)
 ... ... write already reversed raw data in chunk (?)
 ... ... if necessary, dereverse written data
 ... until all raw data written
 ... wait dereversion end, if necessary

I like DNG init task since a 50% of DNG tags don't change on a per picture basis :)

By the way, my code allocate space for tags and thumbnail data (line 475)
« Last Edit: 26 / May / 2013, 13:38:13 by Lebeau »
Logged
Pages: 1 2 [3] 4 5 6 7 8 9   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal