Writing DNG in parallel with reversing - page 5 - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: 1 2 3 4 [5] 6 7 8 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 23838 Views
*

Offline blackhole

  • *****
  • 922
  • A590IS 101b
    • Planetary astrophotography
Re: Writing DNG in parallel with reversing
« Reply #40 on: 27 / May / 2013, 09:48:22 »
Advertisements
a530 ,dngtest.csv file.
SD 512 MB class2
Logged
*

Offline outslider

  • *****
  • 762
Re: Writing DNG in parallel with reversing
« Reply #41 on: 27 / May / 2013, 11:08:57 »
I think that usage of live view for RAW thumbnail should be carefully considered. When shooting is done with overriden shutter of aperture the live view can be simply useless. Especially while making long-shutter photos, for example night shoots. The live view is completely black.

I think that usage of RAW data is better idea. Not to mention that this is much more simple than capturing live view just before it's data get overwritten.
Logged
if (2*b || !2*b) {
    cout<<question
}

Compile error: poor Yorick
*

Offline reyalp

  • ******
  • 14000
Re: Writing DNG in parallel with reversing
« Reply #42 on: 27 / May / 2013, 14:05:00 »
Quote from: outslider on 27 / May / 2013, 11:08:57
I think that usage of RAW data is better idea. Not to mention that this is much more simple than capturing live view just before it's data get overwritten.
I agree about the complexity, but as far as quality goes I'm less convinced. The current thumbnails is pretty horrid.  (edit: but valid point where overrides would make the live view unreadable)

Note that some programs generate a thumbnails from the main image, so the thumbnail shown in windows etc may not reflect the embedded thumbnail. Surprisingly, Adobe DNG codec http://www.adobe.com/support/downloads/product.jsp?product=194&platform=Windows seems to do this.

Irfanview -> view thumbnails does show the actual CHDK thumbnail, verified by generating broken ones when I was messing with the code ;)

There is probably a way to have Digic spit out a very nice thumbnail from the raw data...
« Last Edit: 27 / May / 2013, 14:12:43 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #43 on: 27 / May / 2013, 16:22:28 »
Ok ! Thumbnail data from raw data to insure data pertinence ! And, actually, no benefit initializing half-DNG header prior to any capture !

Therefore, to resume, restrictions are :
 - we cannot reverse bytes until thumbnail is created, in buffer
 - we cannot write reversed raw data until bytes are reversed
upon ( raw ptr == alt raw ptr )
 = we cannot dereverse until reversed bytes are written
 = we cannot leave main task until complete dereversal

The objective is to increase concurrency (or to reduce latency) between write and reversing/dereversing tasks.

You are proposing (and testing) to launch a write task, concurrently to reversed raw data. I am really pleased to see your greatfull efforts and I will be really happy with your results. I am really interested to DNG optimization, therefore, I try to contribute, at my best.

Now, here is another suggestion based on DNG file format.

Since DNG file has a preambule (the little section with the magic 0x4949) that point to IFD0, and IFD0 point toward Thumbnail section (tag 0x0111) and IFD1 point toward Reversed RAW section (tag 0x0111), and since I am trying to optimize DNG upon previous restrictions, I would suggest to resequence DNG sections like the following, with [ underline coprocessing comment ].

--- for (raw <> alt) camera ---
 - write Preambule & Thumbnail data [start reversing task after thumbnail in buffer, before writing]
 - write IFDs [continue reversing task to maximize future chunky size writting]
 - write Reversed RAW data [concurrently to reversing task, if not finish]

--- for (raw == alt) camera ---
 - write Preambule & Thumbnail data [start reversing task after thumbnail in buffer, before writing]
 - write Reversed RAW data [concurrently to reversing task and dereversing task once written]
 - write IFDs [concurrently to dereversing task]
-------------------------------------

Here is the reorganise programming : (assomption, chunky writing increase task concurrency)
 - Patch Pixels b (correct raw data)
 - Fill gamma (or table ?)
 - Fill thumbnail buffer, with preambule and gamma-converted raw data
 - Launch reversing task
 - Do chunky thumbnail buffer write (n*512 ?)
 - if dereversal mandatory [adjust tag 0x0111 of ifd0 & ifd1]
 == Launch dereversal task
 == Do chunky raw writing (n*512 ?)
 == Do chunky IFDs writing (n*512 ?)
 - else [no dereversal but adjust tag 0x0111 of ifd0 & ifd1]
 == Do chunky IFDs writing (n*512 ?)
 == Do chunky raw writing (n*512 ?)
-----------------------------------------------------------------------------

I would appreciate to have your comments on this.
Logged
*

Offline reyalp

  • ******
  • 14000
Re: Writing DNG in parallel with reversing
« Reply #44 on: 27 / May / 2013, 16:38:03 »
Quote from: philmoz on 27 / May / 2013, 05:47:25
Current table values below:
I've added this. It does remove the ~50ms extra delay on the first shot.

Quote
Probably also worth testing if the rounding of the dng header size up to a 512 byte boundary makes any appreciable difference.
I think ewavr did this based on testing but it wouldn't hurt to be sure. Given that we are including the thumb, larger alignments might also be worth trying.

For raw in general, this is an area that might benefit from more investigation. From what I understand, the cards use underlying blocks much larger than 512 bytes, and may be sensitive to alignment on those boundaries. This effect of this might be influenced by FAT cluster sizes, and partition alignment if you are using multiple partitions. (ref https://wiki.linaro.org/WorkingGroups/KernelArchived/Projects/FlashCardSurvey )
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3449
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #45 on: 27 / May / 2013, 18:30:56 »
Based on our IRC discussion yesterday, I have some more thoughts on how the tasks could be overlapped.

The main DNG task takes care of creating the DNG header, opening the file and writing the data, while the new task takes care of everything else.

The new task does (in the following order):
- bad pixel patching
- reverse bytes
- thumbnail generation from the reversed data (requires new version of get_raw_pixel)
- for cameras with one raw buffer, de-reverse ,otherwise exit

To support this the thumbnail data should be saved after the raw data in the DNG file, so the main task does:
- create header
- create new task
- open file
- write header
- write reversed raw data
- write thumbnail
- close file

The exact point at which the new task is started and the task switching points may need some experiments to fine tune.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14000
Re: Writing DNG in parallel with reversing
« Reply #46 on: 27 / May / 2013, 19:19:14 »
I tried putting the reversing in it's own task rather than the writing, patch attached. It makes the code cleaner and doesn't require as much stack, but on D10 performance is worse across the board, 50-175ms, with the biggest penalty on the chunk settings that are the best cases with the write task code. The loss seems to be accounted for in write time.

I suspect this is due to scheduling differences. In the old code I gave the writer task a small priority boost over spytask, which should make it more likely to get scheduled immediately when a write competes. Reversing the roles this doesn't work, because tasks with lower priority than spytask don't seem to run at all (edit: see following post).  I have some test for this in mind and still need to look at the a540 data.

It's worth noting that the best case settings in the write task code on D10 are within ~100ms or less of the raw write time, so the room for additional gain is fairly limited. 30ms of that is the thumbnail.

edit:
the effect is not as pronounced on a540, but it's still there.

edit:
Bumping spytask to priority 0x18 on d10 brings the difference down to a negligible level, possibly worse by ~10-20ms in the fastest cases, but might just be noise.

Doing the same on a540 produces less clear results, some chunk settings that were good with one version are bad with the other, but the overall lowest times are similar. This does not appear to be the an effect of a few slow outliers either.

I'm rather hesitant to bump up spytask globally, my understanding is that the rational for the current 0x19 setting was steal from the canon firmware as little as possible under normal operation. Using a higher priority task for writing should be OK, since that task will spend most of it's time suspended waiting for a DMA to finish. The higher priority just means it be more likely to be scheduled immediately when the write finishes.
« Last Edit: 27 / May / 2013, 21:11:45 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3449
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #47 on: 27 / May / 2013, 19:58:35 »
Quote from: reyalp on 27 / May / 2013, 19:19:14
I tried putting the reversing in it's own task rather than the writing, patch attached. It makes the code cleaner and doesn't require as much stack, but on D10 performance is worse across the board, 50-175ms, with the biggest penalty on the chunk settings that are the best cases with the write task code. The loss seems to be accounted for in write time.

I suspect this is due to scheduling differences. In the old code I gave the writer task a small priority boost over spytask, which should make it more likely to get scheduled immediately when a write competes. Reversing the roles this doesn't work, because tasks with lower priority than spytask don't seem to run at all.  I have some test for this in mind and still need to look at the a540 data.

It's worth noting that the best case settings in the write task code on D10 are within ~100ms or less of the raw write time, so the room for additional gain is fairly limited. 30ms of that is the thumbnail.

edit:
the effect is not as pronounced on a540, but it's still there.

Makes sense.

It might still be possible to generate the thumbnail from the reversed data so its generation overlaps the write (and write it after the RAW bytes). Whether it's worth the effort or not is questionable; but it might allow a better quality or larger thumbnail to be generated without costing any more time.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #48 on: 27 / May / 2013, 21:14:58 »
Philmoz, thumbnailing reversed raw is an excellent idea ! And I agree with you concerning the "beauty" of delegating task to reverse/dereverse bytes.

Reyalp, concerning increasing prioriry of writing task, I would keep writing task concurrently with reversing/dereversing tasks, writing having a higher priority than spytask and reversing/dereversing tasks.

Humm, my best suggestion to date :

In Spy Task :
 - Patch Pixels b (correct raw data)
 - Launch Memory Process Task
 - Launch File Creation Task
 - Wait tasks ending

In File Creation Task (higher priority than other) :
 == Fill IFDs buffer, w/o thumbnail data
 == Write Preambule (small but manage opening process)
 == Launch progressive Reversed Raw writing task (write each reversed chunk [Reversed to Inverted])
 == Launch progressive Thumbnail writing task (write each thumbnailed chunk [Thumbnailed to Thumbnail])
 == Launch IFDs writing task (concurrent to De-reverse, if present)

In Memory Process Task (per memory chunk loop) :
 == Reverse each chunk (chunk state change: Raw to Reversed)
 == Fill Reversed Raw thumbnail buffer (chunk state change: Inverted to Thumbnailed)
 == (if necessary) De-Reverse each chunk (chunk state change: Thumbnail to Raw)

File structure:
 - Préambule
 - Raw
 - Thumb
 - IFDs

I like that thread :D

P.S.: Memory process could be integrated to spytask !)
« Last Edit: 27 / May / 2013, 21:25:16 by Lebeau »
Logged
*

Offline reyalp

  • ******
  • 14000
Re: Writing DNG in parallel with reversing
« Reply #49 on: 27 / May / 2013, 21:27:44 »
OK, I thought I'd tried making a task lower prio (higher number) than spytask and didn't get it to run at all (which seemed weird  :-[), but I just tried putting the reverse task up to 0x1a. Not only did it run, for d10 it returned performance to a similar range to the write task version.

I guess this means when they are the same priority, when a write finishes reverse task might get scheduled again, where if it's lower the write task will get first shot.

I also ran a second run on d10 with identical priorities and the results were *very* similar to the first run, so I'm confident we aren't just chasing noise.

a540 results are similar to running spytask at higher priority.

edit:
added image
« Last Edit: 27 / May / 2013, 21:33:19 by reyalp »
Logged
Don't forget what the H stands for.
Pages: 1 2 3 4 [5] 6 7 8 9   Go Up
« previous next »
 

Related Topics