Writing DNG in parallel with reversing - page 6 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 2 3 4 5 [6] 7 8 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 24758 Views
*

Offline reyalp

  • ******
  • 14034
Re: Writing DNG in parallel with reversing
« Reply #50 on: 27 / May / 2013, 22:01:18 »
Advertisements
I've committed the reversing task code, since it's cleaner and I think we can get performance on a par with the other code.

The image attached to the previous post may help explain why I'm not excited about trying too many variations  :haha

Basically every change needs a run of 70+ photos on both cameras to be sure I have a meaningful understanding of the effects. If the change has unexpected effects (which has happened pretty regularly so far), this has to be done multiple times.

To me, getting DNG within 100-200ms of raw frame buffer dump is an excellent result. Getting it even closer would be nice of course, but the impact for users seems fairly low. If we can make the code easier to maintain, that's worthwhile in it's own right. OTOH, adding a lot of complexity for the last 100ms may not be.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #51 on: 27 / May / 2013, 23:09:54 »
Quote from: reyalp on 27 / May / 2013, 22:01:18
I've committed the reversing task code, since it's cleaner and I think we can get performance on a par with the other code.

The image attached to the previous post may help explain why I'm not excited about trying too many variations  :haha

Basically every change needs a run of 70+ photos on both cameras to be sure I have a meaningful understanding of the effects. If the change has unexpected effects (which has happened pretty regularly so far), this has to be done multiple times.

To me, getting DNG within 100-200ms of raw frame buffer dump is an excellent result. Getting it even closer would be nice of course, but the impact for users seems fairly low. If we can make the code easier to maintain, that's worthwhile in it's own right. OTOH, adding a lot of complexity for the last 100ms may not be.

Sounds good - always best to get version 1 out there and we can work on version 2 later.

A stellar effort on your part to make this happen - big round of applause :)

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline waterwingz

  • ******
  • 12138
    • Waterwingz
Re: Writing DNG in parallel with reversing
« Reply #52 on: 27 / May / 2013, 23:28:18 »
Quote from: philmoz on 27 / May / 2013, 23:09:54
A stellar effort on your part to make this happen - big round of applause :)
2nd that!
Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #53 on: 28 / May / 2013, 06:20:41 »
Quote from: reyalp on 27 / May / 2013, 22:01:18
To me, getting DNG within 100-200ms of raw frame buffer dump is an excellent result. Getting it even closer would be nice of course, but the impact for users seems fairly low. If we can make the code easier to maintain, that's worthwhile in it's own right. OTOH, adding a lot of complexity for the last 100ms may not be.

Did a quick test on the G1X.

Current branch code, best case (512K for both rev & end size) - average of 1293ms.
Increasing thumbnail size to 256x192 (4x) - average 1380ms.

Moving thumbnail generation into reverse task and writing thumbnail last - average 1273ms.
Increasing thumbnail size to 256x192 (4x) - average 1283ms.

So the saving is not huge; but there is certainly potential for an improved thumbnail generation algorithm without adversely affecting save times.

On the G1X (and cams with two raw buffers) the thumbnail still works this way because it is generated from the unreversed buffer. So I haven't tried coding up an alternate get_raw_pixel yet.

Phil.

Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline whim

  • ******
  • 2046
  • A495/590/620/630 ixus70/115/220/230/300/870 S95
Re: Writing DNG in parallel with reversing
« Reply #54 on: 28 / May / 2013, 11:17:23 »
Great job, cuts roughly one third off DNG saving time !

Camera:  ixus70_sd1000  101b (VxWorks)
Card:       Apacer 1 GB  "High Speed 150x"
Script:     dngtest.lua from post #35, default params
CHDK:     dng-async-write branch, rev. 2808

cheers,
wim

edit: added results v2
« Last Edit: 28 / May / 2013, 16:31:05 by whim »
Logged
*

Offline reyalp

  • ******
  • 14034
Re: Writing DNG in parallel with reversing
« Reply #55 on: 28 / May / 2013, 12:48:03 »
Quote from: philmoz on 28 / May / 2013, 06:20:41
Current branch code, best case (512K for both rev & end size) - average of 1293ms.
Increasing thumbnail size to 256x192 (4x) - average 1380ms.
FWIW, end size shouldn't make any difference for cameras with 2 raw buffers. If it does, I've messed something up.

I think generating the thumbnail in parallel is a good idea. I didn't mean my last as a rejection of that or other ideas, just an explanation of how I was focusing my efforts.

In the meantime, I've made a version that puts the file / directory creation and closing in parallel with the reversing. Patched attached, I'll probably commit this later but I haven't finished running the numbers.

The open step isn't worth much, usually only ~10ms except for the first shot which can be a lot longer. Closing can be more significant, and has the benefit that it can run in parallel with the final chunk of de-reversing. This won't help cameras with two raw buffers, but should be an improvement on cams with only one.

A new version of the script (attached) is also required. Note that the numbers aren't directly comparable, because open / close were previously not counted in the total. The save_time number now should be very close to the equivalent of the raw save time shown in the console.

« Last Edit: 28 / May / 2013, 12:50:40 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline dvip

  • ****
  • 451
Re: Writing DNG in parallel with reversing
« Reply #56 on: 28 / May / 2013, 16:25:58 »
With the new dngtest.lua , my SX40HS and branches_dng-async-write-2809.zip.
I get:

:100 attempt to perform arithmetic on field '?' (a nil value)
*** Terminated ***

Could I be missing something in the parameters?

Logged
*

Offline whim

  • ******
  • 2046
  • A495/590/620/630 ixus70/115/220/230/300/870 S95
Re: Writing DNG in parallel with reversing
« Reply #57 on: 28 / May / 2013, 16:34:49 »
Updated post #54 with results after mods in #55

wim
Logged
*

Offline reyalp

  • ******
  • 14034
Re: Writing DNG in parallel with reversing
« Reply #58 on: 28 / May / 2013, 16:43:45 »
Quote from: dvip on 28 / May / 2013, 16:25:58
With the new dngtest.lua , my SX40HS and branches_dng-async-write-2809.zip.
I get:

:100 attempt to perform arithmetic on field '?' (a nil value)
*** Terminated ***

Could I be missing something in the parameters?
You would need a build with the patch applied to use that script. I'll check it in later today.
Logged
Don't forget what the H stands for.
*

Offline dvip

  • ****
  • 451
Re: Writing DNG in parallel with reversing
« Reply #59 on: 29 / May / 2013, 00:14:54 »
Ok, here is the SX40HS, working now with branches_dng-async-write-2810.zip.

Card used: SanDisk 4 GB Class 4 SD. Speed performance rating: Class 4 (based on SD 2.0 Specification).

Attached is the dngtest.csv file.
Logged
Pages: 1 2 3 4 5 [6] 7 8 9   Go Up
« previous next »
 

Related Topics