Writing DNG in parallel with reversing - page 8 - General Discussion and Assistance - CHDK Forum  
« previous next »
Pages: 1 ... 3 4 5 6 7 [8] 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 31428 Views
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #70 on: 30 / May / 2013, 23:14:26 »
Advertisements
Quote from: dvip on 30 / May / 2013, 22:48:03
>heavily used

Could be.


>very old / a fake

Nope, it is new and original.
Never had problems with it and like it because it boots fast.
I should try a class 10 one of these days.

If it's new it's unlikely to have been used heavily enough to cause wear levelling problems.

Do a Google search for fake Sandisk cards - some of them are very good; but there are a lot of them out there.

An old class 2 Sandisk card gives an average speed of ~2900ms with the new code (with variation of 40-50ms).

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline dvip

  • ****
  • 451
Re: Writing DNG in parallel with reversing
« Reply #71 on: 30 / May / 2013, 23:32:52 »
Quote from: philmoz on 30 / May / 2013, 23:14:26
An old class 2 Sandisk card gives an average speed of ~2900ms with the new code (with variation of 40-50ms).
Phil.

Yes, I should try another card one of these days.
Logged
*

Offline reyalp

  • ******
  • 14111
Re: Writing DNG in parallel with reversing
« Reply #72 on: 31 / May / 2013, 01:43:58 »
Some analysis from whims results on the freshly formatted card
- Results are very stable, no slow writes to confuse things.
- Average total time (save_time column) is 1427 for old code, 968 for new. A decent improvement.
- Each reverse of the buffer takes ~310, for 620 total (rev_time and derev_time)
- Average non-reversing time for old code (file open, badpixel, header + thumb creation, writing header, writing data, closing file) is 813.
- note badpixel isn't explicitly timed, but it happens between header creation (hdr_c_end) and thumbnail start (thm_c_start) if using DNG 1.2. I've been testing with DNG 1.3, so haven't really looked at badpixel. There might be room for improvement, but users who really want speed can use 1.3 and fix the badpixels later.

So we are about 150 ms off the theoretical optimum. Where are we losing it?
- Write time (total time spent in the write loop, including sleeps waiting for the reverser) is 710ms in the new code, compared to 665 to just write the buffer all at once.  20ms of that is spent sleeping, waiting for the reverser (write_wait). The rest may be overhead from multiple writes.
- Header create + backpixel + thumbnail all need to happen before reversing start, and take 118 ms.  That only leaves ~700ms of parallelizable time.
- file open starts at 120 ms
- reversing starts at 130 ms, header writing at about the same time.
- file writing starts at 140 ms. This is  too soon for the first chunk to be reversed (full buffer = 310ms, 18 chunks = ~17ms/chunk), so it waits.
- reversing takes 440ms. There is a 10 ms sleep per chunk which should make the total ~490, but I've noticed in other places the N*sleep(10) takes less than N*10 ms.
- de-reversing takes ~390 ms, there's no sleeps except when it's waiting for the write task (finish_wait, average 40ms). The final chunk (17ms) must happen after writing finishes.
- 440+390 = 830. closing adds ~20ms, bringing the write time to 730. So reversing is ~100ms too long.    derev_end - close_end averages 92 ms, confirming this.

Conclusion:
On a camera with a reasonably fast card, smallish sensor and slow processor, there isn't actually "plenty" of time for reversing. Eliminating (some of?) the sleep in the reverse task would seem like the obvious thing to do, but I've tried a few variations of this and always got worse results.

In this situation, making the thumbnail creation parallel wouldn't actually help.
Logged
Don't forget what the H stands for.
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #73 on: 31 / May / 2013, 18:45:41 »
Well, My A650 r2812 with dual partition 16GB C10 AND twin raw buffer drops from, the best case, 2270 ms (4030, 3880, 3110, ...) to 1660 ms, a wonderfull 25%.

Concerning the twin buffer, my r1054 use twin since long time ago without any prob.

I had to modify dngtest.lua since it was too fast and didn't take photos. Here are my compile csv and my lua version.

A650 r2812, 16GBc10dual, twin
Logged
*

Offline genie

  • *
  • 43
Re: Writing DNG in parallel with reversing
« Reply #74 on: 01 / June / 2013, 03:23:48 »
Here is the SX230HS 100c , working with branches_dng-async-write-2822.zip.

Card used: Samsung 16 GB Class 10 SDHC.

Attached is the dngtest.csv file.
Logged
Sorry for my bad English.I am a Bavarian and German is the only foreign language i speak.
*

Offline reyalp

  • ******
  • 14111
Re: Writing DNG in parallel with reversing
« Reply #75 on: 01 / June / 2013, 17:52:27 »
I'm thinking about putting the current version in the trunk. It's not perfect, but it seems stable and significantly faster than the old code.

Some questions

1) Should we have an option for the old code? I wouldn't want it in 1.2 release, but it could be useful for troubleshooting if problems show up.
2) Further development or troubleshooting requires the timing code and associated lua functions, which is messy. Again, not something I'd want to have in a release version.
3) What to do with the chunk sizes? Optimal values will vary depending not just on the camera, but also the card.

I've gone back and forth on these in my head, but my current inclination is
1) No option, just use the new code.
2) Don't include timing code.
3) Use fixed values, probably 512/512 which seems to be a reasonable compromise based on data so far. Given the card dependence, fine tuning individual cameras seems pointless.

For any further development or detailed testing, we can continue to use the test branch, or create a new one with the test code patched in.

Thoughts? Objections?

Logged
Don't forget what the H stands for.
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: Writing DNG in parallel with reversing
« Reply #76 on: 01 / June / 2013, 18:05:06 »
Quote from: reyalp on 01 / June / 2013, 17:52:27
I've gone back and forth on these in my head, but my current inclination is
1) No option, just use the new code.
Agreed - on the assumption you mean 1.2.0 only.

Quote
2) Don't include timing code.
You could leave the code in as #ifdef build option for someone willing to edit the actual source file? (i.e. not even in the buildconf.inc file)

Quote
3) Use fixed values, probably 512/512 which seems to be a reasonable compromise based on data so far. Given the card dependence, fine tuning individual cameras seems pointless.
That works too - any speed up is good.  Can you set things up so a platform_camera.h option exists?
Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline reyalp

  • ******
  • 14111
Re: Writing DNG in parallel with reversing
« Reply #77 on: 01 / June / 2013, 18:27:36 »
Quote from: waterwingz on 01 / June / 2013, 18:05:06
Quote from: reyalp on 01 / June / 2013, 17:52:27
I've gone back and forth on these in my head, but my current inclination is
1) No option, just use the new code.
Agreed - on the assumption you mean 1.2.0 only.
Yes, all of this is only ever going to go in the trunk.

Quote
Quote
2) Don't include timing code.
You could leave the code in as #ifdef build option for someone willing to edit the actual source file? (i.e. not even in the buildconf.inc file)
The current code has pieces in the core, lua module and dng module, so it would really only make sense as a compile time option, and the resulting modules would be incompatible. The timing calls themselves are naturally sprinkled all over the place, which would either be a real mess of ifdefs need to be re-written into macros.

Quote
That works too - any speed up is good.  Can you set things up so a platform_camera.h option exists?
Could, but given the card dependence I have doubts about the value. Even if someone tuned for their camera, it could be worse for other users with different cards. A serious attempt at further optimization would likely have different tunable values.
Logged
Don't forget what the H stands for.
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: Writing DNG in parallel with reversing
« Reply #78 on: 01 / June / 2013, 18:34:02 »
Quote from: reyalp on 01 / June / 2013, 18:27:36
Yes, all of this is only ever going to go in the trunk.
Just checking ..  ;)

Quote
The current code has pieces in the core, lua module and dng module, so it would really only make sense as a compile time option, and the resulting modules would be incompatible. The timing calls themselves are naturally sprinkled all over the place, which would either be a real mess of ifdefs need to be re-written into macros.
Oh .. okay.  Just seems like once its removed,  its unlikely ever to go back without a ton of work 're-inventing the wheel".

Quote
Could, but given the card dependence I have doubts about the value. Even if someone tuned for their camera, it could be worse for other users with different cards. A serious attempt at further optimization would likely have different tunable values.
Never mind then.
Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline reyalp

  • ******
  • 14111
Re: Writing DNG in parallel with reversing
« Reply #79 on: 01 / June / 2013, 18:42:29 »
Quote from: waterwingz on 01 / June / 2013, 18:34:02
Oh .. okay.  Just seems like once its removed,  its unlikely ever to go back without a ton of work 're-inventing the wheel".
Yes, it's annoying, which is why I posted rather than just doing something.

My current thought is to make a patch, which will probably continue to apply cleanly for quite a while, until someone mucks with those areas of the code again. If someone wants to continue mucking with this, it should be easy enough to set up a branch with the timing code again.

The other alternative I see is to just leave all the debug/test code in the trunk for a while.
Logged
Don't forget what the H stands for.
Pages: 1 ... 3 4 5 6 7 [8] 9   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal