Writing DNG in parallel with reversing - page 7 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 2 3 4 5 6 [7] 8 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 33608 Views
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #60 on: 29 / May / 2013, 00:28:49 »
Advertisements
Quote from: reyalp on 28 / May / 2013, 16:43:45
I'll check it in later today.
Checked in. Dryos gets a reasonable improvement (~30ms) but vxworks seems pretty much unchanged. I think the scheduling on vxworks is still less optimal than it was with having the additional task do the writing.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #61 on: 29 / May / 2013, 00:51:21 »
Quote from: dvip on 29 / May / 2013, 00:14:54
Ok, here is the SX40HS, working now with branches_dng-async-write-2810.zip.

Card used: SanDisk 4 GB Class 4 SD. Speed performance rating: Class 4 (based on SD 2.0 Specification).

Attached is the dngtest.csv file.
Interesting, those save times and close times are all over the place, for both new and old code. The average for the new code is lower, but they overlap a lot.

I wonder if a fresh format would smooth it out.

I usually erase the DCIM directory using the "erase all images" in the canon firmware. Formatting might be better but I'm not dedicated enough to do that between every test.
Logged
Don't forget what the H stands for.
*

Offline whim

  • ******
  • 2046
  • A495/590/620/630 ixus70/115/220/230/300/870 S95
Re: Writing DNG in parallel with reversing
« Reply #62 on: 30 / May / 2013, 05:10:43 »
Hi reyalp,

Quote
I usually erase the DCIM directory using the "erase all images" in the canon firmware. Formatting might be better but I'm not dedicated enough to do that between every test.

Tested this on ixus70_sd1000 with Apacer 1 GB  "150x", using latest (2812)
Results attached

wim

PS Cloudflare appears to think the site is down when my 'recent posts' is refreshed, although I can open it
fine manually from bookmark ???
« Last Edit: 30 / May / 2013, 05:16:23 by whim »
Logged
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #63 on: 30 / May / 2013, 13:21:35 »
r2812 should be a bit faster on vxworks cams, but I'm still not entirely satisfied. Getting stuff to actually happen in parallel seems to be trickier than on dryos, which is not surprising given the observed scheduling differences.

Quote from: whim on 30 / May / 2013, 05:10:43
Hi reyalp,

Quote
I usually erase the DCIM directory using the "erase all images" in the canon firmware. Formatting might be better but I'm not dedicated enough to do that between every test.

Tested this on ixus70_sd1000 with Apacer 1 GB  "150x", using latest (2812)
Thanks. My comment was aimed specifically at divp since his saving times appeared to fluctuate a lot more than I've seen on others, but your results are interesting too. The freshly formatted is definitely faster, and both seem to have a bit less variation than your earlier results.

A note for anyone else trying this: "erase all images" will probably leave the DNGs hanging around unless you reboot between shooting and doing the erase.

Did you do a regular format or a low level? Not saying you need to try the other, just for reference.
Logged
Don't forget what the H stands for.
*

Offline Lebeau

  • ***
  • 187
Re: Writing DNG in parallel with reversing
« Reply #64 on: 30 / May / 2013, 15:46:41 »
Concerning formatting, I would get a look at this site
https://www.sdcard.org/downloads/formatter_3/

Concerning r2811, I tried it on my A540 and my A650 with an old 512 MB card (no apparent class) [my beta card for CHDK] My card is not so slow for his age and r2811 cuts 30% (1450 ->1000) in save time. I would expect that save yield on my money :)

I will give a try to my C4-16 GB card and come back. For the moment, here is, attached, my old card results, sorted by save time.

I play a lot on each chunk size. It seams that partitionning rev/derev and write tasks in multiple segments (chunks) is the big part of the optimisation process (minimising save time). That segmentation parameters take care of mem/cpu performance, writing/dma performance, card caracteristics, task scheduler, ... I will analyse data to integrate "learning" period to optimize these 3 parameters. Any idea ?

Note: My A650 has twin raw buffer (raw <> alt). I fix that miss in my code and you would do so in r2812 :)

Reyalp, that's a great job.

Optimal rev/write chink sizes: (for my old card)
A540 : 256 KB / 2 KB, Yes only 2 KB !
A650 : 256 KB / 512 KB
Logged
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #65 on: 30 / May / 2013, 16:36:57 »
Quote from: Lebeau on 30 / May / 2013, 15:46:41
Note: My A650 has twin raw buffer (raw <> alt). I fix that miss in my code and you would do so in r2812 :)
Thanks for that. You've confirmed the images are OK and there are no crashes when you use the alternate buffer for DNG?

Quote
Optimal rev/write chink sizes: (for my old card)
A540 : 256 KB / 2 KB, Yes only 2 KB !
A650 : 256 KB / 512 KB
r2812 changes the behavior of end chunk, so probably this will change for a540. If 2kb is the best value something isn't working very well.

Also, for any test to have meaning, you need a reasonable sample size, there's always variation and some saves are slow. I have been using 7 shots for each setting, but that is still influenced by the occasional slow write.

If there are two raw buffers, then the end size doesn't make any difference.


Logged
Don't forget what the H stands for.
*

Offline dvip

  • ****
  • 451
Re: Writing DNG in parallel with reversing
« Reply #66 on: 30 / May / 2013, 18:21:17 »
Formatted.

The SX40HS(1.00i)  +  branches_dng-async-write-2812.zip.
Card used: SanDisk 4 GB Class 4 SD.

Logged
*

Offline reyalp

  • ******
  • 14128
Re: Writing DNG in parallel with reversing
« Reply #67 on: 30 / May / 2013, 21:58:20 »
Quote from: dvip on 30 / May / 2013, 18:21:17
Formatted.

The SX40HS(1.00i)  +  branches_dng-async-write-2812.zip.
Card used: SanDisk 4 GB Class 4 SD.
Thanks. The write and close times are still all over the place. The baseline run (using the original serial code, with one write() call for the whole buffer) ranges from 2880 ms to 4350 ms. Even if we exclude the first shot (which is often slow on other cameras) it still goes up to 3870. The close time also ranges from 10 ms to 480 ms.

With my d10, on a typical run using a 4 gb class 2 sandisk mircosdhc write times ranging from 3000 to 3330 (first shot after boot), or only to 3080 if the first shot is excluded. Close time ranges from 30ms to 170ms.

I don't know what this means, since it happens with the old code it isn't likely to be related to my changes.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #68 on: 30 / May / 2013, 22:30:07 »
Quote from: reyalp on 30 / May / 2013, 21:58:20
Quote from: dvip on 30 / May / 2013, 18:21:17
Formatted.

The SX40HS(1.00i)  +  branches_dng-async-write-2812.zip.
Card used: SanDisk 4 GB Class 4 SD.
Thanks. The write and close times are still all over the place. The baseline run (using the original serial code, with one write() call for the whole buffer) ranges from 2880 ms to 4350 ms. Even if we exclude the first shot (which is often slow on other cameras) it still goes up to 3870. The close time also ranges from 10 ms to 480 ms.

With my d10, on a typical run using a 4 gb class 2 sandisk mircosdhc write times ranging from 3000 to 3330 (first shot after boot), or only to 3080 if the first shot is excluded. Close time ranges from 30ms to 170ms.

I don't know what this means, since it happens with the old code it isn't likely to be related to my changes.

My previous reply seems to have gotten lost when the forum crashed.

On my SX40, using a class 10 Sandisk card, the old code averages ~1200ms, and the new code ~960ms.
The variation is only 20-30ms in most cases.

I suspect either a very old, heavily used, card (wear levelling problem), or the card is a actually a fake and is not class 4.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline dvip

  • ****
  • 451
Re: Writing DNG in parallel with reversing
« Reply #69 on: 30 / May / 2013, 22:48:03 »
>heavily used

Could be.


>very old / a fake

Nope, it is new and original.
Never had problems with it and like it because it boots fast.
I should try a class 10 one of these days.

Logged
Pages: 1 2 3 4 5 6 [7] 8 9   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal