Writing DNG in parallel with reversing - page 2 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 [2] 3 4 5 6 7 ... 9   Go Down

Writing DNG in parallel with reversing

  • 83 Replies
  • 36484 Views
*

Offline srsa_4c

  • ******
  • 4451
Re: Writing DNG in parallel with reversing
« Reply #10 on: 20 / May / 2013, 00:46:41 »
Advertisements
Quote from: reyalp on 19 / May / 2013, 21:00:47
I would be interested to see what the results are with a sample of cameras. If you run it, please also state your SD card speed rating.
attached, ixus65_sd630, 2+2 steps
2G SD, CL2
Quote
I am still getting hangs on my a540.
Me too (camera is roughly the same generation as the a540). Attachment is from a successful run.
Logged
*

Offline reyalp

  • ******
  • 14138
Re: Writing DNG in parallel with reversing
« Reply #11 on: 20 / May / 2013, 01:51:50 »
Quote from: srsa_4c on 20 / May / 2013, 00:46:41
attached, ixus65_sd630, 2+2 steps
2G SD, CL2
Thanks. 

Some information about the more useful items in the csv file

save_time:
Total time for the write_dng function. This is what we are trying to reduce.

hdr_time:
Time to write the header and thumbnail. Doesn't include creating the thumb image. Strangely, the very long write times often seem to appear in here. In your file, most of the"b" hdr_times were long.  I haven't seen this before on my cams, usually the first one is long, but the rest are normal. This might have something to do with sd card of filesystem fragmentation / usage patterns.

rev_time:
Time spent reversing the data. If you look at the "b" test, this is the actual CPU time required to reverse (in one direction)

write_time:
Time spent writing. In the "b" test, this is the actual time to write the whole buffer to a file, so hdr_time + write_time is the absolute minimum we could achieve, and rev_time*2+write_time+hdr_time is the best case for old code.


Quote
Me too (camera is roughly the same generation as the a540). Attachment is from a successful run.
I suspect this is difference in the way vxworks task scheduling works, but I'm not sure how. I've spent some time turning LEDs on and off at various points, but the results so far are confusing...
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #12 on: 20 / May / 2013, 04:21:10 »
No crashes on the IXUS 700 (vxworks).
Results attached - reasonably consistent 400ms speedup.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Writing DNG in parallel with reversing
« Reply #13 on: 20 / May / 2013, 10:57:42 »
sx100 (dry r23)
shoot() is skipped a few times for some reason ("exp" shows, when). Cam was in M mode and MF. Card is 2GB, no CL rating, was low level formatted by camera before. Two consecutive runs.
Logged
*

Offline reyalp

  • ******
  • 14138
Re: Writing DNG in parallel with reversing
« Reply #14 on: 20 / May / 2013, 15:55:44 »
Quote from: philmoz on 20 / May / 2013, 04:21:10
No crashes on the IXUS 700 (vxworks).
Results attached - reasonably consistent 400ms speedup.
FWIW, I had a run of over 20 shots without hitting the lockup on a540.

Quote from: srsa_4c on 20 / May / 2013, 10:57:42
sx100 (dry r23)
shoot() is skipped a few times for some reason ("exp" shows, when). Cam was in M mode and MF. Card is 2GB, no CL rating, was low level formatted by camera before. Two consecutive runs.
Now that you mention it, I think I've seen the skipped shoot too. I wouldn't think this is related to the dng change.

edit:
Here's a couple plots I made from some of the data on the D10. This run had 5 baseline shots (note I had already done some shots so the first baseline shot is not the super slow first shot) and 6 each of test shots with 256k, 512k, and 1m chunk sizes.
1 is the baseline run
2 is 256/256
3 is 256/512
etc.


In the first one, both scales are in ms, left is the save_time and write_time, right is derev_oh, which is how much time was spent de-reversing after the write finished.

The second one shows average, min and max save times (note the column labels are end chunk, rev chunk, test).

Some takeaways from this
1) The high outliers (+ a few hundred ms over typical) seem to be random behavior of the canon firmware/sd card
2) It isn't hugely sensitive to the parameters. The 256k rev chunk might be marginally worse, but otherwise they are all pretty close.
« Last Edit: 20 / May / 2013, 16:21:37 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Writing DNG in parallel with reversing
« Reply #15 on: 20 / May / 2013, 16:32:51 »
A460 (Vx, 5MP, DIGIC II) on a recently formatted cl2 card. This is the third run, it hung up in the previous attempts.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Writing DNG in parallel with reversing
« Reply #16 on: 20 / May / 2013, 18:22:40 »
Quote from: reyalp on 20 / May / 2013, 15:55:44
Quote from: srsa_4c on 20 / May / 2013, 10:57:42
sx100 (dry r23)
shoot() is skipped a few times for some reason ("exp" shows, when). Cam was in M mode and MF. Card is 2GB, no CL rating, was low level formatted by camera before. Two consecutive runs.
Now that you mention it, I think I've seen the skipped shoot too. I wouldn't think this is related to the dng change.

The trunk version of shoot returns a status value that may indicate why it failed:
 0 - successful shoot
 1 - half-press timed out
 2 - failed to take shot

By default it retries the shot if the first attempt fails.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14138
Re: Writing DNG in parallel with reversing
« Reply #17 on: 22 / May / 2013, 02:26:43 »
Some progress on the hang:
The camera isn't really hung, just spytask. kbd_task seems unresponsive because it's waiting for the scripted shoot to finish, but an LED blinking code in kbd_process runs

ptp also works, as long as it doesn't use lua. Using this, I was able to get the dng_stats structure:

Code: [Select]
0x00000000  write_wait_count;
0x00000003  write_chunk_count;
0x00000097  finish_wait_count;
0x00000008  rev_chunk_count;
0x0005c1d4  save_start;
0x00000000  save_end;
0x0005c238  write_hdr_start;
0x0005c2f6  write_hdr_end;
0x0005c2f6  write_start;
0x0005ca3a  write_end;
0x0005c242  rev_start;
0x0005c382  rev_end;
0x0005c382  derev_start;
0x0005ca6c  derev_end;
The important info from this is that everything but save_end has a tick count. That means the hang occurred somewhere in
Code: [Select]
        dng_stats.derev_end = get_tick_count();

        free_dng_header();
    }
    dng_stats.save_end = get_tick_count();
the dngwriter task at least gets to the ExitTask. This matches results from some LED debugging I did.

This doesn't make much sense to me. I guess memory corruption could theoretically make free_dng_header hang...

I think I've had the camera hang in a different way a couple of times, where kbd_task was hung and spytask was not (based on the keys not working but the voltage display updating, but I'm not completely sure.

edit:
I also had some very long runs (>300 shots over 5 runs of the script) of failing to reproduce the problem with some LED debugging code in place.

My next step is to get the TCBs for spytask and dngwriter (if it still exists), but that will have to wait for tomorrow.
« Last Edit: 22 / May / 2013, 02:30:02 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline mastodon

  • **
  • 93
Re: Writing DNG in parallel with reversing
« Reply #18 on: 22 / May / 2013, 02:55:35 »
The dngtest.lua script hangs up on the SX 200 with build 1.1.0, revision 2798.
Unfortunatelly, I do not know how to use this brunch version...
Logged
(A 410), SX 200, (SX 280), SX 700
*

Offline reyalp

  • ******
  • 14138
Re: Writing DNG in parallel with reversing
« Reply #19 on: 22 / May / 2013, 12:45:32 »
Quote from: mastodon on 22 / May / 2013, 02:55:35
The dngtest.lua script hangs up on the SX 200 with build 1.1.0, revision 2798.
The script requires functions only implemented on the branch, it cannot work with any other version of chdk.
Quote
Unfortunatelly, I do not know how to use this brunch version...
Here's a build for sx200is (100c and 100d both use the same version)

Note the branch is based on the 1.2 trunk, so the menus will be slightly different and it will not use the settings from your 1.1 build.
Logged
Don't forget what the H stands for.
Pages: 1 [2] 3 4 5 6 7 ... 9   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal