Something I've wanted to try for a long time is doing the DNG writes a different task from the reverse bytes, on the assumption that once the DMA for the write is fired off, the OS will yield the task doing the write and we can let the reverse continue in parallel.Finally got around to trying it. The attached patch reduces DNG saving on my D10 from ~3.7 sec to ~3.2 sec. CHDK raw is ~ 3.0 sec, so this gets the overhead of DNG pretty close to negligible. I've tried a whole bunch of variants to get to this point (differing chunk sizes, sleep locations etc), but there's probably more room for improvement. Using proper IPC functions rather than globals and sleeps could potentially improve things.It runs and seems to give a modest improvement on a540, but I haven't tried to optimize it there yet.
It seems that because there are two raw buffers, the loop to re-reverse the buffer is skipped. It returns to the caller where the file handle is closed before the writing is finished.
Updated patch- fixes the off by one and multiple raw buffer issues.- sleep every reverse chunk, seems to give more stable times.- bumped priority of writer task. The idea is that the OS will be more likely to run it again once a write is done. I tried going even lower (16 to be higher priority than spytask) but it didn't seem to make any noticeable difference.a540 seems to do DNG in ~1.3 sec, where the trunk would take about 1.6 sec. CHDK raw is about 1 sec.I had some unexplained hangs on a540 while working on this. I haven't seem them in the current patch, but there may be some issue lurking.
- I moved the header and thumbnail writes into dng_writer, and created the dng_writer task before the reverse loop. Doesn't improve performance; but makes the code cleaner (removes the need for 'task_created').
- I incremented rsleep while waiting for the write to finish - most of the time is spent here.
- Removing the limit on 'size' in dng_writer (to only write a max of DNG_CHUNK_SIZE bytes per loop) reduced the DNG time from 1.5 to 1.4 secs on the G1X.
- Using cached or uncached memory makes no difference to the save times (G1X). Is the call to 'dcache_clean_all' necessary? I commented it out and the DNG files still get saved correctly.
All my cameras also take longer to save the first RAW/DNG file after power up; but this is the case with the existing trunk / release-1.1 code as well.
I created a branch for this https://tools.assembla.com/svn/chdk/branches/dng-async-writeNew version checked in- Writes header in dng_write task, per philmoz suggestion- For cameras with multiple raw buffers, writes as much data as is ready- For cameras with only one raw buffer, try to leave DNG_END_MARGIN to overlap with dereversing- Doesn't clean dcache if not using cached raw- Use msleep(0) immediately after creating the task to let it run. I verified that this does in fact start the task (if it's higher priority), and calling msleep(0) does not wait 10 ms. This sometimes allows the reversing to start before the thumbs are finished.On d10, this gives the best performance so far (~3180 ms)On a540, it seems a bit worse than the work-3 version (1400 ms vs 1300).
Some comments on the debug counterswsleep (ws in misc debug)counts how many times the file writer task waited. This happens if there isn't reversed data past the written data. This can happen if writing the header finishes before the first chunk is completed. Lower is better, since writing is the most time consuming part, you don't want this task waiting around.wcount (wc) how many chunks were written. Generally, lower is better. Optimal will probably be 2 on cameras with multiple raw buffers, or 3 on cameras with only one.rsleep (rs)how many times the main task waited for the write to complete (in the de-reversing process for cams with only one raw buffer, or waiting at the end).adjustable parametersDNG_CHUNK_SIZEhow big a chunk the reversing is done in. Smaller values mean the writer task can get to work sooner, but may cause it to do more writesDNG_END_MARGINThis only applies to cameras with one raw buffer, and is meant to avoid the situation I described in my previous post. This defines the minimum amount of the buffer that will have to be de-reversed after all writes are completed.It's not clear to me how to optimize this for different cameras.
Started by Hacki
General Discussion and Assistance
Started by fudgey
Started by cppasm
Started by larrycafe
RAW Shooting and Processing
Started by dvip
« 1 2 »
General Discussion and Assistance