DNG optimisation for cameras with two raw buffers.

philmoz

3450

« on: 16 / December / 2010, 03:04:08 »

Advertisements

One of the time consuming parts of saving a DNG file is the need to call reverse_bytes_order twice (once to swap the bytes for saving to the DNG file, then again to swap them back for generating the JPEG).

The SX30 I am porting has two raw buffers so it occurred to me that the 'inactive' buffer could be used on each shot to eliminate one call to reverse_bytes_order. Instead of swapping the bytes in place in the active buffer, I re-wrote reverse_bytes_order to store the swapped bytes in the inactive buffer. Then in raw_savefile I save the inactive buffer to the DNG file. This way I don't need to reverse the bytes back for JPEG creation.

As a side effect (and I don't know why) the performance of reverse_bytes_order also improved.

Before I made this change reverse_bytes_order took ~350ms each call.
Now the single call takes ~330ms.

The net result is DNG saving time on the SX30 is down to 1.5 seconds (on a class 10 SD card).

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14080

Re: DNG optimisation for cameras with two raw buffers.

« Reply #1 on: 16 / December / 2010, 23:39:25 »

Very interesting.

With a bit more work, you may actually be able to do MUCH better with this technique... write out the second buffer asynchronously (let raw_hook_here return, then do the save from somewhere else).

With a single buffer you can't do this AFAIK, because the raw buffer gets trashed later in the jpeg develop process.

Logged

Don't forget what the H stands for.

philmoz

3450

Re: DNG optimisation for cameras with two raw buffers.

« Reply #2 on: 17 / December / 2010, 00:05:40 »

Now that's an interesting idea!

Any suggestions as to where the asynch save function could run?
Still getting my head around the whole CHDK model so not sure where would be the best place to do something like this.

Would also need to be able to pause next image until the buffer was free again - I like a challenge

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14080

Re: DNG optimisation for cameras with two raw buffers.

« Reply #3 on: 17 / December / 2010, 00:19:55 »

Look at the loop in core/main.c, and in generic/capt_seq.c

I think you would
call hook_raw_save_complete() <- sets flag for captseq to continue
sleep(10) <- yield, so capt_seq can actually run
write(buffer) <- save the data next time spytask is scheduled

I don't know if this will work.
Note you can abuse the return value of raw_savefile if you want, since raw_postprocess doesn't actually do anything.

Another thing I've thought about is trying to interleave the swapping and writing. So you'd swap say a 32k chunk of the raw buffer, kick a task to write it while you swap the next chunk.

Logged

Don't forget what the H stands for.

Microfunguy

4037

Re: DNG optimisation for cameras with two raw buffers.

« Reply #4 on: 17 / December / 2010, 05:54:40 »

Deleted.

« Last Edit: 17 / December / 2010, 12:30:42 by Microfunguy »

Logged

Bernd R

430

Re: DNG optimisation for cameras with two raw buffers.

« Reply #5 on: 08 / March / 2011, 10:33:29 »

I get now a sandisk Extreme pro 8 GB Card 45 megabyte/sec.cost 31 Eur.

But the chdk benchmark show for write 22 megabyte and read 15 megabyte on Ixus 1000 HS with SDXC controller.
sandisk cards are known as can write as fast as they can read.

My 15 megabyte/sec sandisk write 12 megabyte.so i guess 45 megabyte/sec can then write at least 10 *3 = 30 megabytes.but i get only 22 megaybte.

when i show savetime on RAW i get with DNG values betweeen 1.110 sec and 1.150 sec(with change from philmoz)
without DNG i get 0.750 sec - 0.800 sec on continues shoot

Is there hope to get a savetime of 0.800 sec with DNG ?

The Camera is able to shoot 3 fps /sec with jpg.But so much is not important to me.

On my card reader on USB 2 speed is 18 megabyte read and write.I have a very cheap maxpower 35 in 1 reader since longer time

are there faster readers known, or maybe the Card is a fake and somebody have put a new label on it.I can send back the card in 14 days.
I also write a Mail to sandisk.maybe they can tell more.

If the Canon sdhc controller can not do more as 22 megabytes, a 30 megabyte/sec card seem good enough and is cheaper i guess.

« Last Edit: 08 / March / 2011, 10:41:41 by Bernd R »

Logged

Ixus 1000 HS

philmoz

3450

Re: DNG optimisation for cameras with two raw buffers.

« Reply #6 on: 08 / March / 2011, 18:01:09 »

Quote from: Bernd R on 08 / March / 2011, 10:33:29

But the chdk benchmark show for write 22 megabyte and read 15 megabyte on Ixus 1000 HS with SDXC controller.
sandisk cards are known as can write as fast as they can read.

Looking at the benchmark code it is using open to manage the benchmark files - from the discussions on the startup crash thread this means it is using the 'buffered' IO system.
What this tells me (haven't tested it so I could be wrong) is that the camera SD card hardware is clocked at 15MB/s max, hence the read speed you are seeing. Buffered IO can't improve raw read speed. On slower cards you get slower speeds of course.
The measured write speed is higher because the write function is returning before all of the data is written to the card - some of the data will still be in the buffer and will be written in the background.
To test this change the 'open' calls in the benchmark code to '_open' and see what speed you get - I suspect both read and write will be same 15MB/s.

Quote

when i show savetime on RAW i get with DNG values betweeen 1.110 sec and 1.150 sec(with change from philmoz)
without DNG i get 0.750 sec - 0.800 sec on continues shoot

Is there hope to get a savetime of 0.800 sec with DNG ?

The difference comes from the call to 'reverse_bytes_order' when saving DNG. DNG files require the raw data in big-endian format whereas the camera raw buffer is little-endian - so the data has to be swapped before writing. The optimisation that started this thread is about using the second raw buffer for this - previously the raw buffer had to be reversed, the data written and then the raw data reversed back so the camera could generate the JPG image. So you had to call 'reverse_bytes_order' twice.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Bernd R

430

Re: DNG optimisation for cameras with two raw buffers.

« Reply #7 on: 09 / March / 2011, 04:41:25 »

I test this code.to see raw speed.

Code: [Select]

x = _open("A/BENCH.TMP", O_WRONLY|O_CREAT, 0777);
    if (x>=0) {
        bench.disk_write_raw_bps = 0;
        gui_bench_draw();
        t = get_tick_count();
        s=write(x, hook_raw_image_addr(), hook_raw_size());
        t = get_tick_count() - t;
        close(x);
        bench.disk_write_raw_bps = s*100 / (t/10);
        bench_to_draw = 2;
    }

speed is same.maximum is 21918, this do also after several calls not increase on all open i test

I get too in mind, to try _Open.
But here too same speed.

only last thing i try in the benchmark code, is
swap 2 megabyte
write 2 megabyte
swap 2 megabyte
write 2 megabyte.

etc

if this enhance the speed of the benchmark, because the write to SD card need no CPU power.

If that not help, then maybe no speedup is possible.
that read is slower, is maybe because the read benchmark use 64 kb blocks.
with 64 kb blocks, write is 9256 kb only.

Camera can read 131 megabytes in ram and write 98 megabytes.
this are around 55 megabyte read and write i think.

RAW have 15 megabyte, so swap byte need 0.3 sec.thats also near praxis in savetime.
too bad, that DNG does not support little endian too, as most popular CPU are little endian.

I test this code with embedded byteswap and i use 1 megabyte buffer

Code: [Select]

x = open("A/BENCH.TMP", O_WRONLY|O_CREAT, 0777);
    if (x>=0) {
        int bufsize = 1*1024*1024;
        s=0;
        bench.disk_write_raw_bps = 0;
        gui_bench_draw();
        t = get_tick_count();
        for (i=0;i< 16;i++)
        {
        reverse_bytes_order2(hook_raw_image_addr()+s,hook_raw_image_addr()+s, bufsize);
        s=s + write(x, hook_raw_image_addr()+s, bufsize);
        }
        t = get_tick_count() - t;
        close(x);
        bench.disk_write_raw_bps = s*100 / (t/10);
        bench_to_draw = 2;
    }

write speed i get maximum is 16290 kb /s

now when use 2 megabyte buffers and reduce loop count to 8 write speed is 16865 kb /sec

with 4 megabyte buffers and loop count 4 its 17055 kb /sec

with 1 block save and swap its too 17055

so seem not possible to spead up.

I do more test on RAW.when i remove badpixel patch and thumbnail create, then save time reduce from 1.100 sec to 1.050 sec.so is no important slowdown

when i remove the byteswap for test save time is then 0.900 sec.

All in all my result is i cant speedup anything and must live with the speed ;-)

but ok, after some more test, i think 1.100 sec is ok in real world to use RAW too

« Last Edit: 09 / March / 2011, 05:21:33 by Bernd R »

Logged

Ixus 1000 HS