Cropping the sensor for DNG saving

RaduP

926

« on: 14 / June / 2014, 21:17:51 »

Advertisements

Some cameras (probably all) have some bad sensor areas on the edge, that contain no useful data. Would be nice if there were a feature to be able to crop those areas. Besides for less post processing work, the files would be saved faster and they would require less memory as well.

Logged

waterwingz

12138

Re: Cropping the sensor for DNG saving

« Reply #1 on: 14 / June / 2014, 22:02:47 »

I thought that was already done in platform_camera with the #define CAM_ACTIVE_AREA_xn settings?

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3450

Re: Cropping the sensor for DNG saving

« Reply #2 on: 14 / June / 2014, 22:11:23 »

I think RaduP is suggesting cropping the data saved to the DNG file - e.g. not saving the data outside the active area.

The top & bottom bits can be skipped (with a bit of work) by adjusting the start address and length for the data written to the DNG file.
Skipping the left and right edges would be trickier - the code assumes a continuous buffer for writing. It might also end up being slower for writing the file.

I don't think the savings would be that big 2-3% by my calculation or around 300K - 700K depending on the camera model.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

RaduP

926

Re: Cropping the sensor for DNG saving

« Reply #3 on: 14 / June / 2014, 22:27:48 »

I don't see why cropping the edge would be tricker, it would just be something like fwriting every line to the desired length, then adding a padding value at the end of the line. It should be just a few lines of code, unless if there is something odd about the fwrite implementation of the camera function.

I don't see how it can be slower, since writing the actual data on the SD is much slower than a few extra CPU cycles for the multiple fwrite overhead.

I would implement it myself, but I'd have to install the whole toolkit again and re-familiarize myself with CHDK (it's been almost 5 years).

Logged

philmoz

3450

Re: Cropping the sensor for DNG saving

« Reply #4 on: 14 / June / 2014, 22:42:14 »

Quote from: RaduP on 14 / June / 2014, 22:27:48

I don't see why cropping the edge would be tricker, it would just be something like fwriting every line to the desired length, then adding a padding value at the end of the line. It should be just a few lines of code, unless if there is something odd about the fwrite implementation of the camera function.

I don't see how it can be slower, since writing the actual data on the SD is much slower than a few extra CPU cycles for the multiple fwrite overhead.

I would implement it myself, but I'd have to install the whole toolkit again and re-familiarize myself with CHDK (it's been almost 5 years).

DNG writing now has a seperate task running that interleaves the reverse-bytes of the memory buffer with the actual file IO (thanks to reyalp).

It works on 512K memory chunks, not image scan lines. It's also considerably faster than the old method.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

reyalp

14079

Re: Cropping the sensor for DNG saving

« Reply #5 on: 14 / June / 2014, 23:01:07 »

Quote from: RaduP on 14 / June / 2014, 22:27:48

I don't see how it can be slower, since writing the actual data on the SD is much slower than a few extra CPU cycles for the multiple fwrite overhead.

In fact, doing one write per line would be *much* slower. The Canon IO system does much better with a few large writes than many small ones, presumably because it hands the whole thing off to the SD controller to DMA. SD cards in general prefer large writes too.

This thread http://chdk.setepontos.com/index.php?topic=9970.0 covers the development of the DNG changes Phil mentioned. Much tuning was trying to balance the number of writes vs how much reversing can be done in parallel.

If we wanted to speed it up further, the next thing to do IMO would be to use proper synchronization objects instead of stupid wait loops for the tasks. I haven't tested, but I suspect it could save several tens of milliseconds

We could skip the top and bottom borders, but it would complicate the code that deals with sensor size and have to be tuned for each camera.

Logged

Don't forget what the H stands for.

RaduP

926

Re: Cropping the sensor for DNG saving

« Reply #6 on: 15 / June / 2014, 06:41:12 »

Oh, yes I forgot that it won't do any write caching like desktops do :/
Is there enough memory available in the camera to actually copy the useful sensor info there, and then just write it at once on the card?

Logged

RaduP

926

Re: Cropping the sensor for DNG saving

« Reply #7 on: 15 / June / 2014, 07:29:09 »

Also, is the RAW buffer read only? If not, can't the information be placed in the same buffer, and then written at once as before?

Logged

philmoz

3450

Re: Cropping the sensor for DNG saving

« Reply #8 on: 15 / June / 2014, 07:37:31 »

Quote from: RaduP on 15 / June / 2014, 06:41:12

Oh, yes I forgot that it won't do any write caching like desktops do :/
Is there enough memory available in the camera to actually copy the useful sensor info there, and then just write it at once on the card?

On cameras with two raw buffers, you could use the second buffer.

This would require a lot of change to the DNG code - the RAW buffer has to be byte swapped when saving to the DNG file. You would have to change the byte swapping code to work on image lines, instead of blocks. This could be slower since you will be calling the function many more times. It still has to synch with the file writing code so that enough lines have been swapped before starting to write.

On cameras with only one raw buffer it would be very hard (see below).

Quote from: RaduP on 15 / June / 2014, 07:29:09

Also, is the RAW buffer read only? If not, can't the information be placed in the same buffer, and then written at once as before?

Not easily. As I said above, the memory has to be byte swapped before saving to the DNG file. If there is only one raw buffer it is swapped in place, then swapped back once the DNG file is done. This is needed because the camera has not generated the JPEG image yet.

If you change the layout of the raw buffer, then you have to be able to undo any changes so that the JPEG image generation can happen.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

RaduP

926

Re: Cropping the sensor for DNG saving

« Reply #9 on: 15 / June / 2014, 08:51:54 »

Ok, I can see a problem with one buffer, for the JPG writing. Removing the border would be trivial, but putting them back would be very hard without an extra buffer (although probably not impossible, since we still have the unused area at the end of the buffer, but it would require a lot of memory moves).

Out of curiosity, is there any criteria for which cameras have a second buffer and which do not?

Logged