SX30 IS porting thread

Microfunguy

4037

Re: SX30 IS porting thread

« Reply #30 on: 30 / November / 2010, 18:29:09 »

Advertisements

Quote from: reyalp on 30 / November / 2010, 17:11:07

If you use cached raw buffer, the DNG saving time isn't so bad even on 14mp cams.

In that case why is it not always used instead of being a CHDK option ?

What makes it faster ?

Logged

reyalp

14118

Re: SX30 IS porting thread

« Reply #31 on: 30 / November / 2010, 22:07:03 »

Quote from: Microfunguy on 30 / November / 2010, 18:29:09

Quote from: reyalp on 30 / November / 2010, 17:11:07
If you use cached raw buffer, the DNG saving time isn't so bad even on 14mp cams.
In that case why is it not always used instead of being a CHDK option ?

It probably should be at this point. Theoretically, some incorrect data could be written, but in practice it seems OK. The way we process it should mean that anything but the last 4k (plus however big the write buffer is, but I think it's quite small) would be flushed out anyway.

We could explicitly flushing and draining before at the end of the reverse bytes. I suspect this wouldn't add much delay, and would be a bit safer.

Quote

What makes it faster ?

http://en.wikipedia.org/wiki/Cache

The normal raw address has the CPU data cache and write buffer disabled. This slows any data processing involving that memory down a LOT. Presumably, Canon does this for a reason. My guess is that the DSP part of Digic does most of the normal processing work, and the ARM CPU cache controller doesn't know about this.

The "write" function used to save the raw data also expects to use uncached memory, presumably because it relies on DMA hardware that is also unaware of the caches. Ref http://chdk.setepontos.com/index.php?topic=1910.0

Logged

Don't forget what the H stands for.

Microfunguy

4037

Re: SX30 IS porting thread

« Reply #32 on: 01 / December / 2010, 07:27:16 »

Let us see if I understand this ....

For uncached memory, saving a raw file simply means reading a certain number of bytes starting at a specific location and saving them to a file.

In the case of cached memory, the CPU first checks a relatively small cache to see if any raw data is there.
It will almost certainly not be, so it will have to be fetched from uncached memory many, many times.
Is that correct ?

Why would the uncached memory contain 'junk' ?

Logged

philmoz

3450

Re: SX30 IS porting thread

« Reply #33 on: 01 / December / 2010, 07:45:56 »

Quote from: Microfunguy on 01 / December / 2010, 07:27:16

Let us see if I understand this ....

For uncached memory, saving a raw file simply means reading a certain number of bytes starting at a specific location and saving them to a file.

In the case of cached memory, the CPU first checks a relatively small cache to see if any raw data is there.
It will almost certainly not be, so it will have to be fetched from uncached memory many, many times.
Is that correct ?

Why would the uncached memory contain 'junk' ?

As I understand it the raw buffers are also accessed by the DIGIC IV processor directly. The CPU cache is not linked to this processor so changes made to the raw buffer by the DIGIC processor won't update the CPU cache. So the CPU cache can contain stale data compared to what is actually in the memory.

It's not the uncached memory that's wrong; the possiblity is that the CPU cache is wrong so if you use cached memory you can get incorrect values; but you'll get them a lot faster

Regards,
Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Microfunguy

4037

Re: SX30 IS porting thread

« Reply #34 on: 01 / December / 2010, 08:05:08 »

Quote from: philmoz on 01 / December / 2010, 07:45:56

if you use cached memory you can get incorrect values; but you'll get them a lot faster

Surely the CPU cache is a fraction of the size of the raw buffer ?

Logged

reyalp

14118

Re: SX30 IS porting thread

« Reply #35 on: 01 / December / 2010, 12:51:37 »

Quote from: Microfunguy on 01 / December / 2010, 07:27:16

Let us see if I understand this ....

For uncached memory, saving a raw file simply means reading a certain number of bytes starting at a specific location and saving them to a file.

Note that saving should be almost exactly the same speed whether you use cached or uncached. The speedup in DNG is because we are using the CPU to process the entire raw buffer, by swapping bytes, and also fixing badpixels.

For "CHDK raw" the CPU just uses write() to tell the SD controller to write some area of RAM to the SD card (gross oversimplification

). The SD controller does this (presumably) with DMA, so the data never has to pass through the CPU. The SD controller won't be aware of the cache at all.

For DNG, the CPU reads every byte of the raw buffer, swaps the order, writes it back to memory, and then calls write() as above. You might think this wouldn't be much faster caching enabled, because as you note, the raw buffer is much bigger than the cache, and the entire thing still has to pass through the CPU. But having the caches enabled allows data to be pre-fetched and writes writes to be buffered, which makes it a lot quicker.

The "danger" in doing it this way is that when you call write() not everything might have been written back to actual RAM, so the SD controller might write the unswapped bytes even though the CPU had swapped them. But the data cache is only 4KB, and a lot of time will have passed before write() gets to the last 4kb of the buffer, so it seems it's actually OK.

Logged

Don't forget what the H stands for.

Microfunguy

4037

Re: SX30 IS porting thread

« Reply #36 on: 01 / December / 2010, 14:01:43 »

Thanks, that is very interesting.

In my application (360 degree stereo panoramas with a SINGLE rotating camera) I will not fix bad pixels or swap bytes.

I will simply save raw image strips (that are a fraction the size of the full image anyway).

Should work in continuous shooting mode at normal speed with a suitable card.

On completion, a single header file will be saved.

« Last Edit: 01 / December / 2010, 14:06:44 by Microfunguy »

Logged

philmoz

3450

Re: SX30 IS porting thread

« Reply #37 on: 01 / December / 2010, 16:21:37 »

Quote from: Microfunguy on 01 / December / 2010, 14:01:43

In my application (360 degree stereo panoramas with a SINGLE rotating camera) I will not fix bad pixels or swap bytes.

The lens has significant barrel distortion from 4.3m to ~7mm. You don't see it in the JPEGs because the camera is doing a lot of correction and cropping on the raw data (and then scaling the result back up to 14MP). Below 5mm there is also severe vignetting in the corners.

At 4.3mm, my testing shows that the 14MP camera JPEG image is equivalent to about 8.5MB of RAW data once the barrel distortion is corrected (when I correct for distortion in Lightroom and crop to approx the same as the camera jpeg I get a 3400x2500 resolution image).

Regards,
Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

Microfunguy

4037

Re: SX30 IS porting thread

« Reply #38 on: 01 / December / 2010, 16:25:12 »

I am hoping that PTGui or some command-line tool will correct lens distortions before stitching the TWO sets of image-strips into left/right panoramas.

See my explanation here :-

http://stereo.jpn.org/eng/stphmkr/help/file_27.htm

David

Logged

reyalp

14118

Re: SX30 IS porting thread

« Reply #39 on: 01 / December / 2010, 22:15:55 »

Quote from: Microfunguy on 01 / December / 2010, 14:01:43

Thanks, that is very interesting.

In my application (360 degree stereo panoramas with a SINGLE rotating camera) I will not fix bad pixels or swap bytes.

I will simply save raw image strips (that are a fraction the size of the full image anyway).

Should work in continuous shooting mode at normal speed with a suitable card.

On completion, a single header file will be saved.

This makes sense. I guess you plan to combine and convert them on the PC ?

As an aside, horizontal (with the normal landscape camera orientation) strips should be much simpler than vertical ones, since you can just write() a continuous chunk.

Logged

Don't forget what the H stands for.