I/O Buffer question - General Discussion and Assistance - CHDK Forum supplierdeeply

I/O Buffer question

  • 13 Replies
  • 3805 Views
*

Offline RaduP

  • *****
  • 908
I/O Buffer question
« on: 06 / April / 2009, 16:46:31 »
Advertisements
Since many of the cameras have some sort of burst mode, where you can take pictures at a faster speed than you can write on the SD card, I would guess that there is some buffer where this information is stored while it awaits for the SD card (different from the jpg/raw data buffer).
Do you happen to know if this memory can be directly accessed by the ARM core?

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: I/O Buffer question
« Reply #1 on: 06 / April / 2009, 17:06:52 »
Or do they all require smaller resolution in those (sports?) modes and thus are able to split the RAW buffer?

*

Offline RaduP

  • *****
  • 908
Re: I/O Buffer question
« Reply #2 on: 06 / April / 2009, 17:31:20 »
Well, they would still need the raw buffer for every shot, no?
The idea is, if there is such an independent buffer, it must be pretty big, a few MB maybe. According to the specs, my A570 IS has 1.5 FPS burst mode (I never tried it though). Since people can use slower SD cards, it is reasonable to assume that there is such buffer.

Now, if this buffer can be found, and it is accessible by the ARM core, I think a lot of cool stuff can be done. For example, moving all the ROM in there, to be able to hook functions directly from the ROM.

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: I/O Buffer question
« Reply #3 on: 06 / April / 2009, 18:19:23 »
Well, they would still need the raw buffer for every shot, no?

If they skip lines they don't need all of it, do they? Similarly to video mode... speed up readout by reading less. But maybe someone who has one of these cameras (S series? Others?) tell us how they work.

According to the specs, my A570 IS has 1.5 FPS burst mode (I never tried it though). Since people can use slower SD cards, it is reasonable to assume that there is such buffer.

This is just a spec for the regular continuous shooting mode, available in all shooting modes along with custom timer and the others. This burst speed drops if your card can't handle it. But now that you mention it, I don't think I've ever tested bursts thoroughly on a slow card (hint: those MMC's that come with these cameras are extremely slow). A pretty quick test should reveal whether the camera is able to buffer even one shot. I would think the cam could start shooting the next frame once it has finished JPEG encoding for the first shot, so it's to be expected that the delay between the first and the second shot could be shorter than all consecutive ones even if this extended memory area didn't exist.

The a570 can go on 'forever' with approximately that quoted 1,5 FPS in full resolution as long as no scaling happens, Tv is fast enough (duh), ISO is low (noise reduction processing is slow, if my mind serves me right ISO 200 and up start to slow bursts down...and this is separate from dark frame reduction) and if the card is fast enough.

Maximum burst speed is actually greater than that 1.5 FPS, it's achieved with the digital tele zoom feature cranked up to max 1.9x 'zoom' (==cropping from the center of the CCD) and JPEG resolution reduced accordingly to match the crop. I've measured 2.0 FPS at 1600x1200 this way.

Now, if this buffer can be found, and it is accessible by the ARM core, I think a lot of cool stuff can be done. For example, moving all the ROM in there, to be able to hook functions directly from the ROM.

I think there are some quite old posts you may want to try to find on these forums that I think discussed raw buffers... (no, I don't know where).

Anyway, if regular burst mode uses this area, then it's kind of a bad thing to use it for CHDK since everyday camera operation needs it. But of course sacrificing burst modes temporarily is no biggie if great things can be done this way.


*

Offline RaduP

  • *****
  • 908
Re: I/O Buffer question
« Reply #4 on: 06 / April / 2009, 18:50:11 »
Quote
If they skip lines they don't need all of it, do they? Similarly to video mode... speed up readout by reading less. But maybe someone who has one of these cameras (S series? Others?) tell us how they work.
I guess it depends on how many lines they skip. If they only skip one line (out of 3), the buffer is still not big enough for two shots.

Well, I am a total newbie at ARM and especially at how the Canon cameras work, but would it be possible to have a program that scans all the address ranges to try to find r/w memory? What happens if you read or write to an invalid location? Do you get an exception?
« Last Edit: 06 / April / 2009, 18:52:09 by RaduP »

*

Offline reyalp

  • ******
  • 13448
Re: I/O Buffer question
« Reply #5 on: 06 / April / 2009, 19:31:14 »
Since many of the cameras have some sort of burst mode, where you can take pictures at a faster speed than you can write on the SD card,
Which specific camera do you have in mind ? Most recent (digic-II and later) canon P&S cameras will shoot continuously at the same rate until the card is full. Some do use multiple raw buffers, but the reason for this isn't clear. Perhaps it lets them do shooting and jpeg encoding in parallel.

As for the jpeg buffer CHDK doesn't use it at all, so I don't think anyone has spent much time looking at how it behaves.
Quote
I would guess that there is some buffer where this information is stored while it awaits for the SD card (different from the jpg/raw data buffer).
Do you happen to know if this memory can be directly accessed by the ARM core?
The existence of additional memory is unwarranted speculation on your part. My guess is that it wouldn't be additional memory, it would be multiple locations for of the normal buffers (like the dynamic raw buffer). If this was not the case, a copy would be required, which would be counter productive. Thus it would be in the same 32-128 meg region of memory already known. There may well be more memory we can use, but finding it will require actual reverse engineering, not guessing.
Well, I am a total newbie at ARM and especially at how the Canon cameras work,
I'd suggest that speculation is pointless until you fix that.
Quote
but would it be possible to have a program that scans all the address ranges to try to find r/w memory? What happens if you read or write to an invalid location? Do you get an exception?
I answered this question for you before. http://chdk.setepontos.com/index.php/topic,3360.msg30999.html#msg30999

If you aren't willing to do the basic work required to answer your question, there's little point in speculating.
Don't forget what the H stands for.

*

Offline RaduP

  • *****
  • 908
Re: I/O Buffer question
« Reply #6 on: 06 / April / 2009, 19:49:30 »
Which specific camera do you have in mind ? Most recent (digic-II and later) canon P&S cameras will shoot continuously at the same rate until the card is full. Some do use multiple raw buffers, but the reason for this isn't clear. Perhaps it lets them do shooting and jpeg encoding in parallel.
I was thinking about Canon S3, in some reviews I was reading people claimed a burt mode of about 30 pictures.

Quote
The existence of additional memory is unwarranted speculation on your part. My guess is that it wouldn't be additional memory, it would be multiple locations for of the normal buffers (like the dynamic raw buffer). If this was not the case, a copy would be required, which would be counter productive. Thus it would be in the same 32-128 meg region of memory already known. There may well be more memory we can use, but finding it will require actual reverse engineering, not guessing.
Are you serious? Or are you just trying to flame me?
There is a VERY good reason why there might be extra memory, and the reason is that memory chips usually come in sizes that are powers of 2. If you take the jpeg buffer and the raw buffer, which are the biggest buffers, you will see that there is quite a bit of memory left, especially for cameras with less than 8MP.

I'd suggest that speculation is pointless until you fix that.
Your suggestion is wrong.

Quote
I answered this question for you before. http://chdk.setepontos.com/index.php/topic,3360.msg30999.html#msg30999

If you aren't willing to do the basic work required to answer your question, there's little point in speculating.

No, you did not answer me that at all. You just told me your way of doing it.

*

Offline reyalp

  • ******
  • 13448
Re: I/O Buffer question
« Reply #7 on: 06 / April / 2009, 20:07:34 »
There is a VERY good reason why there might be extra memory, and the reason is that memory chips usually come in sizes that are powers of 2. If you take the jpeg buffer and the raw buffer, which are the biggest buffers, you will see that there is quite a bit of memory left, especially for cameras with less than 8MP.
Huh ? We already know the size of the various memory regions (including the one that corresponds to main memory), and they are significantly larger than the portions whose purpose we know. Figuring out what parts we can re-use requires actually digging in to the disassembly.
Quote
No, you did not answer me that at all. You just told me your way of doing it.
If you are unwilling to take the short amount of time required to get the full answer, it's extremely unlikely you will expend the much greater effort to do something useful with it. Of course, if you'd actually read the link I'd provided, you would have seen detailed information about the memory layout and protection regions of several cameras.
Don't forget what the H stands for.


*

Offline RaduP

  • *****
  • 908
Re: I/O Buffer question
« Reply #8 on: 06 / April / 2009, 20:22:37 »
Huh ? We already know the size of the various memory regions (including the one that corresponds to main memory), and they are significantly larger than the portions whose purpose we know. Figuring out what parts we can re-use requires actually digging in to the disassembly.

Ok, so my original assumption was correct, there is more memory out there than the one taken by the main buffers. Some memory might not even be used, but it would not make sense not to use at least parts of it for i/o buffers To find out if it is used or not, I guess it can all be filled with data at startup, then read later on, after using the camera for a while, and see what memory didn't change. Of course, you can also go manually through 3MB worth of binary dumps and find out which memory is free.

Quote
If you are unwilling to take the short amount of time required to get the full answer, it's extremely unlikely you will expend the much greater effort to do something useful with it. Of course, if you'd actually read the link I'd provided, you would have seen detailed information about the memory layout and protection regions of several cameras.

The reason why I don't like your idea is because you can't know for sure if the camera does some fancy things later on and allocates/changes more of the memory. The code you referenced in that thread is right near the beginning of the dump, but did you look through all the code to see if the camera talks again with the MPU? It is also possible that the camera has other memory areas that may or may not be allocable via the MPU, and it could use some DIGI specific ways, or maybe some I/O, or who knows what.
« Last Edit: 06 / April / 2009, 20:24:30 by RaduP »

*

Offline ewavr

  • ****
  • 1057
  • A710IS
Re: I/O Buffer question
« Reply #9 on: 06 / April / 2009, 20:23:03 »
@ RaduP:

S3IS, sub_FFB13854:

JPEG BUFF: address 10D26E00 (physical address is D26E00), size 12D9200 (19.7 Mb!). So, D26E00+12D9200=2000000=end of RAM.
RAW buff: address 105B8AC0 (physical 5B8AC0), size 75A8F0, end address 5B8AC0+75A8F0=D133B0 (only 86k between RAW and JPEG buffers).
Below RAW buffer (6M)?
1M for variables, heap and CHDK.
0.5-1M for live image buffers, bitmap.
How about space for fonts, FAT copies, I/O buffers?
No more free space. You can make several shoots it continuous mode, them make memory dump and try to find free space.

 

Related Topics