experimental alternate memory allocation - page 9 - General Discussion and Assistance - CHDK Forum  
« previous next »
Pages: 1 ... 4 5 6 7 8 [9] 10 11 12 13   Go Down

experimental alternate memory allocation

  • 128 Replies
  • 49494 Views
*

Offline reyalp

  • ******
  • 14082
Re: experimental alternate memory allocation
« Reply #80 on: 06 / March / 2011, 14:23:14 »
Advertisements
Quote from: f_m_b on 06 / March / 2011, 12:55:39
Hello everybody

this project looks very interesting to me.
I have a SX20 wich are not pleased with lot of memory.

So, if i had understood well this discussion, we are looking for free unused memory regions on the cam and use this instead of the origin heap.
How could it be, that the cam has free memory regions and canon doesn't use it? ???
The heap that malloc comes from is only a small amount of RAM (~1MB out of 64-256MB)

The camera has another memory allocation system for large pieces of memory (called exmem in the canon firmware)

How to find the functions is described in this thread. You need a disassembler to find the entry points, IDA is best, but you can also use gpl tools http://chdk.wikia.com/wiki/GPL_Tools

Logged
Don't forget what the H stands for.
*

Offline f_m_b

  • **
  • 71
Re: experimental alternate memory allocation
« Reply #81 on: 07 / March / 2011, 09:39:00 »
Thank you reyalp.

I found out the address for SX20 1.02b and wrote it into stubs_entry_2.S:
NHSTUB(exmem_alloc, 0xFF889D7C)

Do we need the address for Exmem.FreeCachable too? (0xFF889DF4)

I've activated OPT_EXTMEM_MALLOC and all has compiled well without errors.
After booting the cam shows me 1652960 Bytes of free memory.  :D

So far so good. If i use the cam there are some features which causes a crash. I think, this was expected.

I run the script from reyalp and don't know what i have to do next.

If i've understood well, we have to write into this ext-memory a sort of test pattern DEADBEEF like philmoz described and find out which regions are used from the cam itself? ???

« Last Edit: 07 / March / 2011, 09:43:11 by f_m_b »
Logged
Greetings Frank
SX20 (1.02b)
*

Offline f_m_b

  • **
  • 71
Re: experimental alternate memory allocation
« Reply #82 on: 09 / March / 2011, 16:44:53 »
Ok. Next step.

Finding out the maximal memory amount:
  8 MB ok
16 MB boot crash
12 MB ok
14 MB ok
15 MB boot crash

EXMEM_RAMDISK        [0x431FFFC0] , [0x00E00040]
The start address of EXMEM_RAMDISK is beetween 0x431FFFC0 and 0x43DFFFC0.

I've allocated the memory and wrote the test pattern on it. ok.

Next step is to find out which regions are overwritten by the cam itself.

@philmoz
i have a question to you: why you are using in gui.c first address 0x03FFFFFFC and not the address which comes from EXMEM_RAMDISK? If i try your gui.c code the cam crashes.
So i tried with start address 0x043DFFFC0 and end address 0x044000000. -> cam crashes  too.
The same happens if i look with the memory browser on this memory address. I'am confused ???
Some ideas?



Logged
Greetings Frank
SX20 (1.02b)
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: experimental alternate memory allocation
« Reply #83 on: 01 / April / 2011, 08:38:45 »
Quote from: f_m_b on 09 / March / 2011, 16:44:53
Ok. Next step.

Finding out the maximal memory amount:
  8 MB ok
16 MB boot crash
12 MB ok
14 MB ok
15 MB boot crash

EXMEM_RAMDISK        [0x431FFFC0] , [0x00E00040]
The start address of EXMEM_RAMDISK is beetween 0x431FFFC0 and 0x43DFFFC0.

I've allocated the memory and wrote the test pattern on it. ok.

Next step is to find out which regions are overwritten by the cam itself.

@philmoz
i have a question to you: why you are using in gui.c first address 0x03FFFFFFC and not the address which comes from EXMEM_RAMDISK? If i try your gui.c code the cam crashes.
So i tried with start address 0x043DFFFC0 and end address 0x044000000. -> cam crashes  too.
The same happens if i look with the memory browser on this memory address. I'am confused ???
Some ideas?


Sorry I'm having trouble understanding what you are asking here.
The code I posted previously was what I used to discover where the video buffer addresses were located on the SX30 so I could avoid them when using exmem. This code won't work without modification on the SX20 because it only has 64MB of RAM (the SX30 has 128MB).

If you look at the current code for exmem for the G12 and SX30 you will see I have used some #defined in camera.h to determine how much of the buffer returned from exmem needs to be skipped to avoid overwriting the movie buffers (EXMEM_HEAP_SKIP). The desired amount of exmem memory to allocate is also set here (EXMEM_HEAP_SKIP).

From looking at the SX20 firmware I would guess the value for EXMEM_HEAP_SKIP would be (0x04000000-0x03D08A00) - from sub_FF88A064.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: experimental alternate memory allocation
« Reply #84 on: 01 / April / 2011, 08:46:56 »
In the current version of exmem_malloc_init the call to suba_init is passing a value of 1024 as the last parameter. This parameter sets the minimum cell size to be used in the suba allocator - this means any malloc request smaller than 1K will always allocate 1K.

This is wasting a lot of memory on startup because some of the code is allocating a large number of small chunks. For example the code to load the symbol font file allocates a 30 byte block of memory for each of the 128 symbols in the file. With the min cell size set to 1024 this ends up allocating 128K instead of ~4K it should need.

On my G12 the amount of free memory (using exmem) after startup was 1.5MB even though I allocated 2MB from exmem. Changing the min cell size value to 8 reduces the overhead substantially - I now have 1.98MB of the 2MB still free on startup.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: experimental alternate memory allocation
« Reply #85 on: 01 / April / 2011, 09:41:41 »
My current dev build - G12 using exmem.
- 4MB memory allocation.
- CHDK loaded into exmem and running from there. :D

So far seems stable, although more testing is needed (lots of features to try out).
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14082
Re: experimental alternate memory allocation
« Reply #86 on: 01 / April / 2011, 23:13:44 »
Quote from: philmoz on 01 / April / 2011, 08:46:56
In the current version of exmem_malloc_init the call to suba_init is passing a value of 1024 as the last parameter. This parameter sets the minimum cell size to be used in the suba allocator - this means any malloc request smaller than 1K will always allocate 1K.
Good catch.
Quote
This is wasting a lot of memory on startup because some of the code is allocating a large number of small chunks. For example the code to load the symbol font file allocates a 30 byte block of memory for each of the 128 symbols in the file. With the min cell size set to 1024 this ends up allocating 128K instead of ~4K it should need.
On a somewhat unrelated note, it would be good to avoid using lots of tiny allocations like this in general.
Logged
Don't forget what the H stands for.
*

Offline bdasmith

  • ***
  • 117
Re: experimental alternate memory allocation
« Reply #87 on: 02 / April / 2011, 17:38:25 »
Quote from: f_m_b on 06 / March / 2011, 12:55:39
Hello everybody

this project looks very interesting to me.
I have a SX20 wich are not pleased with lot of memory.

So, if i had understood well this discussion, we are looking for free unused memory regions on the cam and use this instead of the origin heap.
How could it be, that the cam has free memory regions and canon doesn't use it? ???

I have some skills with c, but my asm knowledge lies long years behind me ;-).
Can you give me some hints to find those memory regions? What tools you are using?

Thanks a lot
Frank



I hope as well - that it can be found to use this feature with the sx20 - cause well, you know, any improvements can be a good thing ;)

Not that its a too big a deal i guess, because my sx20 is working wonderfully with CHDK right now as it is.
Love this software!
Logged
sx20is 1.02d
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: experimental alternate memory allocation
« Reply #88 on: 03 / April / 2011, 01:01:23 »
The core CHDK code now include some changes I made for EXMEM (changeset 1124), including the code to build and load CHDK itself into the EXMEM memory space.

I thought I'd share some of what I've learned about using EXMEM on the G12 & SX30 and some instructions on how to go about using it on other recent cameras. This is complex and not for the faint hearted - you've been warned ;)

First here's a simplified memory map for recent dryos cameras like the G12 & SX30 (not to scale). The numbers are in HEX, the question marks are values that change from camera to camera:
Code: [Select]
Without CHDK
============
       variables      heap/stack       buffers (e.g. RAW)    exmem pool           vid buffers
|----|-----------|------------------|----------------------|---------------------|-----------|
0  1900          ?               4000000                   ?                     ?       MAXRAMADDR

With CHDK (default memory)
==========================
       variables  CHDK  heap/stack     buffers (e.g. RAW)    exmem pool           vid buffers
|----|-----------|----|-------------|----------------------|---------------------|-----------|
0  1900          ?    ?          4000000                   ?                     ?       MAXRAMADDR

With CHDK (exmem memory)
==========================
       variables      heap/stack       buffers (e.g. RAW)    exmem pool    CHDK   vid buffers
|----|-----------|------------------|----------------------|-------------|-------|-----------|
0  1900          ?               4000000                   ?             ?       ?       MAXRAMADDR

Moving CHDK into the EXMEM area means that the Canon heap is left at its original size, while still making more memory available for CHDK - everyone wins!

In order to get this working on a camera there are a few steps involved:
1. Find MAXRAMADDR and add to PLATFORM/PLATFORMSUB/makefile.inc if not already there.
2. Find the exmem_alloc routine in the firmware, and enable EXMEM in the build (OPT_EXMEM_MALLOC).
3. Define a value for EXMEM_HEAP_SIZE.
4. Enable OPT_EXMEM_TESTING to determine if the allocated memory is safe to use.
5. If necessary, determine the value for EXMEM_HEAP_SKIP.
6. Turn off EXMEM testing and let CHDK use the EXMEM memory.
7. Enable CHDK to load into EXMEM.

Step 1.
MAXRAMADDR defines the upper limit of the cameras RAM. If it has not been set for your camera you need to determine what the value is and add it to the PLATFORM/PLATFORMSUB/makefile.inc file. For example cameras with 128MB of RAM will have MAXRAMADDR=0x07FFFFFF, cameras with 64MB of RAM will have MAXRAMADDR=0x03FFFFFF.
Search in the firmware for a function that starts like the below (the addresses and values will be different):
Code: [Select]
ROM:FF890270                 EXPORT _sub_FF890270__ExMemMan.c__0
ROM:FF890270 _sub_FF890270__ExMemMan.c__0            ; CODE XREF: sub_FF88FD1C+AC
ROM:FF890270                 STMFD   SP!, {R4-R8,LR}
ROM:FF890274                 LDR     R6, =0x3BFC8
ROM:FF890278                 MOV     R4, R0
ROM:FF89027C                 MOV     R0, #0
ROM:FF890280                 MOV     R5, #0x48000000               <<<<<<<<<<<<<<< want this value
ROM:FF890284
ROM:FF890284 loc_FF890284                            ; CODE XREF: _sub_FF890270__ExMemMan.c__0+38
ROM:FF890284                 LDR     R1, [R6,R0,LSL#3]
We need the value marked in the code above and it will probably be either 0x48000000 (128MB) or 0x44000000 (64MB). The value will have the 'uncached' memory bit set (0x40000000), subtract that and the result is the amount of RAM on the camera. Set the MAXRAMADDR value in makefile.inc to be one less than this value. In the example above MAXRAMADDR will be 0x7ffffff (0x8000000 - 1).

Step 2.
Is covered in reyalp's first post.

Step 3.
The default value for EXMEM_HEAP_SIZE in the code is 2MB if you don't define a value.
You should add a 4MB value into PLATFORM/PLATFORMSUB/makefile.inc.
Starting with 4MB is needed in case the camera has video buffers that need to be skipped (see later).
PLATFORM/PLATFORMSUB/makefile.inc:
Code: [Select]
EXMEM_BUFFER_SIZE=0x400000 # Amount of EXMEM memory to allocate for CHDK = 4MB

Step 4.
Enable OPT_EXMEM_TESTING in buildconf.inc. Rebuild CHDK and load it onto your camera. This enables exmem testing, which allocates the memory; but does not use it. instead it checks the memory for corruption.
You should now have 'OK" displayed at the bottom left of the LCD. Test the camera & CHDK functions as much as possible. If the OK disappears and is replaced by a bunch of numbers then this means that the EXMEM memory has been corrupted by some camera function (most likely recording a video and using the video buffers shown in the memory map above).
The info displayed on two lines:
Code: [Select]
s:XXXXXXXX e:YYYYYYYY
f:AAAAAAAA l:BBBBBBBB c:NN
XXXXXXXX = start address of the EXMEM allocated memory buffer
YYYYYYYY = end address of the EXMEM allocated memory buffer
AAAAAAAA = first corrupt memory location
BBBBBBBB = last corrupt memory location
NN = number of corrupt memory locations found

If the first corrupt location is at the start of the EXMEM allocated memory buffer then reduce the size of EXMEM_HEAP_SIZE to 2MB and try again. If that still fails then you probably won't be able to use EXMEM.
If the first corrupt location is not at the start of the memory then it is most likely a problem with the video buffers, use step 5 below to try and fix this.

If everything works and no corruption occurs you should be good to go to step 6.

Continued in next post.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: experimental alternate memory allocation
« Reply #89 on: 03 / April / 2011, 01:08:41 »
Continued from previous post.

Step 5.
In testing this on the G12 and SX30 it became apparent that the camera was using some of the EXMEM allocated memory. It was found that this memory was for buffers used during video recording and playback (possibly for H.264 encoding/decoding). The exmem_alloc function was not correctly locking this memory out of being allocated to CHDK. At this stage a proper fix for this has not been found and the following is a work around to the problem. These video buffers are approx 3MB in size.
If you performed step 1 you will have already found the place where the video buffer address can be located, if not use the instructions in step 1 to find the function that tells us how much RAM the camera has. A bit later on in this function is the video buffer address:
Code: [Select]
ROM:FF890270                 EXPORT _sub_FF890270__ExMemMan.c__0
ROM:FF890270 _sub_FF890270__ExMemMan.c__0            ; CODE XREF: sub_FF88FD1C+AC
ROM:FF890270                 STMFD   SP!, {R4-R8,LR}
ROM:FF890274                 LDR     R6, =0x3BFC8
ROM:FF890278                 MOV     R4, R0
ROM:FF89027C                 MOV     R0, #0
ROM:FF890280                 MOV     R5, #0x48000000
ROM:FF890284
ROM:FF890284 loc_FF890284                            ; CODE XREF: _sub_FF890270__ExMemMan.c__0+38
ROM:FF890284                 LDR     R1, [R6,R0,LSL#3]
                    ...
ROM:FF8902AC                 CMP     R4, #0xB
ROM:FF8902B0                 CMPNE   R4, #0xC
ROM:FF8902B4                 CMPNE   R4, #0xD
ROM:FF8902B8                 BNE     loc_FF8902C4
ROM:FF8902BC                 CMP     R5, #0x48000000
ROM:FF8902C0                 LDREQ   R5, =0x47D08A00 <<<<<<<<<<<<<<<<<<<< need this value for video buffer address
ROM:FF8902C4
ROM:FF8902C4 loc_FF8902C4                            ; CODE XREF: _sub_FF890270__ExMemMan.c__0+48
ROM:FF8902C4                 LDR     R7, =dword_FFBBB370

On the G12 & SX30 the line marked above tells us the video buffer address, again this has the uncached memory bit set so remove that (in this case the value we want is 0x07d08a00 after removing the uncached bit).
Subtract this value from the memory size (0x08000000 in this case) to get the size of the video buffer memory (the camera uses everything from this address to the top of RAM). Add this size value as EXMEM_HEAP_SKIP to PLATFORM/PLATFORMSUB/makefile.inc. In this case we would add 'EXMEM_HEAP_SKIP=0x2F7600' since 0x08000000 - 0x07d08a00 = 0x2f7600.

Rebuild and try the exmem testing again (step 4).

Step 6.
At this stage you should have a working exmem buffer that does not get corrupted and hopefully at least 2MB in size.
Disable the OPT_EXMEM_TESTING option and rebuild. Load this onto the camera and you should now have CHDK running with a lot more free memory (Miscellaneous stuff --> Show memory info). The load address will still be in the low memory heap area.
Thoroughly test the camera and CHDK functionality. If all is well go onto step 7, otherwise adjust the EXMEM_HEAP_SIZE value and try again.

Step 7.
The final step is to move CHDK itself up into EXMEM.
To do this the value of MEMISOSTART in PLATFORM/PLATFORMSUB/makefile.inc needs to be changed.
The formula for MEMISOSTART when loading CHDK into high (exmem) memory is:
       MEMISOSTART = MAXRAMADDR+1 - EXMEM_HEAP_SKIP - EXMEM_BUFFER_SIZE - 32
You have to calculate the actual value and insert it here, you can't use a formula (at least I haven't been able to).
From my examples above the value would be:
       0x7ffffff + 1 - 0x2f7600 - 0x400000 - 32 = 0x79089e0
(Now you know why the other values are in this file, so you can see them to do the calculation)
The extra -32 at the end is because exmem_alloc allocates 64 bytes extra, with a 32 byte buffer before and after the block returned.
This change to MEMISOSTART should be made conditional so that you can disable all this and easily revert back to the default non-exmem version as shown below (make sure to use your original MEMISOSTART value in the else part, not the value shown here):
Code: [Select]
ifdef OPT_CHDK_IN_EXMEM
MEMISOSTART=0x79089e0 # MAXRAMADDR+1 - EXMEM_HEAP_SKIP - EXMEM_BUFFER_SIZE - 32 (exmem allocates 64 bytes extra, 32 before and 32 after block allocated)
else
MEMISOSTART=0x16E820 # original non-exmem value
endif

A change is also required in the 'boot.c' code for the camera firmware version. This change is to set the built-in heap start address back to the default value when loading CHDK into exmem memory. Otherwise the CHDK startup code will try and put the Canon heap into exmem which will probably crash. Find the code in boot.c that sets the heap start address and change it as shown below (again make sure to use the correct address for your camera and firmware version).

boot.c:
Code: [Select]
#if defined(OPT_CHDK_IN_EXMEM)
               "LDR     R0, =0x16E820\n" // use original heap offset since CHDK is loaded in high memory
#else
               "LDR     R0, =new_sa\n"   // otherwise use patched value
               "LDR     R0, [R0]\n"      //
#endif

This code above also shows the final piece - the OPT_CHDK_IN_EXMEM option needs to be enabled in buildconf.inc. Buildconf.inc should now look like this with OPT_EXMEM_ALLOC and OPT_CHDK_IN_EXMEM both enabled:
Code: [Select]
# experimental - allocate memory from alternate heap. Default 2MB from EXMEM_RAMDISK
# See exmem_malloc_init in generic/wrappers.c
OPT_EXMEM_MALLOC=1
# Set OPT_CHDK_IN_EXMEM to build and load CHDK into EXMEM memory
# Need to define MEMISOSTART and EXMEM values in PLATFORM/PLATFORMSUB/makefile.inc (see G12 or SX30 for example)
# (if setting this must also set OPT_EXMEM_MALLOC !)
OPT_CHDK_IN_EXMEM=1
# set this to enable exmem corruption testing (see wrappers.c). (Don't set OPT_CHDK_IN_EXMEM with this !)
#OPT_EXMEM_TESTING=1

Rebuild and load onto your camera. Show memory info should now show a load address up in the high exmem area plus plenty of free memory.

Enjoy :)

Any suggestions on improving these instructions welcome.

Phil.
« Last Edit: 10 / April / 2011, 20:53:47 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
Pages: 1 ... 4 5 6 7 8 [9] 10 11 12 13   Go Up
« previous next »
 

Related Topics