experimental alternate memory allocation

  • 128 Replies
  • 14808 Views
*

Offline reyalp

  • ******
  • 9810
  • Publish
    Re: experimental alternate memory allocation
    « Reply #30 on: 03 / January / 2011, 17:58:40 »
    Advertisements
    A thought - if we can safely use all this 'exmem' memory why not load CHDK itself into this address space?
    That way there would be more memory available to add features and the default camera heap would be left untouched.

    Regards,
    Phil.

    CHDK isn't relocatable, so the address would have to be known in advance (or the minary made relocatable... a relocatable binary format is something we really need anyway, so we can have loadable modules). Also, CHDK loads before the OS, so the exmem API wouldn't really be available. You might be able to put the code there anyway, but some experimentation would be needed.

    Might be worth looking into, if we figure out how to deal with the movie record problem.
    Don't forget what the H stands for.

    *

    Offline philmoz

    • *****
    • 2936
      • Photos
  • Publish
    Re: experimental alternate memory allocation
    « Reply #31 on: 04 / January / 2011, 04:20:00 »
    Looks like this may not be viable on the G12 (1.00c) at least.

    This is what I've tried:
    - disabled OPT_EXMEM_MALLOC
    - in the boot code (boot.c) I set all memory from 0x00400000 to 0x08000000 to 0xDEADBEEF.
    - in gui_draw_debug_vals_osd (when in Alt mode) scanned all memory in the above range for the largest block that still contained all of the test value 0xDEADBEEF, and displayed the start, end and size.

    After initial power on in playback mode the largest block is almost 19MB; but includes both RAW buffers.
    Power up in record mode the largest block is almost 24MB; but includes the RAW buffers and video buffers (found at the upper end of the memory).

    After using the camera functions I found that the largest untouched block quickly drops to only 930,240 bytes in size (located at 0x00ebca50).

    Unless my testing method is wrong (which I would be happy if it were), it looks like most of the available memory is used at some time or other during normal camera operations.

    Edit: just realised that the above size is in words not bytes, so the actual untouched block is 3,720,960 bytes long (3.5MB). This could still be an improvement; but I have no idea how to reliably find this buffer at startup.

    SX30 is similar, largest untouched block is 995,040 words @ 0x00e91fa0 (3.8MB).

    Phil.

    Code below in case anyone wants to experiment.

    boot.c (boot function), add below just before final branch
    Code: [Select]

    "LDR R3, =0x00400000 \n" // buffer start
    "LDR R1, =0x08000000 \n" // buffer end
    "LDR R2, =0xDEADBEEF \n" // guard value
    "fill_guard: \n"
                     "CMP     R3, R1\n"
                     "STRCC   R2, [R3],#4\n"
                     "BCC     fill_guard\n"

    in gui.c at start of gui_draw_debug_vals_osd
    Code: [Select]
    if (gui_mode==GUI_MODE_ALT) {
    unsigned long* p = (unsigned long*)0x00400000;
    unsigned long *first = 0, *last = 0;
    unsigned long *lstart = 0, *lend = 0;
    long len = 0;
    long cnt = 0;
    int state = 0;
    while (p < (unsigned long*)0x08000000)
    {
    if (*p != 0xDEADBEEF)
    {
    if (state == 1)
    {
    if (cnt > len)
    {
    lstart = first;
    lend = last;
    len = cnt;
    }
    state = 0;
    cnt = 0;
    }
    }
    else
    {
    if (state == 1)
    {
    last = p;
    cnt++;
    }
    else
    {
    first = last = p;
    cnt = 1;
    state = 1;
    }
    }
    p++;
    }
    if (cnt > len)
    {
    lstart = first;
    lend = last;
    cnt = len;
    }
    sprintf(osd_buf, "f:%8x l:%8x c:%d", lstart, lend, len);
    draw_txt_string(2, 10, osd_buf, conf.osd_color);
    }

    « Last Edit: 04 / January / 2011, 04:40:33 by philmoz »
    CHDK ports:
      sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
      g12 (1.00c, 1.00e, 1.00f & 1.00g)
      sx130is (1.01d & 1.01f)
      ixus310hs (1.00a & 1.01a)
      sx40hs (1.00d, 1.00g & 1.00i)
      g1x (1.00e, 1.00f & 1.00g)

  • Publish
    Re: experimental alternate memory allocation
    « Reply #32 on: 04 / January / 2011, 06:42:52 »
    Code: [Select]
    I test now in IX1000

    8 megabyte is what i test and work.16 megabyte and more crash on shoot, (i use this new memory not for chdk).so seem the camera report then too low mem and exit.

    when i records a 19280*1028 video, there are always 1,7 megabyte at front overwritten.when show a video there are 900 kb overwritten.this are the largest overwritten space i get.

    when record video at 1280*720 or show, overwritten memspace is smaller.same happen on highspeed mode.

    nightshoot mode overwrite no memory.also the other shoot modes that take several pictures and calc do not overwrite mem and work with 8 megabyte.

    so in theory can say, when alloc 8 megabyte and set the mem start to + 4 megabyte it should work and we have 4 megabyte for chdk and 2.3 megabyte security gap on beginning.

    I see address of memfree are same as on SX30 when alloc 4 megabyte.
    but for 8 megabyte, i get this addresses 77ffffe0 7fffffe0

    I do more test with that setting 8 megabyte and 2 megabyte gap on start for test worst case, and see if on more use the membounds get overwritten.if that after longer testing not happen, then this seem work stable and even more when the gap is set to 4 megabyte.

    I have a little enhance the testcode by use global var so i need no absolute addresses, to make more easy to test several sizes.



    [code=c]#ifdef OPT_EXMEM_MALLOC
           void *mem_exmem_heap;
      // I set this up to 16 mb and it still booted...
            // on IX1000 1,7 meg max get overwritten on start and 8 megabyte of mem alloc work.
      #define EXMEM_HEAP_SIZE (1024*1024*8)
      // these aren't currently needed elsewhere
      /*
      void * exmem_alloc(unsigned pool_id, unsigned size)
      {
             return _exmem_alloc(pool_id,size,0);
      }

      void exmem_free(unsigned pool_id)
      {
             _exmem_free(pool_id);
      }
      */

      static void *exmem_heap;

      void *suba_init(void *heap, unsigned size, unsigned rst, unsigned mincell);
      void *suba_alloc(void *heap, unsigned size, unsigned zero);
      int suba_free(void *heap, void *p);

      void exmem_malloc_init() {
             // pool zero is EXMEM_RAMDISK on d10
             void *mem = _exmem_alloc(0,EXMEM_HEAP_SIZE,0);
             mem_exmem_heap = mem;



    Code: (c) [Select]
    extern void *mem_exmem_heap;
    void gui_draw_debug_vals_osd() {
    #ifdef OPT_DEBUGGING

        // check exmem allocated memory for corruption (assumes 8MB allocated)
    unsigned long* p = mem_exmem_heap;//(unsigned long*)0x07BFFFE0;
    // strcpy(osd_buf,"OK");
    sprintf(osd_buf,"Start %8x End%8x ok",p,mem_exmem_heap +(1024*1024*8));
    unsigned long *f = 0;
    long cnt = 0;
    while (p <= (unsigned long*)( mem_exmem_heap+ (1024*1024*8)))
    {
    if (p[0] != 0xDEADBEEF)
    {
    if (f == 0) f = p;
    cnt++;
    }
    p++;
    }
    if (cnt != 0)sprintf(osd_buf, "%8x %8x %8x", &f[0], f[0], cnt);


    draw_txt_string(2, 10, osd_buf, conf.osd_color);
    // end of check
    [/code]
     
    « Last Edit: 30 / January / 2011, 07:45:11 by Bernd R »
    Ixus 1000 HS

  • Publish
    Re: experimental alternate memory allocation
    « Reply #33 on: 04 / January / 2011, 11:58:57 »
    I also find the exmem allocation funktion on my a610 100e (VxWorks) at FFC02830.
    Free Memory: 12135264 bytes :)
    It works fine up to 12MB. If I want more ram, the camera hang up when i take a foto.

    CHDKLover
    « Last Edit: 04 / January / 2011, 14:01:00 by CHDKLover »


    *

    Offline reyalp

    • ******
    • 9810
  • Publish
    Re: experimental alternate memory allocation
    « Reply #34 on: 04 / January / 2011, 12:25:31 »
    Unless my testing method is wrong (which I would be happy if it were), it looks like most of the available memory is used at some time or other during normal camera operations.
    In general, I'd expect the canon firmware to use most of the memory. However, just because it's touched doesn't mean that it's all in use simultaneously, or that the camera would necessarily need it all if you allocated some of it.

    Another idea:
    It might be possible to change where the firmware thinks exmem starts/ends, as we do with the malloc heap.

    background:
    CHDK loads where the normal heap would start, the new_sa bit in boot.c modifies that, effectively taking away from malloc memory without ever using malloc. Note that the camera memShow() eventproc does *not* reflect the the CHDK change, because the heap start is presumably a compile time constant in the canon code. It just so happens that we can change it one place and have it work. Depending on how exmem is setup, this may not be possible.
    Don't forget what the H stands for.

    *

    Offline philmoz

    • *****
    • 2936
      • Photos
  • Publish
    Re: experimental alternate memory allocation
    « Reply #35 on: 04 / January / 2011, 16:42:27 »
    Unless my testing method is wrong (which I would be happy if it were), it looks like most of the available memory is used at some time or other during normal camera operations.
    In general, I'd expect the canon firmware to use most of the memory. However, just because it's touched doesn't mean that it's all in use simultaneously, or that the camera would necessarily need it all if you allocated some of it.

    Another idea:
    It might be possible to change where the firmware thinks exmem starts/ends, as we do with the malloc heap.

    background:
    CHDK loads where the normal heap would start, the new_sa bit in boot.c modifies that, effectively taking away from malloc memory without ever using malloc. Note that the camera memShow() eventproc does *not* reflect the the CHDK change, because the heap start is presumably a compile time constant in the canon code. It just so happens that we can change it one place and have it work. Depending on how exmem is setup, this may not be possible.

    Thanks for that, hopefully you're right and my test is not valid.

    I'm concerned that there are other hard wired buffer addresses in the firmware - I was able to find both of the movie buffer addresses mentioned previously in the firmware, referenced from the H264 code. These look to be like the RAW buffers and are fixed values in the firmware code, not allocated with a memory manager.

    How many more are there like this?
    If there is a memory manager why does it not exclude these locations?
    Are we calling exmem_alloc too early in the boot process and it hasn't initialised properly yet?

    What led me to this latest test was I was using my previously posted test to see if there were any other areas like the movie buffers that got modified after we called exmem_alloc.

    On the G12 the second raw buffer starts at 0x06000000 and goes to 0x06ee9200.
    I tried using exmem_alloc to grab 16MB (0x07000000 - 0x08000000), fill it with 0xDEADBEEF and then check in gui.c for any altered locations.
    What I found when I did this was that the camera crashed as soon as I pressed the shutter button. I assumed that there was some other memory after the raw buffer that I was corrupting which was when I thought to try initialising and scanning all the memory to find unused areas.

    After reading your post it occurred to me that if the camera is using a memory manager then the camera may have crashed because I had allocated too much memory and it couldn't get enough when I pressed the shutter. I'll do some more testing to see if this is the case.

    Don't you love a challenge like this  :D

    Regards,
    Phil.

    CHDK ports:
      sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
      g12 (1.00c, 1.00e, 1.00f & 1.00g)
      sx130is (1.01d & 1.01f)
      ixus310hs (1.00a & 1.01a)
      sx40hs (1.00d, 1.00g & 1.00i)
      g1x (1.00e, 1.00f & 1.00g)

    *

    Offline fudgey

    • *****
    • 1705
    • a570is
  • Publish
    Re: experimental alternate memory allocation
    « Reply #36 on: 04 / January / 2011, 17:12:25 »
    After reading your post it occurred to me that if the camera is using a memory manager then the camera may have crashed because I had allocated too much memory and it couldn't get enough when I pressed the shutter. I'll do some more testing to see if this is the case.

    Did you take a look at the romlog to try and find out where it crashed?

    *

    Offline philmoz

    • *****
    • 2936
      • Photos
  • Publish
    Re: experimental alternate memory allocation
    « Reply #37 on: 05 / January / 2011, 03:31:57 »
    Some positive progress on the G12 :D

    The crash was being caused by allocating too much memory in exmem_malloc_init.
    Reducing it back to 4MB fixes that.

    Also it looks like the movie buffer issue may be startup timing related.
    If I move the call to exmem_malloc_init to just before the main loop in core_spytask then when starting up in playback mode the end address of the block returned is just below the movie buffer address found previously. So at some point in the initialisation it is reserving these movie buffers so they get excluded by exmem_alloc.

    Since nothing is ever easy moving the call to exmem_malloc_init doesn't make any difference when starting in record mode - the block returned still overlaps the movie buffers.

    So now the problem is what extra initialisation is needed, and where can the call to exmem_malloc_init be done that works all the time?

    A suggestion for the exmem version of free. If the code is changed to:
    Code: [Select]
    void free(void *p) {
    if(exmem_heap && (p >= exmem_heap))
    suba_free(exmem_heap,p);
    else
    _free(p);
    }
    then if a _malloc allocated address is passed in it will use _free instead of suba_free.

    Phil.


    CHDK ports:
      sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
      g12 (1.00c, 1.00e, 1.00f & 1.00g)
      sx130is (1.01d & 1.01f)
      ixus310hs (1.00a & 1.01a)
      sx40hs (1.00d, 1.00g & 1.00i)
      g1x (1.00e, 1.00f & 1.00g)


  • Publish
    Re: experimental alternate memory allocation
    « Reply #38 on: 05 / January / 2011, 10:39:15 »
    @reyalp
    >In general, I'd expect the canon firmware to use most of the memory.

    maybe on new camera not.its maybe more easy, cheaper and need less power, to add a 64 megabyte mem chip instead add a 32 mb+ 4 mb chip, if you need only 35 MB.

    @philmoz
    >So at some point in the initialisation it is reserving these movie buffers so they get excluded by exmem_alloc.

    I see that on IX1000 too and i think when start the exmem alloc at end of chdk init, then its later as the Firmware alloc the buffer for the video play.

    but it does not help, because when the display on boot does not show a movie, then there is no mem alloc.

    maybe there is a function on top of exmem alloc that handle more mem entries
    « Last Edit: 05 / January / 2011, 10:44:27 by Bernd R »
    Ixus 1000 HS

    *

    Offline reyalp

    • ******
    • 9810
  • Publish
    Re: experimental alternate memory allocation
    « Reply #39 on: 05 / January / 2011, 12:19:51 »
    So now the problem is what extra initialisation is needed, and where can the call to exmem_malloc_init be done that works all the time?
    Your mission, should you choose to accept it is... ;)
    Quote
    A suggestion for the exmem version of free. If the code is changed to:
    Code: [Select]
    void free(void *p) {
    if(exmem_heap && (p >= exmem_heap))
    suba_free(exmem_heap,p);
    else
    _free(p);
    }
    then if a _malloc allocated address is passed in it will use _free instead of suba_free.
    Oh good idea, malloc heap should always be in low memory.

    @reyalp
    >In general, I'd expect the canon firmware to use most of the memory.

    maybe on new camera not.its maybe more easy, cheaper and need less power, to add a 64 megabyte mem chip instead add a 32 mb+ 4 mb chip, if you need only 35 MB.
    I think it's unlikely. Margins are very thin on consumer electronics, and we already know the cheaper cameras have less RAM. Also, if the camera gets bumped up to the next memory size like you suggest, the designers will take advantage of it. That's not to say we can't use some, but I believe it is extremely unlikely we will get tens of megabytes without seriously affecting camera operation. We might be able to use that much for specific, limited times, e.g. in playback mode.
    Don't forget what the H stands for.

     

    Related Topics