supplierdeeply

The difference between malloc() and umalloc()

  • 8 Replies
  • 5601 Views
*

Offline LjL

  • ****
  • 266
  • A720IS
  • Publish
    The difference between malloc() and umalloc()
    « on: 12 / July / 2008, 08:03:19 »
    Advertisements
    The comments and the real function name that the wrapper calls make it clear that umalloc() allocates uncacheable memory, while malloc() doesn't apparently have any such restrictions.

    Fine, but what's the difference in practice? Where would one use one instead rather than the other?

    Even in raw.c, I see a call to malloc() and one to umalloc().

    *

    Online reyalp

    • ******
    • 9962
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #1 on: 16 / September / 2008, 21:54:03 »
    Old thread, but I have the same question.

    It's not at all clear to me why most of the uses of umalloc (conf.c, lang.c, gui_fselect.c) would want uncacheable. They are frequently used with read()/write(), but those functions are also used with cached memory at times. I haven't been able to figure out if these functions behave differently given uncached memory, although it stands to reason they might be able to skip a cache flush or copy.

    FWIW, if you look at the actual code of _AllocateUncacheableMemory, it uses malloc, but then ORs  0x10000000 into the returned pointer to point to the uncached version of main memory. So it isn't even coming from a different pool.

    If I'm not mistaken, Fopen_FileStream (used by Fopen_Fut) allocates 32k (!) of unachacheable memory for each FILE pointer.  I would presume this is for a buffer that's used with DMA ? Since the fut functions ultimately use Read and Write, this would fit the speculation above.
    Don't forget what the H stands for.

    *

    Offline jeff666

    • ****
    • 181
    • A720IS
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #2 on: 18 / September / 2008, 04:02:33 »
    FWIW, if you look at the actual code of _AllocateUncacheableMemory, it uses malloc, but then ORs  0x10000000 into the returned pointer to point to the uncached version of main memory. So it isn't even coming from a different pool.

    See http://chdk.wikia.com/wiki/Adding_support_of_a_new_camera:
    Quote
    Memory layout
    0x10000000-0x1fffffff      - same as 0x0-0x0fffffff but with no caching

    Cheers.

    *

    Online reyalp

    • ******
    • 9962
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #3 on: 18 / September / 2008, 04:27:13 »
    Err yes, that's why I said "ORs  0x10000000 into the returned pointer to point to the uncached version of main memory" :)

    Any thoughts on the original question ?
    Don't forget what the H stands for.


    *

    Offline ewavr

    • ****
    • 1057
    • A710IS
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #4 on: 18 / September / 2008, 06:12:21 »
    My history (see Changeset 403 - chdk - Trac):
    Old version:
    1. Copy partition table to static buffer
    2. Modify buffer contents
    3. Write buffer back to card using WriteSDCard()

    This works for many cameras, but not for IXUS700 - changes have not been written!
    Allocating buffer using AllocateUncacheableMemory() instead of using static buffer has solved this problem.
    Unfortunately I don't remember exactly whether I tried to use malloc() function. Maybe yes, but without success.


    *

    Online reyalp

    • ******
    • 9962
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #5 on: 18 / September / 2008, 16:49:04 »
    You could try  pbuf = &staticbuf | 0x10000000 ;)

    I'm not surprised that WriteSDCard requires uncached memory. However, read/write are frequently used with normal buffers, so it works at least some of the time. If it really needs uncached, then we should go through and fix those.
    Don't forget what the H stands for.

    *

    Online reyalp

    • ******
    • 9962
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #6 on: 25 / October / 2008, 04:20:57 »
    It appears that some cameras do have trouble with read() to cached memory.
    LUA script gives bogus error messages

    I assume write() would also have the same issue.

    Fut should be safe, since it maintains it's own uncached buffer.
    Don't forget what the H stands for.

    *

    Online reyalp

    • ******
    • 9962
  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #7 on: 03 / November / 2009, 16:26:35 »
    Responding to Microfunguys post from http://chdk.setepontos.com/index.php/topic,4465.msg42798.html#msg42798
    This is something I am not clear about ..
    Does malloc reserve memory in the memory cache and a copy in regular memory and write to both ?
    If so, can the cached memory be 'out-of-date' and how would you know ?
    When would you use uncached memory allocation ?
    This is pretty far from the OPs question, so I'm replying in a more appropriate thread.

    First, some background
    Camera has two views of main memory. These address spaces refer to the same physical memory. The differences is how the CPU access it.
    0 through <memory size> has the cache and write buffer bits enabled.
    <uncached start address> through <uncached start + memory size> has the cache and write buffer bits disabled.

    The uncached start address is defined in CHDK with CAM_UNCACHED_BIT. Memory sizes vary between 16 and 128 megs or so. The values for a specific camera can be found from the CP15 instructions at the very start of the firmware.

    If the CPU reads via an address in the cached range, it will check the cache first, and will copy the value from main memory to cache if needed. If it writes a value, the value will be written to the cache, and only updated in main memory later. This is normally completely transparent from the programmers POV. There's also an independent instruction cache, but we'll ignore that for now.

    The specific rules are described in the ARM documentation (see ARM946E-S Technical Reference Manual, chapter 3 "Caches", and chapter  6, section 5 "The write buffer"). Mostly we can ignore the details, but the rules concerning coherency are important:
    Quote
    3.3.4 Data cache validity
    ...
    The ARM946E-S processor does not support external memory snooping. Any shared data memory space therefore, must not be cachable.
    In other words, if an external device modified memory, the CPU won't know. If the CPU happens to have a value for this address in cache, it will happily use that old value. Similarly, if the CPU writes to an address in the cached range, external devices may not see that value until it has made its way through the cache and write buffer.

    Since there are a bunch of external devices that are directly tied to main memory (the sensor frame buffer, the display frame buffers) the canon firmware uses the uncached range to interact with them. It appears that some lower level IO functions (read(), write()) also use DMA, and so only work reliably on uncached addresses.

    CHDK actually breaks this rule with the "raw buffer cached" option for DNG, accessing the raw buffer via the cached range. This is technically wrong (we should flush before we start and flush and drain write buffer when done, or something like that) but appears to work. The data volume is so much larger than the cache (4k on most or all CHDK cameras) that most of it will have been flushed out anyway.

    OK that's very exciting, but what should I use where ?
    • All normal operations should be on cached memory. You don't have to manage the caching, unless you are dealing with external hardware that maps directly to main memory. All the CHDK code and variables is accessed through the cached range.
    • malloc()/free() operate on addresses in the cached ranged. This should be used for any dynamically allocated data, except the following:
    • umalloc()/ufree() operate on uncached addresses. Use them if and only if that memory has to be passed to a low level function that expects uncached memory. Generally in CHDK the only functions with this requirement are read() and write().


    Some of this is also covered in http://chdk.wikia.com/wiki/CHDK_Coding_Guidelines
    Don't forget what the H stands for.


  • Publish
    Re: The difference between malloc() and umalloc()
    « Reply #8 on: 03 / November / 2009, 16:39:25 »
    Thanks, that is very interesting.

    I will read it a few more times  ......


     

    Related Topics