AdditionAgentRAM - an unused piece of memory? - page 2 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 [2] 3 4   Go Down

AdditionAgentRAM - an unused piece of memory?

  • 32 Replies
  • 20118 Views
*

Offline dvip

  • ****
  • 451
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #10 on: 05 / November / 2013, 12:41:55 »
Advertisements
Could that chunk of RAM be used as a buffer to speed up RAW/DNG writing to the SD card?  :xmas
Logged
*

Offline srsa_4c

  • ******
  • 4451
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #11 on: 05 / November / 2013, 16:47:37 »
Quote from: reyalp on 04 / November / 2013, 16:12:02
This could be enough for the main CHDK binary, but 140k is *very* tight (my current d10 trunk build is 147924 bytes). If we were going to use this, we'd probably want a build time check of CHDK size vs ram area size.
Using it for the main binary is the simplest solution (when it fits), but it wastes the rest of that area.
This memory area is right below the "DIGIC mirror", which explains its special start address. Some routines set SP to its start address, but those don't look like normal cam operation. AdditionAgentRAM itself doesn't seem to check the availability of this memory chunk, doesn't seem to depend on a special camera state, and it's registered by System.Create.

Quote
However, I seem to remember some factory mode related binaries being loaded to the same address as a framebuffer, so checking various shooting modes, video recording, ptp would probably be a good idea.
Tried all these.

Quote
Another option would be to make a separate suba pool, which might help avoid some fragmentation issues. Since the addresses are known, you could wrap malloc/free etc to know which pool a pointer came from. In this way, CHDK could allocate from addition ram where possible, and still fall back to regular heap if it filled up.
Sounds like a good idea.

Reading a little of the suba documentation, I found this:
Quote
This allocator is lock-less but reentrant meaning it will be faster but more consideration is necessary for coordinating multiple threads as opposed to the standard C library allocator.
Are we using it in a safe way?

Quote from: dvip on 05 / November / 2013, 12:41:55
Could that chunk of RAM be used as a buffer to speed up RAW/DNG writing to the SD card?
IMHO it's too small for that kind of use.

Quote from: nocaC_230 on 05 / November / 2013, 02:43:16
I have the code "addition Agen tram - on unused piece of memory" transfer to my SX230HS and everything works so far without errors.
The advantage is that I can set "Super Fine" quality as well, without leading to crashes.
This leads to a significant improvement of the JPG-quality .
Glad to hear that.
Quote
Is it possible to set the Canon firmware to another storage path for the JPG images.
I would like to save Panorama-images or HDR-images into separate directories. Any idea?
Sounds possible, but complicated to implement. It would need a user interface, handle two hooks (raw, filewrite) properly, and - depending on the chosen path - the cameras wouldn't recognize some or all images... Additionally, we can't influence native RAW on cameras that support it.
Logged
*

Offline reyalp

  • ******
  • 14128
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #12 on: 05 / November / 2013, 17:54:31 »
Quote from: srsa_4c on 05 / November / 2013, 16:47:37
Using it for the main binary is the simplest solution (when it fits), but it wastes the rest of that area.
Agreed. The "wastes" doesn't bother me, since the size is quite close and we are never able to use this before. If we used it as a malloc pool, we'd lose some to fragmentation etc anyway.

However, the closeness and "when it fits" makes it a bit troublesome to use by default. Ports vary, and binary size also depends on toolchain. A small change in core code could push a port that used to fit just over the limit, which would then either break the port or break the autobuild.

Even if you could automatically kick it out of  AdditionAgentRAM when it got too big, it could cause changes of behavior between builds.
Quote
Are we using it  [suba] in a safe way?
Hmm, on vxworks it should be fine, since the Canon configuration seems to be effectively non-preemptive.  On DryOS it might well not be, would have to look more closely. It seems to work well enough on exmem enabled cams, but we usually don't have more than one task doing a lot of repetitive malloc/free activity. My proposal wouldn't make it any worse...
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #13 on: 06 / November / 2013, 11:02:43 »
Quote from: reyalp on 05 / November / 2013, 17:54:31
If we used it as a malloc pool, we'd lose some to fragmentation etc anyway.
An advantage of this would be that it could delay the fragmentation of the Canon malloc heap (IMHO).

Quote
However, the closeness and "when it fits" makes it a bit troublesome to use by default.
I didn't mean putting the main binary there by default (in all ports). Perhaps we could offer an alternative "stable" build for some troubled ports - but that needs testing. Additionally, I don't think this should be attempted for cameras with only 0x22000 bytes of this memory.
Logged
*

Offline reyalp

  • ******
  • 14128
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #14 on: 06 / November / 2013, 14:23:28 »
Quote from: srsa_4c on 06 / November / 2013, 11:02:43
I didn't mean putting the main binary there by default (in all ports). Perhaps we could offer an alternative "stable" build for some troubled ports - but that needs testing.
What I meant is if we put a "troubled" port there, changes in the core could push it over the limit without being noticed (or alternatively, break the build if we detect it). Though if we don't try to do it for the ones with 0x22000, that should give plenty of margin in the current core code that it isn't an issue.

If it does turn out to be unused, then using it on all the cameras that can support it makes sense to me. Freeing up ~150k of canon heap is pretty significant.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #15 on: 08 / November / 2013, 23:28:55 »
Patch for possible use of AgentRAM.

This adds a new suba memory pool similar to EXMEM with the following settings:
- OPT_ARAM_MALLOC = enable use of AgentRAM
- OPT_CHDK_IN_ARAM = load CHDK at the start of the AgentRAM block
- ARAM_HEAP_START = start address
- ARAM_HEAP_SIZE = block size

If OPT_CHDK_IN_ARAM is defined then whatever is left over in the AgentRAM block after loading CHDK is set up as a suba heap, otherwise the whole block is used.

The memory allocation (malloc / free) will now also use all three potential memory areas, it first tries AgentRAM, if that fails it tries EXMEM, if that also fails it uses the Canon Heap.

Also includes an update to the module inspector to show the AgentRAM info.

Sample G12 files to show usage included.

Requires a change in boot.c for each camera/firmware to set the Canon heap start address correctly.

Note: don't define both OPT_CHDK_IN_EXMEM and OPT_CHDK_IN_ARAM.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline reyalp

  • ******
  • 14128
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #16 on: 09 / November / 2013, 21:07:58 »
Quote from: philmoz on 08 / November / 2013, 23:28:55
Patch for possible use of AgentRAM.
Thanks for doing this, I think it seems like a good approach.

In the attached patch, I've added support for D10 and elph130. I also added some code to check that the generated bin fits if you enable OPT_CHDK_IN_ARAM. This correctly makes the build fail on elph130, which doesn't have enough aram for a full CHDK binary.

On D10, both OPT_CHDK_IN_ARAM and OPT_ARAM_MALLOC appear to work. I haven't exercised them heavily yet.

On elph130, OPT_ARAM_MALLOC crashes shortly after startup in a call to firmware free() from spytask. I haven't looked into this in detail yet.

Some other general comments:
core_get_free_memory used to return the largest free block. In the patched code it returns the sum of the largest free blocks, which isn't really equivalent. For a single value, it might make more sense to return the largest single free block.

lua get_meminfo needs to be updated. Adding "aram" as a heap type to explicitly get the aram heap makes sense. It's less clear what the default should do. The current behavior is to return the heap that CHDK is allocating from. If this returned only the aram value, it could make current scripts think they are running out of memory. It might make sense to make a "combined" value that builds a meminfo from all the pools that are being used. The free size and total size would be the combined values, the largest free block would be the largest among all the pools etc. I'm not sure what you'd do with the allocated peak, but it probably doesn't matter much.

Since we know the canon heap size (from get_meminfo, but it could also be obtained from the firmware dump), it would be possible to detect if the pointer passed to free() falls in any of the heaps. This might be useful for debugging at some point.
Logged
Don't forget what the H stands for.
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #17 on: 10 / November / 2013, 00:52:28 »
Updated patch adds OPT_ARAM_TESTING for checking if the AgentRAM block is actually unused by the Canon firmware (works like OPT_EXMEM_TESTING).

Added a new define 'CHDK_NOT_IN_CANON_HEAP' to simplify the changes that are required to boot.c.

Also includes reyalp's changes, plus updated core_get_free_memory to return the largest block instead of the total.

Note: as with EXMEM, don't define both OPT_ARAM_TESTING and OPT_CHDK_IN_ARAM.

Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline msl

  • *****
  • 1280
  • A720 IS, SX220 HS 1.01a
    • CHDK-DE links
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #18 on: 10 / November / 2013, 09:06:31 »
Great! I'm excited.

A first test with AdditionAgentRAM and the SX220 was very successful.

This was the first time, that the file size of all JPGs were identical with the result without CHDK. In the past not all results were the same like without CHDK. Some files were smaller with CHDK.  At last also works the CHDK picture quality override 'superfine'. This could be the last step to remove the beta status of the SX220.

I have made some other quick tests:

RAW/DNG  -  ok
Video recording - 1 min. full HD ok; time override for 240p slow motion > 30 sec. ok
Scripts - normal user scripts (file size < 10k) ok; big script (~60k) via loader script ok
PTP - download/upload ok; live view ok; remote shoot ok

msl
Logged
CHDK-DE:  CHDK-DE links
*

Offline waterwingz

  • ******
  • 12140
    • Waterwingz
Re: AdditionAgentRAM - an unused piece of memory?
« Reply #19 on: 10 / November / 2013, 11:31:17 »
Tried this with the A1200.    Calls to get_meminfo()  show 30% more free memory at script start when AdditionAgentRAM is used for CHDK rather than exmem.  This should give a lot more headroom for large scripts.    :haha

One small note on the patch file -  the properties on tools/checkbinsize.sh are not set executable. I assume this is because philmoz's patch file was created on a Windows machine?

Logged
Ported :   A1200    SD940   G10    Powershot N    G16
Pages: 1 [2] 3 4   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal