experimental alternate memory allocation

funnel

349

Re: experimental alternate memory allocation

« Reply #110 on: 03 / June / 2011, 13:49:12 »

Advertisements

Thank you philmoz for your great tutorial. I managed to load CHDK into extended memory of SX220 and that fixed a lot of problems for me. With 4MB it looks like it overlaps with the RAW buffers and crashes when shooting. With 2MB I had no problems so far.

Code: [Select]

NHSTUB(exmem_alloc, 0xFF085434)

Code: [Select]

ifdef OPT_CHDK_IN_EXMEM
MEMISOSTART=0x77867E0
else
MEMISOSTART=0x186AE4
endif

MAXRAMADDR=0x7FFFFFF
EXMEM_BUFFER_SIZE=0x200000
EXMEM_HEAP_SKIP=0x679800

Logged

funnel

349

Re: experimental alternate memory allocation

« Reply #111 on: 09 / June / 2011, 17:27:33 »

Can somebody confirm that Quality Overrides work with chdk in exmem? I can't set the superfine quality which works in the normal memory allocation.

Logged

philmoz

3325

Re: experimental alternate memory allocation

« Reply #112 on: 09 / June / 2011, 18:28:09 »

Quote from: funnel on 09 / June / 2011, 17:27:33

Can somebody confirm that Quality Overrides work with chdk in exmem? I can't set the superfine quality which works in the normal memory allocation.

Works on G12 and SX30.
Your EXMEM_HEAP_SKIP value is large (6.4MB) and if 4MB of EXMEM causes problems with RAW buffers (from previous post) it's possible there just isn't enough memory available.

Try enabling OPT_CHDK_EXMEM_TESTING (and disable OPT_CHDK_IN_EXMEM).
Enable superfine and see if you get any EXMEM corruption. If so adjust EXMEM size.

You may need to disable the superfine option override - in my opinion it's of very little value anyway. Canon's fine mode is indistinguishable from superfine under normal viewing conditions. You can see some tests I did using the SX30 here - http://forums.dpreview.com/forums/read.asp?forum=1010&message=37238416

Another option (that I've toyed with; but not implemented fully) is to load CHDK into EXMEM using just enough EXMEM memory for CHDK itself; but then use the normal Canon heap memory for malloc/free.
We would then have four scenarios:
- no EXMEM, CHDK loaded in low memory and uses Canon heap in low memory
- CHDK loaded in low memory; but CHDK heap in EXMEM memory
- CHDK loaded in EXMEM, and CHDK heap in EXMEM
- CHDK loaded in EXMEM, CHDK heap using Canon heap in low memory

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

funnel

349

Re: experimental alternate memory allocation

« Reply #113 on: 10 / June / 2011, 03:02:58 »

I did a thorough exmem_testing and no corruption was found up to 2MB. Now I went for the second option - CHDK loaded in low memory; but CHDK heap in EXMEM memory. I allocated just 256KB to be on the safe side, at 512KB seemed fine too. I never thought about the fourth option. Good to know, thanks.

Logged

ADamb

65
sx200is

Re: experimental alternate memory allocation

« Reply #114 on: 09 / May / 2012, 14:49:16 »

Support for sx200is 100c.
Not throughly tested, but should be OK.

sx200is_100c.zip (5.5 kB - downloaded 48 times.)

Logged

SX200 IS-100C

waterwingz

12083

Re: experimental alternate memory allocation

« Reply #115 on: 09 / May / 2012, 20:30:05 »

Quote from: ADamb on 09 / May / 2012, 14:49:16

Support for sx200is 100c.
Not throughly tested, but should be OK.

Not sure what somebody is supposed to do with the two files you posted ? Are you proposing this as a change to the svn repository / autobuild server ? If so, the normal process is to submit a patch file here : http://chdk.setepontos.com/index.php?topic=650.0 . If you need help with preparing a suitable patch file, just ask.

Logged

Ported : A1200 SD940 G10 Powershot N G16

waterwingz

12083

Re: experimental alternate memory allocation

« Reply #116 on: 27 / August / 2012, 20:52:06 »

I did some experimenting today with EXMEM and am looking for some advice.

Caveat: I set XMEM_BUFFER_SIZE=0x80000 for these experiments. I have not done any OPT_EXMEM_TESTING - not even sure if that's still needed ? (gulp)

In the current CHDK trunk, it seems that every port that uses EXMEM to get around low memory issues enables both of these overrides in platform/camera/sub/fw/makefile.inc :

Code: [Select]

override OPT_EXMEM_MALLOC=1
override OPT_CHDK_IN_EXMEM=1

However, on a cheap low memory camera (A1200), my simple experiments shows :
OPT_EXMEM_MALLOC=0 OPT_CHDK_IN_EXMEM=0 free memory = 122K OPT_EXMEM_MALLOC=1 OPT_CHDK_IN_EXMEM=1 free memory = 291K OPT_EXMEM_MALLOC=0 OPT_CHDK_IN_EXMEM=1 free memory = 399K OPT_EXMEM_MALLOC=1 OPT_CHDK_IN_EXMEM=0 free memory = 517K

So in a low memory camera, enabling both options doubles free memory. But splitting where CHDK loads from where CHDK allocates its memory makes a big difference in the free memory by almost 400%.

Other posts (recently the sx130/150) show that putting things in EXMEM cuts the buffer size available for rapid shooting. But other than that .. why do more cameras not spit where CHDK goes from where it puts its heap ?

Or what else do I need to consider here ? This could be very important to other "low memory" cameras like the A800 / A810.

Logged

Ported : A1200 SD940 G10 Powershot N G16

reyalp

13295

Re: experimental alternate memory allocation

« Reply #117 on: 27 / August / 2012, 21:05:36 »

How are you measuring "free memory" ?

Assuming you are talking about the Canon malloc heap, then something is wrong either in your implementation or measurement. Without CHDK in exmem, the CHDK binary takes away from the camera heap. Your numbers appear to show the reverse.

edit:
Note if you are not using CAM_FIRMWARE_MEMINFO method, you are getting the largest free block, meaning 1 unlucky fragmentation could skew your results a lot.

I suggest implementing CAM_FIRMWARE_MEMINFO (should be found fine by sig finder, on by default in 1.2 for dryos) and using chdkptp:
=return get_meminfo('system')
=return get_meminfo('exmem')

edit for posterity:
the missing not made this confusing.

« Last Edit: 27 / August / 2012, 21:52:47 by reyalp »

Logged

Don't forget what the H stands for.

waterwingz

12083

Re: experimental alternate memory allocation

« Reply #118 on: 27 / August / 2012, 21:09:52 »

Quote from: reyalp on 27 / August / 2012, 21:05:36

How are you measuring "free memory" ?

The numbers are from the Miscellaneous Stuff -> Show Memory Info menu.

I played with
#define CAM_FIRMWARE_MEMINFO 1
but didn't notice a change (not that I repeated every test every way though). It was on for these tests.

If I interpret the results correctly, the best scenerio is CHDK in low memory and CHDK allocating memory in high memory ? Although I could be totally confuse about the semantics of those #defines.

Update : incidentally, I started this when my A1200 port would not load the LARGE badpixel.bin file for the elcheapo sensor (it would load a smaller file from another camera). With the options that seem to free up more memory, I can now use badpixel.bin successfully. And from what philmoz tells me .. I need a large contiguous block to load that file !

« Last Edit: 27 / August / 2012, 21:17:32 by waterwingz »

Logged

Ported : A1200 SD940 G10 Powershot N G16

philmoz

3325

Re: experimental alternate memory allocation

« Reply #119 on: 27 / August / 2012, 21:43:29 »

Don't have time for a detailed reply just now - look at the wiki page for exmem and the diagrams that show how the memory is mapped.

I've added some comments to the options below:
OPT_EXMEM_MALLOC=0 OPT_CHDK_IN_EXMEM=0 free memory = 122K (1) OPT_EXMEM_MALLOC=1 OPT_CHDK_IN_EXMEM=1 free memory = 291K (2) OPT_EXMEM_MALLOC=0 OPT_CHDK_IN_EXMEM=1 free memory = 399K (3) OPT_EXMEM_MALLOC=1 OPT_CHDK_IN_EXMEM=0 free memory = 517K (4)

(1)
This is using the Canon heap only and 'free memory' is showing the size of the largest free block in the Canon heap.

(2)
The Canon heap is mostly untouched in this scenario, CHDK is loaded in EXMEM and uses EXMEM for it's own heap. The 'free memory' value is the largest free block in the EXMEM heap managed by CHDK.

(3)
Don't do this - very very bad
Here you are loading CHDK into EXMEM without first allocating the EXMEM buffer from the firmware - if any firmware function allocates EXMEM it could overwrite CHDK.
(I need to check for this scenario in the makefile and throw an error.)

(4)
Mixed mode. CHDK loaded into the Canon heap so the firmware loses the heap memory used by CHDK. CHDK heap is in EXMEM - the 'free memory' is the largest block in the CHDK EXMEM heap.

You'll need to experiment to see what works best - and yes OPT_EXMEM_TESTING should still be done to see if there are any video buffers that might get corrupted by CHDK EXMEM.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)