CHDK core in multiple fragments?

srsa_4c

4396

CHDK core in multiple fragments?

« on: 23 / January / 2014, 16:38:40 »

Advertisements

Looking at the output of 'extask' (extended task information for DryOS), I noticed that a task named EvShel (event shell) has unusually large stack space (0x8000 bytes aka 32kB). This task is only used for debugging over the UART, but it's sitting idle in all cameras used by ordinary people - waste of RAM.
As an experiment, I have prevented this task from starting (on the a470) - it's usually started from the last BL in the 'CommonDrivers' task. Side effect is that this prevents two other tasks from being started (ConsoleSvr, LowConsole) - these would be started by EvShel itself.
AFAIK tasks' stack space is allocated from the 'system' heap, not the usual 'user' malloc heap.
The above hack leaves 0x8000 + 0x800 + 0x800 bytes of that heap free. Using this amount for CHDK should not cause any problems with the Canon firmware - the resulting system memory allocation would stay the same. Since I don't know the allocator of the system memory, we could launch a simple task with ~32kB 'stack variable', and use that ~32kB for something useful. Since this isn't a whole lot, perhaps it would be better to use it for static storage, such as some part of the CHDK core on cameras that have the lower amount of ARAM. Problem is that it's probably not known (at build time) where this memory will be located.

Now the question: does this (see thread title) sound possible?

Logged

Menu jumping/flickering on M10?

philmoz

3279

Re: CHDK core in multiple fragments?

« Reply #1 on: 23 / January / 2014, 16:47:30 »

Quote from: srsa_4c on 23 / January / 2014, 16:38:40

AFAIK tasks' stack space is allocated from the 'system' heap, not the usual 'user' malloc heap.

Is there a 'system' heap for things like this?
I thought there was just a single heap that everything was allocated from - in which case blocking this task will automatically give the extra 32K back to CHDK & the firmware. I could be wrong of course

The current module system already segments CHDK into pieces; but we need to move more stuff into modules and get the core CHDK down to fit into the smallest ARAM size.

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

srsa_4c

4396

Re: CHDK core in multiple fragments?

« Reply #2 on: 23 / January / 2014, 16:55:14 »

Quote from: philmoz on 23 / January / 2014, 16:47:30

Is there a 'system' heap for things like this?

Yes, some eventproc output from a random DryOS cam:

Code: [Select]

Malloc Information (onetime type)
  Start Address       = 0x0016a898
  End Address         = 0x002f1de0
  Total Size          = 0x00187548 (  1602888)
  Allocated Size      = 0x00100f70 (  1052528)
  Allocated Peak      = 0x00109cb0 (  1088688)
  Allocated Count     = 0x000001ca (      458)
  Free Size           = 0x000865d8 (   550360)
  Free Block Max Size = 0x0007a7a8 (   501672)
  Free Block Count    = 0x00000005 (        5)

System Memory Information
  Start Address       = 0x002f9c08
  End Address         = 0x0034cc00
  Total Size          = 0x00052ff8 (   339960)
  Allocated Size      = 0x0003bee0 (   245472)
  Allocated Peak      = 0x00041f10 (   270096)
  Allocated Count     = 0x00000048 (       72)
  Free Size           = 0x00017118 (    94488)
  Free Block Max Size = 0x00011470 (    70768)
  Free Block Count    = 0x00000005 (        5)

... and from the output of the 'mkcfg' command of drysh (not the same cam):

Code: [Select]

sys_mem_start   0x00559c00
sys_mem_max         471040
user_mem_start  0x0026b310
user_mem_max       3030604
sys_objs_start  0x0054f15c
sys_objs_end    0x00559c00

Some stack space has to be reserved for use by interrupts(?), even on tasks that do nothing.

Logged

Menu jumping/flickering on M10?

reyalp

13118

Re: CHDK core in multiple fragments?

« Reply #3 on: 23 / January / 2014, 23:03:30 »

One possible approach to splitting up CHDK would be to load the code, bss and data segments at different locations. This could significantly complicate the startup code though, and the addresses would need to be known at build time. I'm not entirely sure how you'd get gcc to build it that either, possibly some link-boot.ld hacking?

Modularizing more stuff is probably the better approach...

I bet there are malloc like functions for the "system" heap. malloc and free look like they both call an underlying function that takes the "heap" as an argument, and the system memory info function refers to one of the same addresses.

On elph140
sub_675B80 is the low level malloc, sub_675CD0 is the free, and 0x161A0 seems to be the system heap structure. 0x15FC8 is the normal heap structure.

These are probably somewhat equivalent the memPart functions in vxworks.

Logged

Don't forget what the H stands for.

srsa_4c

4396

Re: CHDK core in multiple fragments?

« Reply #4 on: 24 / January / 2014, 14:50:58 »

Quote from: reyalp on 23 / January / 2014, 23:03:30

On elph140
sub_675B80 is the low level malloc, sub_675CD0 is the free, and 0x161A0 seems to be the system heap structure. 0x15FC8 is the normal heap structure.

These are probably somewhat equivalent the memPart functions in vxworks.

That's good, no need for a dummy task.

Looks like splitting out 'mandatory modules' is the way to go.

Some thoughts on this.
Modules implementing mandatory functions should probably be packaged into diskboot.bin and ps.fi2 somehow. Cameras that don't implement system heap malloc could load the reduced size core and the mandatory modules in one chunk:
[[ CHDK core ][module1][module2][module3]]
and then 'mount' the modules in-place.
Cameras that have sys heap malloc could load the small core, copy the modules to sys heap allocations, and mount them. Their loader will probably need to copy the modules to a temporary 'safe' place like exmem.
Pretty complicated...

The simplest way would be rearranging memory areas (enlarge user heap, shift 'sys objects area', reduce sys heap), but I don't know if it's possible - depends on where the addresses in question are (ROM or RAM).

Logged

Menu jumping/flickering on M10?

reyalp

13118

Re: CHDK core in multiple fragments?

« Reply #5 on: 24 / January / 2014, 23:05:35 »

Quote from: srsa_4c on 24 / January / 2014, 14:50:58

That's good, no need for a dummy task.

Caveat: I haven't tested the stuff I described

Quote

Cameras that have sys heap malloc could load the small core, copy the modules to sys heap allocations, and mount them. Their loader will probably need to copy the modules to a temporary 'safe' place like exmem.

Yeah, this could get tricky with startup order, system malloc not available until some time after chdk has been loaded. I guess system malloc must be ready pretty early though.

Not sure about the need to bundle modules into diskboot / ps.fi2.

Logged

Don't forget what the H stands for.

srsa_4c

4396

Re: CHDK core in multiple fragments?

« Reply #6 on: 25 / January / 2014, 09:15:46 »

Quote from: reyalp on 24 / January / 2014, 23:05:35

Yeah, this could get tricky with startup order, system malloc not available until some time after chdk has been loaded. I guess system malloc must be ready pretty early though.

IMHO it should work by the time the first task is created.

Quote

Not sure about the need to bundle modules into diskboot / ps.fi2.

If it's possible to separate vital parts of CHDK (say, fonts, strings, GUI - just guessing) into modules then having these modules in files (which may be missing) may not work well.

Logged

Menu jumping/flickering on M10?

philmoz

3279

Re: CHDK core in multiple fragments?

« Reply #7 on: 25 / January / 2014, 15:37:39 »

Quote from: srsa_4c on 25 / January / 2014, 09:15:46

Quote from: reyalp on 24 / January / 2014, 23:05:35
Yeah, this could get tricky with startup order, system malloc not available until some time after chdk has been loaded. I guess system malloc must be ready pretty early though.
IMHO it should work by the time the first task is created.
Quote
Not sure about the need to bundle modules into diskboot / ps.fi2.
If it's possible to separate vital parts of CHDK (say, fonts, strings, GUI - just guessing) into modules then having these modules in files (which may be missing) may not work well.

You couldn't easily use the module system and bundle modules into the diskboot.bin

The CHDK loader would need to include all the module relocation logic in order to move and link the modules. It would need a copy of (or a reference to) the address relocation tables. If you could statically link each piece then it might work; but then they aren't really CHDK modules.

At the time the loader code runs, the firmware memory allocation systems have likely been trashed by loading diskboot.bin into memory at 0x1900. So the loader would need pre-determined addresses to put each module into - the location and size of each block would need to be determined by the developer for each port, unless it could be automated somehow with finsig.

It would probably be simpler to split the text/data/bss sections of the code into separate locations as reyalp suggested earlier, using the linker to sort out the addressing. You would still need some way to put different parts in different memory locations in the loader, and work out all the address details during the port.

I think there are still some things in the core CHDK code that might be candidates for modules:
- shot histogram
- auto iso
- usb remote
- CHDK menus (and rbf font stuff)

Moving the CHDK menus to a module would save a lot of core memory; but might cause issues for new users if the module is missing or does not match the core version. It also currently has a lot of conditionally compiled code that would need sorting out.

Phil.

Logged

srsa_4c

4396

Re: CHDK core in multiple fragments?

« Reply #8 on: 25 / January / 2014, 16:30:19 »

Quote from: philmoz on 25 / January / 2014, 15:37:39

The CHDK loader would need to include all the module relocation logic in order to move and link the modules. It would need a copy of (or a reference to) the address relocation tables. If you could statically link each piece then it might work; but then they aren't really CHDK modules.

At the time the loader code runs, the firmware memory allocation systems have likely been trashed by loading diskboot.bin into memory at 0x1900. So the loader would need pre-determined addresses to put each module into - the location and size of each block would need to be determined by the developer for each port, unless it could be automated somehow with finsig.

I'm not sure if those addresses can be pre-determined.
Since I don't know the module loader code well, I have to ask: is additional memory needed for loading a module (I mean, module size + ?) ? Is that additional memory separately allocated, or is it part of the allocated block where the module will be copied? In other words: can the memory size needed to load a module be pre-determined? If so, it could be part of the diskboot/fir, with the correct size - for cameras that are not short of memory. Downside: this would probably increase memory consumed by CHDK, compared to the current state.

For cameras with small ARAM area, the loader would copy the modules to exmem, and then they would be loaded by spytask, using sys-malloc. On other cams, spytask could 'load' the modules in-place.

Still pretty complicated...

Quote

It would probably be simpler to split the text/data/bss sections of the code into separate locations as reyalp suggested earlier, using the linker to sort out the addressing. You would still need some way to put different parts in different memory locations in the loader, and work out all the address details during the port.

My concern is that even if the location of the sys memory is known, we can't just put data there (for example to the top of that area) and expect it to remain untouched. If the area can be shifted (this will have to be researched, I guess), then this whole discussion is almost pointless (that would be my bad

Quote

I think there are still some things in the core CHDK code that might be candidates for modules:
- shot histogram
- auto iso
- usb remote
- CHDK menus (and rbf font stuff)

The first three look like non-mandatory, the menus and fonts are important.

Quote

Moving the CHDK menus to a module would save a lot of core memory; but might cause issues for new users if the module is missing or does not match the core version. It also currently has a lot of conditionally compiled code that would need sorting out.

One of my (distant) plans is a packed CHDK-installer diskboot/fir which includes every needed file.

Logged

Menu jumping/flickering on M10?

philmoz

3279

Re: CHDK core in multiple fragments?

« Reply #9 on: 25 / January / 2014, 16:58:39 »

Quote from: srsa_4c on 25 / January / 2014, 16:30:19

Quote from: philmoz on 25 / January / 2014, 15:37:39
The CHDK loader would need to include all the module relocation logic in order to move and link the modules. It would need a copy of (or a reference to) the address relocation tables. If you could statically link each piece then it might work; but then they aren't really CHDK modules.

At the time the loader code runs, the firmware memory allocation systems have likely been trashed by loading diskboot.bin into memory at 0x1900. So the loader would need pre-determined addresses to put each module into - the location and size of each block would need to be determined by the developer for each port, unless it could be automated somehow with finsig.
I'm not sure if those addresses can be pre-determined.
Since I don't know the module loader code well, I have to ask: is additional memory needed for loading a module (I mean, module size + ?) ? Is that additional memory separately allocated, or is it part of the allocated block where the module will be copied? In other words: can the memory size needed to load a module be pre-determined? If so, it could be part of the diskboot/fir, with the correct size - for cameras that are not short of memory. Downside: this would probably increase memory consumed by CHDK, compared to the current state.

For cameras with small ARAM area, the loader would copy the modules to exmem, and then they would be loaded by spytask, using sys-malloc. On other cams, spytask could 'load' the modules in-place.

Still pretty complicated...

The module files contain relocation tables to allow the module to be linked to the core CHDK at run time. This is loaded into a separate block of memory that is discarded after the module is loaded.

Quote

Quote
It would probably be simpler to split the text/data/bss sections of the code into separate locations as reyalp suggested earlier, using the linker to sort out the addressing. You would still need some way to put different parts in different memory locations in the loader, and work out all the address details during the port.
My concern is that even if the location of the sys memory is known, we can't just put data there (for example to the top of that area) and expect it to remain untouched. If the area can be shifted (this will have to be researched, I guess), then this whole discussion is almost pointless (that would be my bad ).

Another option might be to put text in ARAM and data/bss in the Canon or Exmem heap. These are all known addresses.
If the 'system' free memory is reasonably large, and you can find the malloc/free functions, you might be able to add it to the list of CHDK memory pools. This way you would get access to the memory; but not need to pre-calculate any addresses.

Quote

Quote
I think there are still some things in the core CHDK code that might be candidates for modules:
- shot histogram
- auto iso
- usb remote
- CHDK menus (and rbf font stuff)
The first three look like non-mandatory, the menus and fonts are important.

The menus & rbf fonts (which are only used in menus) aren't needed for booting CHDK. They are important of course; but could be loaded dynamically after startup. The issue is if a user doesn't have the module on the SD card they get an error instead of seeing the menu.

Quote

Quote
Moving the CHDK menus to a module would save a lot of core memory; but might cause issues for new users if the module is missing or does not match the core version. It also currently has a lot of conditionally compiled code that would need sorting out.
One of my (distant) plans is a packed CHDK-installer diskboot/fir which includes every needed file.

Not sure what you would gain with this. Just packing all the files together would be huge (380K for G1X & current modules) and would include all the overhead of each module. You would be better to create a monolithic build that just compiled everything into the core code like we did before the module system.

Phil.

Logged