Exmem - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: [1] 2 3   Go Down

Exmem

  • 26 Replies
  • 8242 Views
*

Offline srsa_4c

  • ******
  • 4451
Exmem
« on: 16 / May / 2019, 19:22:27 »
Advertisements
Exmem provides a way to get larger amounts of RAM for various temporary operations.
I've been working on adding exmem support to CHDK.
edit:
Link to thread discussing (static) use of Exmem as a way to get memory for CHDK on models with low amounts of free RAM (heap).

One of the potential uses I have in mind is providing a way for the user to format the card and keeping CHDK on it. Most cameras don't have a large enough amount of regular heap for keeping CHDK files in RAM.

As a first step, I made a simple module (based on 'module inspector') for displaying active exmem allocations.
To my surprise, it seems that some exmem is always in use.
The module stays loaded if the ALT button is used to exit it. This allows one to examine situations where file operations don't work (e.g. playing or recording a movie).
From what I've seen so far, the firmware's allocation table seems to have remained a list of 'start address' + 'size' pairs, from the first supported models to the last ones.

The core interface and the export(s) are subject to change.

Patch against current trunk attached.

edit2:
Patch updated, exmem_type_count is now a separate export, the module can now log changes of exmem use.

exmem_demo_module_v1.diff (6.96 kB - downloaded 4 times.)
« Last Edit: 30 / June / 2019, 12:53:21 by srsa_4c »
Logged
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #1 on: 08 / April / 2020, 16:48:13 »
Quote from: srsa_4c on 16 / May / 2019, 19:22:27
One of the potential uses I have in mind is providing a way for the user to format the card and keeping CHDK on it.
The attached v3 patch includes a module that backs up CHDK files in RAM (exmem) and restores them on the formatted card. Every step needs to be done manually as the firmware formatting routines are not under CHDK control.

The patch adds further exports to CHDK core. The core routines are kept simple for size considerations. A few ports did not have exmem_alloc and exmem_free, they are added manually. exmem_free is turned into a mandatory stub.
This patch is (IMHO) almost ready for including in trunk. I did not try a batch build yet, but that should work.

The backup/restore module may need a slight modification. I forgot to consider that some cameras (S80 for example) do not like when the keyboard task is held up for several seconds. Moving the responsible routines to spytask would make those models work.
I tested the module on a range of models (ixus40_sd300, ixus65_sd630, a3200, ixus150_elph140, sx280), using a 16MB (not a typo) card.

edit:
S80 added to the tested models, worked without modification.

I forgot to mention that keeping system files while formatting is a Magic Lantern feature (AFAIK it works automatically unlike this CHDK addon).
« Last Edit: 09 / April / 2020, 12:42:33 by srsa_4c »
Logged
*

Offline reyalp

  • ******
  • 14114
Re: Exmem
« Reply #2 on: 09 / April / 2020, 16:38:31 »
Very nice work  :D
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #3 on: 10 / April / 2020, 17:06:08 »
Quote from: reyalp on 09 / April / 2020, 16:38:31
Very nice work  :D
Thanks.

I'm thinking about adding all these (core parts first then the modules). If you'd like to review the core changes (there aren't many), I'll wait with the commits (it's not urgent).

While doing the test runs, I noticed that allocations of certain exmem types are automatically resized when a new type gets allocated. EXMEM_FILESTORE is one of them. This allows the new allocation to be at the top and not fragment the exmem area.
Logged
*

Offline reyalp

  • ******
  • 14114
Re: Exmem
« Reply #4 on: 11 / April / 2020, 01:07:03 »
Quote from: srsa_4c on 10 / April / 2020, 17:06:08
I'm thinking about adding all these (core parts first then the modules). If you'd like to review the core changes (there aren't many), I'll wait with the commits (it's not urgent).
Go ahead and commit, the core stuff seems harmless.

edit:
Used backup and restore on elph180 with no problems. One thing that might be nice is to have some indication the process started. Like "restoring..." in the text area after or press set.
« Last Edit: 11 / April / 2020, 01:35:14 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #5 on: 11 / April / 2020, 16:52:14 »
I noticed that there was an inconsistency regarding the types used in existing code and the parts I'm about to add - signed vs. unsigned for exmem type and size. I decided to turn them into unsigned as none of those can be negative. There's also the issue with existing code using the term 'pool id' and me using 'type', but I think I'll leave them as is and add a short explanation.

Quote
Used backup and restore on elph180 with no problems. One thing that might be nice is to have some indication the process started. Like "restoring..." in the text area after or press set.
Thanks for the test run. I've been doing minor modifications on both modules, I'll add the extra message too.

Current state of changes attached. I'll check in the core stuff tomorrow.

edit:
Done. I chose not to make a separate commit for stubs changes.
« Last Edit: 12 / April / 2020, 15:04:03 by srsa_4c »
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Exmem
« Reply #6 on: 13 / April / 2020, 03:34:45 »
Nice work.

Only tested on G12 so far, here's what I've found:
  • fread & fwrite crash on the G12 with an assert in FsIoNotify. I suspect it may be related to this issue https://chdk.setepontos.com/index.php?topic=6179.0. Changing the code to use open/close/read/write fixes the crash.
  • after a restore all the filenames are in upper case only, the original had mixed case in the names. I have not looked at what might be causing this.
Some suggestions:
  • Some sort of in-progress status would be nice (as reyalp suggested), maybe a file counter.
  • The warning says not to use with cards 32GB or bigger; but for older cameras the limit is 4GB. Formatting a 4GB on the G12 changes it to FAT32 making it unbootable.
  • Change the instructions to tell users to exit with the ALT button, not the Menu button, and to restart from the Tools menu after formatting. Exiting with Menu unloads the module and if you format you can't get back into it.
Phil
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #7 on: 13 / April / 2020, 05:33:50 »
Quote from: philmoz on 13 / April / 2020, 03:34:45
Nice work.
Thanks.
Quote
fread & fwrite crash on the G12 with an assert in FsIoNotify. I suspect it may be related to this issue https://chdk.setepontos.com/index.php?topic=6179.0. Changing the code to use open/close/read/write fixes the crash.[/size]
Hmmm. I thought those FsIoNotify issues were over. Are other parts of CHDK using fopen (such as Lua) not affected? Logging file operations to camera log could give some hints.
Anyway, I'll make a wrapper so that both APIs can be used.
Quote
after a restore all the filenames are in upper case only, the original had mixed case in the names. I have not looked at what might be causing this.
I've never seen that so far. The current code always uses the built-in LFN support regardless of the menu setting, so it should get upper/lower case right. I wonder if the result on this cam depends on whether open() or fopen() was used.
Quote
Some sort of in-progress status would be nice (as reyalp suggested), maybe a file counter.
Already implemented here, will share soon.
Quote
The warning says not to use with cards 32GB or bigger; but for older cameras the limit is 4GB. Formatting a 4GB on the G12 changes it to FAT32 making it unbootable.
I added that warning.
Quote
Change the instructions to tell users to exit with the ALT button, not the Menu button, and to restart from the Tools menu after formatting. Exiting with Menu unloads the module and if you format you can't get back into it.
The module only ever exits if no backup was made. After that it stays loaded (regardless of exiting with MENU) and also keeps the exmem allocation - thus the instruction about restarting the camera.

edit:
typo
« Last Edit: 13 / April / 2020, 05:41:53 by srsa_4c »
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: Exmem
« Reply #8 on: 13 / April / 2020, 06:15:59 »
Quote from: srsa_4c on 13 / April / 2020, 05:33:50
Quote
fread & fwrite crash on the G12 with an assert in FsIoNotify. I suspect it may be related to this issue https://chdk.setepontos.com/index.php?topic=6179.0. Changing the code to use open/close/read/write fixes the crash.
Hmmm. I thought those FsIoNotify issues were over. Are other parts of CHDK using fopen (such as Lua) not affected? Logging file operations to camera log could give some hints.
Anyway, I'll make a wrapper so that both APIs can be used.
The open/close/read/write functions should work correctly on all cameras, why not just change the file save/restore code. Note sure why a wrapper is needed.
Quote
Quote
after a restore all the filenames are in upper case only, the original had mixed case in the names. I have not looked at what might be causing this.
I've never seen that so far. The current code always uses the built-in LFN support regardless of the menu setting, so it should get upper/lower case right. I wonder if the result on this cam depends on whether open() or fopen() was used.

The file browser on the G12 shows the wrong case for most filenames in the CHDK directory. Exceptions are files with names longer than 8 characters and two files in the LUALIB folder that have 6 character names; but still show mixed case  ???  Camera issue, not in your code.
Quote

Quote
Change the instructions to tell users to exit with the ALT button, not the Menu button, and to restart from the Tools menu after formatting. Exiting with Menu unloads the module and if you format you can't get back into it.
The module only ever exits if no backup was made. After that it stays loaded (regardless of exiting with MENU) and also keeps the exmem allocation - thus the instruction about restarting the camera.
Hmmm, the first time I tried to use it I used Menu to exit, formatted the card, then when I came back the module was unloaded so would not run. Have not been able to reproduce this since.


Phil.
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #9 on: 13 / April / 2020, 17:48:30 »
Quote from: philmoz on 13 / April / 2020, 06:15:59
The open/close/read/write functions should work correctly on all cameras, why not just change the file save/restore code. Note sure why a wrapper is needed.
Just in case something unexpected happens during file operations.
Quote
The file browser on the G12 shows the wrong case for most filenames in the CHDK directory. Exceptions are files with names longer than 8 characters and two files in the LUALIB folder that have 6 character names; but still show mixed case  ???  Camera issue, not in your code.
Just tried an a490 (DryOS r43), this one has problems with path length. It can't read the long named scripts in CHDK/SCRIPTS/TEST. No problems with filename upper/lowercase though.
VxWorks does not seem to like files that only have an extension (I had one such file in CHDK/DATA).

Work-in-progress version attached. I changed the exmem size calculation (the 64 byte extra was ignored previously which caused crashes on a490 due to over-reading physical memory). File reading failures are now noticed.
For files, open() is used by default which can be changed to fopen() by half pressing (blue asterisk in title line = fopen).

edit:
patch updated
« Last Edit: 14 / April / 2020, 18:36:04 by srsa_4c »
Logged
Pages: [1] 2 3   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal