Exmem - page 2 - General Discussion and Assistance - CHDK Forum
« previous next »
Pages: 1 [2] 3   Go Down

Exmem

  • 26 Replies
  • 8164 Views
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #10 on: 18 / April / 2020, 18:57:53 »
Advertisements
I have attached the diff for the 2 modules I'm about to add.
Some parts of the backup/restore module have been rewritten.
The current version of these modules has been tested successfully on all my cameras ranging from Ixus40 to M100.
If anyone has issues or suggestions, I'm listening.
Logged
*

Offline reyalp

  • ******
  • 14110
Re: Exmem
« Reply #11 on: 18 / April / 2020, 20:40:30 »
Quote from: srsa_4c on 13 / April / 2020, 05:33:50
Quote
fread & fwrite crash on the G12 with an assert in FsIoNotify. I suspect it may be related to this issue https://chdk.setepontos.com/index.php?topic=6179.0. Changing the code to use open/close/read/write fixes the crash.[/size]
Hmmm. I thought those FsIoNotify issues were over. Are other parts of CHDK using fopen (such as Lua) not affected? Logging file operations to camera log could give some hints.
Yes, this seems like it could use more interrogation, because AFAIK fopen etc *should* work.

Looking at the history, the related changes were mostly around 3392, 3409, 3760

open etc have their own problems with cached memory on some platforms.

edit:
The above isn't an objection to checking in the modules. I'm fine with having them checked in.

I tested backup/restore on a elph130, a540, sx160 without issue.

A couple features that would be nice:
Restore file timestamps. It looks like it's saved, but not restored
Warn in the status message area on the >2gb and >32gb cases (but I'm not sure we have a way to know from code which applies)
« Last Edit: 18 / April / 2020, 23:49:19 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #12 on: 19 / April / 2020, 17:34:20 »
Quote from: reyalp on 18 / April / 2020, 20:40:30
open etc have their own problems with cached memory on some platforms.
Do you have links to related posts/discussions?
Quote
A couple features that would be nice:
Restore file timestamps. It looks like it's saved, but not restored
Yes, planned feature, but no implementation yet.
Quote
Warn in the status message area on the >2gb and >32gb cases (but I'm not sure we have a way to know from code which applies)
Might be possible to compile lists of platformIDs, I guess.

Quote
The above isn't an objection to checking in the modules. I'm fine with having them checked in.
I was about to, but...
It turned out (and I should have noticed that earlier), on most VxWorks models there is no cached version of exmem_alloc. That was introduced around the last Vx generation. So, what's currently identified as exmem_alloc returns uncached address on old cameras and a cached one on newer cams. The difference between cached and uncached routines is that the cached exmem_free flushes the data cache, plus the allocation has that 32+32 bytes overhead.

So, we could either leave everything as is now and expect both kind of exmem. Or, we could switch to the uncached version of exmem_alloc/free - but in that case the cached exmem_alloc must remain as it's used in OPT_EXMEM_MALLOC ports. I say "we", but currently I'm the only one who has use for exmem allocations.

I'll probably need to review those older Vx routines.
Logged
*

Offline reyalp

  • ******
  • 14110
Re: Exmem
« Reply #13 on: 19 / April / 2020, 17:57:40 »
Quote from: srsa_4c on 19 / April / 2020, 17:34:20
Do you have links to related posts/discussions?
IIRC, that's the reason the module code has the b_ wrappers. Also https://chdk.setepontos.com/index.php?topic=6261.0

edit: Another early thread about that, with some specific models mentioned https://chdk.setepontos.com/index.php?topic=2483.0

A post-write CRC check might be a good idea, though obviously would slow things down add significant complexity.

Quote
Quote
Warn in the status message area on the >2gb and >32gb cases (but I'm not sure we have a way to know from code which applies)
Might be possible to compile lists of platformIDs, I guess.
That seems messy, unless there's a clean cutoff  like PID > X boots from FAT32) but I suspect there isn't. CAM_MULTIPART might be a proxy, since the cameras just before the transition should have it, but look out for very old ones that don't support SDHC won't.

Not a big deal in any case.
« Last Edit: 19 / April / 2020, 18:13:49 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #14 on: 25 / April / 2020, 20:33:12 »
New patch with significant change in exmem stubs.
It turned out that current vx exmem_alloc stubs were actually what I called exmem_ualloc (allocating uncached allocation). The same named DryOS stubs are okay. As old vx models (12 of them) don't have a cached version of exmem_alloc, I decided to switch the CHDK exmem_alloc and exmem_free to use the uncached stubs.
The patch
- removes exmem_free from stubs_entry.S
- adds exmem_ualloc and exmem_ufree to stubs_entry.S
- adds exmem_ualloc and exmem_ufree stubs to the 3 newest models manually
- makes the CHDK core version of exmem_alloc and exmem_free use the uncached fw functions
- keeps the cached exmem_alloc stubs as they are used by OPT_EXMEM_MALLOC code

So, the exmem allocator now consistently returns an uncached allocation on all models.
I changed the backup/restore module accordingly.
Also removed the card related warnings from help text and made them red warnings that show up on first launch of the module when needed.

I have not done extensive testing yet.

Quote from: reyalp on 19 / April / 2020, 17:57:40
edit: Another early thread about that, with some specific models mentioned https://chdk.setepontos.com/index.php?topic=2483.0
Models mentioned there happen to be cams without cached exmem_alloc, interesting coincidence.
Quote
Warn in the status message area on the >2gb and >32gb cases (but I'm not sure we have a way to know from code which applies)
r47 cameras have P-ID above 0x3222, so this is implemented (but untested).
Logged
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #15 on: 02 / May / 2020, 17:53:16 »
The latest version of the backup/restore module checks the crc of all restored files (by reading them back from card). It also restores the file modification times.

I have done a batch build but did not re-check all my cameras yet.

I'd like to get the changes in, but I have some concerns.
One is that the new exmem allocator allocates uncached memory, purely because of those early Vx models that have no cached version.
Another is that that I'm still naming the new CHDK functions exmem_alloc and exmem_free - that is, the names don't indicate that the memory is uncached. Should they?
Logged
*

Offline reyalp

  • ******
  • 14110
Re: Exmem
« Reply #16 on: 02 / May / 2020, 18:31:55 »
Quote from: srsa_4c on 02 / May / 2020, 17:53:16
One is that the new exmem allocator allocates uncached memory, purely because of those early Vx models that have no cached version.
IMO, uncached is desirable if using open rather than fopen. If it had an unacceptable impact on performance, then it would need some additional buffer for read/write (like modules) but I expect it's not a concern.
Quote
Another is that that I'm still naming the new CHDK functions exmem_alloc and exmem_free - that is, the names don't indicate that the memory is uncached. Should they?
Seems like it might be a little clearer to call them exmem_ualloc and exmem_ufree instead (edit: or _uncached to follow the same pattern as exmem_alloc_cached).
« Last Edit: 02 / May / 2020, 19:00:56 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #17 on: 02 / May / 2020, 20:07:14 »
Quote from: reyalp on 02 / May / 2020, 18:31:55
Seems like it might be a little clearer to call them exmem_ualloc and exmem_ufree instead (edit: or _uncached to follow the same pattern as exmem_alloc_cached).
Used _uncached in this patch. It would then be the "final" one.

The allocated memory could still be used as the application requires, but it's up to the caller to use the appropriate cache operation(s) and safety margins.
The module IMO still has an acceptable speed with uncached memory.
Logged
*

Offline reyalp

  • ******
  • 14110
Re: Exmem
« Reply #18 on: 02 / May / 2020, 20:27:45 »
Quote from: srsa_4c on 02 / May / 2020, 20:07:14
The allocated memory could still be used as the application requires, but it's up to the caller to use the appropriate cache operation(s) and safety margins.

The module IMO still has an acceptable speed with uncached memory.
Agree with all.

Tested on the previous version:
elph130, 32GB FAT32 card, backup / restore OK
D10, 4GB FAT16 card, warns correctly autoboot will be lost
SX710, 64GB FAT32, warns correctly autoboot will be lost
SX710,  32GB FAT32, backup / restore OK
A540, 2GB FAT16, backup / restore OK
Logged
Don't forget what the H stands for.
*

Offline srsa_4c

  • ******
  • 4451
Re: Exmem
« Reply #19 on: 03 / May / 2020, 14:38:35 »
Quote from: reyalp on 02 / May / 2020, 20:27:45
Tested on the previous version:
elph130, 32GB FAT32 card, backup / restore OK
D10, 4GB FAT16 card, warns correctly autoboot will be lost
SX710, 64GB FAT32, warns correctly autoboot will be lost
SX710,  32GB FAT32, backup / restore OK
A540, 2GB FAT16, backup / restore OK
Thanks.
I checked in every change except for the two modules, r5481...5483. I'll probably add those too, soon.
Logged
Pages: 1 [2] 3   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal