How to write binary files? - page 2 - LUA Scripting - CHDK Forum  
« previous next »
Pages: 1 [2]   Go Down

How to write binary files?

  • 19 Replies
  • 16209 Views
*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: How to write binary files?
« Reply #10 on: 05 / February / 2009, 16:48:57 »
Advertisements
Quote from: mattkime on 05 / February / 2009, 16:25:01
>>100 kiB is a huge amount of RAM for a570is IMO.

I'm curious what that is based on. From your wiki page on memory it seems that there should be plenty of room to do everything and have more than 100k to spare. Anyway, i'm interested finding a good universal number and i'm willing to take suggestions.

Well...if the script is a standalone file mover not intended to be uses as a module, 100k is feasible unless it fragments (this size is similar to zebra's RAM requirement while it's being displayed, so a camera that doesn't crash with zebra should have 100k to spare).

But when starting with a clean (of jpegs) SD card, sunsetF15 starts with about 190k free RAM (that's from the top of my head, real value depends on CHDK build and settings) and it then starts decreasing due to file table increasing. I've already ran out of RAM more than once running that script. I think I've gotten about 7500 shots maximum. And due to RAM fragmentation, if I stop and start the script carelessly, it soon reports less than 80k free. Now imagine a710 which has 100k less RAM to begin with....

sunsetF15 is here as an example of a fairly complex Lua script of which I have usage experience from, but it actually could benefit from a file copying feature: the script desperately needs a mass JPEG rename (if that proves to be enough; if not files have to be moved) + reboot camera os + script state resume from autostart feature (you already more or less implemented that I believe) to clear the SD card from jpegs Canon firmware can find (to keep file table RAM allocation to minimum and to work around possible bugs in Canon firmware related to this).

I'd go with something like 8k or 32k as a default, but it all depends a whole lot on how much buf size affects the speed and how fast does garbage collection act (or how much does it slow it down if it's done after each block write).
Logged
*

Offline mattkime

  • ****
  • 299
Re: How to write binary files?
« Reply #11 on: 05 / February / 2009, 17:36:00 »
heh, i had a moment of panic when you said that the script runs out of memory if left running long enough. fortunately for me, the eye-fi card requires a reboot every 200 pics or so (taking a pic every five minutes) so that shouldn't cause a problem for me.

still, why does the memory usage steadily increase?

a big part of my refactoring of sunsetF15 was to localize as many variables as possible. perhaps there are negative consequences to this but its good programming practice.

mass rename would be easy enough. move is possible as well but thats really copy + delete.

do you know how effective garbage collection is at freeing memory?
Logged
*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: How to write binary files?
« Reply #12 on: 05 / February / 2009, 17:47:18 »
Quote from: mattkime on 05 / February / 2009, 17:36:00
still, why does the memory usage steadily increase?

http://chdk.wikia.com/wiki/CHDK/Camera_RAM_memory_usage#How_to_make_Canon_firmware_use_less_RAM

Quote from: mattkime on 05 / February / 2009, 17:36:00
mass rename would be easy enough. move is possible as well but thats really copy + delete.
Yep... but reboot is a prerequisite for either to help me (just deleting images doesn't remove them from the camera's table as we know; play mode gets confused and shows that broken pic thumbnail for those).

Quote from: mattkime on 05 / February / 2009, 17:36:00
do you know how effective garbage collection is at freeing memory?
No.
Logged
*

Offline mattkime

  • ****
  • 299
Re: How to write binary files?
« Reply #13 on: 05 / February / 2009, 17:52:35 »
>>Yep... but reboot is a prerequisite for either to help me

I'm rebooting by using an arduino to control the power. i connected wires to the power button on the camera. this is very easy to do on the 570 and similar cameras.

its possible if you're willing to modify your camera.
Logged
*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: How to write binary files?
« Reply #14 on: 05 / February / 2009, 18:18:27 »
Quote from: mattkime on 05 / February / 2009, 17:52:35
I'm rebooting by using an arduino to control the power. i connected wires to the power button on the camera. this is very easy to do on the 570 and similar cameras.

its possible if you're willing to modify your camera.

Will have to keep that in mind...
Logged
*

Offline reyalp

  • ******
  • 14117
Re: How to write binary files?
« Reply #15 on: 05 / February / 2009, 18:53:17 »
Quote from: mattkime on 05 / February / 2009, 07:44:20
oh, one annoyance - files created by lua don't get their date/time set correctly.
use os.utime http://chdk.wikia.com/wiki/LUA#LUA_standard_libraries:_io.2C_os_and_string and os.date

I suspect the reason the firmware doesn't do this by default is for efficiency, since it requires an extra sector write.

As for reading/writing large blocks, I'd say there's only about 4 values worth considering:
1) 1 byte at a time. Very slow, but no memory issues because there are only 256 unique 1 byte strings
2) 1 hardware sector at a time (512 bytes AFAIK)
3) 1 FAT sector at a time (variable, 64K or less)
4) 32 KB (size of Fut IO buffer)
Beyond the last option, I doubt you will see any increase in performance. If the performance between 2) and 4) is equivalent, I'd stick with 2.

Quote
a big part of my refactoring of sunsetF15 was to localize as many variables as possible. perhaps there are negative consequences to this but its good programming practice.
Making variable local does not force the values it referenced to be collected when the variables go out of scope. The garbage collector acts on values, not variables. Using locals is fine, but if you want to understand memory usage, you have to understand how the garbage collector operates.

As a start, I'd suggest logging the "count" value from collectgarbage. Then try forcing a full collection, and logging the new count value and free RAM. If actual memory usage is still going up, there is a problem. Of course, you will probably get quite a bit of fragmentation from long running lua scripts.
Logged
Don't forget what the H stands for.
*

Offline mattkime

  • ****
  • 299
Re: How to write binary files?
« Reply #16 on: 05 / February / 2009, 21:08:54 »
my tests show that accessing between 2k and 16k of data results in roughly the same copy speed with a slight edge to 4k. 4k was about 15% faster than 512 bytes.

can you explain more about the garbage collector? i have a beginner's general understanding of them. i know that declaring vars as local doesn't collect them, but it should help make them available for collection. does lua garbage collect on its own or does it need to be told?
Logged
*

Offline reyalp

  • ******
  • 14117
Re: How to write binary files?
« Reply #17 on: 05 / February / 2009, 23:48:50 »
2.10 - Garbage Collection

The important thing to remember is that collectible values (strings, tables, functions) are what gets allocated and collected. Variables just refer to values, and values are collectible when no variables refer to them. So
Code: [Select]
x = "foo" -- x refers to the string value "foo"
x = "bar" -- x refers to the string value "bar", the value "foo" is no longer referenced and can be collected.
Another thing to remember is that strings are unique and immutable. So there is only ever one instance of the string "foo" in your lua program. You cannot modify a string: you can only a make a new string which is a modification of the old one. You can see that if you copy a file by reading/writing strings, the number of unique strings could pile up quite quickly, despite all but one of them being collectible at a given time.

Generally, garbage collection should be automatic, but it is possible to run out of memory due to uncollected objects. The lua garbage collector doesn't know how much memory is available, and doesn't attempt to force a collection when a malloc fails*.

As I suggested before, monitor usage with collectgarbage("count"), and tune the GC parameters or force collection as needed.

Useful links http://lua-users.org/wiki/GarbageCollection http://lua-users.org/wiki/GarbageCollectionTutorial

* This feature is available in the http://lua-users.org/wiki/EmergencyGarbageCollector patch, and may also be implemented in future versions of lua. Might be a useful addition to CHDK lua.
Logged
Don't forget what the H stands for.
*

Offline mattkime

  • ****
  • 299
Re: How to write binary files?
« Reply #18 on: 06 / February / 2009, 00:44:13 »
>>use os.utime http://chdk.wikia.com/wiki/LUA#LUA_standard_libraries:_io.2C_os_and_string and os.date

that fixes the modified date but not the creation date, for what its worth.
Logged
*

Offline reyalp

  • ******
  • 14117
Re: How to write binary files?
« Reply #19 on: 06 / February / 2009, 02:37:56 »
Ah so it does. If you actually need the creation time for some reason, you are SOL.
Logged
Don't forget what the H stands for.
Pages: 1 [2]   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal