preliminary lua iolib and oslib port - General Discussion and Assistance - CHDK Forum

preliminary lua iolib and oslib port

  • 36 Replies
  • 22585 Views
*

Offline reyalp

  • ******
  • 14077
preliminary lua iolib and oslib port
« on: 09 / September / 2008, 23:51:42 »
Advertisements
I've got the lua iolib mostly ported. This lets you read and write files from lua. The official iolib documentation is here: Lua 5.1 Reference Manual
This is not finished. Currently, a540 is only supported camera because I haven't done the gensig stuff yet, and it requires several additional functions be exposed. Thanks to fudgeys docs getting the sigs for the other cameras hopefully won't take me too long.

I had to reverse engineer the Fut FILE structure, because as far as I can tell there's no ftell that is compatible with Fut. If someone knows different, let me know.

I also made wrappers that behave more like stdio than the normal CHDK Fut wrappers (take a FILE * instead of an int, return values when ANSI expects them to). If I'm not missing something, the rest of CHDK should probably just use this, but for the moment everything outside of lua uses the old way.

My port leaves out things that are meaningless or unavailable in CHDK
Missing features:
- popen: no other processes, no pipes. Someone can revisit this when ELF is done ;)
- standard file handles: stdin/stdout/stderr all start out closed, as do io.input() and io.output().
- tmpfile isn't implemented yet, but should be possible.
Other differences:
- Error detection and reporting is more limited: no errno, no ferror, no strerror. errno and strerror support may be possible, but it's not clear to what extent the Fut API actually sets it. ferror doesn't appear to exist for Fut ?
- file:read("*n") will behave slightly differently because it doesn't use scanf. Shouldn't matter for sanely structured files, but corner cases will be different.
Added:
- a new file method file:_getfptr() which returns the actual FILE * associated with the file object. This is strictly for development/debugging, but combined with peek makes verifying FILE structure hack much simpler.

Patch (against juciphox) is attached, along with a lua script I used for testing/debuging.

This adds a noticeable amount to CHDK image size, but it's worth it IMO.

os and string are next :)
« Last Edit: 19 / September / 2008, 03:34:23 by reyalp »
Don't forget what the H stands for.

*

Offline PhyrePhoX

  • *****
  • 2254
  • make RAW not WAR
    • PhyreWorX
Re: preliminary lua iolib port
« Reply #1 on: 11 / September / 2008, 08:55:56 »
congratz on such a nice feature. perhaps you should also add some possible uses of it, i guess not all the people get what iolib is (well they could read the link you provided, but still woulnt know a practical reason).
so, how can one help ya finding the adresses for the other cams?

*

Offline reyalp

  • ******
  • 14077
Re: preliminary lua iolib port
« Reply #2 on: 11 / September / 2008, 16:04:26 »
congratz on such a nice feature. perhaps you should also add some possible uses of it, i guess not all the people get what iolib is (well they could read the link you provided, but still woulnt know a practical reason).
so, how can one help ya finding the adresses for the other cams?
I'm working on getting gensig/finsig going, which should find most of them automatically.

As far as uses, it means you can read, write, or modify any file from lua. This would allow this: Instant video uploads using CHDK and Eye-fi to be done from script, and also offer a much more flexible alternative to the print_screen stuff. Want to write a csv file that logs all the settings used for every exposure ? Easy. Combined with peek, you could also save any part of camera memory to disk (the live view or raw buffer even ;)) If you enable poke, you could write to them too. This has both interesting creative applications as well as being useful for development.

Various forms of file access have been requested for script many times, this solves almost all of them in a flexible, well documented way.

Once I get oslib ported (it includes remove and rename) you'll be able to do stuff like this Deleting images with interval shooting from lua too. That will also allow you to do all kinds file management from script.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14077
Re: preliminary lua iolib port
« Reply #3 on: 12 / September / 2008, 03:14:09 »
Here is a patch that gets the required entry points from gensig/finsig. This replaces the earlier patch. This compiles for all the cameras that I have dumps for (everything in juciphox except a450 100d and tx1 100g) but I certainly haven't checked the disassembly on all of them. The three Fut functions should detect pretty reliably.

It also isn't certain that the FILE structure is the same on all cameras, although I have verified that it is compatible in the reference firmwares. Given that it's the same on dryos and vxworks for several different cameras, it's probably OK.

Any exceptions to either of the above will become obvious whens someone tries it.

I can provide binaries for any of the above mentioned cams.

I've also attached a patch of the generated files. Note that
- some of the checked in stubs_entry.S have entry points that aren't in the checked in sig_ref_*.txt. Unless you define CAM_CONSOLE_LOG_ENABLED none of the missing entry points seem to be required.
- some of the checked in stubs_entry.S are empty
- At least one is in a different newline format.
Don't forget what the H stands for.


*

Offline reyalp

  • ******
  • 14077
Re: preliminary lua iolib and oslib port checked in
« Reply #4 on: 19 / September / 2008, 03:29:47 »
Checked in lua io and os libraries port on the trunk, changeset 517. See the lua documentation for details of how to use these libraries, but be sure to read below for important differences, limitations and warnings.
WARNING:
These features are experimental and untested on most cameras. They might corrupt your memory card, causing all data to be lost. Even if they work correctly on your camera, incorrect use may cause data loss or filesystem corruption.
.
Note this is only a risk if you try to use these features from lua. Just having them in your build doesn't pose a risk.

These are compile time optional. If you don't want to build them, see root makefile.inc. Together, they add about 4kb to the memiso useage.

Test script:
llibtst.lua added to /script in svn, and also attached to this post.

If the libraries have not already been tested on your camera and firmware version, you should run the script before attempting to use them.

This script tests the various io and os functions, generating A/llibtst.log and reporting pass or fail in the console. If not all tests are run, or there are failures, it may leave other files or directories in the root. Failures could crash your camera, or potentially corrupt your SD card. Be sure you don't have anything wouldn't mind losing before you run it.

The pass/fail in the console only provides a general indication. The logfile should be examined to be sure everything is working. Please post the log if your camera/firmware has not been tested already. An example log from my camera is attached.

The log is formatted such that most lines end with OK or ERR followed by possible return values, and then PASS, FAIL, or NA
PASS or FAIL indicate whether the test recieved the result it expected.  OK and ERR indicate whether the function being tested reported success or failure, which may represent a PASS or FAIL depending on circumstances.
NA means the script doesn't count the result of that particular call as a pass or fail, because the results are variable or not meaningful.

The script also serves as an example of how to use the libraries.

Differences from standard lua iolib:
Missing features:
- popen: CHDK has no other processes, no pipes. Someone can revisit this when ELF is done ;)
- standard file handles: stdin/stdout/stderr all start out closed, as do io.input() and io.output().
- tmpfile isn't implemented yet, but may be added later.
Other differences:
- Error detection and reporting is more limited:
  vxworks reports errno, but no text description
  dryos has no errno (that I was able to find), and simply reports "error"
- file:read("*n") will behave slightly differently because it doesn't use scanf. Shouldn't matter for sanely structured files, but corner cases will be different.
Added:
- a new file method file:_getfptr() which returns the actual FILE * associated with the file object. This is strictly for development/debugging, but combined with peek makes verifying FILE structure hack much simpler.

Differences from standard lua oslib:
Missing features:
- os.clock: not implemented. Note this relates to CPU time used by a process, not time of day (which is handled by os.time and os.date)
- os.execute: no processes, revisit for ELF
- os.exit: not meaningful for CHDK
- os.getenv: environment variables exist in vxworks, but usefulness is doubtful
- os.setlocale: functions may exist, but it's unlikely the rest of the firmware uses them
- os.tmpname: TODO, not implemented yet
Added features:
- os.mkdir("name"): not standard, but useful
Other differences:
- Similar error reporting limitations to io
- os.date doesn't distinguish between local and GMT. It's possible support could be added for some cameras
- os.difftime just returns t1 - t2
Still to do:
- Add methods to list directories and stat files.


General notes from vxworks testing:
os.remove works on files and empty directories. Attempting to remove a non-empty directory fails gracefully.
os.rename works on files and empty directories. Attempting to rename a non-empty directory may result in filesystem corruption!.
os.rename only works within a given directory. You can not use rename to move a file to another directory. Attempting to do so may cause unpredictable results.
os.rename may produce unpredictable results if the target name already exists.
os.mkdir will not create a directory if the name has a trailing /, but fails gracefully
io functions are very likely limited to files of less that two gigabytes, and certainly less than four.
unlike some OSes, file:seek cannot be used to extend a file

Future versions should attempt to handle some of these situations more gracefully. Dryos may behave differently.


New entry points used, all added to required sig_ref_*.txt
DeleteFile_Fut
Feof_Fut
Fflush_Fut
Fgets_Fut
MakeDirectory_Fut
RenameFile_Fut
errnoOfTaskGet - vxworks only, stubbed in wrappers.c for dryos
mktime - vworks only, version in wrappers.c does the right thing for each OS
strftime
mktime_ext - dryos only. Not called mktime because it doesn't behave quite like mktime

Although they all get good matches from finsig, these entry points are only verified on a few cameras. Running the test script should quickly verify each camera.

Added two #ifdef controlled options to stdlib.h
STDLIB_COMPAT_FILE - cause fopen et al use a FILE* instead of an int, and generally behave like C stdio. Also adds reverse engineered FILE structure and ftell
FS_USE_FUT - cause rename, mkdir and remove to use Fut.
Both of these can be defined on a per file basis at compile time, and are currently used for lua. In the future we may want to make this the default for the rest of CHDK

Unsupported cameras:
a450/100d
tx1/100g
I don't have dumps for these cameras, and so cannot support them. New entry points are stubbed to nullstub in stubs_entry_2.S
Attempting to actually use these stubs will probably result in crashes.
lua libs are #ifdef'd out for these, see each platforms makefile.inc

Miscelanious changes:
Some stubs_entry.S had entry points not in any of the sig_ref_*.txt files. These were not used in the code, and so are removed by this checkin: fprintf fputs puts

Added Remove to sig_ref_vxworks_3.txt. This causes the function to be incorrectly detected on some cameras (with a low match %) but all those cams already have it in stubs_entry_2.S
Added iosDevFind to sig_ref_vxworks_1.txt, not required for libs but existed in some old stubs_entry.S, used if CAM_CONSOLE_LOG_ENABLED is defined

Replace the empty stubs_entry.S for s5is with the one generated in my build. All entrypoints are unverified.

Fixed ROMBASEADDR for ixus65_sd630 100a. Now 0xff810000. If your dump has leading zeros, you'll need to fix it.

« Last Edit: 19 / September / 2008, 03:34:53 by reyalp »
Don't forget what the H stands for.

*

Offline Jucifer

  • *****
  • 251
  • [A710IS]
Re: preliminary lua iolib and oslib port
« Reply #5 on: 19 / September / 2008, 04:13:37 »
a710 tested, seems to work as a540

*

Offline msl

  • *****
  • 1280
  • A720 IS, SX220 HS 1.01a
    • CHDK-DE links
Re: preliminary lua iolib and oslib port
« Reply #6 on: 19 / September / 2008, 05:22:13 »
It seems also to work for the a720.

Thanks for the great job.
CHDK-DE:  CHDK-DE links

*

Offline fe50

  • ******
  • 3147
  • IXUS50 & 860, SX10 Star WARs-Star RAWs
    • fe50
Re: preliminary lua iolib and oslib port
« Reply #7 on: 19 / September / 2008, 06:22:01 »
#518 successfully tested on SD870 1.00C.
Thx reyalp, it's like a very early xmas gift  :xmas ++
Now we have almost endless possibilities with lua scripting (e.g. self-modifying code  :o) - soon we need a virus scanner in CHDK to verify whether a script does contain "unwanted functionality"  8)


*

Offline PhyrePhoX

  • *****
  • 2254
  • make RAW not WAR
    • PhyreWorX
Re: preliminary lua iolib and oslib port
« Reply #8 on: 19 / September / 2008, 08:10:19 »
Darn,i'm in the middle of nowhere,cant test this right now on my cams :( will do so next week.
Btw reyalp,please update version.inc and version.txt when u committ something,so the autobuilds reflect the changes,no?
« Last Edit: 19 / September / 2008, 08:16:54 by PhyrePhoX »

*

Offline reyalp

  • ******
  • 14077
Re: preliminary lua iolib and oslib port
« Reply #9 on: 19 / September / 2008, 14:30:43 »
#518 successfully tested on SD870 1.00C.
soon we need a virus scanner in CHDK to verify whether a script does contain "unwanted functionality"  8)
lua keylogger, see attached :D (unfortunately, lua only sees key presses when you are in alt mode)

Btw reyalp,please update version.inc and version.txt when u committ something,so the autobuilds reflect the changes,no?
I'm a noob  :( Would be nice to have the build number updated automatically somehow.
Don't forget what the H stands for.

 

Related Topics