we already have universal dumper... why dont we have a universal loader?

PhyrePhoX

2254
make RAW not WAR

« on: 02 / July / 2008, 19:56:58 »

Advertisements

i don't know if it is possible, but here's the deal:

the universal dumper is pretty good at "guessing things" and dump the firmware.
a universal loader doesnt need to guess since it only needs to load firmware of cameras we already ported.
what i'm trying to say is: is there any technical reason that we cant have an SDcard crammed full with all the different chdk versions for different cameras. the loader then checks which cam it is running on, then moves the right binary to rootfolder, reboots the cam - done. right?
so i can have one sdcard to fire off chdk on all powershots i come across (if they have been ported of course).
forgive my ignorance, maybe it is impossible.

anyhow, as soon as the ELF edition is done, i guess something like a universal loader will be the future... at least in my imagination

Logged

Flickr*
www.PhyreWorX.de
CHDK News @ Twitter

cyril42e

111
SD1000/Ixus70 1.02a

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #1 on: 03 / July / 2008, 16:04:17 »

Well, I never have dumped any firmware, so I may be wrong, but some suppositions:
- technically, is it possible to move primary.bin while it is in use? it doesn't seem possible using CHDK filebrowser

- cons: from what I've understood the camera already have to reboot to load the loader/chdk, so loading chdk afterward would need a second reboot. Time to copy primary.bin + time to reboot = significant additional boot time

. And more place on the card to store all firmwares.
- pros are very weak I think, everyone knows what camera model they have, and finding the firmware version is not that hard. Well, ok, firmware independant CHDK would be easier to use. I don't know how much different firmware version differ, maybe it could be possible to detect it at the beginning, and then executing different code blocks depending on that... But I'm still not sure it is worth the pain...

Logged

PhyrePhoX

2254
make RAW not WAR

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #2 on: 28 / October / 2008, 20:40:11 »

reviving, had a chat in irc

Quote

[01:17:15] morris: The only project I have thought about doing for CHDK is looking for a way to do a single version that sniffs the camera version, does a table lookup to find the right routine intercepts, and has all code dfiferences using runtime flags rather than compiler flags.
[01:17:31] morris: Sound feasible?
[01:18:04] morris: i.e. even if the version is fatter, it would be a whole lot easier to get people to get hooked in it.
[01:18:19] PhoXy: morris, well, this sounds feasible, and it already is a feature request (i had the same idea), however it seems to be difficult
[01:18:41] Hacki: and i supposed it would slow down the boot process quite a bit
[01:19:18] morris: BTW: I am a developer (I have even done embedded programming) so it is just a matter of finding the time to hack, rather than lacking the skills.
[01:19:33] Hacki: cool.
[01:19:45] PhoXy: i am looking for the original thread or request i made just now
[01:19:47] morris: PhoXy: what was the difficulty?
[01:19:51] Hacki: really. experienced devs is exactly what chdk needs
[01:20:39] PhoXy: difficulty? a) i dont speak asm. i dont even speak C if it boils down to effectiveness and sluggishness b) it seems to be difficult & tedious
[01:20:56] morris: The problem is that expeienced developers also tend to do it during the day (or I should be LOL) so perhaps don't have the same desire to do dev after work.
[01:21:45] Hacki: probably
[01:21:53] Hacki: we need an experienced unemployed developer
[01:22:13] morris: Yeah Go global meltdown! GO!
[01:23:04] PhoXy: lol
[01:23:50] morris: PhoXy - Hmmm - a) I did some 6502 asm as a kid b) I have done work in C, c) boring tedious work is the crux -- sounds too much like my job ;(
[01:24:15] PhoXy: found the thread: we already have universal dumper... why dont we have a universal loader? didnt raise a lot of attention though
[01:25:08] PhoXy: btw it wouldnt even be needed to include all adresses in ONE binary
[01:25:15] PhoXy: like in my request in that thread
[01:25:33] PhoXy: we just need a small program that can read/write/move files and set the boot flasg
[01:25:35] PhoXy: flag
[01:26:52] PhoXy: "universal loader" boots, finds the "camera info string + firmware version string", then moves diskboot.bin & ps.fir from for example "bins\s3is\101a" (locations are in the table) to the root of the card, sets boot flag - reboots
[01:27:26] morris: Sounds like a reasonable solution...
[01:27:27] PhoXy: the udumper already can write to the sdcard, so i guess it should be possible, more than feasible
[01:28:03] PhoXy: also it sounds extremely un-tedious and very exciting, the more i think about it ^^
[01:28:04] pixeldoc2000: at least this will took some space on your memory card to have a bunch of chdk versions on it...
[01:28:40] PhoXy: and it would work only once.
[01:28:55] PhoXy: but this may be overcome
[01:29:14] PhoXy: by a chdk option like "set card to uloader"
[01:29:56] pixeldoc2000: maybe a nice setup or gui to choose and install chdk on the memory card would be a better "solution"
[01:30:28] pixeldoc2000: like to extend cardtricks...
[01:30:45] PhoXy: of course what might be the better solution would be to only limit the "universal" part to certain firmware versions, not different camera hardwares. meaning there would be a universal bin for the g7 for example ( i think 4 firmware versions are known)
[01:31:12] pixeldoc2000: know thats a better idea
[01:31:35] PhoXy: this would rid us of the need to do the vers.req thing, i mean, the people know what cameras they have
[01:31:48] morris: Without looking at the technical details of each of the ideas it is pretty difficult to know which is easiest, or which is best.
[01:33:09] PhoXy: mind if i add this conversation to the thread so it doesnt get lost and also revives it, making room for other devs to join in on the fun?
[01:33:17] pixeldoc2000: the question is, if its easy to determ firmware version early at the boot process
[01:33:17] morris: PhoXy : about space: 500kb and 50 versions is only 25MB - so I would ignore that issue.
[01:33:39] morris: It should be possible to sniff it safely.
[01:34:35] PhoXy: btw morris perhaps you can invest time & brains in the ELF loader? any experience regarding that?
[01:34:47] PhoXy: this covers almost the same subject
[01:34:54] pixeldoc2000: yes, stick it to the post!
[01:35:03] PhoXy: as with the elf loader we would have much more room for the uloader
[01:35:09] morris: pixeldoc2000 We know the universal dumper can safely access ROM memory, so it can search (or sniff) the version within that.
[01:35:20] morris: Or even jsut use an MD5 hash of the memory.
[01:35:41] morris: I don't know anything about the "ELF" loader feature.
[01:36:09] morris: Sure, stick it in the post.
[01:36:17] pixeldoc2000: but md5 will take some time on "limit" hardware... but could be an option
[01:36:58] morris: Well, you probably only need to md5 every 100th byte..
[01:37:00] PhoXy: new branch - CHDK : Elf Edition - Developers wanted ELF loader is one of the most needed features of chdk
[01:37:23] PhoXy: basically if this thing runs, we will never have any space or memory restrictions
[01:37:42] morris: Got you.
[01:37:58] pixeldoc2000: morris: maybe this would be enough
[01:39:17] morris: I was wondering about a loadable libraries feature too!

Logged

Flickr*
www.PhyreWorX.de
CHDK News @ Twitter

reyalp

14079

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #3 on: 28 / October / 2008, 22:52:17 »

This is another way of saying automate the porting process. We can't yet do that offline on a PC, which has gobs of CPU, memory and all kinds of nice tools available. Doing it on the camera, without even the OS functions will be much harder.

We do have an automated way of finding entry points. It's called finsig. It is frequently wrong, and there are quite a few types of things that it cannot find. You could write more sophisticated tools, but we are currently a long way off.

Remember, udumper basically trashes the OS. Starting from zero, finding everything you need, and then booting the camera would be very difficult, and not very efficient.

Writing better automation for the porting process is definitely a good idea. If you ever get to the point where you can just put in a dump, and it spits out a working port, then you could consider doing it on the camera.

Logged

Don't forget what the H stands for.

robocat

33

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #4 on: 29 / October / 2008, 00:55:48 »

reyalp, the suggestion was not to automate entry point detection for unknown cameras.

The suggestion was to detect/sniff existing camera versions (e.g. search firmware for version, or perhaps make a hash, hash using every 100th byte if speed an issue), and then do one of:

a) rename/copy/vector the appropriate CHDK build for the firmware from a complete list on the card and reboot into it (PhyrePhoX suggestion), or

b) have a table of entry points for all the known camera firmware versions, and have a single CHDK build that dynamically configures itself to use the appropriate entry points from the table for the camera the card is placed in. The CHDK code would also need to use normal control statements wherever code is currently statically #ifdef'd (conditionally compiled)

c) perhaps limit "autoversioning" by some sensible subsets (by OS, by model, by family, by Digic processor?)

d) any other solutions?

The reason I was interested in this feature is that I think it would significantly increase the uptake of the CHDK firmware, and would lead to getting more users and developers - which means more features for everyone. Each extra step required for people to get up and running reduces usage significantly. A single build that could be preloaded on SD cards and given to anyone to use would be fantastic!!!

Even if it could jsut remove the need for different builds for different versions of firmware on the same camera, that would remove a barrier to people using CHDK.

I am sure the existing "optimised" builds for specific cameras would still be kept.

All up, I am interested in any known technical issues of doing this (Although perhaps I should just get stuck in and find out!) or if there are any other smart solutions to the issue. e.g. are there differences in CPU targets, base object code offsets?

Morris

« Last Edit: 29 / October / 2008, 01:05:04 by robocat »

Logged

reyalp

14079

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #5 on: 29 / October / 2008, 02:51:19 »

OK, that sounds more possible. I'm still not sure I really get the point.

Is it just so people don't have to figure out which diskboot.bin to put on their camera ? It may sound harsh, but I'm not sure that the people who can't figure how to use vers.req and download the right file are going to benefit much from CHDK. Things are currently confusing to new users because the documentation is a mess, but the actual process is reasonably simple.

That said, figuring out which camera you are on shouldn't be too hard (there's strings you could look for, or you could CRC/hash some of the ROM), but at that point you won't have enough of an OS to actually load it. I guess you could store the minimal set of offsets required for every known camera.

Quote

a) rename/copy/vector the appropriate CHDK build for the firmware from a complete list on the card and reboot into it (PhyrePhoX suggestion), or

Possible. IMO, it would be much more desirable to have this act as an installer rather than run every time the camera boots. In other words, you copy something to your SD card, the first time it loads, it figures out what camera you have, and replaces itself with the correct diskboot.bin. If you were clever, you could have it do this every time the camera changes.

Still, the whole idea of putting every build of CHDK on the card just strikes me a giant kludge.

Quote

b) have a table of entry points for all the known camera firmware versions, and have a single CHDK build that dynamically configures itself to use the appropriate entry points from the table for the camera the card is placed in. The CHDK code would also need to use normal control statements wherever code is currently statically #ifdef'd (conditionally compiled)

There's a lot more to CHDK than just entry points. Large parts of the canon firmware are disassembled and modified. You'd have to do the same process at runtime, or have loadable objects for those files. Moving all the #ifdefs and and #defines to variables would negatively impact the memory footprint at the very least.

If you want to take this idea further, you'll definitely have to dig into the CHDK code. As I've expressed above, it strikes me as a waste of effort, and likely to negatively impact both function and maintainability.

There is one direction that strikes me as possibly desirable: If you implement some sort of loadable module system (like linux kernel modules) you might be able to reasonably break things down into platform specific and not, and then load whichever bits you need for the current camera. This would be major surgery, but there would be many other benefits to such a system, so it wouldn't be wasted effort in any case.

Handling different firmware revisions of the same camera might not be too bad. You'd have to fix up all the address and the asm boot code, but that should be fairly straightforward using something kind of like a relocation table.

Logged

Don't forget what the H stands for.

fudgey

1705
a570is

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #6 on: 29 / October / 2008, 03:45:34 »

Two small things to keep in mind:

1) Anything that slows down camera startup or CHDK startup is to be avoided, especially if that startup makes the camera slower to start (from button press to ready to shoot status). I already compile my own CHDK, currently only because multipartition support makes CHDK startup significantly slower and I don't need it.

2) I often use those tiny16MB or 32MB MMC cards for testing, so 25MB on the SD card actually is slightly significant.

Logged

fe50

3147
IXUS50 & 860, SX10 Star WARs-Star RAWs

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #7 on: 29 / October / 2008, 05:10:27 »

Quote from: fudgey on 29 / October / 2008, 03:45:34

1) Anything that slows down camera startup or CHDK startup is to be avoided, especially if that startup makes the camera slower to start (from button press to ready to shoot status). I already compile my own CHDK, currently only because multipartition support makes CHDK startup significantly slower and I don't need it.

2) I often use those tiny16MB or 32MB MMC cards for testing, so 25MB on the SD card actually is slightly significant.

...me too !

*Edit: i want a small, fast, powerful & intuitive useable CHDK to shoot great photos - no need for a big, fat & slow 'Universal ~~256~~ 257-All-In-One-Super-Special-XXL-MegaCHDK power machine' here

[/sarcasm]

« Last Edit: 29 / October / 2008, 05:21:22 by fe50 »

Logged

-->FIRMWARE DUMPS<-- * Manual * Lua * me

robocat

33

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #8 on: 29 / October / 2008, 06:40:07 »

Hmmmmm, it sounds like the idea is mostly in the too hard basket.

However, to clariify:

A fat 'easyboot' version is *not* mutually exclusive with keeping the existing builds.

Quote

reboot. Time to copy primary.bin + time to reboot = significant additional boot time

Quote

1) Anything that slows down camera startup or CHDK startup is to be avoided

Quote

2) I often use those tiny16MB or 32MB MMC cards for testing

I wasn't advocating getting rid of the existing builds - there is no way I want bloat and slowness on *my* camera!!! Avoiding any extra complication in the source code is to be avoided too.

Sorry I was unclear - I was trying to advocate a separate easy-to-use build.

Quote

Is it just so people don't have to figure out which diskboot.bin to put on their camera ? It may sound harsh, but I'm not sure that the people who can't figure how to use vers.req and download the right file are going to benefit much from CHDK

I just think that an easyboot option would be a great thing to make it as easy as possible for non-developer users to get hooked on CHDK.
I think that there are *plenty* of photographers that would love to use the features, and get great benefit from it, that would find the technical issues a barrier to initially trying it. Not all photography enthusiasts technically capable with cameras are technically literate with computers.
Fundamentally, I just like the idea of being able to give anyone with a Canon a single SD card to try out CHDK (perhaps stripped of any unstable/complicated features!). I would love to be able to show someone on their own camera CHDK working (e.g. a friend with a CHDK compatible camera the other day visited, but I didn't have time to get the right version and put it onto their card).

Finally - I have 2 different Canon cameras that I want to use CHDK on. This concept would allow me to use the same cards between them.

Thanks for all the comments.

« Last Edit: 29 / October / 2008, 06:49:57 by robocat »

Logged

whim

2046
A495/590/620/630 ixus70/115/220/230/300/870 S95

Re: we already have universal dumper... why dont we have a universal loader?

« Reply #9 on: 29 / October / 2008, 07:37:46 »

I see a major obstacle to creating a uniloader:

how do you propose to create a DISKBOOT.BIN that gets accepted by all cams ?
DryOS wants them 'plain', while NewDryOS prefers them 'with filter'

wim

Logged

*Wikia FAQ *GCC Compiler Shell for Windows