the sx10 porting thread

[nimrod]

Some *.S files for 1.01B firmware for future developers  - http://ewavr.nm.ru/chdk/trash/sx10-101b.zip
Sorry, I can't continue this work.

Thanks for all your work on this! I just bought a new SX10 which seems to have come with 1.02B, but rather than post another "WhEn is 1.02b gonna be done!!!1!!!1!" reply, I'm actually looking into what it will take to port this to 1.02B.

Luckily I have some asm and IDA experience, so I might actually be able to accomplish something. Can't say that I've written anything for the ARM, but I read an overview of the architecture once (maybe 3 years ago, lol.) Seems to be a pretty popular core for SOC ASICs, and sounds like lots of people/companies have developed their own cores using the same instruction set, etc. Even if I didn't know anything about ARM, the instruction set looks RISC-like and similar to m68k which I'm probably most familiar with. Luckily this is nothing like Intel x86 (yuck).

I haven't used IDA in 2-3 years either. When I did use it I used it for malware reverse engineering (though I can't say I did a massive amount of that since it wasn't my primary job). Given the obfuscation techniques used in malware code, hopefully this fw will be easy in comparison. (I'm assuming that there's no real execution-based obfuscation, just the whole-image XOR "encryption" or whatever.)

Ok, so questions:

*  First you need either the original firmware or a firmware dump of your camera.

Ok, that's been done, but somewhere I saw some suggestion that 1.01B was the same as 1.02B or something? But then when I searched around I couldn't seem to figure out where I saw someone claim this. Am I smoking crack or is someone else smoking crack?

Reason it matters is because I'm wondering how useful your .S files would be for 1.02B. I suppose I'll end up finding out though.

* Create a platform subdirectory for a new camera model/fw version (you can just copy an existing one).

Question here: Since you've already ported a 1.01A for this camera, I'm guessing that maybe some of the work necessary for 1.02B etc has already been done? If nothing else I guess I can look at the .S files you have, compare them, then get some idea. Is it mostly just a matter of finding a bunch of function/subroutine offsets?

    * Modify a couple of addresses and constants.
    * Check the keyboard "driver" (button constants and behaviour).

So I'm guessing that the constants will be the same....

    * Find the adresses of the RAW-, video- and frame-buffers.
    * Check the addresses of autofound functions. Manually correct/find the right addresses for some of them with a disassembler like IDA Pro.

Sounds like this is the main work? How likely is it that the boot procedure has changed and will need to be copied/modified?

I installed CHDK shell. Nice automation. (Nice idea to automate all that given the need to port this for every firmware version. It increases the chance that ports get done.)

Stuff I'm reading:
http://chdk.wikia.com/wiki/For_Developers
http://chdk.wikia.com/wiki/SX10
http://chdk.wikia.com/wiki/DryOS_Porting
http://chdk.wikia.com/wiki/Loading_dump_to_IDA
http://chdk.wikia.com/wiki/Adding_support_of_a_new_camera

Looked at this: CHDK\trunk\trunk772\platform\sx10\sub\101a
Put your 101b files in 101b dir
looked at everything in sub briefly, as well as loader and platform

Ok, so it looks like I shouldn't need to do anything with sx10 loader or platform and everything I'd need to change is in a new platform/sx10/sub/102b dir? (Can any be copied from 101a?)

Sorry for being a bit verbose and thinking out loud, but I just want to see if I'm on the right track here and what I know so far so you can see what sort of tips you might have for me that I could actually understand. Doesn't look entirely trivial to port (especially since I haven't done stuff like this in a while) but so far everything I've looked at generally makes sense even if I don't yet understand exactly how it all fits together.

[reyalp]

Nimrod, looks like you've found pretty much everything you need. Porting a new sub version of an existing firmware is a lot easier than a new port. You will need to dump your specific firmware version.

For the most part, all that will change is function and variable addresses. All the hardware addresses are (AFAIK) always the same between different versions of the same camera. Same goes for things like the constants in kbd.c. As you guessed, you shouldn't need to change the loader, or anything outside of the 'sub' directory (except for the top level makefile etc)
The majority of function addresses are automatically found, as described in http://chdk.wikia.com/wiki/Signature_finder (that page is aimed at people adding new function signatures, which you don't need to do, but should help you understand the process) So what you are left with is fixing up the asm in boot.c, capt_seq.c, movie_rec.c, and filling in the addresses in stubs_entry_2.s (including any that aren't found correctly in the automatic stubs_entry.s) and stubs_min.s

You can start by copying an existing platform, but for some of the inline asm it's probably easier to start from ida disassembly instead, since you would have to go through line by line to fix all the offsets anyway. For the .s files, you should start from scratch.

The 1.01b files will probably not help you with 1.02b, unless they happen to be compatible. Since there aren't ports for either one yet, we don't know if they are.

The online arm docs are pretty good, you can find links to the most important ones in http://chdk.wikia.com/wiki/Developer_Technical_Documents

If you need real time advice, there's frequently a few of us idling in #chdk on freenode.

[nimrod]

Nimrod, looks like you've found pretty much everything you need. Porting a new sub version of an existing firmware is a lot easier than a new port. You will need to dump your specific firmware version.

Awesome. I just downloaded the 1.02b dump on the wiki page for the sx10. Hopefully that one is correct.

The majority of function addresses are automatically found, as described in http://chdk.wikia.com/wiki/Signature_finder (that page is aimed at people adding new function signatures, which you don't need to do, but should help you understand the process) So what you are left with is fixing up the asm in boot.c, capt_seq.c, movie_rec.c, and filling in the addresses in stubs_entry_2.s (including any that aren't found correctly in the automatic stubs_entry.s) and stubs_min.s

Ok, got it. That was the page that I looked at way too briefly earlier, but I get the basic idea. Right now I'm just using it as part of the normal build process having put the PRIMARY.BIN into my new sub directory.

You can start by copying an existing platform, but for some of the inline asm it's probably easier to start from ida disassembly instead, since you would have to go through line by line to fix all the offsets anyway. For the .s files, you should start from scratch.

Ok, that's what I did, for now, just to make 'make' happy so it would generate the stubs_entry.S, and that seems to be working.

The 1.01b files will probably not help you with 1.02b, unless they happen to be compatible. Since there aren't ports for either one yet, we don't know if they are.

Well, comparing what evwar uploaded in his zip file for 1.01b .S files, it looks like his stubs_entry.S looks suspiciously like what the build process just generated from the 1.02b image:

1.01b from evwar:

Code: [Select]

// !!! THIS FILE IS GENERATED. DO NOT EDIT. !!!
#include "stubs_asm.h"

NSTUB(AllocateMemory, 0xffa67c20)
// Best match: 56%
NSTUB(AllocateUncacheableMemory, 0xff827fd8)
NSTUB(Close, 0xff823a64)
NSTUB(CreateTask, 0xff81aef4)
NSTUB(DeleteFile_Fut, 0xff823310)
// ERROR: EnterToCompensationEVF is not found!
... etc

1.02b generated by me running build with 1.02b image from wiki sx10 page:

Code: [Select]

// !!! THIS FILE IS GENERATED. DO NOT EDIT. !!!
#include "stubs_asm.h"

NSTUB(AllocateMemory, 0xffa67c20)
// Best match: 56%
NSTUB(AllocateUncacheableMemory, 0xff827fd8)
NSTUB(Close, 0xff823a64)
NSTUB(CreateTask, 0xff81aef4)
NSTUB(DeleteFile_Fut, 0xff823310)
// ERROR: EnterToCompensationEVF is not found!
... etc

Doing an actual diff between these (evwar's 101b and the 102b I just generated) gives me:

Code: [Select]

C:\CHDK\trunk\trunk772\platform\sx10\sub>diff 101b\stubs_entry.S 102b\stubs_entry.S
99,102c99,102
< NSTUB(_log, 0xffb01a10)
< NSTUB(_log10, 0xffb0069c)
< NSTUB(_pow, 0xffb0080c)
< NSTUB(_sqrt, 0xffb0366c)
---
> NSTUB(_log, 0xffb01a40)
> NSTUB(_log10, 0xffb006cc)
> NSTUB(_pow, 0xffb0083c)
> NSTUB(_sqrt, 0xffb0369c)
How often does this turn out to be the case? Funny that it's all the same except for these 4 math functions. Of course I have to keep in mind that some of the ones that weren't found automatically may also be similarly different, but it sounds like evwar's work might apply and thus save time.

If you need real time advice, there's frequently a few of us idling in #chdk on freenode.

Cool, thanks.

(I'm surprised that I've gotten this far so fast, but I suppose that most of it is thanks to whim's CHDK shell and all the people who contributed to the wiki docs! That's saved a lot of time. Thanks guys!)

[reyalp]

How often does this turn out to be the case? Funny that it's all the same except for these 4 math functions.

It's not uncommon to have a bunch the the same, and the ones that are different only be off by some small fixed offset.  Basically canon adds some code or data somewhere, and everything else moves down a bit (48 bytes in the above case).

The math functions are near the end of ROM code, which explains why they are the only ones affected. This also means that functions in stubs_entry_2.s that are at lower address than whatever the last identical one you in stubs_entry.S will almost certainly be the same too. If you wanted, you could do a binary compare to figure out what offset the changes started.

Variable addresses (stubs_min.S, lib.c and and a few others scattered around) will follow a similar pattern, either being the same, or offset by some small amount.

[GAJoe]

1.02B owners have reason to celebrate!!
Nimrod you're our hero!
Let us know if we can help!
Some where I read that one of the monotonous tasks is comparing lines and lines of code looking for small differences. If so let me know where to go to help.
A thread specific to the 1.02B has been started here: http://chdk.setepontos.com/index.php/topic,3540.0.html so it may be the place to get help / delegate tasks.
Again, thanks for stepping up to the plate Nimrod!
GAJoe

[escobar]

Above all, sorry for the bad english:

I understand all these requests for a 1.02b are quite annoying. But I also understand people have no other place to tell that they wish it. The problem is that there is to few people with the knowledge to do it and that people may be not interested on challenged to do it right now.

Well, I'm not a developer (unless Clipper Summer 87 still counts), but, working for the government for 25 years, I'm no strange to tedious tasks :-)

I would be more than glad to contribute with my almost noob knowledge and skills to port the firmware. I don't know if it's possible to assign some tasks to non-developers - like getting some pizza :-) - to speed or easy up the job, but, if so, count on me.

[redeyes3000]

Please make CHDK for FW: 1.02b

BIG THX 

   
I am very disappointed that no one takes the job. For once I can test your product. Yet I am not the only person to expect a version 1.02b

Please MTV.... Pip my Camera !! lol

Pleeeeeasee!! make chdk ver. 1.02b pleeeeeeeeeeease!!!!!!
?? ? ?? ? ?? RAW ? ?? ??(((

Please read the thread, and think about what you are asking for a second.

What would you do if someone on the street walks up to you and asks you to do a couple days of complex but not very interesting technical work for free ? If they say OMG PLEASE PLEASE PLEASE does that make you more likely to do it ?

   
I am a junior programmer and I would do this program myself if I had only knowledge required. For now, I contribute to society by freely giving of my time on open source project. I know it is not always pleasant to work on some aspect of a project. For my part, I face some challenge that gives me great satisfaction, at least when I can contribute.

   
Who will be my hero on CHDK ?

[nimrod]

Well, don't get too excited yet. I haven't done this kind of stuff in a while, but quite a bit of stuff has already been automated for me. It's mostly a matter of how long this will take me.

The thing is that this kind of work falls into the realm of Embedded Systems which is not your usual sort of software engineering because it usually involves working closer to the hardware than in application programming. Basically it's more like developing a Windows device driver than writing an application in Visual Basic or something.

In addition, the vendor doesn't intend for people to do any of this so they provide no documentation. So you have to take stuff apart (reverse engineer) in order to figure out what to do, whereas a vendor that expects you to develop software extensions will just give you exact instructions on what to do. So in addition to understanding the architecture and assembly language, you also have to understand DISassembly and analysis well enough to take the firmware apart.

That's why it's hard for someone who knows C, Java, VB, etc to catch on to all this because those languages are most often used to write applications for operating systems that insulate you from the hardware with many layers of abstraction, and come with nice documentation for the uppermost layers. Embedded systems are small though so there is usually not enough space and power for all those abstraction layers to make things easy for the programmer. As a result you usually end up with something like a layer of function calls written in assembly which can then be called by C code. Then at the very least you get to write the rest in C instead of asm.

As far as what other people can do....

It looks like 1.01b is almost exactly the same as 1.02b. I'm going to do 1.02b first since I have a camera with 1.02b, but once I do that it should be pretty easy (from what I can tell) to generate a 1.01b image with very little work. It may be a while, but when I get the 1.01b image then I'll need someone with a 1.01b camera to test it.

[Beber]

Hi Nimrod,

I'm a beginner developper in embedded systems (CE) but I am interesting on how to program with this platform. At the time I don't know anything about how to program with powershot.
Also, I have a 1.02b SX10IS.

Can I help you in your work and how?

Thanks,

[redeyes3000]

At least I encourage you guys

[nimrod]

Can I help you in your work and how?

Well, it might be possible to split up tasks but you'll probably need to spend some time understanding all this stuff. Right now I'm working on understanding and then updating the code in trunk\trunk772\platform\sx10\sub\102b\boot.c.

Do you have IDA Pro? (You need ARM instruction support.) Basically you'll need that to do much of anything other than help test the code on your camera.

Stuff you'll need to read:
http://chdk.wikia.com/wiki/For_Developers
http://chdk.wikia.com/wiki/SX10
http://chdk.wikia.com/wiki/DryOS_Porting
http://chdk.wikia.com/wiki/Adding_support_of_a_new_camera
http://chdk.wikia.com/wiki/Signature_finder
http://chdk.wikia.com/wiki/Loading_dump_to_IDA
http://chdk.wikia.com/wiki/Developer_Technical_Documents

And the page on For Developers on how to compile CHDK for your platform. (I recommend using CHDK shell if you have windows because it just does everything, which is nice and saves time.)

Once you can build sx10/sub/101a I can send you what I have for 102b, but right now I haven't really done much of anything yet. (Mostly still trying to understand everything in boot.c)

[Grin]

Hello!

I'm newbie in chdk & arm's too.

but...
as I have already bought SX10 IS with firmware 1.02b...
in three nights i'm finished porting from 1.01a to 1.02b

Testers are welcome!!! http://rapidshare.de/files/47479851/sx10-102b-0.9.8.rar.html

P.S.: sorry for my bad english
P.S.S.: how may(must) i publish code?

[reyalp]

Grin: a patch is preferred. You can zip it and upload to rapidshare since forum attachments are currently disabled. Or just zip the new/changed files.

[Grin]

sources: http://rapidshare.de/files/47479952/sx10-102b-sources.rar.html

[reyalp]

sources: http://rapidshare.de/files/47479952/sx10-102b-sources.rar.html

Excellent.

I've checked this in, svn 774. This should make it easier for you guys to work together. You can post patches (svn or tortoise diff preferred) against that.

Grin: please check that the stubs_entry.s and stubs_auto.s that I've generated and checked in match what you are using.