Signature finder

  • 7 Replies
  • 3420 Views
*

Offline fudgey

  • *****
  • 1705
  • a570is
  • Publish
    Signature finder
    « on: 31 / August / 2008, 10:28:14 »
    Advertisements
    Hi... documentation I could find about the signature finder in the tools directory was sort of scarce, so I wrote this while fiddling with it a bit. I had my mind set on adding new features, not porting new cameras:

    Signature finder (finsig/gensig) usage:

    1) Put all firmware dumps of every camera into the CHDK source tree. Each firmware version needs to be called PRIMARY.BIN and be located in it's own subdirectory, such as platform/a570/sub/100e/PRIMARY.BIN for the a570is 1.00e dump.

    Once you have done that, signatures will be automatically searched for during the build process.

    You should try batch-build at this point, because it's likely that some platforms will not build due to missing stubs. This is because your dumps may not be perfect (incomplete or not starting from the correct address) or because someone changed something in CHDK source tree that breaks building with firmware dumps in place. If you have a bad firmware dump, fix it (removing garbage from the start is often enough; compare to dumps of other similar cameras and you'll see how it should start) or remove it from the source tree (to skip signature search for this platform), or just don't build for that platform (commenting it out in the Makefile works).

    Tweak until the build process goes through cleanly.

    2) Go CHDK source tree tools/ directory. See readme.txt. At the time of writing this, it says:

    sig_ref_vxworks_1.bin : should contain a610-100e primary
    sig_ref_vxworks_2.bin : should contain a710-100a primary
    sig_ref_vxworks_3.bin : should contain a570-100e primary
    sig_ref_dryos_1.bin   : should contain a720-100c primary

    Copy those firmware dumps to the tools/ directory with those names.

    3) See sig_ref_*.txt. These are simple lists with function names and their entry points for each of these four reference firmware dumps.

    Using a disassembler, find entry points to your new functions for each of these four reference firmwares and add them to the end of those files.

    4) Build.

    If all goes perfectly, your new stubs are now found as NSTUBs in plaform/*/sub/*/stubs_entry.S files among all other the automatically generated stubs.

    It's likely that some platforms will not build. The signatures may not be good enough to find your function from all cameras. Your function may not even exist on all cameras, or versions of it may be too different to match the signatures.

    To get past an error, you can of course delete the unlucky PRIMARY.BIN or skip building for this particular firmware, but that won't fix the fact that your new signature breaks building for this platform if the firmware dump is in it's place.

    What you should do is manually find the entry point for this platform and add it as an NHSTUB to stubs_entry_2.S. If it is not found, you need to add it as a null stub (a stub that has a correct name, but which points to a part of code which will just return without doing anything) to stubs_entry_2.S to keep 'make' happy.

    The problem with this approach is that sometimes you may not want to bind this function to a nullstub. Instead, you may want to implement a dummy function in lib.c (only for platforms that don't have the real firmware function) that returns something more sane than a null function. But you can't add that dummy function if the name is taken by the stub!

    The alternative is not to add your new function signatures to CHDK svn, but instead provide NHSTUBs in stubs_entry_2.S for all platforms, with dummy stubs or lib.c dummy functions for all unsupported platforms. This doesn't help people porting for a new camera, though (quite the contrary since there's more work to be done in porting).

    If the function exists but isn't found, you could find it manually and add this firmware as another new reference firmware, so that this signature would be created and matched. I have never done this, but if I'm not mistaken this is how it's done:

    1) Create a new signature input file (for example tools/sig_ref_vxworks_4.txt if that's not taken and if this camera was a vxworks camera) containing your function name and entry point, just like in the other txt files. There is no need to add any other entry points in this file, but for the sake of porting new cameras, it would probably be a good idea to add functions that are listed in stubs_entry_2.S, because those are the ones that weren't found automatically using the old signatures.

    2) Copy the firmware dump where gensig can find it (in our example, tools/sig_ref_vxworks_4.bin).

    3) Modify tools/Makefile in the numerous occasions where additions are required ('clean:' and 'signatures_vxworks.h:' need to be modified in our example, and 'sig_ref_vxworks_4.bin:' must be added).

    4) Update tools/readme.txt.

    *

    Offline fudgey

    • *****
    • 1705
    • a570is
  • Publish
    Re: Signature finder
    « Reply #1 on: 31 / August / 2008, 10:37:23 »
    And to continue the already lengthy rant, I tested this with EnterToCompensationEVF and ExitFromCompensationEVF. I had already manually determined entry points for 35 firmwares from disassemblies. Two of those failed to build with firmwares in place in ways I didn't know how to fix quickly.

    Out of the remaining 33, EnterToCompensationEVF was correctly found in 23 dumps and ExitFromCompensationEVF 16 dumps. Four of the successful ones were reference firmwares. So, we got success ratios of 19/29 and 12/29 (66 % and 41 %). It _is_ possible that some of the incorrectly found addresses would work, since the same function can be entered in more than one way. But it is also possible that they wouldn't work properly (see explanation below *). I'm also assuming that the entry points I found manually are correct (and the ones that one could expect the signature finder to be able to find).

    A candidate for ExitFromCompensationEVF was found in all firmwares, but 17 of 29 (59 %) were incorrect. For two of the incorrect ones, finsig gave a correct alternative (commented out in stubs_entry.S). EnterToCompensationEVF was not found in 6 firmwares, meaning only 4 out of 29 (14 % were) incorrectly found.

    Some of this could be handled by selecting more reference firmwares (to better represent all incarnations of the functions and their calling mechanisms). This may still not be enough (unless finsig is really smart and follows branches?).


    (*) The way these functions are laid out in the firmware can be divided into groups that look similar (this is one reason why we need more than one reference firmware). When finding functions manually using strings, some have strings located just before the function (e.g. a570is), and a good entry point is right after the string, on an instruction that loads a pointer to that string for debug purposes.

    On many other firmwares, there are blocks of strings with no code in between. Near those blocks there are lists that map these strings to code.

    But these lists often point to a very short code wrapper, which just branches to the real entry point of the function. So -- unless finsig is smarter than I think it is -- it's not a good idea to use the listed address as an entry point for a reference firmware, because that will result in a bad signature. It can be used as an entry point for a manually found function, altough it will run slower by a few instruction cycles. S3IS is an exemple for this category.

    For some firmwares this short code pointed from the list actually does something. It may be the actual function, but often it is just some simple input or output variable manipulation that only takes a few instructions before or after a branch to where most of the function code resides. This short code snippet may still be something that is necessary in order to make the function work correctly (it's possible that two functions share code this way, for instance?).

    These kinds of layouts are more difficult to find automatically (unless finsig is smart and follows branches), since the actual code may be only a few instructions long and the real functionality happens elsewhere. So, using this kind of a firmware as a reference is not going to work (a very short signature would create lots and lots of matches, and a signature extending beyond a function is just plain wrong), but not using one may result in finsig creating a stub that points to the real code without the (possibly cruicial) wrapper.


    *

    Offline fe50

    • ******
    • 3034
    • IXUS50 & 860, SX10 Star WARs-Star RAWs
      • fe50
  • Publish
    Re: Signature finder
    « Reply #2 on: 02 / September / 2008, 05:19:05 »
    Hi fudgey,
    congratulations for this wonderful, really great stuff !   (added it to the wikia: For Developers: Signature finder)

    *

    Offline PhyrePhoX

    • *****
    • 2253
    • make RAW not WAR
      • PhyreWorX
  • Publish
    Re: Signature finder
    « Reply #3 on: 02 / September / 2008, 09:02:03 »
    i guess this cant be used for something like this? i gather the adress for conti_shooting can only be found in memory, right?


    *

    Offline reyalp

    • ******
    • 9961
  • Publish
    Re: Signature finder
    « Reply #4 on: 10 / September / 2008, 03:57:19 »
    I have found that gensig.sh likes to hang under XP (using the GCC + tools package found at Compiling CHDK under Windows - CHDK Wiki) in this:
    Code: [Select]
        cat sig_ref_$2_${v}.txt | \
        while read line; do
        # workaround - multiword read does not work in Windows shell!
    It's not really clear why it hangs, it just ends up with an sh sitting around not using much CPU. It doesn't happen every time, but doing a batch-zip hits it after a few firmwares (but not always on the same one).

    Replacing the cat | while ... with following lovely bit of awk appears to avoid the problem
    Code: [Select]
    awk -v os=$2 -v ver=$v -- '{system(sprintf("gensig sig_ref_%s_%s.bin 0xffc00000 %s_%s %s %s",os,ver,$1,ver,$2,$3?$3:32))}' < sig_ref_$2_${v}.txt
    If you want the above to work on *nix, you'll need a ./

    edit:
    This looks like a good place to note the broken dumps etc I find while getting this running. Building against the latest juciphox tree. So far:
    ixus65_sd630/sub/100a/
    The makefile.inc has ROMBASEADDR=0xff800000, but the actual start (MOV     R0, #2 b ... "Copyright...") is at 0xFF810000. The dump found in Collection 2 - zSHARE - primaries.bin.rar starts with the MOV, so running finsig makes offsets that are 0x10000 too low compared to the checked stubs_entry.S. The dump is also short, ending at 0xFFB35351. There's another dump in this post [DOWNLOAD LINKS] Firmware dumps available which is 8MB, but appears to start at FF759000

    Found in SVN stubs_entry.S, but not generated:
    Code: [Select]
    NSTUB(kbd_p2_f, 0xff8270d0)
    NSTUB(fprintf, 0xffb0a35c)
    NSTUB(fputs, 0xffb0a47c)
    NSTUB(puts, 0xffb11634)

    ixus50_sd400/sub/101a/
    Found in SVN stubs_entry.S, but not generated:
    Code: [Select]
    NSTUB(iosDevFind, 0xffabcd4c)
    NSTUB(fprintf, 0xffab2414)
    // ALT: NSTUB(fprintf, 0xffab3bd8) // 15/0
    NSTUB(fputs, 0xffab2534)
    NSTUB(puts, 0xffab981c)

    s5is/sub/101a/
    checked in stubs_entry.S is empty.

    Aside from those, all the other cams included in the two dump collections from [DOWNLOAD LINKS] Firmware dumps available seem to work perfectly (they produce files identical to what is checked in).

    edit:
    Here's some more that weren't in either of the collections (I'll post a new rar of them once they are all cleaned up and working)
    s5is/sub/101b/
    checked in stubs_entry.S is also empty.

    ixus50_sd400/sub/101a/
    ixus50_sd400/sub/101b/
    both missing iosDevFind,fprintf,fputs,puts
    ixus55_sd450/sub/100d/
    missing as above except puts.

    several dumps had extra stuff at the start. I made a tool to clean these up, which I'll post later.

    edit:
    The "missing" above mostly appear to be caused by someone checking in the generated files, but not the corresponding changes to sig_ref_*.txt
    « Last Edit: 11 / September / 2008, 20:35:37 by reyalp »
    Don't forget what the H stands for.

    *

    Offline philmoz

    • *****
    • 2936
      • Photos
  • Publish
    Re: Signature finder
    « Reply #5 on: 31 / December / 2010, 03:00:50 »
    (Hope this is the right place to post this.)

    In the current version of signatures_dryos.h the signature for sig_UpdateMBROnFlash_3 is the same as func_sig_strncmp_1.

    As a result finsig is finding a 100% match for UpdateMBROnFlash but it points to strncmp.
    So the 'Make card bootable...' menu option doesn't work.

    I read the earlier posts; but not quite sure how to fix this.

    Regards,
    Phil.
    CHDK ports:
      sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
      g12 (1.00c, 1.00e, 1.00f & 1.00g)
      sx130is (1.01d & 1.01f)
      ixus310hs (1.00a & 1.01a)
      sx40hs (1.00d, 1.00g & 1.00i)
      g1x (1.00e, 1.00f & 1.00g)

  • Publish
    Re: Signature finder
    « Reply #6 on: 31 / December / 2010, 06:54:54 »
    Hallo philmoz,
    I'm not sure, but I think the address UpdateMBROnFlash in sig_ref_dryos_3.txt should be: 0xFF86B984 and not 0xFF813F74.

    CHDKLover

    *

    Offline reyalp

    • ******
    • 9961
  • Publish
    Re: Signature finder
    « Reply #7 on: 31 / December / 2010, 12:33:04 »
    (Hope this is the right place to post this.)

    In the current version of signatures_dryos.h the signature for sig_UpdateMBROnFlash_3 is the same as func_sig_strncmp_1.

    As a result finsig is finding a 100% match for UpdateMBROnFlash but it points to strncmp.
    So the 'Make card bootable...' menu option doesn't work.
    Good find.

    Actually, UpdateMBROnFlash is one instruction longer, but finsig takes the first 100% match edit: oops, wrong function, these are the same, but AFAIK comment below still applies :(
    Fixing this (so it takes the longest correct match instead of the first should be possible, but might break some existing matches. The whole system is rather fragile.

    But fixing as CHDKlover says might fix this case :)
    « Last Edit: 31 / December / 2010, 12:49:10 by reyalp »
    Don't forget what the H stands for.


     

    Related Topics