New stubs/signature finder (gensig2/finsig2) for DryOS cameras. - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: [1] 2 3 4 5 6 ... 11   Go Down

New stubs/signature finder (gensig2/finsig2) for DryOS cameras.

  • 103 Replies
  • 43327 Views
*

Offline philmoz

  • *****
  • 3450
    • Photos
New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« on: 26 / June / 2011, 00:53:27 »
Advertisements
Attached is the new V2 version of the firmware address signature matching and stub finding code that I've been working on for a while.

Thanks to waterwingz for validating the results along the way and helping to test this.

I think this is ready for beta testing now; but before I post it as a patch I'd like to have wider usage & testing to make sure there are no serious problems.

I've implemented this as a new version of gensig and finsig (gensig2 & finsig2) and associated files so it can work in parallel with the existing version. There is a compile time option to switch between them (see below).

Note: This is for DryOS cameras only at the moment. VxWorks based cameras will continue to use the existing system.

To use this, unpack the attached .zip file into your trunk directory.
This will replace platform/makefile_sub.inc and tools/Makefile with updated versions for the new stuff.
It should also add a bunch of new files to the tools directory (gensig2.c, gensig2.sh, finsig2.c, ...).

You will also need to have your firmware dump (PRIMARY.BIN) in the 'platform/camera/sub/fwversion' directory for your camera/firmware in order to trigger the rebuild of stubs_entry.S below.

First step is to do a normal clean build to make sure you have a valid stubs_entry.S file for the camera/firmware you want to test.
Then add the option OPT_SIGS_V2=1 to either buildconf.inc or localbuildconf.inc and do another clean build.

The first time the V2 version is used it will rename stubs_entry.S to stubs_entry.S.orig and then use the new V2 code to build a new stubs_entry.S.

The V2 code reads the existing stubs_entry.S.orig, stubs_entry_2.S and stubs_min.S files so that it can compare the addresses it finds against the old addresses. Any discrepancies are reported in the new stubs_entry.S file to be investigated.

The recompiled CHDK should run as before; but it is worth opening up the new stubs_entry.S file to check for any issues that may need to be looked at. Also the V2 code tries to find some of the other addresses that might be useful for a new port (stubs_min.S, kbd.c and lib.c values).

This is a work in progress so it's likely that I may have some bugs in there. I've validated the results against the exisitng values for quite a few cameras and it seems to be pretty solid. If you find something that looks wrong please post details here.

So what's new in V2:
- Includes some camera info strings from the firmware (dryos version, firmware version, camera name).
- Will include a warning message if the firmware dump looks incorrect (too small, etc).
- Includes new signature matching algorithms (primarily based on finding function name strings in the firmware with nearby function pointers).
- Looks for some of the addresses that are needed for stubs_min.s, kbd.c and lib.c
- Addresses found are checked against the existing stubs_entry_2.s, stubs_min.s and the old stubs_entry.S files. Any discrepancies are reported.
- Close to 100% signature matching for all existing camera and firmware versions in the core trunk.
- Slightly faster than the current system.
- For each address found includes the reference location in the firmware where it was found.

If you want to rebuild the 'signatures2_dryos.h' file you will need to copy the 10 firmware dump files to the tools directory (see the readme_gensig2.txt file for details).

Hopefully the format and layout of the new stubs_entry.S file will be self explanatory; but a couple of notes.
- The first section is the camera info stuff.
- This is followed by stubs that may have issues, e.g. they did not match 100% or the new address found does not match the one in stubs_entry_2.s or the old stubs_entry.S
- Next is the stubs_min.S values found along with comparison to the existing stubs_min.S values.
- Then comes the lib.c values (note these are not validated against the existing lib.c stuff so should be checked manually).
- After that is the kbd.c values (again not validated so please check manually).
- Finally are all the stubs that matched 100% and also match existing values.

Edit:
- 2nd July. Updated version now finds kbd.c values for IXUS 960 (SD950)
- 7th July - Fixed kbd.c values for A1100. Added platform_camera.h values (CAM_DRYOS, CAM_DRYOS_2_3_R39, CAM_RAW_ROWPIX, CAM_RAW_ROWS, CAM_UNCACHED_BIT)
- 10th July - Now finds the shooting mode table in the firmware and uses it to do some checks on the modemap data in platform/CAMERA/shooting.c. Note because the mode numbers change from camera to camera I can't validate that the mode names in 'modemap' are correct. This version looks for modes in the firmware table that are missing from 'modemap' and for values in 'modemap' that aren't in the firmware.
- 12 July - Updated tools/Makefile to include the new font_8x16_pack tool (changeset 1234).
- 14 July - Updated with missing functions for G10.
- 18 July - Updated output format to make it easier to see discrepancies (thx waterwingz). Updated for R47 cameras which don't have UpdateMBROnFlash function; but have MakeSDCardBootable function instead.
- 24 July - Removed old version from post, fixed crash on A2200, handle NSTUB entries in stubs_entry_2.S

Phil.
« Last Edit: 23 / July / 2011, 20:42:58 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline funnel

  • ****
  • 349
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #1 on: 26 / June / 2011, 05:01:24 »
Incredible. Good job. I found two errors in stubs_min.S I didn't know about.
Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #2 on: 26 / June / 2011, 06:34:20 »
The following firmware dumps that I downloaded from the box.net repository will not work correctly with the new V2 code (for the reasons listed). The will still work; but the matching will not be as accurate.

The following dumps are too small (so are missing important tables in the data section):

ixus200_sd980   100c  4MB should be 8MB
s90                    101c  7.1MB should be 8MB
s90                    101a  7.0MB should be 8MB
sx20                  102b  4MB should be 8MB
sx200is              100c  4MB should be 8MB
sx10                  101a  4MB should be 8MB   thx fe50
ixus870_sd880   100e  4MB should be 8MB
ixus960_sd950   100c  4MB should be 8MB
ixus960_sd950   100d  4MB should be 8MB   thx hwntw
ixus860_sd870   100c  2.9MB should be 8MB   thx fe50
ixus90_sd790     100d  3.1MB should be 8MB
ixus90_sd790     100c  4MB should be 8MB
ixus80_sd1100   100c  4MB should be 8MB
g9                     100i  3.2MB should be 8MB
a590                 101b  2.9MB should be 4MB   thx blackhole

Also the s90 100c firmware dump appears to be corrupt. There is a block of data in the middle of the code segment that is a duplicate of data in the data section, and the camera id string is missing from the end of the dump.

If anyone has full dumps for these camera/firmware versions please post links.

Phil.
« Last Edit: 03 / July / 2011, 05:22:56 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline blackhole

  • *****
  • 937
  • A590IS 101b
    • Planetary astrophotography
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #3 on: 26 / June / 2011, 08:17:40 »
@philmoz

a590 101b 4MB dump

http://www.megaupload.com/?d=TYM2QTV3
Logged
*

Offline waterwingz

  • ******
  • 12138
    • Waterwingz
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #4 on: 26 / June / 2011, 09:27:35 »
Congratulations on this "beta" release philmoz !
Its really hard to describe how big an advance this is.  Almost 100% match in stub_entry.S is a huge step all by itself, but the addition of stub_min.S and other addresses would have saved me weeks of time during my first port.

As funnel found out and I can verify,  its probably worth developers having a look at their old (DryOS) ports with this.


« Last Edit: 26 / June / 2011, 09:29:28 by waterwingz »
Logged
Ported :   A1200    SD940   G10    Powershot N    G16
*

Offline fe50

  • ******
  • 3147
  • IXUS50 & 860, SX10 Star WARs-Star RAWs
    • fe50
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #5 on: 27 / June / 2011, 01:17:17 »
Thx phil for this great stuff !

Quote from: philmoz on 26 / June / 2011, 06:34:20
Code: [Select]
ixus200_sd980   100c  4MB should be 8MB
s90             101c  7.1MB should be 8MB
s90             101a  7.0MB should be 8MB
sx20            102b  4MB should be 8MB        --> replaced 18-Aug-2011
sx200is         100c  4MB should be 8MB
sx10            101a  4MB should be 8MB        --> replaced 03-Jul-2011
ixus870_sd880   100e  4MB should be 8MB
ixus960_sd950   100d  4MB should be 8MB        --> replaced 01-Jul-2011
ixus860_sd870   100c  2.9MB should be 8MB      --> replaced 03-Jul-2011
ixus90_sd790    100d  3.1MB should be 8MB
ixus90_sd790    100c  4MB should be 8MB
ixus80_sd1100   100c  4MB should be 8MB
g9              100i  3.2MB should be 8MB
a590            101b  2.9MB should be 4MB      --> replaced 27-Jun-2011
s90             100c  corrupt
I'll update this table when replacing these dumps in the box repository...
« Last Edit: 19 / August / 2011, 06:04:25 by fe50 »
Logged
*

Offline funnel

  • ****
  • 349
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #6 on: 27 / June / 2011, 11:56:35 »
I think I spotted a small bug. In stubs_min.S it found the zoom_status incorrectly. The correct address is 0xF5A4

Code: [Select]
//DEF(zoom_status                             ,0x0000f5c4) // Found @0xff31c3ec,          stubs_min = 0x0000f5c4 (0xF5C4)
Code: [Select]
FF31C3DC sub_FF31C3DC                            ; CODE XREF: sub_FF1BAE40:loc_FF1BAE90p
FF31C3DC                 STMFD   SP!, {R4,LR}
FF31C3E0                 ADR     R1, aTerminatedeliv ; "TerminateDeliverToZoomController"
FF31C3E4                 MOV     R0, #0x60
FF31C3E8                 BL      sub_FF087AC8
FF31C3EC                 LDR     R1, =0xF5A4
FF31C3F0                 MOV     R0, #0
FF31C3F4                 STR     R0, [R1]
FF31C3F8                 STR     R0, [R1,#4]
FF31C3FC                 LDMFD   SP!, {R4,LR}
FF31C400                 MOV     R1, R0
FF31C404                 LDR     R0, =0x315C
FF31C408                 B       sub_FF31C398

Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #7 on: 27 / June / 2011, 18:47:31 »
@funnel - I presume this is for the SX220?

I use three methods to try and find zoom_status, the 'TerminateDeliverToZoomController' string is the last one I try if the other two fail. I add 0x20 to the value extracted from the code after the reference to the above string - this seems to match with the other cameras I compared to that already had zoom_status defined in stubs_min.S.

It's a bit of a hack so could well be wrong for your camera.

Can you please try displaying the value at each of the two locations (0xF5A4 & 0xF5C4) and see how they change while zooming in and out (include digital zoom range as well as optical). I find the gui_draw_debug_vals_osd function in gui.c the best place to do this.

Phil.

(O.T. anyone else having problems with the 'Quote' function for replying to messages?)
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
*

Offline funnel

  • ****
  • 349
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #8 on: 28 / June / 2011, 02:32:39 »
I guess you're right. I had a crash when zooming that happened shortly after I changed this value so I though that could be wrong.

F5A4:
-standyby 0
-zoom_in 1
-zoom_out 0
-digital in, out 0

F5C4:
-standby 2
-zoom_in 20002
-zoom_out 30002
-digital_zoom_in 2000102
-digital_zoom_out 3000102

Logged
*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: New stubs/signature finder (gensig2/finsig2) for DryOS cameras.
« Reply #9 on: 28 / June / 2011, 04:42:28 »
Quote from: funnel on 28 / June / 2011, 02:32:39
I guess you're right. I had a crash when zooming that happened shortly after I changed this value so I though that could be wrong.

F5A4:
-standyby 0
-zoom_in 1
-zoom_out 0
-digital in, out 0

F5C4:
-standby 2
-zoom_in 20002
-zoom_out 30002
-digital_zoom_in 2000102
-digital_zoom_out 3000102



Those values for 0xF5C4 are consistent with what I get on the G12 and SX30 for what I believe is the zoom_status location.

Of course it could be completely wrong :)

The zoom_status seems to be handled slightly differently for the S90 and S95 in gui.c (other_kbd_process) and I made the G12 and SX30 do the same since the G12 and S95 are very similar (and the SX30 firmware code seems to match the G12 in this area). This might be related to your crash.

It looks like something that needs some closer investigation - if I find anything I'll let you know.

Edit:
I'm pretty sure that the 0xF5C4 value is correct.
On the G12 when this value goes to 0x102 in movie record mode (digital zoom range), the code in other_kbd_process sets it back to 0x2 when the digital zoom position goes to 0. This unlocks the optical zoom so you can zoom back into the optical range when recording movies. On the SX30 none of this is needed because the camera supports optical zooming in movie record mode natively - in this case the CAM_CAN_UNLOCK_OPTICAL_ZOOM_IN_VIDEO setting is cleared in the platform_camera.h file.

The zoom_status value is also set in the lens_set_zoom_point function in generic/wrappers.c. Again I'm using custom code for the G12 and SX30 here because the original did not work correctly. I need to review the usage of zoom_status here to see if it is needed / correct.

Hope this helps,

Phil.
« Last Edit: 28 / June / 2011, 08:09:12 by philmoz »
Logged
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)
Pages: [1] 2 3 4 5 6 ... 11   Go Up
« previous next »
 

Related Topics