finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world) - page 3 - General Discussion and Assistance - CHDK Forum

finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

  • 77 Replies
  • 60948 Views
*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #20 on: 03 / January / 2016, 16:23:28 »
Advertisements
An incomplete list of open items. Mostly to get my own thoughts in order, but if anyone has suggestions or wants to work on a particular part feel free to chime in. I will have less time to work on this with the ending, but will continue to chip away on it as I work on the g7x port.

ARM support/thumb:
Currently, the firmware_load_ng.c only operates in either ARM or Thumb mode. findsig_thumb2 essentially assumes everything is thumb. Following calls requires supporting both. As far as I can tell, capstone requires separate handles for ARM and thumb.

Matching:
I stripped out a lot of stuff just to get something running
* Unlike the old sig finder, finsig_thumb2 has not concept of multiple or partial matches. It just stuffs the most recent one in func_names. It also doesn't report the match method
* Finding things the current match depends on. In the old sig finder, get_saved_sig will run the find func for for the named function if there it isn't already found.
* Better finding of code refs. Because disassembly is relatively expensive, scanning the whole firmware multiple times for e.g. branches targets constant refs isn't viable. I've been thinking about an initial scan that looks for all the b, bl, blx, adr, ldr etc and puts them in easily searchable arrays. For backtracking this would need to store some address or instruction size history, or some way of syncing up disassembly e.g backtrack X words, try to disassemble, see if you get valid instructions and end up on the same thing the references was from, if not adjust by half word, try again.

Find more stuff:
Enough code should be in place to write matches for specific functions and useful constants.
Mode map checking etc should be re-added.


Test / add support for additional cameras.
* I made a zip with a dummy platform trees for the D6 cams I have dumps for, and script to run the new sig finder on all of them https://app.box.com/s/v4y9wn172drai63lx76gkliw9it3rq8p (note I've assumed all the ROMs start at FC000000). I haven't spent much time looking at the results yet.
* The additional copied RAM area srsa noted looks like it might be important, e.g. for g5x
add_event_proc: CreateCountingSemaphore disassembly failed at 0xbfe15351
At this point should probably have generic way of dealing with multiple RAM/ROM regions


capdis
* Load known function names from csv, use in listing
* Show string refs
* Eventually get it to the point where it can be used on a whole firmware, like thumb2dis.pl

codegen:
It should be possible to re-use a lot of the capdis code. Would be nice to refactor the processing / parsing code such that old and new codgen don't duplicate too much. I will probably hold off working on this until the firmware_load and capdis code are more mature.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #21 on: 04 / January / 2016, 02:34:11 »
Quote
The additional copied RAM area srsa noted looks like it might be important, e.g. for g5x
add_event_proc: CreateCountingSemaphore disassembly failed at 0xbfe15351
I added code to detect and list these in stubs_entry, but it doesn't yet use it for sig finding / disassembly.
Quote
At this point should probably have generic way of dealing with multiple RAM/ROM regions
I started working on this, but it's not functional yet.

I wonder what these new regions are. Some observations:
* It's small, ~28k on g5x
* There's no cache clean/flush after the copy, unlike the main code.
Some kind of TCM maybe?
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #22 on: 05 / January / 2016, 01:11:56 »
In 4329, finsig_thumb2 handles the second RAM range. This is still preliminary, and the new code is quite a bit slower. edit: fixed.
« Last Edit: 05 / January / 2016, 02:14:57 by reyalp »
Don't forget what the H stands for.

*

Offline msl

  • *****
  • 1280
  • A720 IS, SX220 HS 1.01a
    • CHDK-DE links
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #23 on: 05 / January / 2016, 16:57:35 »
Thanks for this good basic work.

I was able to generate the stubs for the SX280.
Code: [Select]
Entering to ./platform/sx280hs/sub/102b
boot.c -> boot.o
stubs_min.S -> stubs_min.o
stubs_auto.S -> stubs_auto.o
-> stubs_entry.S
failed to get export/register eventproc args at 0xfc1a7ef6
failed to get CreateTask args at 0xfc1abec8
failed to get export/register eventproc args at 0xfc29abce
failed to get RegisterEventProcTable arg 0x00000000 at 0xfc29b0e8
task name name not string at 0xfc29ea54
task name name not string at 0xfc29f0ac
failed to get RegisterEventProcTable arg 0x00000000 at 0xfc3b81e2
Time to generate stubs 6.89 seconds
stubs_entry.S -> stubs_entry.o
lib.c -> lib.o
stubs_entry_2.S -> stubs_entry_2.o
i5-5257U Win10 with the gcc toolchain from the CHDK shell

Btw., there is a small c&p error in the description in reply#13: CAPSTONE_TOOLS_LINK=-Lc:/capstone/lib -lcapstone

msl
CHDK-DE:  CHDK-DE links


*

Offline srsa_4c

  • ******
  • 4451
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #24 on: 05 / January / 2016, 17:17:56 »
I was able to generate the stubs for the SX280.
Code: [Select]
Time to generate stubs 6.89 seconds
FWIW, my result is
Code: [Select]
Time to generate stubs 4.53 secondson 32 bit linux, C2D @ 3GHz.
The g5x dump took ~10.5 seconds on Vista x86, ~9.5 seconds on linux (done a few days earlier and using reyalp's capstone lib on Windows).
Really wonder about the low performance on those modern CPUs...

*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #25 on: 05 / January / 2016, 20:53:45 »
Btw., there is a small c&p error in the description in reply#13: CAPSTONE_TOOLS_LINK=-Lc:/capstone/lib -lcapstone
Thanks, edited.
Quote from: srsa_4c
The g5x dump took ~10.5 seconds on Vista x86, ~9.5 seconds on linux (done a few days earlier and using reyalp's capstone lib on Windows).
Really wonder about the low performance on those modern CPUs...
Yeah, that's odd. I'm not worrying much about performance right now, there's room to optimize and will be a long before we have enough d6 ports to be a real problem, but the spread is strange.

FWIW, sx280 takes ~9.5 seconds on my  system: win7 64 bit, 32 bit tool chain, core i5 2400
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #26 on: 06 / January / 2016, 01:43:48 »
I added ctypes detection in r4331 and use the cytpes location as the upper limit for disassembling normal ROM code. This seems to be a valid assumption in current firmwares, and cuts the sx280 time down to 5.5 sec on my system. There is other code after ctypes, but from what I can tell it's all copied or the other mysterious dryos firmwares.
Don't forget what the H stands for.

*

Offline philmoz

  • *****
  • 3450
    • Photos
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #27 on: 07 / January / 2016, 16:45:23 »
The g5x dump took ~10.5 seconds on Vista x86, ~9.5 seconds on linux (done a few days earlier and using reyalp's capstone lib on Windows).
Really wonder about the low performance on those modern CPUs...

Mint LMDE 64 bit (VM running on OS X, 2.8GHz i7) - g5x takes 2.29 seconds.

Phil.
CHDK ports:
  sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
  g12 (1.00c, 1.00e, 1.00f & 1.00g)
  sx130is (1.01d & 1.01f)
  ixus310hs (1.00a & 1.01a)
  sx40hs (1.00d, 1.00g & 1.00i)
  g1x (1.00e, 1.00f & 1.00g)
  g5x (1.00c, 1.01a, 1.01b)
  g7x2 (1.01a, 1.01b, 1.10b)


*

Offline reyalp

  • ******
  • 14080
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #28 on: 11 / January / 2016, 02:29:39 »
A warning for anyone trying to use this. I found a fairly serious bug in the capstone disassembly:
Code: [Select]
objdump  fc065e7a: f269 ea6c blx loc_fc2cf354
capstone fc065e7a: f269 ea6c blx sub_fc2cf358
(above from sx60 100b)
I've verified that the objdump version is correct. The bug is still present in the latest source from the capstone github. It seems to have to do with blx from thumb where the calling address isn't word aligned, but I'm still trying to understand exactly where/how. There was an issue that looked somewhat related to this https://github.com/aquynh/capstone/pull/158

Also, the function identified as CreateTask_arm in the current svn finsig_thumb2 is actually CreateTaskStrictly. I have a bunch of changes I was almost ready to commit when I ran into this bug.
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4451
Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)
« Reply #29 on: 11 / January / 2016, 14:15:26 »
A warning for anyone trying to use this. I found a fairly serious bug in the capstone disassembly:
Code: [Select]
objdump  fc065e7a: f269 ea6c blx loc_fc2cf354
capstone fc065e7a: f269 ea6c blx sub_fc2cf358
(above from sx60 100b)
From the armv7 ref. manual:
Quote
For BLX (encodings T2, A2), the assembler calculates the required value of the offset from
the Align(PC,4) value of the BLX instruction to this label, then selects an encoding that sets
imm32 to that offset.
So the following would be my guess for a fix (arch/ARM/ARMInstPrinter.c, this is from the next branch but this part of the code hasn't changed since 3.0.4):
Code: [Select]
--- ARMInstPrinter_orig.c       2016-01-10 22:07:30.000000000 +0100
+++ ARMInstPrinter.c    2016-01-11 20:01:41.653172595 +0100
@@ -838,13 +838,15 @@
                if (ARM_rel_branch(MI->csh, opc)) {
                        // only do this for relative branch
                        if (MI->csh->mode & CS_MODE_THUMB) {
-                               imm += (int32_t)MI->address + 4;
                                if (ARM_blx_to_arm_mode(MI->csh, opc)) {
                                        // here need to align down to the nearest 4-byte address
-#define _ALIGN_DOWN(v, align_width) ((v/align_width)*align_width)
-                                       imm = _ALIGN_DOWN(imm, 4);
+#define _ALIGN_DOWN(v, align_width) (((v)/align_width)*align_width)
+                                       imm += (int32_t)_ALIGN_DOWN(MI->address + 4, 4);
 #undef _ALIGN_DOWN
                                }
+                               else {
+                                       imm += (int32_t)MI->address + 4;
+                               }
                        } else {
                                imm += (int32_t)MI->address + 8;
                        }
I only verified the code snippet from your example, so it's possible that I'm wrong.

 

Related Topics