finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #30 on: 12 / January / 2016, 02:06:31 »

Advertisements

Quote from: srsa_4c on 11 / January / 2016, 14:15:26

So the following would be my guess for a fix (arch/ARM/ARMInstPrinter.c, this is from the next branch but this part of the code hasn't changed since 3.0.4):
...
I only verified the code snippet from your example, so it's possible that I'm wrong.

Thanks. That seems to work for me. So does replacing the whole _ALIGN_DOWN thing with imm &= ~3;

I'll put something on the capstone issue tracker.

I suspect this has gone unnoticed because it only affects addresses that have the high bit set.

I've checked in my lastest finsig_thumb2 work. This finds a lot more tasks and includes more code to handle arm/thumb transitions, but requires a patched capstone. I also fixed a bug get_call_const_args that caused a mis-identified eventproc on some firmwares.

FWIW, if you are building capstone, you can make an ARM (including thumb) only library using.
CAPSTONE_ARCHS="arm" ./make.sh default

I've uploaded a patched windows library to https://app.box.com/s/sshu7dv0mebvnee6gvpsex46y2m18w51

Logged

Don't forget what the H stands for.

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #31 on: 19 / January / 2016, 00:32:16 »

Quote from: reyalp on 03 / January / 2016, 16:23:28

capdis
* Load known function names from csv, use in listing
* Show string refs

These are both implemented. If -stubs is used, functions are loaded form funcs_by_name.csv, stubs_entry.S and stubs_entry_2.S

It also uses the firmware_load code to identify RAM regions (by default, can be disabled with -nofwdata), so you can specify RAM addresses directly.

The output is pretty usable now. I expect there are quite a few bugs still lurking, but it can now be used to disassemble large chunks of code with string refs and functions named. e.g., to get the g7x RAM code disassembled at the right location, I used

./capdis.exe ../../dumps/g7x/sub/100d/PRIMARY.BIN 0xfc000000 -stubs=../platform/g7x/sub/100d -s=0x10e1001 -e=0x110dc1c -f=objdump -d-const -d-addr -d-bin > ../../dumps/g7x/sub/100d/RAMCODE.DIS

Note capdis now sets the initial arm/thumb mode based on the start address.

Logged

Don't forget what the H stands for.

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #32 on: 12 / November / 2016, 20:13:49 »

I put some documentation on the wiki http://chdk.wikia.com/wiki/Capdis_Disassembly_Tool

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #33 on: 19 / November / 2016, 11:40:52 »

@reyalp
Attached is the finsig_thumb2 version of the levent table dumper patch (changeset 4723 for the other 2 finsig's). I decided to not check it in, because I figured it may not be desirable to output the additional file (by default).

edit:
Looks like there already is a utility in tools for this purpose, so this work is kinda redundant...

finsig_thumb2_levent_dumper.diff (1.5 kB - downloaded 39 times.)

« Last Edit: 19 / November / 2016, 19:27:14 by srsa_4c »

Logged

axman

145

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #34 on: 30 / May / 2017, 01:19:44 »

Apologies in advance.. Before I further compound recent blunders, I could use some feedback;

Until the build and install of capstone source 3.0.5-rc2, I could (using my distro toolchain, and unpatched capstone) make any CHDK trunk/ target I wanted; tools worked properly. Eg, dumpchk -fix did not core dump, capstone could read inside the dumps. The CHDK's that I built booted and ran in the cams.

Since the build and install of capstone source 3.0.5-rc2, I can't build dumpchk or capdis correctly. Compiler spits warnings, (perhaps relevant) but makes the files. dumpchk core dumps. Getting this on ubuntu 16.04 and 17.04, both x86_64.

Does not seem to matter now if I use distro toolchain for gcc-arm-none-eabi or not. Does not seem to matter if source capstone or distro capstone is used.. any clue helps, thanks.

Compiler err

Code: [Select]

:~/code/new_trunk/tools$ make OPT_USE_GCC_EABI=1 extras
rawconvert.c -> rawconvert.o
rawconvert.o -> rawconvert
yuvconvert.c -> yuvconvert.o
yuvconvert.c: In function 'main':
yuvconvert.c:212:9: warning: pointer targets in assignment differ in signedness [-Wpointer-sign]
   p_yuv = in_data;
         ^
yuvconvert.o -> yuvconvert
find_levent.c -> find_levent.o
dumputil.c -> dumputil.o
find_levent.o dumputil.o -> find_levent
find_eventproc.c -> find_eventproc.o
find_eventproc.c: In function 'find_event_proc':
find_eventproc.c:23:32: warning: pointer targets in passing argument 2 of 'find_word_aligned' differ in signedness [-Wpointer-sign]
   while(find_word_aligned(dump,&i,str_addr)) {
                                ^
In file included from find_eventproc.c:7:0:
dumputil.h:16:5: note: expected 'unsigned int *' but argument is of type 'int *'
 int find_word_aligned(dump_t *dump, unsigned *index, uint32_t word);
     ^~~~~~~~~~~~~~~~~
find_eventproc.o dumputil.o -> find_eventproc
dumpchk.c -> dumpchk.o
dumpchk.c: In function 'main':
dumpchk.c:111:3: warning: ignoring return value of 'fread', declared with attribute warn_unused_result [-Wunused-result]
   fread(buf, 1, size, dumpfile);
   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
dumpchk.o -> dumpchk
extract_error_table.c -> extract_error_table.o
extract_error_table.o dumputil.o -> extract_error_table
capdis.c -> capdis.o
capdis.c: In function 'do_dis_range':
capdis.c:866:17: warning: format not a string literal and no format arguments [-Wformat-security]
                 printf(comment_start);
                 ^~~~~~
capdis.c:905:25: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
                         printf("%04x     ",*(unsigned short *)is->insn->bytes);
                         ^~~~~~
capdis.c:907:25: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
                         printf("%04x %04x",*(unsigned short *)is->insn->bytes,*(unsigned short *)(is->insn->bytes+2));
                         ^~~~~~
stubs_load.c -> stubs_load.o
firmware_load_ng.c -> firmware_load_ng.o
firmware_load_ng.c: In function 'firmware_load':
firmware_load_ng.c:1608:5: warning: ignoring return value of 'fread', declared with attribute warn_unused_result [-Wunused-result]
     fread(fw->buf8, 1, fw->size8, f);
     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
capdis.o stubs_load.o firmware_load_ng.o -> capdis

and informing about the core dump

Code: [Select]

:~/code/new_trunk/tools$ ./dumpchk -fix ../../chdk_trunk/tools/PRIMARY.BIN.M10
../../chdk_trunk/tools/PRIMARY.BIN.M10 0x2000000 (33554432) bytes no sig found
*** Error in `./dumpchk': double free or corruption (!prev): 0x000055cdc3d6d010 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f5f5457a08b]
/lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f5f545836fa]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f5f5458712c]
/lib/x86_64-linux-gnu/libc.so.6(fclose+0x132)[0x7f5f5456f522]
./dumpchk(+0xc23)[0x55cdc30b1c23]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f5f545213f1]
./dumpchk(+0xeaa)[0x55cdc30b1eaa]
======= Memory map: ========
55cdc30b1000-55cdc30b3000 r-xp 00000000 08:04 336693                     /home/ajm/code/new_trunk/tools/dumpchk
55cdc32b2000-55cdc32b3000 r--p 00001000 08:04 336693                     /home/ajm/code/new_trunk/tools/dumpchk
55cdc32b3000-55cdc32b4000 rw-p 00002000 08:04 336693                     /home/ajm/code/new_trunk/tools/dumpchk
55cdc3d6d000-55cdc3d8e000 rw-p 00000000 00:00 0                          [heap]
7f5f50000000-7f5f50021000 rw-p 00000000 00:00 0 
7f5f50021000-7f5f54000000 ---p 00000000 00:00 0 
7f5f542ea000-7f5f54300000 r-xp 00000000 08:04 5247577                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f54300000-7f5f544ff000 ---p 00016000 08:04 5247577                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f544ff000-7f5f54500000 r--p 00015000 08:04 5247577                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f54500000-7f5f54501000 rw-p 00016000 08:04 5247577                    /lib/x86_64-linux-gnu/libgcc_s.so.1
7f5f54501000-7f5f546be000 r-xp 00000000 08:04 5247537                    /lib/x86_64-linux-gnu/libc-2.24.so
7f5f546be000-7f5f548be000 ---p 001bd000 08:04 5247537                    /lib/x86_64-linux-gnu/libc-2.24.so
7f5f548be000-7f5f548c2000 r--p 001bd000 08:04 5247537                    /lib/x86_64-linux-gnu/libc-2.24.so
7f5f548c2000-7f5f548c4000 rw-p 001c1000 08:04 5247537                    /lib/x86_64-linux-gnu/libc-2.24.so
7f5f548c4000-7f5f548c8000 rw-p 00000000 00:00 0 
7f5f548c8000-7f5f548ed000 r-xp 00000000 08:04 5247509                    /lib/x86_64-linux-gnu/ld-2.24.so
7f5f54acb000-7f5f54acd000 rw-p 00000000 00:00 0 
7f5f54ae9000-7f5f54aed000 rw-p 00000000 00:00 0 
7f5f54aed000-7f5f54aee000 r--p 00025000 08:04 5247509                    /lib/x86_64-linux-gnu/ld-2.24.so
7f5f54aee000-7f5f54aef000 rw-p 00026000 08:04 5247509                    /lib/x86_64-linux-gnu/ld-2.24.so
7f5f54aef000-7f5f54af0000 rw-p 00000000 00:00 0 
7fff23597000-7fff235b8000 rw-p 00000000 00:00 0                          [stack]
7fff235b8000-7fff235ba000 r--p 00000000 00:00 0                          [vvar]
7fff235ba000-7fff235bc000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

Logged

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #35 on: 30 / May / 2017, 12:58:36 »

Quote from: axman on 30 / May / 2017, 01:19:44

Since the build and install of capstone source 3.0.5-rc2, I can't build dumpchk or capdis correctly. Compiler spits warnings, (perhaps relevant) but makes the files. dumpchk core dumps. Getting this on ubuntu 16.04 and 17.04, both x86_64.

FWIW, dumpchk is a very old tool which probably doesn't work on modern firmware dumps. It shouldn't crash, but you shouldn't need it, and whatever the problem is shouldn't be related to capstone or the thumb2 tools.

The warnings all look normal to me and shouldn't be related to the crash.

The OPT_USE_GCC_EABI=1 should only affect building camera side code. Unless there's something very weird in your configuration, host programs like capdis should always be built using your distro gcc.

If you can describe what problem you have with capdis, I'll take a look at it.

Logged

Don't forget what the H stands for.

axman

145

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #36 on: 30 / May / 2017, 13:59:43 »

Quote from: reyalp on 30 / May / 2017, 12:58:36

FWIW, dumpchk is a very old tool which probably doesn't work on modern firmware dumps. It shouldn't crash, but you shouldn't need it, and whatever the problem is shouldn't be related to capstone or the thumb2 tools.

Thanks for that info.

Quote

The warnings all look normal to me and shouldn't be related to the crash.

The OPT_USE_GCC_EABI=1 should only affect building camera side code. Unless there's something very weird in your configuration, host programs like capdis should always be built using your distro gcc.

Makes sense now, thanks.

Code snip below shows the "Warning" about incorrect disassembly, using patched capstone source libs. PRIMARY.BIN is freshly dumped from ixus160 using universal method. And capdis was built fresh using the 'host' gcc, and not gcc-arm-none-eabi.

I'm trying to follow the 160's filewrite process through the code.

Code: [Select]

:~/code/new_trunk$ tools/capdis -v -s='0xffab8cf0' -e='0xffab8d3f' -stubs=platform/ixus160_elph160/sub/100a/ -d ../ixus160/PRIMARY.BIN 0xff810000
WARNING code start offset not found, assuming 0
WARNING! Incorrect disassembly is likely
; ../ixus160/PRIMARY.BIN size:0x7efffc start:0xffab8cf0 instructions:39 opts:0x1f00c7
; task_FileWrite 0xffab8cf0
; 0xffab8cf0 fe 40 2d e9
; OPERANDS 8:
...

I don't understand. Using patched capstone. Capdis complains like it is using non-patched, yet it seems to return the correctly disassembled stuff..

I say "correctly disassembled" not because I know what I'm doing, but because in trunk/ ixus160, the filewrite source code comments identified the markers, and I think I see them in what capdis returns..

Logged

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #37 on: 30 / May / 2017, 17:20:10 »

Quote from: axman on 30 / May / 2017, 13:59:43

Code snip below shows the "Warning" about incorrect disassembly, using patched capstone source libs.

The code that generates that warning is a test case for the actual bug, so if the warning appears, then the bug is present in the linked capstone library, or there's a bug in my test case.

The bug only affects very specific cases in thumb/thumb2 code, so your ixus160 disassembly should not be affected at all, and digic 6 firmware will *look* reasonable, it will just have incorrect addresses in certain cases.

Did you set CAPSTONE_TOOLS_INC and CAPSTONE_TOOLS_LINK in localbuildconf.inc to point at your self built capstone? If not, it might still be picking up distro packages. If you haven't already, you might consider uninstalling any other capstone packages. It's possible that you are linking to the right code but picking up the wrong shared library. You can use ldd to find out what shared library it's using.

The "WARNING code start offset not found, assuming 0" is not related to the capstone version. It's caused by the firmware loading code not being set up to fully handle recent pre-digic 6 firmwares like like ixus160. Disassembly should still work (I think) but these tools were mostly written for digic 6 firmware.

Logged

Don't forget what the H stands for.

axman

145

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #38 on: 30 / May / 2017, 19:48:34 »

Quote from: reyalp on 30 / May / 2017, 17:20:10

The code that generates that warning is a test case for the actual bug, so if the warning appears, then the bug is present in the linked capstone library, or there's a bug in my test case.

I'm not qualified to eval capdis code.. So I made sure that capstone patch is applied, built and installed fresh. I hacked on capstone /tests/test_arm.c and changed what was being tested for Thumb: From some string To ffab8cf0 fe 40 2d e9

which is the entry point to ixus160 filewrite task, along with first instruction, PUSH.

I compared capdis disassembly to capstone test_ARM disassembly. Capstone test results do not match up with arm instructions I get from capdis for the same disassembly range.

capstone test_arm.c results, hacked to evaluate ffab8cf0 fe 40 2d e9

Code: [Select]

****************
Platform: Thumb
Code:0xff 0xab 0x8c 0xf0 0xfe 0x40 0x2d 0xe9 
Disasm:
0x80001000:    add    r3, sp, #0x3fc
    op_count: 3
        operands[0].type: REG = r3
        operands[1].type: REG = sp
        operands[2].type: IMM = 0x3fc

0x80001002:    eor    r0, ip, #0x7f000000
    op_count: 3
        operands[0].type: REG = r0
        operands[1].type: REG = ip
        operands[2].type: IMM = 0x7f000000

0x80001006:

Clearly, capstone test does not disassemble the same way capdis does. I don't know any other way to eval the situation.. Learning much but not capable to judge what this means..

Quote

Did you set CAPSTONE_TOOLS_INC and CAPSTONE_TOOLS_LINK in localbuildconf.inc to point at your self built capstone?

No, I set them in buildconf.inc . Is localbuild intended to reside in /tools or /trunk?

Quote

If not, it might still be picking up distro packages. If you haven't already, you might consider uninstalling any other capstone packages. It's possible that you are linking to the right code but picking up the wrong shared library. You can use ldd to find out what shared library it's using.

Distro pkgs removed. ldd identifies the source-built libcapstone in /usr/lib .

Quote

The "WARNING code start offset not found, assuming 0" is not related to the capstone version. It's caused by the firmware loading code not being set up to fully handle recent pre-digic 6 firmwares like like ixus160. Disassembly should still work (I think) but these tools were mostly written for digic 6 firmware.

Thanks Super Much for the guidance, apologies for time consumed with capstone AGAIN. I'm embarrassed about this, just trying to get past the setup and on to the real work, and believing that what the tools put out is correct is critical.

Now I can hopefully move on from messing with toolchain and utilities, and get to the sigfinding and arm decoding.

Logged

reyalp

14079

Re: finsig and other tools for thumb2 (was Re: chdk in the DIGIC6 world)

« Reply #39 on: 30 / May / 2017, 22:22:34 »

Quote from: axman on 30 / May / 2017, 19:48:34

I'm not qualified to eval capdis code.. So I made sure that capstone patch is applied, built and installed fresh. I hacked on capstone /tests/test_arm.c and changed what was being tested for Clearly, capstone test does not disassemble the same way capdis does. I don't know any other way to eval the situation.. Learning much but not capable to judge what this means..

Sorry, I really don't understand what you are trying to do.

The code that checks for the bug is in tools/firmware_load_ng.c do_blx_check

It defines the specific sequence of instructions that would trigger the bug and then checks the results. The bug specifically relates to BLX instructions from thumb, where the instruction isn't word aligned, and target address has the highest bit set. Again, since this is specific to thumb code, it would not affect disassembly of pre-digic 6 firmwares.

Quote

Quote
Did you set CAPSTONE_TOOLS_INC and CAPSTONE_TOOLS_LINK in localbuildconf.inc to point at your self built capstone?
No, I set them in buildconf.inc . Is localbuild intended to reside in /tools or /trunk?

Using buildconf.inc is fine. localbuildconf.inc does not exist by default, but can be used to override buildconf.inc settings without changing a file that is in version control.

Logged

Don't forget what the H stands for.