eos M10 port - DryOS Development - CHDK Forum supplierdeeply
« previous next »
Pages: [1] 2 3 4 5 6 ... 23   Go Down

eos M10 port

  • 221 Replies
  • 98042 Views
*

Offline axman

  • ***
  • 145
eos M10 port
« on: 15 / February / 2016, 15:08:02 »
Advertisements
As time permits, I'm having a go at porting this cam.  Let's see.  Thanks, reyalp for guidance on toolchain here.   I believe Thumb2 imposes requirement for for capstone?  I read all I could find.  first mention then finsig stuff here then the patched Win lib here   Which all looks promising, but not immediately useful to me with build env = ubuntu Precise.  I will read more about that patch and see if I can apply it to my copy of capstone src before building chdk_src.

No shame in saying it;  I lack some of the fundamental skills needed to attempt this port.  Probably the most challenging task I've taken on in 25+ years in and around computing.  Apologies in advance for stupid questions bound to occur.  Progress will be slow.  Books are on the way; ARM & C stuff.  I have zero expectations of hand-holding time from gurus and experts, but any guidance is appreciated when offered.

edit: will begin chdk_src build with PLATFORM=sx280hs PLATFORMSUB=102c, just a starting point until it compiles nicely.
« Last Edit: 18 / January / 2017, 14:44:24 by acseven »
Logged
*

Offline reyalp

  • ******
  • 11923
Re: eos M10 port
« Reply #1 on: 15 / February / 2016, 16:12:23 »
Quote from: axman on 15 / February / 2016, 15:08:02
As time permits, I'm having a go at porting this cam.  Let's see.  Thanks, reyalp for guidance on toolchain here.   I believe Thumb2 imposes requirement for for capstone?  I read all I could find.  first mention then finsig stuff here then the patched Win lib here   Which all looks promising, but not immediately useful to me with build env = ubuntu Precise.  I will read more about that patch and see if I can apply it to my copy of capstone src before building chdk_src.
You should be able to apply the patch to the capstone 3.0.4 source no problem, it's included in the windows library zip.

I ran your dump through the current finsig_thumb2 and it seems to work reasonably well, although there's a few things that aren't found.

Ant apparently has an at least semi-working EOS M3 port. AFAIK the source hasn't been made public, but collaboration would probably be useful since CHDK hasn't previously supported interchangeable lens cameras.

If you want realtime assistance, you can sometimes get it in the IRC channel (#chdk on freenode). I'm mostly on evenings and weekends US Pacific time.

Quote
edit: will begin chdk_src build with PLATFORM=sx280hs PLATFORMSUB=102c, just a starting point until it compiles nicely.
Note the current sx280 code in the trunk doesn't use finsig thumb2. You can use g7x port as an example to see how it's set up. edit: enabled in trunk 4451

You will also probably want to be able to build the ixus160, since you already have that camera. Having a working port can be very useful to understand how things are supposed to work.
« Last Edit: 19 / February / 2016, 13:42:32 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline axman

  • ***
  • 145
Re: eos M10 port
« Reply #2 on: 15 / February / 2016, 17:50:15 »
Hello, thanks for the feedback & gracious offer to assist.  Trying not to beat you up too bad with dumb stuff..
Quote from: reyalp on 15 / February / 2016, 16:12:23
You should be able to apply the patch to the capstone 3.0.4 source no problem, it's included in the windows library zip.
  Used the code snip from this post
Quote from: srsa_4c on 11 / January / 2016, 14:15:26
..
  i modded capstone arch/ARMInstPrinter.c according to that post, and built capstone.  Included capstone with $CHDK_SRC/trunk/buildconf.inc

Code: [Select]
OPT_CURVES=1
OPT_DEBUGGING=1
OPT_PTP=1
OPT_GEN_SIGS=1
OPT_GEN_STUBS=1
OPT_FIRMWARE_PC24_CALL=1
OPT_USE_GCC_EABI=1
OPT_DISABLE_COMPAT_CHECK=1
OPT_CAPSTONE_TOOLS=1
CAPSTONE_TOOLS_INC=-I/$HOME/code/capstone-3.0.4/include
CAPSTONE_TOOLS_LINK=-L/$HOME/code/capstone-3.0.4 -lcapstone
DEF_SVN_REF=0

CHDK build from src completes with this cmd;
Code: [Select]
make PLATFORM=sx280hs PLATFORMSUB=102b firResults in a DISKBOOT.BIN, caminfo.txt  I'm happy to continue.

Re: collaboration, happy to share.  Will also borrow and adapt existing stuff from elsewhere, where-ever possible.
Quote
You will also probably want to be able to build the ixus160, since you already have that camera. Having a working port can be very useful to understand how things are supposed to work.
Great advice, thanks.  Setting it up in separate build area now.  Also saw a recent post from srsa_4c re: a visual tool to assist with the firmware..  Hope it's ok, I will snag & try it.  It looked like an invaluable learning aid.
Logged
*

Offline srsa_4c

  • ******
  • 3920
Re: eos M10 port
« Reply #3 on: 19 / February / 2016, 13:13:58 »
Quote from: reyalp on 15 / February / 2016, 16:12:23
Ant apparently has an at least semi-working EOS M3 port. AFAIK the source hasn't been made public, but collaboration would probably be useful since CHDK hasn't previously supported interchangeable lens cameras.
It is available publicly, it just wasn't advertised. No visible activity since months, though.

Quote from: axman on 15 / February / 2016, 17:50:15
Also saw a recent post from srsa_4c re: a visual tool to assist with the firmware..
To clarify: that tool is mostly useful for finding framebuffers (and their parameters) in a camera RAM dump.
Logged
*

Offline axman

  • ***
  • 145
Re: eos M10 port
« Reply #4 on: 19 / February / 2016, 19:37:53 »
Quote from: srsa_4c on 19 / February / 2016, 13:13:58
[To clarify: that tool is mostly useful for finding framebuffers (and their parameters) in a camera RAM dump.
 
Thanks for clarification.  I used it on firmware dump anyway, it's helpful for me seeing hex and ascii in a split window.  Also used the perl you've provided.  Goes without saying, thanks for all your efforts.  I'm learning, immensely, by reading what you guys do and trying to understand it, then trying to mimic and apply that to what's in my hands.  I'd be absolutely nowhere without what you've shared.

Re: M10     Spent 3 days getting a toolchain which can build ixus160 to a working .BIN state.  Proved my toolchain is good by compiling 160 from source, then booting what I built-from-src in my 160.  @Reyalp, the recommended prebuilt toolchain didn't work for me, nor did their OS-specific-pkg toolchain.  Working toolchain was made by following gcc 4.6.4 version on Ubuntu 15.10, x86_64.  Took a lot of notes from the various and numerous failures.  My toolchain also builds the "patched" capstone libs.  Used capstone's test_ARM code, results good.  I had a lot of trouble with capstone..

Spent a few more days reading the source and looking at how things are done in the 160.  Read posts from 160 port thread, many sx280 port threads.  Eyes bleeding, but #stiptic_pencil. 

Copied g7x stuff as starting point into $chdk_src/trunk/platform/m10 and $/m10/sub/100a.  Rationale for using g7x; Less likely to damage stuff right away.  Edited tiny parts of makefile.inc, platform_camera.h with what I've learned so far.  Compiled m10 platform using the 'fir' target; even tho I know it can't build, I wanted to see what would happen.  It dies in the $modules section, but $chdk_src/trunk/tools are built.  I *think* the /tools are ok.  I used the resulting finsig_thumb2 against m10 PRIMARY.BIN ok.  I think I see what reyalp was referring to re: missing sigs.  So..  hunting for addresses..

Using these tools, experienced devs and other informed ppl are able to determine that the M10 has DryOS v 57, and the sx280 & g7x are something else. 

While I take reyalp at his word, I need to learn for myself how that value is determined within the BIN file - what address or sig represents the DryOS version?  Same for rowpix vals, and all other critical vals.  Not asking for gurus to solve it for me.  Gotta learn this (and any pre-req's  ! ) before proceeding.  Using $DRYOSVERSION and $ROWPIX as learning examples.
Logged
*

Offline reyalp

  • ******
  • 11923
Re: eos M10 port
« Reply #5 on: 19 / February / 2016, 22:16:10 »
Quote from: axman on 19 / February / 2016, 19:37:53
Re: M10     Spent 3 days getting a toolchain which can build ixus160 to a working .BIN state.  Proved my toolchain is good by compiling 160 from source, then booting what I built-from-src in my 160.  @Reyalp, the recommended prebuilt toolchain didn't work for me, nor did their OS-specific-pkg toolchain.  Working toolchain was made by following gcc 4.6.4 version on Ubuntu 15.10, x86_64.
I am surprised that the pre-built toolchain didn't work out of the box. If you can give more detail about what specifically didn't work, that might other avoid the pain in the future. That said, if you don't want to dive back into this it's understandable ;)

Quote
OPT_FIRMWARE_PC24_CALL=1
I would NOT suggest using that. I'm not sure if it works at all in the current source, let alone for thumb 2,  but in any case known thumb2 firmwares don't meet the requirements specified in the buildconf.inc comment.

Quote
Took a lot of notes from the various and numerous failures.  My toolchain also builds the "patched" capstone libs.
You should be able to build capstone with the regular x86/x64 gcc provided by your distro. I hope this is what you mean.

Quote
Using these tools, experienced devs and other informed ppl are able to determine that the M10 has DryOS v 57, and the sx280 & g7x are something else. 
Other known r57 cams are G3x and sx710.

Philmoz has done quite a bit of work on g5x, which is r58.

sx280 is r52, and g7x is r55p6

Quote
I need to learn for myself how that value is determined within the BIN file - what address or sig represents the DryOS version? 
For that one, I just open up the .bin file in vim and search for DRYOS ;) (finsig_thumb2 finds it too, using essentially the same method)

Quote
Same for rowpix vals, and all other critical vals.  Not asking for gurus to solve it for me.  Gotta learn this (and any pre-req's  ! ) before proceeding.  Using $DRYOSVERSION and $ROWPIX as learning examples.
It depends on the value. For raw dimensions on G7x, I used exiftool to dump a camera CRW  and used the "Sensor Width" and "Sensor Height" values. (only applicable on cameras with native raw)

I checked this by searching the dump for "CRAW BUFF SIZE" (which I know from other cameras refers to the actual sensor data size), and finding the value printed with that.
Code: [Select]
fc4f1ffe: 492d      ldr r1, [pc, #180] ; 0xfc4f20b4: (01de3e00)
fc4f2000: a02d      add r0, pc, #180 ; 0xfc4f20b8: (57415243) *"CRAW BUFF SIZE  %p"
fc4f2002: f63b fb06 bl loc_fc32d612
0x1de3e00 = 31342080 = (5632*3710*12)/8

This is the general method we use to find a lot of stuff:
Find some "landmark" that is somehow associated with the value in a known firmware (e.g. the sx280).
Find the equivalent landmark in your own firmware.

If you are lucky, the landmark is a unique string that is used with the value as in the example above.

If you are less lucky, it's something less straightforward, like the function that references some string is called by some other function, and that function also calls another function that is somehow related to the thing you are looking for. Or maybe you just know that two functions appear close together in the firmware, and you look at the disassembly to recognize the one you want.

From this it should be clear that it's a good idea to have several already known firmwares disassembled, in addition to the one you are working on. For the g7x, I used the sx280 and ixus140 mostly. Even though ixus140 is a different instruction set, much of the code is easily recognizable, and I already have it well analyzed because I did that port.

Some ports have comments on how things are found, but beware that comments are often blindly copied. I've been trying to leave a lot of comments on g7x (also my ixus140_elph130 and d10 ports). The comment usually refer to an address in the camera firmware, or some other landmark.

The "landmark" approach is essentially what finsig_thumb2 does. The bad news for you is that most of the things that are easy to find this way are already found. The stuff that it doesn't find is mostly tricky in some way, though in some cases it's easy for a human but hard to specify in code. There are also a few things I just haven't gotten around to yet.

If you are unlucky, the code related to thing you are looking for isn't similar to any known port. Then you have to do original research. How you do this really depends on what you are looking for.

Sorry for the wall of text, hopefully it's useful.
« Last Edit: 19 / February / 2016, 22:17:59 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline axman

  • ***
  • 145
Re: eos M10 port
« Reply #6 on: 20 / February / 2016, 16:41:44 »
Quote
I am surprised that the pre-built toolchain didn't work out of the box. If you can give more detail about what specifically didn't work, that might other avoid the pain in the future. That said, if you don't want to dive back into this it's understandable ;)
  Attached txt file.  Happy to help others avoid the trouble, but not sure if value in what I wrote..  Best audience may be noobs and novices.
Quote
OPT_FIRMWARE_PC24_CALL=1
I would NOT suggest using that. I'm not sure if it works at all in the current source, let alone for thumb 2,  but in any case known thumb2 firmwares don't meet the requirements specified in the buildconf.inc comment.
Thanks for that info.  So, now I infer that M10 (rather thumb2 firmwares), CHDK does not load in the lowest 16MB.  True?  Haven't followed the code far enough yet to learn where/how that happens.  Probably not ready to absorb more details about it yet.
Quote
You should be able to build capstone with the regular x86/x64 gcc provided by your distro. I hope this is what you mean.
Yes, finally, by way of attached txt file.  Initially, I mis-understood the purpose and method of application of capstone.  Over-thunk it.  I thought the purpose was including capstone or a portion into what is built into CHDK source - actively utilizing capstone dis-assem capabilities in what runs *on the cam.*  Finally realized how to apply it - just as you said above - built with standard stuff on my distro.  Doh, wasted 2 days Learned A Lot.   :haha

Quote
For that one, I just open up the .bin file in vim and search for DRYOS ;) (finsig_thumb2 finds it too, using essentially the same method)
  :)  Thanks.  Learning when to use a hammer instead of an OBD diag box is part of becoming a good mechanic.

Quote
Sorry for the wall of text, hopefully it's useful.
You just wrote me a clear and detailed how2.  Thanks!  :)  But you shouldn't have spent that time..  Grateful for it, tho.  Can't say thanks big enough really, for the time and effort you, and other spend to educate me and the others who attempt porting.  I read a lot here.  I'm getting it Slowly, by reading the code, the code comments, forum posts.  Working hard at this.  I will make more big errors (a `la capstone above) I'm afraid.  Going away for a bit, to read, try and form good questions, try to find answers.
Logged
*

Offline reyalp

  • ******
  • 11923
Re: eos M10 port
« Reply #7 on: 20 / February / 2016, 18:39:43 »
Quote from: axman on 20 / February / 2016, 16:41:44
Thanks for that info.  So, now I infer that M10 (rather thumb2 firmwares), CHDK does not load in the lowest 16MB.  True?  Haven't followed the code far enough yet to learn where/how that happens.  Probably not ready to absorb more details about it yet.
CHDK loads at MEMISOSTART. Where this is depends on the configuration. The "original" location is replacing the start of the canon malloc heap. This is in the low 16 megs for known cameras (AFAIK), but CHDK can be loaded in ARAM or EXMEM, which may not be.

In this case, it doesn't matter though, because all known thumb2 firmwares have their ROM in FC000000-FE000000, which is >=32 megs from the end of ROM, and thus not reachable by wrapping around from low RAM.

Quote
But you shouldn't have spent that time..  Grateful for it, tho
Being able to help people is one of the best parts of this hobby :)
I'm happy to give advice about techniques or where to look for things, or help if you are stumped on some specific problem. I do get grumpy with people who say "just give me step by step instructions" for something that requires actual work and understanding on their part (to be clear, not the case with your posts.)

Some general advice, since I'm not sure what your programming skill set is:
To do this successfully, you need to be able to read ARM and thumb2 assembler, and have at least basic proficiency in C. If you don't yet have these skills you certainly can teach yourself, but it's a substantial undertaking, equivalent to at least several semesters of CS classes.

You don't need to know assembler well enough to write from scratch, but you do need to understand the concepts (registers, addressing modes, conditionals, calling conventions) and be able to understand the specifics from the ARM documentation. The ARM documentation is all free and generally pretty good. (you may have to register to download some documents)

I recommend "ARM Compiler toolchain Assembler Reference" (ARM DUI 0489C ) as a reference for instructions on digic 6 firmwares. For older firmwares, I recommend "ARMv5 Architecture Reference Manual"

For information about the specific CPUs (caches, MPU etc) I suggest "Cortex™-R4 and Cortex-R4F
Technical Reference Manual" for digic 6, and "ARM946E-S™ Technical Reference Manual" for earlier digic. You shouldn't need to refer to these for most normal porting work.
Logged
Don't forget what the H stands for.
*

Offline Hardware_Hacker

  • ****
  • 318
Re: Eos M10 port - Progress Report
« Reply #8 on: 23 / May / 2016, 04:41:55 »
Eos M10 Port - Progress Report

The Sig-Finder is working, see the Demo's in the attached Files.
The IDA_6v8-Demo only partially worked, for me.

Note that the attached files are just a comparison between Sig-Finder generated files,
so as to try to understand whats going on between the M10-100a, Eos M3-100d and  Eos M3-101a.

The Eos M3 Port - Progress Report is here
https://chdk.setepontos.com/index.php?topic=12542.msg128699#msg128699

H-H
« Last Edit: 23 / May / 2016, 04:52:09 by Hardware_Hacker »
Logged
*

Offline reyalp

  • ******
  • 11923
Re: Eos M10 port - Progress Report
« Reply #9 on: 23 / May / 2016, 13:27:40 »
Quote from: Hardware_Hacker on 23 / May / 2016, 04:41:55
Eos M10 Port - Progress Report
Do you have an EOS M3 or M10 camera?
Do you have anything running on these cameras?

If not, then there is not really progress to report, and posting these files only serves to confuse others about the actual state of progress. You have posted this kind of things many times, yet you have never submitted a working port for inclusion in CHDK. Before attempting to work on a digic 6 port, I would strongly recommend finishing one of the older cameras you have. Digic 6 is far more difficult.

Incomplete, non-working "ports" are not really helpful. Anyone with the ability to finish them is likely to be better off starting from scratch.

Running the sig finder is not "progress", anyone capable of doing CHDK development can do it, and we already know it works reasonably well on the EOS M3 and M10 dumps.

There is absolutely no point in posting screenshots of diffs of csvs that anyone can generate and diff themselves.

Posts with actual working code, or specific development information are always welcome, but please, no more pointless "teaser" screenshots and attachments.
Logged
Don't forget what the H stands for.
Pages: [1] 2 3 4 5 6 ... 23   Go Up
« previous next »
 

Related Topics