CHDK PTP interface

reyalp

13624

Re: CHDK PTP interface

« Reply #1210 on: 15 / July / 2019, 00:41:49 »

Advertisements

Quote from: reyalp on 04 / July / 2019, 19:08:10

Next step I guess is to either hook and log PTP functions and/or capture USB traffic.

I went with hooking. Canon UploadFirmware.exe doesn't trigger the crash because it uses OC 0x9019, identified as "PTP_OC_CANON_SendPartialObject" in the gphoto source.

Attached is a log from uploading a 262645 byte file with and corresponding patch. The patch is only set up for A540, but provides a generic mechanism for hooking PTP handlers which could be useful in the future.

Most of the chatty stuff in the log is initializing the hooks and the Canon program querying storage and object handles. The actual upload is
SendObjectInfo

Code: [Select]

00011830 DBG:ptp h:0x001fb6ac d:0x00008d94 oc:0x100c s:0x1 t:0x21 0x0 0x0 0x0 0x0 0x0
00011870 DBG:recv h:0x001bb09c b:0x0021e490 s:0x8d (141) u1:0x0 u2:0x0

2x PTP_OC_CANON_SendPartialObject

Code: [Select]

00011960 DBG:ptp h:0x001fb6ac d:0x00008d94 oc:0x9019 s:0x1 t:0x22 0x0 0x40000 0x1 0x0 0x0
00012000 DBG:recv h:0x001bb09c b:0x11f80000 s:0x40000 (262144) u1:0x0 u2:0x0
00012220 DBG:ptp h:0x001fb6ac d:0x00008d94 oc:0x9019 s:0x1 t:0x23 0x40000 0x1f5 0x3 0x0 0x0
00012270 DBG:recv h:0x001bb09c b:0x11f80000 s:0x1f5 (501) u1:0x0 u2:0x0

Note only object related calls are logged (the ones in the table at 0xffe5bacc on A540) are logged. The transaction ID indicates some other operations were done.

Given that Canon uses this approach, the it seems likely our only fix would be to extend the CHDK protocol to allow similarly splitting uploads over multiple transactions.

This is somewhat complicated for CHDK since with the current code, we don't know in advance how big a chunk can be sent at a time. If we switched to using the Canon PTP buffer, it would be simpler.

I'm not sure this is worth fixing given the lack of complaints in the last 10 years, but at least we have an option.

Quote from: srsa_4c on 06 / July / 2019, 12:26:49

May or may not be related: the low end a4xx series has crippled USB ("full speed"), up to and including a470.

Wow, I thought that was only the really old ones. Interesting.

ps_uploader-ptp-log.txt (3.85 kB - downloaded 24 times.)

ptp-hook-std-ops-work-3.patch (7.33 kB - downloaded 25 times.)

Logged

Don't forget what the H stands for.

reyalp

13624

Re: CHDK PTP interface

« Reply #1211 on: 22 / July / 2019, 02:10:51 »

Quote from: reyalp on 15 / July / 2019, 00:41:49

Given that Canon uses this approach, the it seems likely our only fix would be to extend the CHDK protocol to allow similarly splitting uploads over multiple transactions.

Thinking about this more, I realized it should be possible to work around without resorting multiple transactions or protocol changes. My previous observation was

Quote

The crash happens when
1) data->recv_data is called more than once
2) total size is (512*n) - 11 to (512*n) - 1 i.e. 0x...1f5 through 1ff.

Additionally, when using the native buffer (maybe any uncached) a stalled connection happens for something like 512*n to 512*(n+1)-13. Larger sizes appear to be OK

So if the size is in the magic range (0x...1f4 < size < 0x...3f4), reducing the size of the next-to-last read to ensure the final one is > 0x3f3 should avoid the problem.

From initial testing, this appears to work.

My current test code is attached. I have more testing and cleanup to do, so this is just in case anyone is interested.

canon-ptp-buf-work-7.patch (16.1 kB - downloaded 19 times.)

Logged

Don't forget what the H stands for.

reyalp

13624

Re: CHDK PTP interface

« Reply #1212 on: 04 / August / 2019, 23:44:38 »

I finally checked this in, trunk 5235

The recv_data logic is completely re-worked. I found that problem size varies in a complicated way depending on the size of the preceding chunks (see attached csv), so for simplicity, I just forced the buffer size to be a multiple of 512. The current code also errors if less than 2kb is available for the buffer. This shouldn't be problem in practice because a camera with free memory that low will be unstable anyway.

PTP host to cam transfers now work as follows:
For cases where you want to transfer all the data into a buffer for use later (e.g. script messages, script content) allocate the final buffer yourself and use recv_ptp_data. It will take care of reading correctly sized chunks and buffering through the Canon buffer if CAM_PTP_USE_NATIVE_BUFFER is defined

For cases where you want process the data in the PTP task without copying elsewhere (like file upload)
* recv_ptp_data_init initializes a recv_ptp_data_state_t and gets/allocates a transfer buffer if needed
* recv_ptp_data_chunk receives a chunk of data, as determined by buffer size and other size requirements
* when it's all done (can be detected with recv_ptp_data_state_t total_read == total_size) use recv_ptp_data_finish to free any temp buffer and clear the struct.

CAM_PTP_USE_NATIVE_BUFFER causes the above functions to use 256KB of the Canon firmware file transfer buffer for PTP transfers. This buffer is uncached, so avoids problems like the D10 has. It also appears to be a bit faster on old cameras like A540, and may be be useful on cameras with low free RAM. It should be available on all pre-Digic 6 cams, although it's probably a good idea to check whether the PTP buffer overlaps the EXMEM_HEAP_SKIP video region that causes problems with CHDK exmem (it does not appear to on D10 or SX160)

Other DryOS cams of similar era to D10 are likely to need it. If the chdkptp test xferbug_0x23f4 fails, it should be enabled.

I'd appreciate anyone who can test with the current svn chdkptp running the following:

Code: [Select]

 chdkptp -e"exec require'camtests'.runbatch{bench=true,xfersizebugs=true,filexfer=true}"

Note that the xferbug_0x1f5 test may take something like a minute on some cameras.

ptp-0x1f5-borksizes.csv (0.29 kB - downloaded 21 times.)

« Last Edit: 03 / November / 2019, 17:17:37 by reyalp »

Logged

Don't forget what the H stands for.

srsa_4c

4437

Re: CHDK PTP interface

« Reply #1213 on: 06 / August / 2019, 15:23:28 »

Quote from: reyalp on 04 / August / 2019, 23:44:38

I'd appreciate anyone who can test with the current svn chdkptp running the following:
Code: [Select]
chdkptp -e"exec require'camtests'.runbatch{bench=true,xfersizebugs=true,filexfer=true}"

a410 result attached (I used the GUI). I get the same log (with execution times differing somewhat) with or without CAM_PTP_USE_NATIVE_BUFFER. The ixus65_sd630 passes the test.

transfertest_a410_5235.log (6.99 kB - downloaded 22 times.)

Logged

Menu jumping/flickering on M10?

reyalp

13624

Re: CHDK PTP interface

« Reply #1214 on: 06 / August / 2019, 16:01:52 »

Quote from: srsa_4c on 06 / August / 2019, 15:23:28

a410 result attached (I used the GUI). I get the same log (with execution times differing somewhat) with or without CAM_PTP_USE_NATIVE_BUFFER. The ixus65_sd630 passes the test.

Thanks. The initial failure is not in one of my new tests, but on sending a 52 byte message in the basic message test. Can you try

Code: [Select]

!m=require'extras/msgtest'
!m.test{size=52,count=1}

If that works, try

Code: [Select]

!m=require'extras/msgtest' m.test{size=1,sizeinc=1,count=100}

It would also be good to know if builds before 5235 have the same problem.

The following errors in your log seem to be related to a stalled or disconnected camera rather than a crash, since it picks up again at the reconnect test. I should really make the test try to reconnect / reset between tests if needed.

Quote

I used the GUI

Do you mean you ran it using an executable with GUI support in CLI, or actually ran it in the GUI?

Logged

Don't forget what the H stands for.

srsa_4c

4437

Re: CHDK PTP interface

« Reply #1215 on: 06 / August / 2019, 17:44:50 »

Quote from: reyalp on 06 / August / 2019, 16:01:52

Can you try
Code: [Select]
!m=require'extras/msgtest' !m.test{size=52,count=1}

Code: [Select]

___> !m=require'extras/msgtest'
___> c
connected: Canon PowerShot A410, max packet size 64
con> !m.test{size=52,count=1}
testing 1 messages size 52
send 1 size:52 0x34...I/O error
aborted at send 1 size:52 0x34, communication error
quit failed I/O error
ran 1 fail 1 time 10.0164 send time 0.0109

Quote

It would also be good to know if builds before 5235 have the same problem.

I get the same behaviour with r5204.

I don't actually expect you to fix all ptp problems on this model, I only picked it for this test because it has a history of ptp issues.

Quote

Do you mean you ran it using an executable with GUI support in CLI, or actually ran it in the GUI?

I copied this

Code: [Select]

exec require'camtests'.runbatch{bench=true,xfersizebugs=true,filexfer=true}to the gui window and pressed execute. chdkptp.sh starts the GUI by default.

Logged

Menu jumping/flickering on M10?

reyalp

13624

Re: CHDK PTP interface

« Reply #1216 on: 06 / August / 2019, 18:07:38 »

Quote from: srsa_4c on 06 / August / 2019, 17:44:50

I don't actually expect you to fix all ptp problems on this model, I only picked it for this test because it has a history of ptp issues.

Understood. It would be good to fix or understand if we can, but this seems like a different problem from the other cameras. You could run without the 'bench' option to see if the others pass, like

Code: [Select]

exec require'camtests'.runbatch{xfersizebugs=true,filexfer=true}

Quote

I copied this
Code: [Select]
exec require'camtests'.runbatch{bench=true,xfersizebugs=true,filexfer=true}to the gui window and pressed execute. chdkptp.sh starts the GUI by default.

FWIW, if you pass any command line arguments without -g, it should start in CLI mode as long as the .sh script passes the arguments through like the one in SVN does. I really doubt this has anything to do with the error, but it should be easy to try.

Logged

Don't forget what the H stands for.

srsa_4c

4437

Re: CHDK PTP interface

« Reply #1217 on: 07 / August / 2019, 18:25:32 »

Quote from: reyalp on 06 / August / 2019, 18:07:38

You could run without the 'bench' option to see if the others pass, like
Code: [Select]
exec require'camtests'.runbatch{xfersizebugs=true,filexfer=true}

That executes with no failures.

Quote

I really doubt this has anything to do with the error

Me too. My previous post shows the results of the msgtest, run via cli.

Not completely off-topic as it is about PTP, taken from here.
https://www.usa.canon.com/internet/portal/us/home/support/product-advisories/detail/the-vulnerability-in-canon-digital-cameras
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5994
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5998
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5999
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6000
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-6001

Plus (off-topic), firmware update related vulnerability in both EOS and PowerShot firmwares...
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5995

Logged

Menu jumping/flickering on M10?

reyalp

13624

Re: CHDK PTP interface

« Reply #1218 on: 07 / August / 2019, 20:57:09 »

Quote from: srsa_4c on 07 / August / 2019, 18:25:32

That executes with no failures.

Thanks. One more

Code: [Select]

!m=require'extras/msgtest'
!m.test{size=513,sizeinc=-1,count=512}

Quote

here.

Hmm. I've always operated on the assumption that it's game over if the PTP host is compromised. CHDK is obviously vulnerable: PTP_CHDK_CallFunction PTP_CHDK_SetMemory, Lua poke and likely a lot of vulnerable Lua functions are all readily available. These could all be executed over PTP/IP too.

I would be very surprised if there aren't more vulnerabilities in the Canon PTP code too.

We could offer an option to disable CHDK PTP functionality, checking a conf setting and returning an error in handle_ptp would be easy.

Logged

Don't forget what the H stands for.

srsa_4c

4437

Re: CHDK PTP interface

« Reply #1219 on: 08 / August / 2019, 17:34:17 »

Quote from: reyalp on 07 / August / 2019, 20:57:09

One more

Code: [Select]

___> !m=require'extras/msgtest'
___> c
connected: Canon PowerShot A410, max packet size 64
con> !m.test{size=513,sizeinc=-1,count=512}
testing 512 messages size 513-1, inc -1 gc=step
send 1 size:513 0x201...ok
send 2 size:512 0x200...ok
(...)
send 461 size:53 0x35...ok
send 462 size:52 0x34...I/O error
aborted at send 462 size:52 0x34, communication error
quit failed I/O error
ran 462 fail 1 time 26.5785 send time 6.2933

Quote

We could offer an option to disable CHDK PTP functionality, checking a conf setting and returning an error in handle_ptp would be easy.

We could, although I'm not sure if it's worth it. The cameras only activate their wireless interface when the user enters the Wifi related dialogs. I find it hard to imagine that anyone (other than some 3-letter agencies and the like) will spend their time exploiting these firmware bugs. And only a very low percentage of cameras run CHDK.

I posted the links because the descriptions mention the problematic PTP handlers by name. And that we'll get to see the fixes (M3 and M10 are on the list).

Logged

Menu jumping/flickering on M10?