CHDK PTP interface

reyalp

14118

Re: CHDK PTP interface

« Reply #1240 on: 21 / February / 2022, 00:35:34 »

Advertisements

In chdkptp r1141, I added a devutil command 'dptpdevinfo' to display PTP information like supported opcodes, events, properties and so in.
Constants and names are largely from current (as of Feb 2022) gphoto source with Canon opcodes supplemented from @coon's list (I also used this combined list to update the CHDK sig finders in trunk r6071)

As usual, devutil is activated using

Code: [Select]

!require'extras/devutil'.init_cli()

After that, dptpdevinfo without options gives reasonable output. Adding -ptpevp calls InitiateEventProc0 before querying deviceinfo, which will add the related opcodes to the supported list. However, it won't work on cameras that expect a different handshake, or non-canon devices.

Output for my cameras and an iphone is attached.

Some observations:
- Cameras starting with early DryOS (D10, r31, at least) support MTP, explaining why they report a Microsoft extension ID (6) rather than Canon's own ID (11). Gphoto comments suggest that 0xffffffff may also identify MTP, but I haven't seen it. The 0x980x opcodes are MTP. The PTP native extension system doesn't really support multiple / layered extensions, so there's no PTP standard way to identify a Canon's approach of supporting MTP and a bunch of their own opcodes, aside from theirs being outside the 0x980x range. MTP actually does define a mechanism (using the VendorExtensionDesc string) but Canon doesn't use it.
- Most Canon operation codes are identified, although I don't know how reliable the names/descriptions from Gphoto are.
- Some of the known opcodes and properties might be useful or interesting for reverse engineering
- Aside from minor variations in order, returned info is essentially identical Wifi or USB.
- Calling MTP.GetObjectPropsSupported puts the camera in "black screen" state, like GetObjectHandles

ptpdevinfo-20220220-1.zip (15.58 kB - downloaded 37 times.)

« Last Edit: 15 / January / 2023, 14:36:20 by reyalp »

Logged

Don't forget what the H stands for.

reyalp

14118

Re: CHDK PTP interface

« Reply #1241 on: 01 / February / 2024, 01:23:29 »

Over in @RaduP's Camera Connect thread, I noted that when using PTP/IP on SX730 through the mobile phone option

Quote from: reyalp on 23 / December / 2023, 02:33:12

Oddly, the CHDK PTP opcode isn't reported as supported on the initial connection, although it's available subsequently and works.

The cause of this appears to be
- CHDK starts init_chdk_ptp_task in spytask at startup.
- init_chdk_ptp_task sits in loop sleeping for 250 ms until it successfully adds the CHDK handler, or more strictly, until add_ptp_handler doesn't return 0x17
- The 0x17 return is based on a variable that isn't set until PTP is initialized for the first time.

When a USB cable is used, the firmware initializes PTP when it detects a USB host (more or less), so the odds of starting a PTP connection within 250ms are extremely low, at least if connecting manually (could possibly be an issue if anyone makes a CHDK client that runs from udev).

For wifi in the cases where the UPnP handshake is required, PTP seems to be initialized based on the UPnP handshake. Again, very hard to connect within 250ms

For the direct connect without the UPnP handshake, PTP appears to be initialized when it sees the first PTP/IP packet, so CHDK has a very good chance of not adding its opcode before the client fetches device info.

It would be "better" for CHDK to hook into the stuff that registers all the other PTP handlers, but it's easy enough to work around in chdkptp by waiting 250ms and refreshing the devinfo if the CHDK opcode isn't detected the first time. This will slightly slow down connections to devices not running CHDK, but that should be a rare case.

Logged

Don't forget what the H stands for.

reyalp

14118

Re: CHDK PTP interface

« Reply #1242 on: 13 / July / 2024, 00:13:41 »

Another host -> camera transfer bug:
I discovered on SX730 HS (dryos r59p4), PTP/IP uploads larger than PTP buffer size (meaning multiple calls to data->recv_data) hang for a minute and close the connection.

This only affects PTP/IP (wifi) and does not affect my earlier wifi enabled cams: elph130 (dryos r52), g7x (r55p6) and sx710 (r57).

The cause appears to be a bug in the Canon firmware. Given that we've seen several other bugs related to large or specifically sized uploads (see here and here for example), it seems likely Canon's own software doesn't use large uploads in a single transaction and their testing doesn't exercise it.

The SX730 PTP/IP code appears to have been refactored significantly compared to the earlier cams. In particular, it adds tasks called PtpipController and PtpipPacketDetector which are directly involved in the affected code. I haven't checked in detail yet, but other cameras with the tasks may well be affected. These include G7X2 (r58p10) G9X2 (r59p3), ixus190_elph200 (r59p3) M100 (r59p4) M5 (r59p3) M6 (r59p3) SX430IS (r59p3) SX620HS (r58p7)

(edit: Actually on the r58 cams, the task is "PtpipContoroller" [sic], not "PtpipController"

but it appears functionally the equivalent)

The digic 7 cameras may be different since some network stuff is handled on a different CPU core ("Lime") but on g7x2 at least, the structure of PtpipController and PtpipPacketDetector seems very similar to sx730.
SX740 (Digic 8, r60) and xf605 firmware do not have these task names, suggesting the new unified EOS/PowerShot/DV codebase is unlikely to be affected.

The bug itself appears to be due to the call to FUN_fc162848 (SX730 firmware 100d) in FUN_fc21fe8e. FUN_fc162848 resets a structure used by PtpipPacketDetector, including removing the command socket from the fd_set used by the select() call in PtpipPacketDetector. Since the socket is removed, the next call to select blocks indefinitely waiting for data, and the eventflag wait at fc2212ac in PTP/IP data->recv_data times out after 60s, triggering an error which closes the connection. As far as I can tell this is not a deliberate error case (which typically either assert or immediately trigger an error state that closes the connection), and the Canon code is simply wrong. This is supported by the fact a standard SendObject call without CHDK running fails in a similar way (though it hangs until the battery is removed rather than disconnecting).

The error case is only hit when a call to data->recv_data reads the specified data size, and there is still more data in the PTP transaction. For CHDK commands, the trigger size will usually be around half of free memory, which ends up around 250K for SX730 (a wifi connection reduces free mem significantly). For native Canon commands (or CHDK ports configured with CAM_PTP_USE_NATIVE_BUFFER) it should be around 256K.

There is clearly logic in the code that is supposed to handle the multiple recv_data call case. In fact, bypassing the problematic call to FUN_fc162848 and twiddling a couple other things appears to completely fix the problem, both for my chdkptp tests and SendObject. So I've done that for SX730 100d (and 100e since it uses the same build) in trunk r6279.

I haven't yet implemented the fix for the other SX730 firmware, since it involves a bunch of firmware specific addresses and I suspect chdkptp PTP/IP usage is very rare, but I can probably add if anyone needs it, or implement the same for other affected models.

You can test if a CHDK supported camera is affected using chdkptp (from recent svn, built with PTPIP support, not the binary builds) by initiating a PTP/IP connection on the camera side and then using

Code: [Select]

chdkptp -e'exec require"camtests".tests:do_test{connect_dev="-h=<camera IP address>",skip_names={"shoot"}}'

Note this will take several minutes. If it fails, there will be error messages and the final output will include a failure count.

I've attached my initial fix with my final debugging / logging code for future reference, since it includes some potentially useful hooking that isn't need in the final fix. This includes hooking socket recv() as well as the various callbacks used by PtpipController

ptpip-packet-debug-work-17.patch (17.05 kB - downloaded 2 times.)

« Last Edit: 13 / July / 2024, 02:10:54 by reyalp »

Logged

Don't forget what the H stands for.

reyalp

14118

Re: CHDK PTP interface

« Reply #1243 on: 13 / July / 2024, 01:21:15 »

One other thing I came across working on this:

The two final arguments to ptp_data->recv_data are always zero in our code (and most/all canon code). These appear to be a callback and an argument for it respectively. For a PTP/IP connection on SX730 100d, I believe it would be called at the end of ptpip_recv_data_cb (fc221104), the same function I hooked for the recv_data fix. I considered trying to use this for the fix (potentially avoiding the need to hook PtpipController), but the callback being present affects logic in other places, for example in data->recv_data at fc22129a
That change in the logic looks like it might be used to make recv_data asynchronous, but I didn't investigate further.

Logged

Don't forget what the H stands for.

Stregone

5

Re: CHDK PTP interface

« Reply #1244 on: 17 / August / 2024, 09:18:05 »

I took a look at your PTP extension for CHDK—nice work! I’ve been messing around with remote control and memory manipulation for my cameras too, and your tool seems like it could really streamline things. The command-line tool sounds especially useful. I’ll check out the documentation and see how it might fit into my workflow.

Logged