CHDK PTP interface - page 123 - General Discussion and Assistance - CHDK Forum supplierdeeply

CHDK PTP interface

  • 1244 Replies
  • 528840 Views
*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1220 on: 08 / August / 2019, 18:37:16 »
Advertisements
send 461 size:53 0x35...ok
send 462 size:52 0x34...I/O error
aborted at send 462 size:52 0x34, communication error
Thanks.
So it fails at exactly 52 (64 - 12 overhead?) and not nearby multiples. Not sure what we could do about this. Could try two reads, like 48 and 4.

Quote
We could, although I'm not sure if it's worth it. The cameras only activate their wireless interface when the user enters the Wifi related dialogs. I find it hard to imagine that anyone (other than some 3-letter agencies and the like) will spend their time exploiting these firmware bugs. And only a very low percentage of cameras run CHDK.
Yeah, this doesn't seem a high risk. OTOH, someone making a stink about "CHDK undoes Canon security fix" could be unfortunate.

Thinking about it more, we could check the USB bit (in physw_status, not the hardware state) and only allow CHDK operations if its set. Since there's no working PTP/IP client for CHDK, having it off when USB is not present shouldn't be a problem (unless there's threading issues that make it drop out momentarily when kbd_task runs  :-[). This could also be an option: CHDK PTP: [USB only, USB+wifi, off]

If there's exposure when the camera is paired with a phone, that could be a bigger risk, but my impression is that's totally separate.

Quote
I posted the links because the descriptions mention the problematic PTP handlers by name. And that we'll get to see the fixes (M3 and M10 are on the list).
That could be interesting.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1221 on: 22 / September / 2019, 16:59:27 »
In trunk 5274, I re-organized the code that uses send_data (camera->host transfers). This shouldn't affect the actual operations. It mostly cleans up the logic around the old global buf_size. Some needless complexity in send_ptp_data goes away, and core_get_free_memory is only called when a buffer is actually needed. This could be extended to use native ptp buffers like recv_data, but there currently doesn't seem to be need.

The main motivation is to simplify the code before trying to address issues with >2gb files reported in https://chdk.setepontos.com/index.php?topic=13896.0
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1222 on: 22 / September / 2019, 20:26:59 »
Here's my non-working attempt at a patch to make the CHDK code behave correctly for >2GB files.

Tested on sx710

camtests all pass, so sub 2 GB stuff seems fine.

Used a 2260826276 zip file to test.

Attempting to upload after running camtests resulted in IO Error on the chdkptp side. The camera appears to be running normally, but it is not possible to reconnect, even with reset_device.

I tried this twice. In one instance, powering off the camera seemed normal, but I was unable to power it on without pulling the battery. In another, the camera became unresponsive while still on.

Trying to upload on a fresh boot without running camtests first, I got an immediate "general error"

Copying the file to the SD card with a card reader trying to download in chdkptp, I get
read_data failed 0x2ff
ERROR: I/O error

The file from the failed download is 2145386496 bytes. This is 2^31 - 2 MB. chdkptp uses a 2MB buffer for file transfers, so it seems likely that the read request which hit 2^31 failed.

While it's quite possible there are bugs in my code, my suspicion is the Canon code uses signed values. I haven't tried standard ptp getobject or sendobject with a >2gb file yet.

If we want to support this, it would probably make more sense to add operations to do partial uploads and downloads: Read X bytes from offset, append X bytes or write X bytes at offset.

I'm not  sure this is worth doing. The case of downloading > 2gb files seems useful, but fairly rare.

There are additional issues in CHDK lua, since the stat size is negative for > 2gb files. chdkptp side code can work around this, but it would be on a case by case basis.

It may become more of an issue in the future, @srsa_4c suggested DryOS 59 extended the stat structure to support 64 bit sizes. >4 GB files would require a lot of changes, though probably not too common for CHDK until exFAT boot or multi-partition is supported.
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4451
Re: CHDK PTP interface
« Reply #1223 on: 24 / September / 2019, 17:25:13 »
It may become more of an issue in the future, @srsa_4c suggested DryOS 59 extended the stat structure to support 64 bit sizes. >4 GB files would require a lot of changes, though probably not too common for CHDK until exFAT boot or multi-partition is supported.
I don't think there is a PowerShot firmware based camera that records movies exceeding the 4GB barrier - but I can't be sure as this limitation is often omitted from various literature.


*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1224 on: 24 / September / 2019, 21:43:31 »
I don't think there is a PowerShot firmware based camera that records movies exceeding the 4GB barrier - but I can't be sure as this limitation is often omitted from various literature.
Agreed, my impression is that they don't. My thought was extending the stat structure might be a hint that it was coming in future models, though of course it could be for other products that use DryOS, and of course it's possible future models will not be supportable by CHDK anyway.

My current thinking on >2gb files is just to provide a clean error, probably GeneralError in the protocol, something like "file too large" in chdkptp in cases where it would know.
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4451
Re: CHDK PTP interface
« Reply #1225 on: 25 / September / 2019, 17:00:45 »
My current thinking on >2gb files is just to provide a clean error, probably GeneralError in the protocol, something like "file too large" in chdkptp in cases where it would know.
It's not like I see 2...4GB files very often, but they can be created on camera. I checked SendObject, it uses Open() rather than one of the fopen() variants. Download of >2GB files does work via libgphoto based software.
How about using open/read instead of fopen/fread when dealing with big files? I agree that files bigger than 4GB don't need to be supported (64bit versions of open etc. may not even exist in firmware).

*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1226 on: 26 / September / 2019, 00:14:29 »
How about using open/read instead of fopen/fread when dealing with big files? I agree that files bigger than 4GB don't need to be supported (64bit versions of open etc. may not even exist in firmware).
My impression was that the problems were in the PTP layer, not file IO, but I didn't dig deeply. If sendobject works, it might be worth investigating further.
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4451
Re: CHDK PTP interface
« Reply #1227 on: 28 / September / 2019, 14:34:41 »
My impression was that the problems were in the PTP layer, not file IO, but I didn't dig deeply. If sendobject works, it might be worth investigating further.
It's possible that you're right and the problem is indeed caused by transferring >=2GB of data in one go. I guess this could be addressed by transferring them in multiple pieces, but that would not work with clients not implementing that addition to the protocol (in case that's a problem).


*

Offline srsa_4c

  • ******
  • 4451
Native ExecuteEventProc over PTP
« Reply #1228 on: 06 / February / 2022, 17:50:24 »
Continuing petabyte's work, I further reverse engineered a few Canon-specific PTP operations that can be used executing code on camera.
These could be useful for implementing card preparation without a card reader, and more. The operations are available on PowerShots running DryOS r31 (not all r31 cameras) and later. They are also available on EOS cameras (don't know which models). An additional check will prevent (ab)using these operations on DIGIC 8 and later cameras - until someone reverses the new 0x9050/0x905c handlers.

The operations are (names come from EOS debug strings, following is from here):
#define PTP_OC_CANON_InitiateEventProc0 0x9050
#define PTP_OC_CANON_TerminateEventProc_051 0x9051
#define PTP_OC_CANON_ExecuteEventProc 0x9052
#define PTP_OC_CANON_GetEventProcReturnData 0x9053
#define PTP_OC_CANON_IsEventProcRunning 0x9057
#define PTP_OC_CANON_QuerySizeOfTransparentMemory 0x9058
#define PTP_OC_CANON_LoadTransparentMemory 0x9059
#define PTP_OC_CANON_SaveTransparentMemory 0x905a
#define PTP_OC_CANON_QuickLoadTransparentMemory 0x905b
#define PTP_OC_CANON_InitiateEventProc1 0x905c
#define PTP_OC_CANON_TerminateEventProc_05D 0x905d

InitiateEventProc 0x9050, 0x905c
TerminateEventProc 0x9051, 0x905d

These are for enabling and disabling operations between 0x9052 ... 0x905b. Some EOS cameras seem to register those handlers by default.

ExecuteEventProc 0x9052

Executes a registered event procedure. Parameter passing and more are detailed later. Returns the eventproc's one-word result as PTP param.

GetEventProcReturnData 0x9053

Returns data from a fixed buffer, but only after an appropriate ExecuteEventProc operation.

IsEventProcRunning 0x9057

Self-explanatory name, but I did not try calling it. Can return up to 3 params. reyalp's findings

QuerySizeOfTransparentMemory 0x9058
LoadTransparentMemory 0x9059
SaveTransparentMemory 0x905a
QuickLoadTransparentMemory 0x905b

Not researched, transparent memory is something like a named allocation from regular heap.



Now, ExecuteEventProc in details.

Eventproc name and its arguments are sent in the payload (data phase).

This operation has 2 PTP params that I know of:
- 1st parameter: async execution flag. When nonzero, the eventproc is executed in a separate task (so that it can run for longer).
- 2nd parameter: return data flag. When nonzero, the eventproc is expected to create data that can be returned in a GetEventProcReturnData operation.
  When this flag is set, the eventproc will get 3 fixed arguments first, followed by those specified by caller.
  - 1st arg will contain the available buffer space in bytes
  - 2nd arg will point to the return size (the eventproc can set the actual return data amount which defaults to zero)
  - 3rd arg will point to the return data buffer

Eventproc name and arguments are to be sent in an unaligned stream of bytes.
Encoding:
- evproc name as string of chars, terminated by 0 char
The rest comes right after, is not word-aligned, words are in LE byte order.
- 32bit value: num of args (10 or less)
- for each arg:
 - 5*4 bytes data:
  - word1: type, 4 for long arg (0=uint8, 1=int16, 2=(u)int32, 3=(u)int64), PS and EOS treat every non-4 type as (u)int32
  - word2: arg value when word1 isn't 4
  - word3: (ignored, treated as 0) (high-order word of 64-bit integer arg, but not on PS and EOS)
  - word4: (? picked up but not used?) index of long arg
  - word5: (size of data when word1=4)
- 32bit value: num of long args (not limited here, but has to be 10 or less)
- for each long arg:
 - 32bit value (? picked up, but unknown purpose) index of long arg, should match word4 above
 - variable length data (length specified by word5, must be larger than zero, size only limited by camera heap memory)

I expect the above to work on all (< DIGIC 8) cameras that implement these operations.
There might be details that I missed, corrections are welcome.

A demo is attached, in python (I have 3.6), along with a patch against https://github.com/petabyt/sequoia-ptpy . I'm not very familiar with python, it just seemed the easiest choice to make a demo.

edit:
description is fixed, plus link added that explains 0x9057 (also, read on for other findings)
« Last Edit: 12 / February / 2022, 17:40:57 by srsa_4c »

*

Offline reyalp

  • ******
  • 14121
Re: CHDK PTP interface
« Reply #1229 on: 06 / February / 2022, 19:13:30 »
Very nice work. I'll add support in chdkptp.

Interesting Canon issued a fix for https://chdk.setepontos.com/index.php?topic=13872.0 and left this ;)

I guess we don't need to worry too much about the security of the CHDK extension :haha
Don't forget what the H stands for.

 

Related Topics


SimplePortal 2.3.6 © 2008-2014, SimplePortal