CHDK PTP interface - page 115 - General Discussion and Assistance - CHDK Forum

CHDK PTP interface

  • 1167 Replies
  • 240280 Views
*

Online reyalp

  • ******
  • 11791
Re: Re: chdkptp - alternative ptp client
« Reply #1140 on: 16 / April / 2017, 14:44:43 »
Advertisements
I wasn't convinced of the need for code changes to read address zero, but a general buffered option for GetMemory seems like it would be useful. I'll try to add something a bit less hacky.
Don't forget what the H stands for.

*

Offline Ant

  • ****
  • 423
Re: Re: chdkptp - alternative ptp client
« Reply #1141 on: 16 / April / 2017, 14:58:33 »
I wasn't convinced of the need for code changes to read address zero
Every time I work with RAM dumps made with chdkptp I need to recalculate addresses. This is inconvenient.

Re: Re: chdkptp - alternative ptp client
« Reply #1142 on: 17 / April / 2017, 11:09:11 »
Just need to output a place holder. 

*

Online reyalp

  • ******
  • 11791
Re: Re: chdkptp - alternative ptp client
« Reply #1143 on: 17 / April / 2017, 13:26:31 »
Quick hack that lets getting camera memory from places that are not PTP friendly (I wanted to get a copy of the 0xbfe10000 TCM content). It uses buffered read. Still not suitable for reading MMIO because it uses memcpy.
One slight complication http://pubs.opengroup.org/onlinepubs/009695399/functions/memcpy.html

Quote
If copying takes place between objects that overlap, the behavior is undefined.
This will obviously happen if you are dumping the whole RAM using a buffered copy. Whether this applies to dryos or actually matters is unclear. Obviously no RAM dump is going to be fully consistent instantaneous snapshot.
Don't forget what the H stands for.


*

Offline srsa_4c

  • ******
  • 3832
Re: Re: chdkptp - alternative ptp client
« Reply #1144 on: 17 / April / 2017, 14:02:31 »
Quote
If copying takes place between objects that overlap, the behavior is undefined.
This will obviously happen if you are dumping the whole RAM using a buffered copy. Whether this applies to dryos or actually matters is unclear. Obviously no RAM dump is going to be fully consistent instantaneous snapshot.
TBH, I did this because I needed it, and wanted to get it done fast. As you say, this (in its current form) may not be what Ant wants, because it can mess up the copied version of the Canon heap.
A more proper implementation would only use memcpy for areas that can't be safely transmitted directly via the PTP send function. An even more proper implementation would also use a single-word-at-a-time copy routine for MMIO areas.

*

Online reyalp

  • ******
  • 11791
Re: Re: chdkptp - alternative ptp client
« Reply #1145 on: 17 / April / 2017, 16:32:55 »
TBH, I did this because I needed it, and wanted to get it done fast. As you say, this (in its current form) may not be what Ant wants, because it can mess up the copied version of the Canon heap.
Yes, understood and thanks for posting it. I wasn't criticizing the hack, just trying to figure out what to add to the protocol.

My first thought was to just add a "buffered" option to GetMemory , but as you say this isn't great for whole memory dumps. Then again, even with DMA whole memory dumps aren't going to be totally consistent. I suspect the "undefined" is refers to the content copied in overlapping regions, which really isn't a problem since looking at the PTP transfer buffer in a PTP memory dump wouldn't make sense.

Ideally you'd want GetMemory to automatically identify the different regions and use DMA, memcpy or word by word as required, but analyzing the CP15 values at run time seems excessive.
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 3832
Re: Re: chdkptp - alternative ptp client
« Reply #1146 on: 17 / April / 2017, 20:24:22 »
Ideally you'd want GetMemory to automatically identify the different regions and use DMA, memcpy or word by word as required, but analyzing the CP15 values at run time seems excessive.
Perhaps allowing the user to specify the method would be the optimal solution.
Except for RAM dumps, where we'd want to copy the zero mapped TCM buffered (RAM between 0..MEMBASEADDR, to avoid detection).

*

Online reyalp

  • ******
  • 11791
Re: CHDK PTP interface
« Reply #1147 on: 24 / April / 2017, 00:51:15 »
I moved this to the PTP interface thread, since it's was more of a protocol / camera side issue than a chdkptp client issue.

In r4798, I committed the following:
ptp - updates to PTP_CHDK_GetMemory, based on discussion in https://chdk.setepontos.com/index.php?topic=6231.msg132332#msg132332
* Automatically work around Canon data->send_data failing on NULL source, by sending any bytes in first word buffered if source address is < 4
* Update to use data->send_data instead of send_ptp_data. Better performance, same strategy already used for live view, remote shoot
* Update protocol to 2.8: PTP_CHDK_GetMemory input param4 is option which defines whether transfer should be buffered or unbuffered, independent of address 0 workaround. Default (0) is unbuffered, compatible with previous versions

I went through a few different approaches before I settled on this.

On my cameras, the problem with NULL really seems to be only that the Canon function treats NULL as an error. So I made the default GetMemory function just detect that and transfer any bytes in the first word separately. (First word rather than first byte because of https://chdk.setepontos.com/index.php?topic=13101.0)

This means that existing clients should now be able to do RAM dumps from address zero, although it's possible some will still fail because due to the first 4 or 16 kb being TCM.

I also changed the unbuffered GetMemory to use a single send_data call, unlike the original version which sent in buffer_size chunks despite not being buffered. This is noticeably faster (33 MB/s vs 28 MB/s on D10) I believe the original logic was based on the idea that the firmware might try to allocate internal buffers for the whole transfer, but this is clearly not true, because live view and remote capture send all at once.

Separate from this, I added the mode option to PTP_CHDK_GetMemory to do the entire transfer buffered. This requires client support, chdkptp r741 or later

It currently uses memcpy. Originally, I was going to have another option for word-by-word copy, but in my testing memcpy worked fine for MMIO addresses. If it turns out word-by-word is needed, it's easy to add. Note that many MMIOs are write only, and reading them may have side effects, so dumping the whole MMIO space may not be a great idea in any case.

Buffered is noticeably slower than unbuffered, ~17 vs 35 MB/s for G7X. Oddly, D10 suffers less of a penalty (18 vs 33)

While mentioning speed
On a540 cached addresses are much slower than uncached, 5 vs 15 MB/s. ROM is in between at 12. I these old cameras buffer internally if the address isn't in uncached RAM. This was noted long ago.

Interestingly on newer cameras (d10 onward) there seems to be little difference between cached an uncached. ROM is somewhat slower than RAM (26 MB/s on G7x, 13  on D10)
« Last Edit: 24 / April / 2017, 00:59:12 by reyalp »
Don't forget what the H stands for.


Re: CHDK PTP interface
« Reply #1148 on: 27 / June / 2017, 16:41:18 »
Has there been any progress on the PTP event front beyond mweerden's findings?

The reason I'm asking is I've been looking into implementing some debugging capabilities on the Initiator side, and polling CHDK_PTP_GetMemory doesn't seem like an optimal approach.
Author of CHIMP, Canon Hack Installation and Management Platform

*

Online reyalp

  • ******
  • 11791
Re: CHDK PTP interface
« Reply #1149 on: 27 / June / 2017, 23:58:24 »
Has there been any progress on the PTP event front beyond mweerden's findings?
Note that I am aware of.
Don't forget what the H stands for.

 

Related Topics