CHDK PTP interface

reyalp

11862

Re: CHDK PTP interface

« Reply #1190 on: 30 / April / 2019, 00:04:28 »

Advertisements

Quote from: srsa_4c on 29 / April / 2019, 18:32:40

Quote
It does not appear to have the n*512 - 11 problem our code has.
How can I test this?

To test if it happens with CHDK code, you can use the attached patch, which buffers reads in TBUF_SIZE chunks. You can switch between malloc/free and umalloc/ufree.

On a540, running msgtest with

Code: [Select]

m.test{size=1,sizeinc=1,count=10000,gc='step'}

will fail at 1013 with a stalled connection and IO error on the chdkptp side. If you change the size to 1024, it will then fail at 1525, and so on for multiples of 512.

You could of course just test around that size (start with size=1000) but given the different magic values on different cameras, covering a larger range seems like a good idea.

To reproduce this in unmodified, stock code, you'd have to arrange a file upload that was n*512-11 (including filename and size int) and larger than half the largest free block. I haven't done this for a540.

To see if the Canon code was affected, I tried SendObject with sizes that would result in multiple reads and fall on a 512-11 boundary, like

Code: [Select]

!newf={ Filename="TEST.DAT", ObjectFormat=0xBF01, ObjectCompressedSize=256*3*1024-11 } con:ptp_send_object_info(newf) con:ptp_send_object(('x'):rep(newf.ObjectCompressedSize))

which did not fail.

On d10, the patch works fine with with 512 buffer size, albeit slowly with either malloc or umalloc. I expected this to fail at 0x23f4 or 0x43f4 but ...

With a buffer size of 0x3000 in cached memory, it fails at 0x23f4. Uncached works fine.

With a 0x23f0 buffer, both cached and uncached crash with an ASSERT!! BulkTrns.c Line 1549 at 0x23f5

Sorry for the random notes on d10, I went to confirm it behaved as I expected from previous tests and it didn't

ptp-buffered-test-1.patch (1.14 kB - downloaded 2 times.)

Logged

Don't forget what the H stands for.

reyalp

11862

Re: CHDK PTP interface

« Reply #1191 on: 04 / May / 2019, 23:24:39 »

I looked at the PTP buffer functions some on D10.

As srsa_4c mentioned earlier, the first ComMemMan function (d10 sub_FF8C9114) allocates a 1 MB uncached exmem buffer.
This appears to be done in PTP startup, it's already allocated once a chdkptp connection is established.

On d10 (without CHDK exmem enabled) the buffer address is ~~0x43E42E00~~ 0x43E02E00
Comments in makefile.inc indicate the video buffer starts at 0x43F02E00, exactly 1MB after the start of the PTP buffer. This raises the question of whether some exmem types automatically stay clear of the video buffer.

The second ComMemMan (d10 sub_FF8C91A4) function frees the exmem buffer.

The third (d10 sub_FF8C9228) is called by individual PTP functions that need a buffer. My impression is the big exmem buffer is split up into 7 "slots" used for different things. The main buffer for SendObject is slot 4.

sub_FF8C9374 appears to return a size for each slot

Code: [Select]

con 1> =return call_func_ptr(0xFF8C9374,0)
2:return:8192
con 2> =return call_func_ptr(0xFF8C9374,1)
3:return:2048
con 3> =return call_func_ptr(0xFF8C9374,2)
4:return:16384
con 4> =return call_func_ptr(0xFF8C9374,3)
5:return:235520
con 5> =return call_func_ptr(0xFF8C9374,4)
6:return:524288
con 6> =return call_func_ptr(0xFF8C9374,5)
7:return:0
con 7> =return call_func_ptr(0xFF8C9374,6)

sub_FF9F6B28 is used by the SendObject code. It allocates slot 4 if needed (using the size from sub_FF8C9374 above), and returns the address. In my testing it was always ~~the start of the exmem buffer~~ 0x43e42e00.

This suggests that it would be fairly easy to use the stock Canon buffer, at least where it doesn't conflict with video stuff. Only one PTP operation can be in progress at a given time, so I think there's little risk of conflicting with Canon firmware code for slot 4. (other slots are used by other parts of the PTP code, e.g. something in BulkTrns.c uses slot 1)

Test patch for D10 attached. Unsurprisingly, the various msgtest tests work fine with this, since the buffer is uncached and far larger than usable message sizes.

It's probably not practical switch all ports to this approach. My current thought platform specific functions that fall back to malloc in the default case, but for cameras without a dedicated buffer, it's desirable to transfer directly to the destination address for script messages.

edit:
The same approach works for A540 using

Code: [Select]

NHSTUB(get_ptp_buf_size,            0xFFE38264)
NHSTUB(get_ptp_file_buf,            0xFFE57498)

d10-canon-ptp-buf-work-1.patch (2.39 kB - downloaded 2 times.)

« Last Edit: 05 / May / 2019, 20:38:06 by reyalp »

Logged

Don't forget what the H stands for.

srsa_4c

3890

Re: CHDK PTP interface

« Reply #1192 on: 05 / May / 2019, 18:42:53 »

Quote from: reyalp on 04 / May / 2019, 23:24:39

On d10 (without CHDK exmem enabled) the buffer address is 0x43E42E00
Comments in makefile.inc indicate the video buffer starts at 0x43F02E00, exactly 1MB after the start of the PTP buffer.

If there's no typo here, the difference of the two addresses is only 0xc0000.

Logged

Menu jumping/flickering on M10?

reyalp

11862

Re: CHDK PTP interface

« Reply #1193 on: 05 / May / 2019, 20:37:03 »

Quote from: srsa_4c on 05 / May / 2019, 18:42:53

Quote from: reyalp on 04 / May / 2019, 23:24:39
On d10 (without CHDK exmem enabled) the buffer address is 0x43E42E00
Comments in makefile.inc indicate the video buffer starts at 0x43F02E00, exactly 1MB after the start of the PTP buffer.
If there's no typo here, the difference of the two addresses is only 0xc0000.

Good catch. It was mistake. The exmem buffer is at 0x43e02e00. The
0x43e42e00 is the address returned for slot 4, which I mistakenly thought was at the start.

Logged

Don't forget what the H stands for.

reyalp

11862

Re: CHDK PTP interface

« Reply #1194 on: 12 / May / 2019, 21:32:38 »

Quote from: reyalp on 28 / April / 2019, 18:48:30

A540 (and likely other vxworks cameras) appear to have another, unrelated problem, present in the current trunk. Rapid fire messages like
Code: [Select]
!m=require'extras/msgtest' !m.test{size=1,sizeinc=1,count=10000,gc='step'}occasionally crash (often after 3000-5000 iterations, but variable, sometimes it goes 10k without problems)

I worked on this a bit more. It appears that protecting Canon malloc/free/GetMemInfo with a semaphore avoids the issue.
It didn't crash in ten runs of m.test{size=1,sizeinc=1,count=10000,gc='step'} with the attached patch, where it usually crashes within one or two with the unpatched code.

The patch is just a proof of concept. DryOS malloc, free and GetMemInfo are clearly protected by their own semaphore internally, so the real fix would only need to apply to vxworks. Where to initialize the semaphore is also a bit unclear.

There are a couple things that make this a bit weird:
* Scheduling on vxworks cameras appears to mostly non-preemptive. However, my impression from the vxworks docs is that an interrupt can still result in a task switch (ie interrupt handler changes a semaphore etc that makes a different task the highest priority ready to run), which would make sense with it happening in PTP.
* The semaphore only protects CHDK calls, so presumably the Canon firmware routines could suffer the same problem. Maybe the Canon firmware doesn't do enough rapid fire malloc/frees for it be an issue, or maybe it's only memPartInfo that gets confused when it gets called in the middle of a malloc/free. I didn't come up with a simple way to test this. The (mostly) non-preemptive scheduling makes it hard to come up with a test case that really hammers it.

My inclination is to implement this for vxworks. It's pretty rare in normal usage, but it makes testing the other transfer issues difficult, and I think I have run into with the chdkptp image download commands and lots of files.

canon-mem-sem-work-1.patch (2.2 kB - downloaded 1 times.)

« Last Edit: 15 / May / 2019, 01:28:15 by reyalp »

Logged

Don't forget what the H stands for.

reyalp

11862

Re: CHDK PTP interface

« Reply #1195 on: 13 / May / 2019, 00:38:30 »

Proposed patch. I'll check this in if there aren't objections.

I built all the vx cams and checked the stubs for the semaphore functions seem reasonable (take and give are already used for FileIO, create probably wasn't used in CHDK on most vx cameras)

edit:
Added in trunk 5205

canon-mem-sem-work-3.patch (4.16 kB - downloaded 4 times.)

« Last Edit: 15 / May / 2019, 01:30:43 by reyalp »

Logged

Don't forget what the H stands for.

reyalp

11862

Re: CHDK PTP interface

« Reply #1196 on: 21 / May / 2019, 02:39:28 »

Here's my initial patch to find the PTP handlers in pre-digic 6 cams (corresponding to the d6 one in https://chdk.setepontos.com/index.php?topic=12714.msg140348#msg140348

It turns out that on vxworks cams, the compiler canon used pretty much always put 0x1000 or 0x9000 in one register, and then used ORR or ADD to make the operation ID for a bunch of subsequent add_ptp_handler calls. So the finsig_vxworks code only finds handlers that are in a table of object / storage related operations (GetObject, SendObject etc) rather than trying to scrape all the calls.

This is OK since my immediate use for this is to get the PTP buffer related functions.

The dryos firmwares have a few instances of the ADD/ORR logic, but mostly use LDR.

find-ptp-handlers-pre-d6-work-1.patch (8.63 kB - downloaded 2 times.)

Logged

Don't forget what the H stands for.

srsa_4c

3890

Re: CHDK PTP interface

« Reply #1197 on: 22 / May / 2019, 18:50:56 »

Quote from: reyalp on 21 / May / 2019, 02:39:28

It turns out that on vxworks cams, the compiler canon used pretty much always put 0x1000 or 0x9000 in one register, and then used ORR or ADD to make the operation ID for a bunch of subsequent add_ptp_handler calls.

Attached modified patch adds (incomplete) support for this - vx only. Might be duplicated effort.
I found an ID (0x9028) that seems to have two different handlers in some firmwares.

find-ptp-handlers-pre-d6-work-1mod.patch (12.12 kB - downloaded 2 times.)

Logged

Menu jumping/flickering on M10?