CHDK PTP interface

reyalp

13878

Re: CHDK PTP interface

« Reply #1200 on: 27 / May / 2019, 23:25:58 »

Advertisements

Here's a patch that finds the ptp buffer size function for pre-digic 6 dryos and vxworks.

It seems to work for all but the very oldest vxworks (ixus30 - ixus50, ixus700 and s2is) which seem to use a constant 0x40000 instead of a function.

The function to get the buffer address (d10 FF9F6B28, a540 FFE57498) seems a bit more complicated to match, so any clever ideas for that are welcome.

find-get_ptp_buf_size-work-2.patch (5.39 kB - downloaded 18 times.)

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: CHDK PTP interface

« Reply #1201 on: 01 / June / 2019, 16:21:55 »

Quote from: reyalp on 27 / May / 2019, 23:25:58

The function to get the buffer address (d10 FF9F6B28, a540 FFE57498) seems a bit more complicated to match, so any clever ideas for that are welcome.

No code this time, but
Vx:
- [ObjMethdRW]Error:PrepareSubstancePTPObject, Open[%s] reference though not all cams have it
- ObjMethdRW.c string reference can still help with those cams that don't have the above string
both:
- get_ptp_buf_addr references get_ptp_buf_size
- get_ptp_buf_addr also references one of the commemman routines
- bic instruction

Logged

reyalp

13878

Re: CHDK PTP interface

« Reply #1202 on: 02 / June / 2019, 02:24:10 »

Thanks. I ended up matching the instruction sequence with get_ptp_buf_size and bic. It seems work on all cams except the very old vxworks that don't have get_ptp_buf_size (which is fine, there's few enough that they could be dealt with manually if needed)

find-get_ptp_buf_funcs-work-4.patch (9.63 kB - downloaded 13 times.)

Logged

Don't forget what the H stands for.

reyalp

13878

Re: CHDK PTP interface

« Reply #1203 on: 03 / June / 2019, 02:22:06 »

Here's an initial patch that allows use of either the canon PTP buffers or heap malloc'd buffers. It also includes the earlier sig finder patches.

* ptp.c is compiled as platform code, instead of generic
* camera.h define CAM_PTP_USE_NATIVE_BUFFER uses the canon buffer. Requires get get_ptp* functions
* New ptp.c function get_ptp_temp_buf takes the total transfer size, and returns a buffer and buffer size. If using CAM_PTP_USE_NATIVE_BUFFER, the max buffer size is from get_ptp_file_buf_size, otherwise it's the half the largest free block.
* free_ptp_temp_buf frees the buffer, if using heap buffer.
* Code that *only* needs a temporary buffer (flush_recv_ptp_data and PTP_CHDK_UploadFile) uses get_ptp_temp_buf and data->recv_data directly
* Other code uses recv_ptp_data, which uses get_ptp_temp_buf only if the camera is using CAM_PTP_USE_NATIVE_BUFFER and otherwise uses the destination buffer.

The last two items are to unnecessary buffering that could be problematic on low memory cameras that don't use the native buffer.

There are still some loose ends to clean up, but it does seem to solve the problems on D10, and not break cameras using the heap buffer. I still need to make some easy to run tests. I'd also like to get rid of the last of the global buf_size logic for camera->pc transfers too.

edit:
Oops, missed some files in the first patch. Update attached.

canon-ptp-buf-work-3a.patch (19.82 kB - downloaded 15 times.)

« Last Edit: 03 / June / 2019, 02:25:57 by reyalp »

Logged

Don't forget what the H stands for.

reyalp

13878

Re: CHDK PTP interface

« Reply #1204 on: 17 / June / 2019, 02:31:50 »

I spent some more time trying to understand the issues on A540

The crash is "Assert BulkTrns.c line 1544", which happens in sub_FFE4DBDC
This is called from sub_FFE50B50, which comes from some table of BulkTrns related functions at FFE50BAC. The condition being checked in the assert isn't obvious to me.

The crash happens when
1) data->recv_data is called more than once
2) total size is (512*n) - 11 to (512*n) - 1 i.e. 0x...1f5 through 1ff.

Additionally, when using the native buffer (maybe any uncached) a stalled connection happens for something like 512*n to 512*(n+1)-13. Larger sizes appear to be OK, though I haven't checked exhaustively.

These problems happen even using native PTP operations, so

Code: [Select]

!newfi2={ Filename="BAD.DAT", ObjectFormat=0xbf01, ObjectCompressedSize=1024*256+512-11}
!return con:ptp_send_object_info(newfi2)
!return con:ptp_send_object(('x'):rep(newfi2.ObjectCompressedSize))

Triggers the assert, and size 1024*256+512 triggers the stalled connection, while 1024*257 works fine. I previously thought native operations weren't affected, but my test was incorrect. This is reproducible without CHDK running.

Note as I posted earlier some cameras require A/ on the path and crash with an different assert if it is not present. Others, including A540 (and likely all vxworks cameras) crash if it is present.

Also, although get_ptp_buf_size(4) returns 512K, the camera divides this value by 2 for transfers. I previously verified by hooking data->recv_data and logging the values that it's really doing 256K chunks. Based on the total ComMemMan size and other code, it seems likely the actual buffer is 512K, so why the firmware uses size/2 is unclear.

With CHDK, the problem will only occur if the total transfer is larger than 1/2 free memory and one of the magic sizes. Using a fixed 256K buffer, the following reproduce it

file upload

Code: [Select]

!fh=io.open('TEST.DAT','wb') fh:write(('x'):rep(1024*255+1525-14)) fh:close()
u test.dat

The -14 accounts for the file name and file name length.

Flushed script message (no script running):

Code: [Select]

!con:write_msg(('x'):rep(1024*255+1525))

The problem also occurs if the first read isn't a multiple of 512, e.g. if the first recv_data is 4000, it fails at 4085.

tl;dr
1) I'd be interested to see results on other vxworks cameras for the SendObject test.
2) This doesn't seem fixable for A540, and the current trunk style code doesn't seem worse than other options

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: CHDK PTP interface

« Reply #1205 on: 17 / June / 2019, 19:00:26 »

Quote from: reyalp on 17 / June / 2019, 02:31:50

Code: [Select]
!newfi2={ Filename="BAD.DAT", ObjectFormat=0xbf01, ObjectCompressedSize=1024*256+512-11} !return con:ptp_send_object_info(newfi2) !return con:ptp_send_object(('x'):rep(newfi2.ObjectCompressedSize))Triggers the assert, and size 1024*256+512 triggers the stalled connection

Same experience (crash, hang) on s80 (older vx).
No problem executing with either size on a460 (late vx), file gets created in both cases.

Logged

reyalp

13878

Re: CHDK PTP interface

« Reply #1206 on: 01 / July / 2019, 00:22:50 »

I checked in the sigfinder changes to ad get_ptp_buf_size and get_ptp_file_buf in r5231-5232

I added tests to chdkptp camtests to check for these issues in chdkptp r864.

original d10 bug (recv_data to uncached at 0x23f4 +N*512)

Code: [Select]

 chdkptp -c -e'exec require"camtests".run("xferbug_0x23f4")'

Unresolved A540 bug (multiple recv_data calls, total size = the first 512*N-11 that results in multiple recv_data)

Code: [Select]

 chdkptp -c -e'exec require"camtests".run("xferbug_0x1f5")'

However, the above test fails on ALL my cameras, with a similar assert (I think it's the same but the line number vary slightly). Like a540, I was able to reproduce it with SendObject, so this isn't an unrelated problem with the test. (The test uses CHDK messages rather than SendObject, because I don't know which cameras require A/ or not on the SendObject path)

This does raise the question why a460 apparently wasn't affected, and also gets me back to wondering if it is something in our our PTP/USB code. If anyone can get another program to PTP upload a 262645 byte file, that would be interesting.

a540
Assert BulkTrns.c line 1544

d10
ASSERT!! BulkTrns.c Line 1549

sx160
ASSERT!! BulkTrns.c Line 1596

elph130
ASSERT!! BulkTrns.c Line 1595

elph180
ASSERT!! BulkTrns.c Line 1595

g7x
ASSERT!! BulkTrns.c Line 1595

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: CHDK PTP interface

« Reply #1207 on: 01 / July / 2019, 19:10:54 »

Quote from: reyalp on 01 / July / 2019, 00:22:50

original d10 bug (recv_data to uncached at 0x23f4 +N*512)
Code: [Select]
chdkptp -c -e'exec require"camtests".run("xferbug_0x23f4")'
Unresolved A540 bug (multiple recv_data calls, total size = the first 512*N-11 that results in multiple recv_data)
Code: [Select]
chdkptp -c -e'exec require"camtests".run("xferbug_0x1f5")'

Both of these tests execute successfully on a460.

Quote from: reyalp on 01 / July / 2019, 00:22:50

If anyone can get another program to PTP upload a 262645 byte file, that would be interesting.

I don't know whether gphoto qualifies. Tried on ixus65_sd630 (old vx with working gphoto upload), cam crashed with that size. Assert is the same as on your a540.

Code: [Select]

gphoto2 -u /tmp/camou.fir --folder=/store_00010001/DCIM/100CANON

Logged

reyalp

13878

Re: CHDK PTP interface

« Reply #1208 on: 04 / July / 2019, 19:08:10 »

Quote from: srsa_4c on 01 / July / 2019, 19:10:54

Both of these tests execute successfully on a460.

Weird. Early DryOS like sx100 and a470 might be interesting.

Quote

I don't know whether gphoto qualifies. Tried on ixus65_sd630 (old vx with working gphoto upload), cam crashed with that size. Assert is the same as on your a540.
Code: [Select]
gphoto2 -u /tmp/camou.fir --folder=/store_00010001/DCIM/100CANON

I think the gphoto code is related to ptpcam, but it's better than nothing.

There was a Canon uploader that worked for some old vxworks A series cams (ps_uploader on https://drive.google.com/drive/folders/0B08pqRtyrObjSmFqSV8yRUlqbnM note the exe/dlls are the same in each folder, it's just premade config files for different cams)
I'll see if I can make that work with A540.

edit:
I was able to upload a 256K + 501 size file to A540 using Canon UploadFirmware.exe on windows 10.

Pretty much just need to edit the FirmInfo.txt to have the file name and camera name you want to upload, and make sure libusb drivers are removed.

Uploading the same file with SendObject in chdkptp triggers the expected BulkTrns.c 1544 assert.

Next step I guess is to either hook and log PTP functions and/or capture USB traffic. Reversing the uploader program could also maybe be interesting.

« Last Edit: 04 / July / 2019, 19:59:16 by reyalp »

Logged

Don't forget what the H stands for.

srsa_4c

4451

Re: CHDK PTP interface

« Reply #1209 on: 06 / July / 2019, 12:26:49 »

Quote from: reyalp on 04 / July / 2019, 19:08:10

Early DryOS like sx100 and a470 might be interesting.

sx100: xferbug_0x23f4 executes normally, xferbug_0x1f5 causes ASSERT!! BulkTrns.c Line 1568 .
a470: both tests execute normally

May or may not be related: the low end a4xx series has crippled USB ("full speed"), up to and including a470.

Logged