CHDK PTP interface

srsa_4c

I have also experienced some weirdness when I tested the PTP liveview stuff on the A410, but I haven't tested it thoroughly. In short: I've been unable to upload the whole (trunk) installation package over CHDK-PTP (using the latest chdkptp GUI, with neither CHDK 1.0 nor CHDK 1.1 on the camera). A particular file which always caused assert was CHDK/LUALIB/drawings.lua . More investigation would be needed, I guess. The romlog shows an assert in BulkTrns.c line 0x5d8, task: tPtpTrsnpV.
Separate (long standing) issue is the extreme slowness of the file upload on USB1.1 cameras (pause of multiple seconds after each file).

reyalp

I'm not getting an assert with either messages or file upload. The ptp task gets stalled.

However, I wrote a script that sends messages from with every length from 1 to N and that seems to be crashing in a different way.

edit:
Here's the script. Getting several unexplained results, but it looks like D10 gets a similar issue at length=500

chdkptp -c -i
!m=require'msgtest'
!m.load()
!m.test(<start lenght>,<optional max length>,<optional increment>)

edit:
Updated, you can now specify increment

edit:
this appears to be related to the bits of the data length ending in 1f4 😛

msgtest.lua

812b

reyalp

Filename length may be a red herring.

On my a540, both PTP_CHDK_WriteScriptMsg and PTP_CHDK_UploadFile seem to hang for ~5 sec in recv_ptp_data

Stuffing the filename length into a global displayed with misc debug vals shows the right.

Attached is patch with debug code I'm using to check this.

Another oddity: on D10 0x1f4 is bad, on a540 is not. Subsequent values (3f4, 5f4 etc) are bad on both.

ptp-0x1F4-debug-1.patch

3.91KB

srsa_4c

reyalp
However, I wrote a script that sends messages from with every length from 1 to N and that seems to be crashing in a different way.

Just a quick test: problematic lengths on the A410: 0x74, 0xb4, 0xf4, 0x134 (and so on, I guess). The assert (same as above) comes on the next ptp attempt.

reyalp

srsa_4c
Interesting the interval is smaller. IIRC, usb1 uses smaller packets for bulk data ?

Some other random observations:
Examining ptp_data->recv_data, it looks like it returns 0 on success, and various values on error. When this particular problem happens, it returns 1. recv_ptp_data in ptp.c doesn't check the status

On d10
00116180: [PTPState] PTP_Unexpected Type=0 STATE=5 EVENT=16
seems to appear in camera log when this happens. Unfortunately, it's not clear where the function that produces this string is called from.

There are references in the firmware (both a540 and d10) to "A/BulkTrns.Log". On a540 it looks like the eventprocs

PTPRspnd.StartTransportLog
PTPRspnd.StopTransportLog
PTPRspnd.ShowTransportLog
PTPRspnd.ClearTransportLog

should relate to this. However, after registering them with RegComEventProc, I wasn't able to get them to do anything. They don't seem to exist on d10.

msl

reyalp
Minor versions (like 2.0 to 2.3) are supposed to be backward compatible, so it should be safe for ptpCamGui to start if the minor version is greater than what it knows about.

Yes, I know. I've made a little update for ptpCamGui. Now the gui will only check the major version.

msl

reyalp

I had the idea that if the problem is triggered by reading a particular size, we could detect that size and force multiple calls to data->recv_data, so no single call uses the magic size. Turns out, this doesn't help 🙁

Another observation: On D10, size 0x1F4 behaves slightly differently from the larger problem values. It appears to hang in data->get_data_size, which then returns 0. For larger values, get_data_size returns the correct value and the hang is in recv_data. I suppose this relates to the fact that ox1F4 is ok on a540...

edit:
The [PTPState] PTP_Unexpected Type=0 STATE=5 EVENT=16 message comes from the get_data_size case, not recv_data

reyalp

On D10, I narrowed down where the error is returned from in data->recv_data (sub_FF9FA25😎 to the bne at FF9FA37C. The hang seems to happen in the preceding call to sub_FF869A1C__KerFlag_c (some kind of synchronization object timing out, I think)

Some of the preceding code seems to be related to the unknown parameter to recv_data that we always set to zero. ~~I tried putting in a 1, but that didn't help.~~

edit:
testing error, setting the unknown param to 1 crashes very promptly when you try to use ptp. No surprise there 😉

philmoz

reyalp
On D10, I narrowed down where the error is returned from in data->recv_data (sub_FF9FA25😎 to the bne at FF9FA37C. The hang seems to happen in the preceding call to sub_FF869A1C__KerFlag_c (some kind of synchronization object timing out, I think)

Some of the preceding code seems to be related to the unknown parameter to recv_data that we always set to zero. ~~I tried putting in a 1, but that didn't help.~~

edit:
testing error, setting the unknown param to 1 crashes very promptly when you try to use ptp. No surprise there 😉

Out of curiosity, does the problem still happen using the Rasberry PI version you posted about in another thread?
If so that would pretty much make it a firmware problem, as your research seems to indicate. Or at least a problem in the way CHDK is using the firmware.

Phil.

reyalp

philmoz
Out of curiosity, does the problem still happen using the Rasberry PI version you posted about in another thread?
If so that would pretty much make it a firmware problem, as your research seems to indicate. Or at least a problem in the way CHDK is using the firmware.

It does the same thing from the pi. Not sure it's really any different than the linux laptop I already tested though...

reyalp

Another thought: Can we reproduce the problem with ps_uploader mentioned here: https://chdk.setepontos.com/d/1632/5

File is available in ps_uploader at https://www.box.com/chdk#/chdk/1/55270403

edit:
We don't know the canon protocol of course, a range of files sizes near one of the magic numbers might be worthwhile.

edit:
I can't seem to get it to recognize my a540 though...

reyalp

Other ideas
http://biot.com/blog/usb-sniffing-on-linux
http://lxr.linux.no/#linux+v2.6.28.8/Documentation/usb/usbmon.txt

edit:
http://wiki.wireshark.org/CaptureSetup/USB

philmoz

reyalp
Other ideas
http://biot.com/blog/usb-sniffing-on-linux
http://lxr.linux.no/#linux+v2.6.28.8/Documentation/usb/usbmon.txt

edit:
http://wiki.wireshark.org/CaptureSetup/USB

Found it 😀

While researching how USB PTP transfers work I came across this:
A bulk transfer is considered complete when it has transferred the exact amount of data requested, transferred a packet less than the maximum endpoint size, or transferred a zero-length packet.

When the chdkptp code sends a file/message that is exactly a multiple of the maximum end point size (512) the camera is still waiting for the bulk transfer to be completed. In this case it is necessary to send another 0 length packet to terminate the transfer.

The modified ptp_usb_senddata code below works on my cameras.

uint16_t
ptp_usb_senddata (PTPParams* params, PTPContainer* ptp,
			unsigned char *data, unsigned int size)
{
	static uint16_t ret;
	static PTPUSBBulkContainer usbdata;

	/* build appropriate USB container */
	usbdata.length=htod32(PTP_USB_BULK_HDR_LEN+size);
	usbdata.type=htod16(PTP_USB_CONTAINER_DATA);
	usbdata.code=htod16(ptp->Code);
	usbdata.trans_id=htod32(ptp->Transaction_ID);
	memcpy(usbdata.payload.data,data,
		(size<PTP_USB_BULK_PAYLOAD_LEN)?size😛TP_USB_BULK_PAYLOAD_LEN);
	/* send first part of data */
	ret=params->write_func((unsigned char *)&usbdata, PTP_USB_BULK_HDR_LEN+
		((size<PTP_USB_BULK_PAYLOAD_LEN)?size😛TP_USB_BULK_PAYLOAD_LEN),
		params->data);
	if (ret!=PTP_RC_OK) {
		ret = PTP_ERROR_IO;
/*		ptp_error (params,
		"PTP: request code 0x%04x sending data error 0x%04x",
			ptp->Code,ret);*/
		return ret;
	}
    if (size > PTP_USB_BULK_PAYLOAD_LEN)
    {
        /* if everything OK send the rest */
        ret=params->write_func (data+PTP_USB_BULK_PAYLOAD_LEN,
                    size-PTP_USB_BULK_PAYLOAD_LEN, params->data);
        if (ret!=PTP_RC_OK) {
            ret = PTP_ERROR_IO;
    /*		ptp_error (params,
            "PTP: request code 0x%04x sending data error 0x%04x",
                ptp->Code,ret); */
        }
    }
    if ((usbdata.length & 0x1FF) == 0)
        params->write_func(data, 0, params->data);
    return ret;
}

Phil.

srsa_4c

philmoz
Found it 😀

While researching how USB PTP transfers work I came across this:
A bulk transfer is considered complete when it has transferred the exact amount of data requested, transferred a packet less than the maximum endpoint size, or transferred a zero-length packet.

When the chdkptp code sends a file/message that is exactly a multiple of the maximum end point size (512) the camera is still waiting for the bulk transfer to be completed. In this case it is necessary to send another 0 length packet to terminate the transfer.

Nice find. 😀

I've tested it with 0x3f as the magic value (still testing with a USB1.1 (or full speed ?) camera), and it really fixes the issue in question. (I now get "unexpected msg type" sporadically while running reyalp's msgtest)
[hr]
Am I correct that the current implementation assumes "high speed" connection? I've found a define named PTP_USB_BULK_HS_MAX_PACKET_LEN in ptp.h . According to my web search this originates from some libgphoto related code(?). I've also found PTP_USB_BULK_FS_MAX_PACKET_LEN on the web in a project named arducam-osd which assumes full speed instead (which is understandable since it's Arduino-related).

I guess the connection speed could be detected somehow, and that magic value could be based on the connection type. I do not know the libusb code enough to tell how to do that...

reyalp

philmoz
Found it 😀

While researching how USB PTP transfers work I came across this:
A bulk transfer is considered complete when it has transferred the exact amount of data requested, transferred a packet less than the maximum endpoint size, or transferred a zero-length packet.

Excellent 😀

My impression (not knowing much about USB...) is that the ptpcam code is pretty rough if a number of ways.

philmoz

srsa_4c

philmoz
Found it 😀

While researching how USB PTP transfers work I came across this:
A bulk transfer is considered complete when it has transferred the exact amount of data requested, transferred a packet less than the maximum endpoint size, or transferred a zero-length packet.

When the chdkptp code sends a file/message that is exactly a multiple of the maximum end point size (512) the camera is still waiting for the bulk transfer to be completed. In this case it is necessary to send another 0 length packet to terminate the transfer.

Nice find. 😀

I've tested it with 0x3f as the magic value (still testing with a USB1.1 (or full speed ?) camera), and it really fixes the issue in question. (I now get "unexpected msg type" sporadically while running reyalp's msgtest)
[hr]
Am I correct that the current implementation assumes "high speed" connection? I've found a define named PTP_USB_BULK_HS_MAX_PACKET_LEN in ptp.h . According to my web search this originates from some libgphoto related code(?). I've also found PTP_USB_BULK_FS_MAX_PACKET_LEN on the web in a project named arducam-osd which assumes full speed instead (which is understandable since it's Arduino-related).

I guess the connection speed could be detected somehow, and that magic value could be based on the connection type. I do not know the libusb code enough to tell how to do that...

Attached is a patch to the current chdkptp trunk that attempts to get the maximum endpoint size from the libusb data and use that to determine if the dummy packet needs to be sent.

Can you try this with your camera and see what happens.

I have no idea if this is the right way to do this sort if thing - I'm still trying to understand the whole USB PTP interface stuff.

Phil.

bulk_transfer_fix.patch

2.8KB

srsa_4c

philmoz
Attached is a patch to the current chdkptp trunk that attempts to get the maximum endpoint size from the libusb data and use that to determine if the dummy packet needs to be sent.

Can you try this with your camera and see what happens.

The recognition of the max packet size works well for the two "full speed" cameras I tried.
After testing them with reyalp's msgtest, the results are:
A410 (DIGIC II, full speed)
Fails at length=0x34 (camera does not become unstable), works with any other length (tried up to 520). Note that as I reported earlier, length=0x34 was not problematic on this camera. (Hmmm, looks like a need for special casing some models...)
A420 (DIGIC II, full speed)
Max packet size recognition correct, msgtest runs correctly with any length.
A470 (DIGIC III, full speed)
Works with any length, but I get random "unexpected msg type" showstoppers while running msgtest. These start to be an issue over length>200. Camera does not become unstable, the test can be restarted. Previously, all lengths of (n*0x40)-0xc were problematic.[hr]
Ixus65 (DIGIC II, high speed)
Max packet size recognition correct, msgtest runs correctly with any length.

edit: host controller USB2.0, OS Linux 32bit

A separate note: the test with msgtest runs much slower on DIGIC II cameras, no matter full or high speed.

I have no idea if this is the right way to do this sort if thing - I'm still trying to understand the whole USB PTP interface stuff.

Looks good enough to me.

reyalp

Great work.

One thing I noticed, PTP_USB_BULK_PAYLOAD_LEN was based on the old hard coded max_packet_size. I'm not clear if this actually matters anywhere, the first send in send_data might send more than one packet, but I'm not sure there's anything wrong with that.

ptp_usb_getdata looks similar.

I think the "unexpected msg type" error is an unrelated problem. I get it on D10 as well. That is probably the something failing in the msgtest script, which returns an error rather than user message, which msgtest doesn't understand.

edit:
I checked in philmoz patch in chdkptp changeset 258. Leaving the debug print in for now so people can see what size it's detecting. I think getting wMaxPacketSize is correct.

edit:
the "unexpected" message I get is "not enough memory"

edit:
Fixed in trunk 1950/release 1951. This should make anything that relies on sending ptp messages a lot more stable 😖

edit:
with this change, a540 runs msgtest from 1 to 10000 with no problems
d10 died with a lost ptp connection after over 9000, not clear why
sending 9204 (0x23f4)...unexpected return code 0x2ff

cgott4242

I have 2 SD1100IS cameras and have ptpgui working. It enables me to shoot photos (remote shutter release) and d/l files.
I don't see a live view function
or a way to control multiple cameras

Does the PTPcam do these things? if not, which software does?

thx

msl

cgott4242
I don't see a live view function
or a way to control multiple cameras

https://chdk.setepontos.com/d/6285/55

« Previous Page Next Page »