CHDK PTP interface - page 101 - General Discussion and Assistance - CHDK Forum

CHDK PTP interface

  • 1227 Replies
  • 381358 Views
*

Offline srsa_4c

  • ******
  • 4426
Re: CHDK PTP interface
« Reply #1000 on: 17 / June / 2013, 15:05:54 »
Advertisements
One oddity I noticed is that the jpeg code will sometimes return a zero size chunk as the last chunk. This isn't specifically a problem by itself, but I'm not sure how it crops up.
Looks like a check inside filewrite_get_jpeg_chunk() was not complete. Following patch should correct that.
Code: [Select]
Index: platform/generic/filewrite.c
===================================================================
--- platform/generic/filewrite.c (revision 2877)
+++ platform/generic/filewrite.c (working copy)
@@ -34,6 +34,9 @@
             return 0; // last chunk
         }
     }
+    else {
+        return 0; // last chunk
+    }
     return 1; // not last chunk
 #else
     if ( jpeg_chunks == NULL ) { //do we have a valid queue?

*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1001 on: 22 / June / 2013, 21:38:18 »
Thanks srsa, I added that fix.

In chdkptp changeset 375 I added an option to the rs command to shoot in continuous mode. This is mainly just to test that the remote capture code works in continuous mode. It appears to on both my cameras.

Use
rs -cont=<n>
to shoot N shots. You must set the camera to continuous mode in the canon UI yourself. It should work with any combination of raw/dng and jpeg.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1002 on: 23 / June / 2013, 22:43:23 »
In changeset 2896 I moved the raw / dng part out of capt_seq into raw.c (raw_process, formerly raw_savefile)

There's still plenty of ugly but I think it's an improvement overall.

Bracketing, curves, manual bad-pixel should all happen normally for remote cap now. You can now use the continuous mode remotecap with bracketing if you want.

Trying to do DNG badpixel generation or raw develop will make remote capture raw/dng fail, but I think that's OK.

This was the last significant code reorganization I wanted to do before merging to the trunk.

One bit of script api / protocol that I'm still not sure about is how file writing is handled:

Currently, if any remote cap options are used, all writing to the card is suppressed, except for jpeg on cameras that don't have filewrite task (edit: or canon raw on cams that have it). Being able to turn off jpeg is something that has been requested in the past, unrelated to remote capture. So it might be tempting to make it an option and/or scriptable setting.

I can also imagine situations where someone would want to use remoteshoot for some data and keep some on the card. Right now, if you request only dng header, all the image data is thrown away. OTOH, maybe people who data on the card should just use shoot+download?

Another minor thing:
The timeouts for the camera waiting for the PTP client to fetch data are currently hard coded. Does this need to be adjustable?


There's also a some non-PTP specific stuff I ran into while mucking around in the raw code, but I'll start another thread for that.

edit:
One more thing I've been thinking about: Is it worth sending the filename? It would be simpler to just send the exposure number as a param with each chunk (or as part of the RemoteCaptureIsReady response?), and that's essentially all we are using right now anyway. It would include the directory name/number, but that can be obtained lua if someone really wants it.
« Last Edit: 29 / June / 2013, 20:23:24 by reyalp »
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4426
Re: CHDK PTP interface
« Reply #1003 on: 24 / June / 2013, 19:05:31 »
You can now use the continuous mode remotecap with bracketing if you want.
I was afraid of continuous mode when remote shooting, so I never even tried that before. Now, after seeing what a dryos r51 camera is doing, it looks like there's a caching layer between filesystem operations (open, close, read, write) and real card accesses. The whole thing can change if Canon decides to cache the raw data before/during processing, which is not currently happening on P&S models AFAIK.

Quote
This was the last significant code reorganization I wanted to do before merging to the trunk.
Thanks for doing that.

Quote
Being able to turn off jpeg is something that has been requested in the past, unrelated to remote capture. So it might be tempting to make it an option and/or scriptable setting.
Makes sense, although I think that throwing away the jpeg in favour of dng is not a good idea for the average shooter.

Quote
I can also imagine situations where someone would want to use remoteshoot for some data and keep some on the card.
I think there may be problems with simultaneous sd card AND usb transfer (in a series of early tests nafraf's a810 was acting unstable when I managed to enable file operations while remote shooting on that test build), so we should avoid that.
Quote
Right now, if you request only dng header, all the image data is thrown away. OTOH, maybe people who data on the card should just use shoot+download?
That should still work, so yes. I think Canon's official remote shooting support included both scenarios (remote file only, remote+local file).

Quote
The timeouts for the camera waiting for the PTP client to fetch data are currently hard coded. Does this need to be adjustable?
I can imagine situations where it could be needed (slow usb host, slow transfer, 16MP raw + jpeg).

Quote
One more thing I've been thinking about: Is it worth sending the filename? It would be simpler to just send the exposure number as a param with each chunk (or as part of the RemoteCaptureIsReady response?), and that's essentially all we are using right now anyway. It would include the directory name/number, but that can be obtained lua if someone really wants it.
Makes sense, one round trip less. I originally thought that this task is responsible for writing native raw too, but that assumption turned out to be false (according to an experiment done by SticK last year).

Once these changes get into trunk, there are some other possible uses for that filewritetask hook (I have collected some ideas here - note that the list is old).


*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1004 on: 29 / June / 2013, 20:41:06 »
You can now use the continuous mode remotecap with bracketing if you want.
I was afraid of continuous mode when remote shooting, so I never even tried that before. Now, after seeing what a dryos r51 camera is doing, it looks like there's a caching layer between filesystem operations (open, close, read, write) and real card accesses. The whole thing can change if Canon decides to cache the raw data before/during processing, which is not currently happening on P&S models AFAIK.
Of course. I was a bit surprised that continuous mode worked, but it does on my cams at least. I think in general it should work with raw, because we hold up the whole shooting process until it's done. jpeg... I'm less sure.
Quote
Quote
Being able to turn off jpeg is something that has been requested in the past, unrelated to remote capture. So it might be tempting to make it an option and/or scriptable setting.
Makes sense, although I think that throwing away the jpeg in favour of dng is not a good idea for the average shooter.
Agreed. To save space, you could just use a small size / low quality jpeg. Time saved by skipping jpeg isn't very significant. People have asked for it a few times, but maybe it doesn't need to be added with remotecap.

Quote
I can imagine situations where it could be needed (slow usb host, slow transfer, 16MP raw + jpeg).
Good point. I'll add this. I think one timeout value for all the phases is sufficient.

Quote
Quote
One more thing I've been thinking about: Is it worth sending the filename? It would be simpler to just send the exposure number as a param with each chunk (or as part of the RemoteCaptureIsReady response?), and that's essentially all we are using right now anyway. It would include the directory name/number, but that can be obtained lua if someone really wants it.
Makes sense, one round trip less.
On the other hand, nothing requires the client to ask for the file name.
edit: but the current code doesn't actually give you the full path (since the jpeg name isn't available yet when we do raw), so I'm going to take it out.
« Last Edit: 29 / June / 2013, 20:49:55 by reyalp »
Don't forget what the H stands for.

*

Offline srsa_4c

  • ******
  • 4426
Re: CHDK PTP interface
« Reply #1005 on: 30 / June / 2013, 18:53:00 »
@reyalp

I'm a bit concerned about calling get_target_file_num() directly in PTP_CHDK_RemoteCaptureIsReady. Some cameras have a flaky counter (or the implementation of getting that number is not optimal), file numbering may become less reliable (my method was to store the counter at the start of the raw hook).

Tried to play with rs a bit (a470), it's still working, even in continuous mode.

*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1006 on: 30 / June / 2013, 19:25:43 »
I'm a bit concerned about calling get_target_file_num() directly in PTP_CHDK_RemoteCaptureIsReady. Some cameras have a flaky counter (or the implementation of getting that number is not optimal), file numbering may become less reliable (my method was to store the counter at the start of the raw hook).
You're right, it probably shouldn't call this when no datatypes are available yet. I'll change this in the next version.

Calling after the first one becomes available should be OK, since that always has to happen after the raw hook is called.

I'm currently in process of re-organizing the hook wait / timeout / cancel code, since after I implemented the adjustable timeout I found the behavior wasn't well defined. I'll probably break something...

Another thought:
PTP_CHDK_RemoteCaptureIsReady currently returns a special value if you call when no target is selected. I wonder if it should use PTP_RC_GeneralError instead. On the other hand, remotecap can get turned off by various errors, so maybe it should be different.
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1007 on: 30 / June / 2013, 22:38:16 »
I'm currently in process of re-organizing the hook wait / timeout / cancel code, since after I implemented the adjustable timeout I found the behavior wasn't well defined. I'll probably break something...
This is done (or at least started...) in changeset 2917

I've tried to make it so any error condition including timeouts will free all the hooks, clear all the chunk types, and clear the remotecap settings. I also tried to clean up the logic in PTP_CHDK_RemoteCaptureGetData a bit.

One thing that is not very well defined is what happens if the timeout happens in the middle of a download. What will happen is that the (spytask/filewrite) hook will be unblocked as soon as the timeout expires, and continue on it's way. The ptp task will continue transferring data, which could potentially get corrupted. There will be no error status, unless the client expected to get more chunks or data types. I'm not really sure how we could make this safe. I don't think this is likely to crash or corrupt memory on the cam, but I could be wrong.

We do want the timeout to be possible even if the transfer has started, because the client could lose connection or crash etc. We might be able to make the ptp command return an error status, but it would be convoluted.

There's a fair chance I broke CAM_FILEWRITETASK_SEEKS (dryos >= r50) cams. I'd appreciate if anyone with those cams can test.

Some things to check

rs with the various format options. Be sure to check the actual jpeg and dng to make sure they are valid.

Timeout error status:
Code: [Select]
=set_remotecap_timeout(10)
rs
This should shoot a shot without any delay and return something like
ERROR: remote shoot error

Timeout, data not downloaded
Code: [Select]
=set_remotecap_timeout(10000)
=init_remotecap(1) ; shoot()
This should shoot, and then wait 10 seconds, doing whatever LED activity would be normal when writing a jpeg.

Cancel remotecap from script:
Code: [Select]
=set_remotecap_timeout();init_remotecap(1);press('shoot_half');repeat sleep(10) until get_shooting();click('shoot_full');sleep(3000);init_remotecap(0)
This should shoot, wait 3 seconds and then finish. No files should be written.

Note that set_remotecap_timeout state is global, and stays set until you change it. To reset back to default, you can use set_remotecap_timeout() with no arguments.
Don't forget what the H stands for.


*

Offline srsa_4c

  • ******
  • 4426
Re: CHDK PTP interface
« Reply #1008 on: 02 / July / 2013, 12:10:36 »
Calling after the first one becomes available should be OK, since that always has to happen after the raw hook is called.
This counter is currently only used for raw (in vanilla CHDK), and is probably 'optimized' for that. I think that on at least a few Vx ports a delay of a few tens of msecs will produce a much less reliable result (you can take a look at the respective platform function of my ports for example (a410..) and the note beside it)
Code: [Select]
long get_file_next_counter() { //looks like this hack is needed for old vxworks
    return ((get_file_counter()>>4)+1)<<4;
}
You could put in a duplicate file (or counter) check in the client - at least for debug reasons.
Quote
PTP_CHDK_RemoteCaptureIsReady currently returns a special value if you call when no target is selected. I wonder if it should use PTP_RC_GeneralError instead. On the other hand, remotecap can get turned off by various errors, so maybe it should be different.
Since errors usually produce PTP_RC_GeneralError, maybe this one should also. Would it be possible (at some point) to return a more descriptive error sub-code?

The ptp task will continue transferring data, which could potentially get corrupted. There will be no error status, unless the client expected to get more chunks or data types. I'm not really sure how we could make this safe. I don't think this is likely to crash or corrupt memory on the cam, but I could be wrong.
I don't have a clear overview over this ATM, but:
In case of jpeg, the chunk descriptor structure is part of the filewritetask's message structure. If the task continues, it may get corrupted. We could either make a copy of it and use that, or we could plant some kind of an 'emergency brake' ... uhmm ... somewhere. We never send information about the file size, so it's up to the user to recognize a truncated file...

Can't help with r50+, but your examples appear to work on the a470.

edit:
You have a comment in remotecap.c:
remotecap_jpeg_chunks_done(); // make jpeg_chunks NULL, immediately. TODO needed?
This was (and probably is) needed for r50+ cameras, due to multitasking issues (can't be done from another task, because it's too late by then).
« Last Edit: 02 / July / 2013, 12:17:38 by srsa_4c »

*

Offline reyalp

  • ******
  • 13457
Re: CHDK PTP interface
« Reply #1009 on: 02 / July / 2013, 13:18:44 »
Calling after the first one becomes available should be OK, since that always has to happen after the raw hook is called.
This counter is currently only used for raw (in vanilla CHDK), and is probably 'optimized' for that. I think that on at least a few Vx ports a delay of a few tens of msecs will produce a much less reliable result (you can take a look at the respective platform function of my ports for example (a410..) and the note beside it)
It's also exposed in script as get_exp_count(). But you've convinced me, I'll save it in the raw hook so it will behave the same as raw.
Quote
Since errors usually produce PTP_RC_GeneralError, maybe this one should also. Would it be possible (at some point) to return a more descriptive error sub-code?
Good idea. Something to look into.

Quote
I don't have a clear overview over this ATM, but:
In case of jpeg, the chunk descriptor structure is part of the filewritetask's message structure. If the task continues, it may get corrupted. We could either make a copy of it and use that, or we could plant some kind of an 'emergency brake' ... uhmm ... somewhere. We never send information about the file size, so it's up to the user to recognize a truncated file...
The current code gets the address and size once for each chunk. If the timeout expires while that chunk is being transferred, the transfer will continue using that address/size (which may no longer contain valid data). If the chunk wasn't the last chunk, it will get an error because the remotecap state has been cleared.

Quote
Can't help with r50+, but your examples appear to work on the a470.
Nafraf reported it worked on sx160. But I'm thinking that the situation above may have additional problems, because there could be calls to the filewrite hook after the remotecap state is cleared... but maybe it's ok because the ignore write flag is still set? I'll have to look at this again.

Quote
You have a comment in remotecap.c:
remotecap_jpeg_chunks_done(); // make jpeg_chunks NULL, immediately. TODO needed?
This was (and probably is) needed for r50+ cameras, due to multitasking issues (can't be done from another task, because it's too late by then).
I'm not sure I follow. This would get called from the ptp task at the end of it's chunk. Meanwhile, filewritetask is waiting. When filewritetask is finished waiting, it will immediately set jpeg_chunks null. I guess there could be another call from ptp before it gets there?

Thanks very much for looking at these changes. I didn't really want to change this much code again, but I started trying to simplify things and then realized I hadn't understood how it was really working before...
Don't forget what the H stands for.

 

Related Topics