CHDKPTP - PC Remote Control Performance Analysis - page 27 - RAW Shooting and Processing

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #260 on: 22 / September / 2012, 16:33:09 »

Advertisements

Quote from: SticK on 22 / September / 2012, 11:38:05

OOPS! ... CANON MENU TRAP // #81 //-- Can you please read my comments in the console file.

How can I past this so we can get the test results?

I really don't see how you expect me to diagnose anything from this.

Code: [Select]

ERROR: a script is already running

Means that some preceding command didn't finish before you issued the next command. Maybe if you'd waited longer, it would have finished. Or maybe it was hung due to some other problem. Since I have no idea what command was actually running and no insight into the state of the camera at the time, it's impossible for me to know.

There's no way I can magically look at some random error message with negligible context and tell you what is wrong.

Logged

Don't forget what the H stands for.

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #261 on: 22 / September / 2012, 16:45:01 »

Quote from: SticK on 22 / September / 2012, 01:41:15

Yes I have tried it a few times and the only buttons that work are camera PLAY and Alt, whether I send the event or not. Pressing camera SHOOT runs a script (flash in lower left corner) but does nothing else.

Oops, I missed this in the wall of text. This means that you are in the semi-alt mode I've described several times before.

Every time you run a script on the camera (this includes all camera buttons in the gui, any = or . commands, and con:exec* functions) the camera goes into this state, if it wasn't already in normal "alt" mode when you started.

To get out of this state, press the alt button until the <alt> sign appears in the UI, and press it once more. Then you aren't in alt mode. Then, if you've sent the unlock event, the canon buttons should probably work. It's possible the number for the unlock event has changed. If this is the case, you might find the right one by trying nearby numbers. note sending random levents might crash or shutdown the camera or cause other odd behavior.

I would suggest NOT using the play button on the camera to switch between play and rec, as in my experience this can leave the camera in a confused state and can kill the ptp connection. Use the play/rec buttons in the gui. (or =switch_mode_usb(0|1) in the cli)

Logged

Don't forget what the H stands for.

SticK

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #262 on: 22 / September / 2012, 17:59:45 »

Quote "This means that you are in the semi-alt mode I've described several times before. "

I recall asking you only once about getting CHDK menus to work again. You told me to push Alt a few times. I did not know this was also related to Canon buttons, and yes it works. Thanks. I am still learning, please be patient.

Quote "Assuming this "leak" is actually normal firmware behavior (which appears likely at this point), there's really very little alternative."

Although I could not confirm the above with a good test because the menu got stuck, sadly, I am concluding the same."

Quote "The most plausible approach would seem to be suppressing the file creation step completely, and downloading the data over USB."

That's how Canon's RemoteCapture works is why I haven't seen any issues in transfers of tens of thousands of files between scheduled maintenance intervals. This would be the best and correct solution.

Quote "Although I haven't tested yet, I expect srsa's remote capture system will not prevent this "leak", since it gets the data in the file write task, and still creates "ghost" files in the Canon UI."

Hmmmm, too bad // seems so. But ...... assuming "Canon UI" in this case must be the SD card, hence I am guessing what srsa is doing may be something like the following (CHDK RAWs notwithstanding) ....
a) shoot
b) instruct Canon to create file (or file pair) on SD with a new arbitrary file sequence number;
c) transfer the file (or file pair) to CHDKPTP;
d) delete the file (or file pair) from SD;
e) decrement the file counter in the camera;
f) increment a local CHDKPTP file sequence counter based on the last file number currently present in the target PC directory (Canon does Capture_NNNNN, ie five not four Ns);
g) save the file in the PC.
This sound generally right to you? This mechanism would preserve files already on the SD unaltered (what Canon RemoteCapture does too). It's not useful for me but I can see it has general use for photographers. There are two failure modes that have surfaced in this work over the last few weeks: post-shoot delay related and memory leak related. Given the limitations of Canon heap management, I think overall it is a good result and gives us some assurances and new knowledge. Sure, according to how you explain it here, we would quite definitely not see improvement on the memory leak. However ... if srsa can implement the "write-detect" hold-off in (a), then that would be a major improvement in my view. Why? Couple of reasons: 1) it would prevent random failures when the post-shoot delay is not worked out by the user for the worst-case in a specific hardware configuration (good for general use, not just me), leaving the memory leak to be handled externally by the user, and 2) leak-related failures can be made predictable and mitigated (reboot). For example, the srsa mechanism can sniff memory and simply report freemem to box on the GUI. It could be a bit more sophisticated, and allow user-settable warning threshold to turn ON a red LED on the CHDKPTP GUI when memory is getting dangerously low (reboot).

Also ...... could you use his write-detect holdoff in your shoot ?

What do you say?

« Last Edit: 22 / September / 2012, 18:06:22 by SticK »

Logged

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #263 on: 22 / September / 2012, 20:18:37 »

Quote from: SticK on 22 / September / 2012, 17:59:45

That's how Canon's RemoteCapture works is why I haven't seen any issues in transfers of tens of thousands of files between scheduled maintenance intervals. This would be the best and correct solution.

Sure, that would be nice, but we don't currently know how to do it. If you can find a skilled programmer who wants to dedicate a few hundred hours to reverse engineering, maybe they can figure it out for you.

Quote

assuming "Canon UI" in this case must be the SD card,

Canon UI means what the Canon firmware displays on the screen. By I "ghost file" I mean if you for example go to playback mode, you will see the ? unknown image icon for each picture taken. This doesn't mean a file was ever actually written, just that some unknown / not fully reverse engineered part of the firmware thinks one was.

Quote

hence I am guessing what srsa is doing may be something like the following (CHDK RAWs notwithstanding) ....
a) shoot
b) instruct Canon to create file (or file pair) on SD with a new arbitrary file sequence number;
c) transfer the file (or file pair) to CHDKPTP;
d) delete the file (or file pair) from SD;
e) decrement the file counter in the camera;
f) increment a local CHDKPTP file sequence counter based on the last file number currently present in the target
PC directory (Canon does Capture_NNNNN, ie five not four Ns);
g) save the file in the PC.
This sound generally right to you?

No, it's completely wrong, most of that stuff we have no control over. We just tell the camera to shoot, and modify the task (similar to thread/process) that would write the data out to disk. Between these two steps, a few dozen other tasks are probably involved in the shooting process, and most of them are not understood at all.

Speculation without some understanding of the details is just a waste of time. You can look at the code yourself http://trac.assembla.com/chdk/changeset?reponame=&new=2163%40branches%2Fptp-remote-capture-test&old=2114%40branches%2Fptp-remote-capture-test and discussion starting at http://chdk.setepontos.com/index.php?topic=4338.msg88931#msg88931

You still don't seem to have really get the fact that the vast majority of the Canon firmware is totally unknown to us. There are only a few small areas we interact with, and much of that is based on guesswork, trial and error rather than a full understanding.

Quote

leaving the memory leak to be handled externally by the user, and 2) leak-related failures can be made predictable and mitigated (reboot). For example, the srsa mechanism can sniff memory and simply report freemem to box on the GUI

Huh ? You don't need any remote capture stuff to do this, just call get_meminfo() and deal appropriately with the result. The "leak" accumulates slowly over time, there's no need to check inside of shoot. How little is safe is not well defined and depends on what you are doing, but if it gets under a hundred KB or so, it's definitely time to reboot.

Logged

Don't forget what the H stands for.

SticK

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #264 on: 22 / September / 2012, 20:54:50 »

My quote "assuming "Canon UI" in this case must be the SD card"
Quote "Canon UI means what the Canon firmware displays on the screen."

I was referring to the direct-USB mechanism srsa might be using that seems obvious to me: the only way you can transfer a 10MB file is either via an intermediate write to SD through a temporary file (my (a) to (g)), --or- he intercepts the Canon write and redirects it to USB? Possible?

Quote "No, it's completely wrong,"

I see my a-to-g is no good, so then how is he doing it, in general terms?

========

This is where I stand after the new knowledge gained. Overall, I think it might be faster and simpler if you could implement the write-detect holdoff (PT_CompleteFileWrite?) in shoot. That would go a very long way to prevent unwanted hardware contentions from causing a random error (like from the SD write time tails in the histogram I showed you), and, with a simple memory condition calculation reboot warning, shoot+dcimdl as-is could work very well. I'd be for the solution *acceptable*, and could move to Phase II.

Can we try it? If you can code it, I can pour in a few good days to test it. Sound good?

« Last Edit: 22 / September / 2012, 21:02:28 by SticK »

Logged

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #265 on: 22 / September / 2012, 22:12:52 »

Quote from: SticK on 22 / September / 2012, 20:54:50

he intercepts the Canon write and redirects it to USB? Possible?

This is sort of how it works. We modify "filewritetask", which is responsible for actually writing the data out. Instead of writing the data, we block until a PTP command received in another task reads the data, and then skip writing the actual file. This happens after some unknown number of other tasks have taken part in the shooting process. Implementing the filewrite hook needs to be done for each camera and is not trivial.

Note that this has only been done for jpeg and YUV data, not Canon raw, neither srsa nor myself has a camera with Canon raw. CHDK raw is done in a different place, and doesn't need any file trickery since that's all in our code.

Quote

Overall, I think it might be faster and simpler if you could implement the write-detect holdoff (PT_CompleteFileWrite?) in shoot.

Yes, now I see, it would be faster and simpler if I would just write whatever code you want

I don't see why this is urgent for you though. The fact that you could repeatedly hit the out of memory error after > 1000 shots suggests that you have a delay long enough to avoid timing issues. The FsIoNotify error is the only error we know is timing related. It seems like you have hit some other problems, but I don't have any evidence that detecting when the jpeg is done would help.

Logged

Don't forget what the H stands for.

SticK

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #266 on: 22 / September / 2012, 23:14:40 »

My quote "Overall, I think it might be faster and simpler if you could implement the write-detect holdoff (PT_CompleteFileWrite?) in shoot."
Quote "Yes, now I see, it would be faster and simpler if I would just write whatever code you want

"

You don't seem to understand ... especially now having read the RC link between you srsa, that is exactly my point. I am not asking for a CHDK feature like zebra, histogram, battery voltage or CCD temp, which I'm sure were no simple task. I am referring to a fundamental operational caveat. While I value the effort srsa is making and will be the first in line to give it a thorough shakeout with the final hope of moving his solution into the apparatus, in the interim to begin high-speed compatibility testing, fixing shoot is something that could be comparatively quicker than what srsa is doing, and, it would be useful for all users of CHDK who would no longer need to put in a delay for their scripts // that's what I'm saying. We know SD cards can delay an occasional write by up to several seconds. When doing compatibility testing in the apparatus, I would not want to be duped by an error that can be controlled ... and ... in order to do high-speed testing, I cannot set the post-shoot delay to several seconds to mitigate those controllable errors: if I run into a failure, it *has* to be on my side so *I* could handle it // it cannot be from CHDK+CHDKPTP side which could confuse me, get it?

Furthermore, from your conversation with srsa ... he is not considering CanonRAW because he does not have a camera that supports it, so development of his full solution may take even longer. Do you see why I favor fixing the shoot command?

« Last Edit: 22 / September / 2012, 23:33:19 by SticK »

Logged

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #267 on: 23 / September / 2012, 03:09:18 »

Quote from: SticK on 22 / September / 2012, 23:14:40

it would be useful for all users of CHDK who would no longer need to put in a delay for their scripts // that's what I'm saying.

It would certainly be good to fix, but you are the only user I'm aware of for whom it's a significant issue. The vast majority aren't trying to do a file operation immediately after a shot.

I currently have limited time to work on this stuff. I've already spent a great deal of it trying to help you, I'm not interested in being told what to do or arguing about my priorities.

Quote

in order to do high-speed testing, I cannot set the post-shoot delay to several seconds to mitigate those controllable errors: if I run into a failure, it *has* to be on my side so *I* could handle it // it cannot be from CHDK+CHDKPTP side which could confuse me, get it?

If you set requirements which can't be met with existing capabilities, that's your own choice.

Logged

Don't forget what the H stands for.

SticK

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #268 on: 23 / September / 2012, 08:55:51 »

I left a loop running last night that was intended to find a workable freemem limit. I came into the lab this morning and found your message and this error, attached.

I don't know if 645K (down from 830K initial value) on an S90 is actually the limit or this is a write time issue, nonetheless I cannot do anything more but to implore you to fix shoot to make it considerably more robust.

1.6s delay is a very long time for this SD card because the write time I measured is 3/4s for the 2 files. This could have been a case of memory card garbage collection write stall, since it has been written over several times by now. Here the camera did not crash but continues to give "ERROR A script is already running" if I try to control it from CHDKPTP.

Can you please give me your take on the console file?

Thanks.

[84] S90-100c 1.6s - 1000 myshoots - soft error at 514 ref DCIM [JPG&CR2].txt (126.31 kB - downloaded 44 times.)

Logged

reyalp

13878

Re: CHDKPTP - PC Remote Control Performance Analysis

« Reply #269 on: 23 / September / 2012, 15:35:09 »

Quote from: SticK on 23 / September / 2012, 08:55:51

I don't know if 645K (down from 830K initial value) on an S90 is actually the limit

Definitely not, many cameras don't have that much free at startup. 300k should be plenty, down to 150 is probably ok unless you are using something really memory heavy.

Quote

or this is a write time issue,

You should not assume those are the only two possibilities.

Quote

Here the camera did not crash but continues to give "ERROR A script is already running" if I try to control it from CHDKPTP.

I already explained what this means, and why by itself it does not provide enough information to make a useful diagnosis.

Quote

Can you please give me your take on the console file?

It's very strange, somehow a message from mdownload got treated as an error message. I don't have any idea how this could happen, and again the log really doesn't provide me with enough insight into the state of chdkptp and the camera to do any actual debugging. It's possible if you post the code you are currently using that something will be apparent, although the fact it got though more than 500 images suggests something odd happened on the camera.

Logged

Don't forget what the H stands for.