CHDK PTP interface

reyalp

srsa_4c

- hook_raw_save_complete() is only called after all the chunks are downloaded, meaning the the raw hook will be blocked until the jpeg is complete. This seems like it might have unwanted impacts, perhaps depending on where the raw hook is on a specific camera.

I assumed the two hooks to never overlap. If they do, the port is IMHO broken (imagine what would happen if FileWriteTask's open stage found an open file). That was the reason I used the same variable and status codes for the two hooks.

OK, the current issue is probably a problem in my understanding of the old code and change then. filewrite now doesn't use the raw status variable at all (more clear, I think), so hook_raw_save_complete can probably be called when the last raw chunk is collected. ~~I'm actually surprised it works the way I have it.~~

edit:
More misunderstanding on my part, I was thinking all the chunks could be read before the end case, but of course raw has to be finished before jpeg starts. hook_raw_save_complete will get called redundantly for non-raw chunks, but as you said, raw hook should not be running when jpeg or yuv chunks are being handled. Still need to look at state_shooting_progresss

edit:
You had a comment

hook_raw_size() is sometimes different than CAM_RAW_ROWS*CAM_RAW_ROWPIX*CAM_SENSOR_BITS_PER_PIXEL/8

Do you know which cameras have this ? It seems like it should not ever be true. DNG will probably be corrupt if this happens (more / less data than given in exif...)

srsa_4c

reyalp
hook_raw_size() is sometimes different than CAM_RAW_ROWS*CAM_RAW_ROWPIX*CAM_SENSOR_BITS_PER_PIXEL/8

Do you know which cameras have this ? It seems like it should not ever be true. DNG will probably be corrupt if this happens (more / less data than given in exif...)

A470, the difference is 6 lines. I haven't tested RAW shooting on it too much 😉

raw has to be finished before jpeg starts

Now that I think of it... What happens in continuous mode? There has to be some caching somewhere. Could it be a filesystem cache (meaning Open(), Write(), Close() finish before the data actually ends up on the card)?

reyalp

srsa_4c
A470, the difference is 6 lines. I haven't tested RAW shooting on it too much 😉

OK, this should be investigated. It may be that someone thought it was a good idea to discard the blank lines at the bottom of the sensor, but the values should be consistent.

Now that I think of it... What happens in continuous mode? There has to be some caching somewhere. Could it be a filesystem cache (meaning Open(), Write(), Close() finish before the data actually ends up on the card)?

When we write raw in continuous mode, I'm pretty certain it blocks until the write is finished, and very little if any caching is done with write().

srsa_4c

reyalp

Now that I think of it... What happens in continuous mode? There has to be some caching somewhere. Could it be a filesystem cache (meaning Open(), Write(), Close() finish before the data actually ends up on the card)?

When we write raw in continuous mode, I'm pretty certain it blocks until the write is finished, and very little if any caching is done with write().

😳 Canon DSLRs do this sort of thing (and maybe compacts of some other manufacturers). My mistake. Haven't found any PowerShot with this feature (caching).

I have committed some YUV related code to the test branch. It will have to be improved:
- Either a new #define in camera.h will be needed for this ability, or the client has to handle the case where YUV data has 0 length.
- As discussed earlier, YUV data is not always available. Picture sizes with one pass JPEG creation include "L" and "W" (every picture size with full width).
- The review time should last longer than the time required for transfer.

A question:
Shouldn't these

void filewrite_set_discard_jpeg(int state);
void filewrite_get_jpeg_chunk(char **ardr,unsigned *size, unsigned n);

declarations in core/remotecap.c be extern?

A470:
the values found by sigfinder are good.

About [tt]SHOOTING_PROGRESS_DONE[/tt]:
core/action_stack.c
one of the last actions of [tt]AS_SHOOT[/tt] is [tt]AS_WAIT_SAVE[/tt]
[tt]AS_WAIT_SAVE[/tt] will not finish until [tt](state_shooting_progress == SHOOTING_PROGRESS_DONE)[/tt]
state_shooting_progress normally gets this value in core/main.c, which is not happening when remote shooting is active.

It's true that

state_shooting_progress=SHOOTING_PROGRESS_DONE;

in remotecap_get_data_chunk() is in an unfortunate place though...

reyalp

srsa_4c
A question:
Shouldn't these
[code]void filewrite_set_discard_jpeg(int state);
void filewrite_get_jpeg_chunk(char **ardr,unsigned *size, unsigned n);[/code]declarations in core/remotecap.c be extern?

For functions, it doesn't make any difference, but I guess we mostly use extern elsewhere (including in the headers), so it would be more consistent.

state_shooting_progress normally gets this value in core/main.c, which is not happening when remote shooting is active.

It's true that
[code]state_shooting_progress=SHOOTING_PROGRESS_DONE;[/code]in remotecap_get_data_chunk() is in an unfortunate place though...

That makes sense, and I agree that it's not ideal at the moment. Especially in the case where it could get set at an arbitrary time to clear hooks.

It may be wroth tracking what stage we are in (waiting for raw to be read, waiting for jpeg/yuv) and only having the appropriate once canceled. Something like state_remotecap_stage = IDLE | RAW | FILEWRITE

srsa_4c

Still experimenting with state_shooting_progress. Also inspected core/main.c again. Unlike what I wrote above, some shooting related conditional parts of spytask are effective even in remote target mode. Many - raw related - parts of CHDK set state_shooting_progress to SHOOTING_PROGRESS_PROCESSING. Doing this in remotecap.c (or at the start of the raw hook in generic/capt_seq.c) works, with one drawback: shoot() won't finish until review time is over (issue observed on the A410, and is probably unrelated to these experiments). The block is caused by PROPCASE_SHOOTING, see shooting_in_progress(). This doesn't prevent another rs from executing, but other scripts will not run until PROPCASE_SHOOTING becomes 0 again.

#1

Index: core/remotecap.c
===================================================================
--- core/remotecap.c   (revision 2141)
+++ core/remotecap.c   (working copy)
@@ -212,7 +212,7 @@
         // allow raw hook to continue
         // TODO could do like filewrite wait
         hook_raw_save_complete(); 
-        state_shooting_progress=SHOOTING_PROGRESS_DONE; //shoot() needs this... TODO
+        state_shooting_progress=SHOOTING_PROGRESS_PROCESSING;
     }
 }

or
#2

Index: platform/generic/capt_seq.c
===================================================================
--- platform/generic/capt_seq.c   (revision 2141)
+++ platform/generic/capt_seq.c   (working copy)
@@ -50,10 +50,12 @@
     }
     else if ( (remotecap_get_target() & 2) > 0 ) //chdk raw requested
     {
+        state_shooting_progress=SHOOTING_PROGRESS_PROCESSING;
         remotecap_raw_available();
     }
     else
     {
+        state_shooting_progress=SHOOTING_PROGRESS_PROCESSING;
         raw_save_stage = RAWDATA_SAVED;
     }
 #else

[hr]

reyalp
- should the remote capture selection only apply to the next shot, like exposure overrides, or should you need to explicitly cancel it ?

My initial code used the latter (rs sets it up at start and resets it when it has finished). Probably both approach should deal with the possibility of PTP connection problems.

If it's left on after the PTP connection goes away, that could confuse users. We don't explicitly know whether there is a PTP connection, but I guess we could check USB power is present (and clear remotecap if it's not found ?).

Could be good enough, until somebody decides to allow usb remote and ptp at the same time if that's possible at all (physw bit override or eventprocs like ConnectUSBCable, etc.)

Setting it once per shot shouldn't be much of a burden if you are using script to shoot, but I guess you could potentially set it once and have the user manually shooting with the shutter ?

Maybe. There are also some G series cams with proper remote control socket.
[hr]
Committed my changes (changeset 2144), chose #1 from above.
The client (the code I know) was already prepared, so it will continue to work. RemoteCaptureGetData's Param2 returned parameter has slightly changed in its interpretation: it no longer shows the transferred chunk's type. I chose to increment PTP_CHDK_VERSION_MINOR to 104 (the hundreds shall reflect test revisions 😀 )
[hr]Tried and failed miserably to split the code part marked with [tt]//TODO turn this into a function[/tt] in core/remotecap.c, remotecap_get_data_chunk(). With that code in a separate function (6 params), called by the raw and yuv cases, the asm code size almost doubled. Leaving it as it is, it's ugly but saves space...

reyalp

In chdkptp r301, I've started making a lua-based remoteshoot command. It's pretty messy at the moment, but it seems to work. This uses remotecap_init() on the camera to set up.

I understand the "script is already running" problem a bit a better now, of course the 'shoot()' command doesn't finish until the remote capture is done, so you can't cancel with script. The jpeg timeout works at least 😉 In the new command, I get around this by using button presses instead of shoot, so the script can end as soon as shoot_full has been clicked.

Eventually, the rs2 command should be rolled into options for the regular shoot command, and the supporting code should be moved into re-usable functions in one of the lua modules.

srsa_4c

reyalp
In chdkptp r301, I've started making a lua-based remoteshoot command. It's pretty messy at the moment, but it seems to work. This uses remotecap_init() on the camera to set up.

Will try it soon.

I understand the "script is already running" problem a bit a better now, of course the 'shoot()' command doesn't finish until the remote capture is done, so you can't cancel with script. The jpeg timeout works at least 😉 In the new command, I get around this by using button presses instead of shoot, so the script can end as soon as shoot_full has been clicked.

If this approach is the way to go, can we get rid of setting [tt]state_shooting_progress[/tt]?

I plan to commit a few changes soon (raw hook, consistent file number request), if testing confirms that they work.

There may be a new issue: right now it's assumed that there is only one filewritetask activity after shooting. Cameras with Canon RAW are usually capable of storing a separate jpeg beside the native RAW (when the camera is set up to do so). In that case, only the first Canon file (my guess: RAW) will be retrieved and/or prevented from being written. Should we ignore this or support it? I don't have access to such camera, so can't try myself.
[hr]
Changes committed (changeset 2163). rs2 works well if it encounters no problems.
I'm considering returning some useful data for debug use (chunk's memory address, could provide useful info about the jpeg or native raw buffers).

FREE Spirit

Hi
I have sx40hs and I installed the latest CHDK version: sx40hs-100i-1.2.0-2171-full_BETA
I tried both (ptpCamGui2.0.131) and (chdkptp-r291-win32)
in CHDK PTP, everything seems to be fine, even live view. But unfortunately, there are 2 problems:

1) when I press "shoot" to capture a picture, the shot is taken but the camera freezes and the viewfinder on PC becomes white noise (pic attached). The only thing can be done is to shut the camera off manually.
Same thing occurs in PTP Cam, except of course, there's no viewfinder.

2) I can't view files under the files tab. I tried copying the contents and got the following error message:

...hy\CHDK\LiveCapture\chdkptp-r291-win32\lua\chdku.lua:162: attempt to index local 'opts' (a nil value)
stack traceback:
	...hy\CHDK\LiveCapture\chdkptp-r291-win32\lua\chdku.lua:162: in function 'mdownload'
	...CHDK\LiveCapture\chdkptp-r291-win32\lua\gui_tree.lua:110: in function 'do_dir_download_dialog'
	...CHDK\LiveCapture\chdkptp-r291-win32\lua\gui_tree.lua:273: in function <...CHDK\LiveCapture\chdkptp-r291-win32\lua\gui_tree.lua:272>
	(tail call): ?
	[C]: in function 'Popup'
	menu.lua:15: in function 'popup'
	...CHDK\LiveCapture\chdkptp-r291-win32\lua\gui_tree.lua:288: in function <...CHDK\LiveCapture\chdkptp-r291-win32\lua\gui_tree.lua:241>
	(tail call): ?
	[C]: in function 'MainLoop'
	...aphy\CHDK\LiveCapture\chdkptp-r291-win32\lua\gui.lua:673: in function <...aphy\CHDK\LiveCapture\chdkptp-r291-win32\lua\gui.lua:659>
	(tail call): ?
	...phy\CHDK\LiveCapture\chdkptp-r291-win32\lua\main.lua:226: in main chunk
	[C]: in function 'require'
	[string "require('main')"]:1: in main chunk

However, on PTP Cam, I can copy the files

any solution?

reyalp

FREE Spirit
1) when I press "shoot" to capture a picture, the shot is taken but the camera freezes and the viewfinder on PC becomes white noise (pic attached). The only thing can be done is to shut the camera off manually.
Same thing occurs in PTP Cam, except of course, there's no viewfinder.

Do you have review mode set to "hold" on the camera ? Or a long time ?

2) I can't view files under the files tab. I tried copying the contents and got the following error message:
[code]...hy\CHDK\LiveCapture\chdkptp-r291-win32\lua\chdku.lua:162: attempt to index local 'opts' (a nil value)
stack traceback:
[/code]

This bug is fixed in the latest chdkptp svn. If you replace lua/chdku.lua with the one found here http://trac.assembla.com/chdkptp/browser/trunk/lua/chdku.lua?rev=293 it should be fixed.

I haven't put up another snapshot because I'm in the middle of some messy changes for remotecapture...

FREE Spirit

Thanx for the reply
yes I had my review mode set to "hold", disabling this got things working 😀

This bug is fixed in the latest chdkptp svn. If you replace lua/chdku.lua with the one found here http://trac.assembla.com/chdkptp/browser/trunk/lua/chdku.lua?rev=293 it should be fixed

I replaced the file, but it didn't solve the problem. Now I get this error message:
"error: A/ÚºÃ«Â«Â§ÚºÂ§~1.PPT: error"

reyalp

FREE Spirit
Thanx for the reply
yes I had my review mode set to "hold", disabling this got things working 😀

This bug is fixed in the latest chdkptp svn. If you replace lua/chdku.lua with the one found here http://trac.assembla.com/chdkptp/browser/trunk/lua/chdku.lua?rev=293 it should be fixed

I replaced the file, but it didn't solve the problem. Now I get this error message:
"error: A/ÚºÃ«Â«Â§ÚºÂ§~1.PPT: error"

That is very strange. Are you using non-latin character set on the camera or PC ?

If you enter
ls -l
in the console, what does it do ?

FREE Spirit

sorry for the delay
I have non-latin character on my laptop *only* as typing language.
Both my camera and my laptop's OS (Windows 7 x64 Home Premium) have English UI

I tried entering
ls -l
in the console. Nothing happened

srsa_4c

FREE Spirit
I tried entering
ls -l
in the console. Nothing happened

You need to enter that line in the text box beside the Execute button, and of course hit enter afterwards (or click Execute).

Now I get this error message:
"error: A/ÚºÃ«Â«Â§ÚºÂ§~1.PPT: error"

Are you sure you don't have a PowerPoint file on that card (copied there with a card reader)?

FREE Spirit

yea that's what I did... nothing happened

srsa_4c

My experience with a different camera
https://chdk.setepontos.com/d/8600/12
The camera was not prepared to correctly handle a long filename.
What happens if you execute this:

=return os.listdir('A/')

?
In your case the problem might be caused by a filename with non-latin characters.

FREE Spirit

I entered the code
=return os.listdir('A/')
and it displayed the list of files on the card root... and indeed there were2 files with non-latin names which I forgot about... and even if I remembered them I wouldn't have thought they can cause total failure of the whole thing

Thanx for taking the time to solve the problem 😀

srsa_4c

As a result of the recent filewritetask investigation on the S90 (thx to SticK) and A810 (thx to nafraf), it looks like many of my previous assumptions were wrong.
It looks like that
- filewritetask only writes jpeg files (!), native raw is written elsewhere
- earlier cameras (including the S90) usually write the files in two chunks, first one is the exif part, second one is the main jpeg. On the A810 this is different, the whole file appears in one or two chunks, the exif is presumably embedded in the first chunk.
- another oddity on the A810: the chunks of the jpeg seem to come from non-contiguous buffers
edit:
~~... and of course there's no such thing as multiple filewritetask invocations for a single file, according to the collected data.~~
edit, A810: Now it looks like that Canon has started doing some parallel processing. Sometimes, the jpeg file is starting to get written when only a part of the (L sized) compressed image is present. When an additional part of the image becomes available, it gets appended to the partial file on card!

srsa_4c

@reyalp

Changeset 2203:
My (uncommitted) version was like this:

int register_pt_hooks() {
#if (!CAM_DRYOS) || defined(CAMERA_s5is)
    /*
     * System.Create -> SystemEventInit
     * SS.Create -> ?
     * PT_CompleteFileWrite -> no equivalent?
     */
    return -1;
#else
    if(_ExecuteEventProcedure("System.Create") == -1) {
        return 1;
    }
    if(_ExecuteEventProcedure("SS.Create") == -1) {
        return 2;
    }
    if(_ExecuteEventProcedure("ExportToEventProcedure","PT_CompleteFileWrite",overridden_PT_CompleteFileWrite) == -1) {
        return 3;
    }
    //_LogPrintf(0x120,"pt hook(s) registered");
    return 0;
#endif
}

All DryOS >= r23 cameras seem to have System.Create, SS.Create, PT_CompleteFileWrite. This is based on searching through a few dozen firmware files I have on my HDD.

Which (kind of) functions are not supposed to be called from thumb?

reyalp

srsa_4c
All DryOS >= r23 cameras seem to have System.Create, SS.Create, PT_CompleteFileWrite. This is based on searching through a few dozen firmware files I have on my HDD.

Calling the eventprocs in the sequence I used and checking for -1 should be OK, the romlog code does the same thing.

Edit:
Oops, I didn't realize (or forgot, since I listed them before) until now that the older cams didn't have PT_CompleteFileWrite. That explains why this code didn't work right on a540 😉

So in this case, we don't need to fall back to the older registration functions, unless we decide to use other eventproc "hooks".

Which (kind of) functions are not supposed to be called from thumb?

The firmware code doesn't always return with a BX LR, so it might return to your thumb code without switching back to thumb mode. This is most common on vxworks cameras, but happens in some cases newer cameras do it too. So generally you shouldn't call a firmware function directly without a wrapper.

« Previous Page Next Page »