is it possible to download images while a script is running? - General Discussion and Assistance - CHDK Forum supplierdeeply

is it possible to download images while a script is running?

  • 56 Replies
  • 1543 Views
*

Offline Mlapse

  • ****
  • 368
  • S95 S110
is it possible to download images while a script is running?
« on: 24 / October / 2021, 15:40:05 »
Advertisements
I have this setup that takes one shot every 30 seconds.
downloading and erasing the same sd card would loose a fair amount of time.
hence, i have 2 identical sd cards that I exchange.
but I would like to minimize contact with the camera.

remote shoot with chdkptp is naturally an option.

getting even larger sd cards would also be an option, but i don't feel much for this one.

and so I was wondering
would it be possible to make a setup with chdkptp or some other way where i connect a laptop to the usb and that files would be automatically downloaded and erased between shots..
that would give me some 25 seconds between shots that the script is waiting.
i assume it involves switching to playback and back...so that might cost some time.

don't know how that should work and/or if that is possible, maybe per 5 or 10 at a time and then wait until after the next shot has been saved by the script?
« Last Edit: 24 / October / 2021, 16:14:29 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 13454
Re: is it possible to download images while a script is running?
« Reply #1 on: 24 / October / 2021, 18:30:53 »
remote shoot with chdkptp is naturally an option.
If you decide to go that route, the "glue script" approach used by dolomiti_timelapse lets you use existing chdk scripts mostly unmodified https://chdk.setepontos.com/index.php?topic=14302.0
Your mileage may vary depending on the specific script

manoweb's webcams are an example of an an alternative approach, using a custom script driven by the host side (https://chdk.setepontos.com/index.php?topic=14302.msg145505#msg145505https://chdk.setepontos.com/index.php?topic=14302.msg147074#msg147074)

The choice between these is mostly personal preference, IMO.
Quote
would it be possible to make a setup with chdkptp or some other way where i connect a laptop to the usb and that files would be automatically downloaded and erased between shots..
that would give me some 25 seconds between shots that the script is waiting.
i assume it involves switching to playback and back...so that might cost some time.
It should be possible, but coordinating between the script and download process will require some thought and experimentation.

You do not have to switch to playback to download files with chdkptp. Downloading while the camera shoots might even be OK, although I wouldn't recommend it.

You can technically download while a (non-ptp aware) script is running using dl -nolua or con:download(), but you must know the exact filename and path beforehand. The multiple download commands mdl and imdl need to run a camera-side script to list the files.

An alternative is to make your own camera side script provide the list of files, like multicam.lua download_images.

Quote
don't know how that should work and/or if that is possible, maybe per 5 or 10 at a time and then wait until after the next shot has been saved by the script?
A few considerations:
The "connection error" "communication error" (edit: corrected message) limit. As described in https://chdk.setepontos.com/index.php?topic=14302.msg147190#msg147190 the camera may not allow a USB connection if it thinks too many files are present. One image every 30s for one day = 2880 which may or may not be OK. More than one day would likely fail. If the camera isn't switched to play back, this error may be avoided, but some cameras crash when you try to switch to play or shutdown after deleting files in rec. Dealing with this will require experimentation with your particular setup.

If the camera isn't connected to USB all the time, you need to be careful with play/rec switching. By default, when you connect USB, the camera will attempt to switch to play mode. Additionally, if you switch with normal switching functions after switching while USB is present, switching may fail.

You could hide the USB status until the camera script is ready using the USB remote setting (with set_config_value, clearing it when the script is ready after detecting usb power), or force the camera to think it's always present using usb_force_present(1) (see https://chdk.fandom.com/wiki/Lua/PTP_Scripting)

To download images, your script needs provide the PC side with a list, or you need to exit and restart the script between downloads so you can use something like imdl/mdl.

Since you want to download a few images at a time, and need support in the script to handle USB showing up anyway, I'd lean to making the script provide the file list.

Providing a file list also needs some thought. It will likely be too large to for the camera side script to keep in RAM. The chdkptp multi-file download scripts handle by sending batches of filenames in multiple messages, with the PC side collecting the full list. Re-using this code should be straightforward (take ff_imglist and its dependencies from chdkptp rlibs, and borrow the multicam download_images code), but if you have a lot of files, just building the list could take longer than your shot interval.

Another alternative would be to have your script log each shot to a file, and have the PC side download that.

Putting this together, I'd probably go for something like this:
script uses usb_force_present(1) usb_force_active(1) (edit: oops, originally posted wrong function name) at startup to prevent USB connections from messing up play/rec state, and uses switch_mode_usb to ensure mode switching is compatible

Script main loop checks for messages with read_usb_msg. Using the timeout option, you can use this as a replacement for the regular sleep in your interval script, since it just returns nil if no messages are received in the specified time.

When you connect the PC, the chdkptp script sends a message, and the camera side script responds with file info when it's ready for downloads to begin. If necessary, you can use additional message to tell the script when a batch of files is done etc.

Another alternative approach would be to use a wifi SD card.

OK, that was a long rambling post, I hope some of it is useful  :-[
« Last Edit: Yesterday at 13:34:52 by reyalp »
Don't forget what the H stands for.

*

Offline Mlapse

  • ****
  • 368
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #2 on: 25 / October / 2021, 03:09:32 »
it most definately is usefull.
now see if i can be equally usefull by putting all you wrote to good use.

although my feeling is that remote shoot is very professional and the fact that you can see "live" what the cam is doing is very appealing.
i think i like the approach of downloading during a running script better in my case because the camera setup can remain lean.

a file list seems fairly easy, as you wrote: i could use the log file. after adding get_image_dir() with each line it should have all i need.
the only trouble is naturally that the log file keeps growing after you downloaded it, but it's not that important at this moment.

looked into the wifi card approach but then i think the remote shoot would be more reliable...and with a wifi card there still has to be a dedicated pc since card sizes are small and range is limited.

the connection error might be the real trouble, i was thinking of using cards that allowed me to take 2 weeks worth of pictures, giving me some flexibility in when to download......that's around 40.000 jpgs, but planning to remove them once a week when the card is only half full. I'm guessing i'll have to test that first. because if that does not work the whole idea stops.......1000 pictures is not much in my setup....each day just after 8:15 AM I'm there i'd say
hopefully that's as conservative as the number of shots a camera can make according to canon.
but I have had a similar experience with my a480..that i could not download from the cam after long shoots and had to use a card reader to get the pictures off....and those were around 20.000

either way i'm going to work on a chdkptp setup this winter.  8) ??? :P

« Last Edit: 25 / October / 2021, 04:33:54 by Mlapse »
frustration is a key ingredient in progress

*

Offline Caefix

  • *****
  • 652
  • Sorry, busy deleting test shots...
Re: is it possible to download images while a script is running?
« Reply #3 on: 25 / October / 2021, 12:18:44 »
the only trouble is naturally that the log file keeps growing after you downloaded it,
Code: [Select]
function oneline(file,line)
file=file or "A/oneline.log"
line=line or string.format("%s/IMG_%04d.JPG",get_image_dir(),get_exp_count())
   w=io.open(file,"w")
   w:write(line)
   w:close()
end

oneline()
Or You use a Oneline.log  :), could be enhanced to a Batchfile.bat, called from Hostlua with os.execute()

connect a laptop to the usb and that files would be automatically downloaded and erased between shots..
"Erased" is not the same for CHDK & Canon until reboot().  :-[
Maybe script could reboot to a sleep(100000) at the end of daywork before it awakes(tomorrow) 50secs earlier with a sleep(50000) first...
« Last Edit: 25 / October / 2021, 13:37:14 by Caefix »
All lifetime is a loan from eternity.


*

Offline Mlapse

  • ****
  • 368
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #4 on: 25 / October / 2021, 14:30:12 »
i dunno, 20.000+ oneline logs are a bit much for me, but i could always give it a try if it becomes an issue.

i understand that erase will only take effect after a reboot, but that has to happen anyway, so that's all sweet.
probably best to let the reboot be triggered after the last files & folders are removed, so it starts clean with a 100_XXXX folder.


frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 13454
Re: is it possible to download images while a script is running?
« Reply #5 on: 25 / October / 2021, 19:03:55 »
the connection error might be the real trouble, i was thinking of using cards that allowed me to take 2 weeks worth of pictures, giving me some flexibility in when to download......that's around 40.000 jpgs, but planning to remove them once a week when the card is only half full. I'm guessing i'll have to test that first. because if that does not work the whole idea stops.......1000 pictures is not much in my setup....each day just after 8:15 AM I'm there i'd say
hopefully that's as conservative as the number of shots a camera can make according to canon.
Yes, I think this is a real concern, you'll want to test before investing too much time in the other parts.  My impression is that 2000-ish is generally OK, but I haven't tested sufficiently to know if the limit is consistent for a given camera.

This would be a worthwhile area for reverse engineering. I assume the limit has something to do with handling standard PTP "objects". Since CHDK PTP doesn't depend on them, it's possible we could bypass it somehow. But it's annoying to test...

Note the actual error is "communication error" rather than "connection" as I stated earlier.
Quote
but I have had a similar experience with my a480..that i could not download from the cam after long shoots and had to use a card reader to get the pictures off....and those were around 20.000
Yes, I would definitely expect 20,000 to be a problem, but as I mentioned above, you may not encounter it if you don't switch to playback.
Don't forget what the H stands for.

*

Offline Mlapse

  • ****
  • 368
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #6 on: 26 / October / 2021, 02:35:39 »
while SUIx is filling it's card at 1 frame per 5 seconds, i can do my first test this evening with < 10.000.
If that works out i'll ramp up the # of frames.

i'm just wondering if i should start fresh after that or if i can power down and up with the first 10.000 still on the card...but i'll test that too if it doesn't fail.
since canon states a max of 9999 images for any large card i thought it might be a good starting test.

note to self: kill the shuttersound for the next run  :)

« Last Edit: 26 / October / 2021, 03:23:03 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 13454
Re: is it possible to download images while a script is running?
« Reply #7 on: 26 / October / 2021, 13:58:01 »
since canon states a max of 9999 images for any large card i thought it might be a good starting test.
FWIW, the "communication error" limit is described like this in the canon manuals (with similar wording from at least 2006 to ~2013 (DryOS R52):

"The computer was unable to download the image due to the large amount of images (approx. 1000) stored on the memory card."

This wording does not appear in the manuals of later cameras, starting around DryOS 54 from a quick look, so I suspect they finally addressed it.
Don't forget what the H stands for.


*

Offline Mlapse

  • ****
  • 368
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #8 on: 26 / October / 2021, 15:11:15 »
i had some good hope on my s110, R51 since it supports fat32, and those larger cards would hold >1000images.
to bad that took a few revisions to sink in  :)
your mail was a -spoiler alert- Yes, confirmed: "connection error" at 9261 images on disk

so, the idea is off, it might be possible but not with the s110. too bad, alas a set of g7x is out of my reach.
i guess i have to go for a pc that saves the images..so the next question i'll have to post is in the chdkptp tread?
« Last Edit: 26 / October / 2021, 15:19:13 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 13454
Re: is it possible to download images while a script is running?
« Reply #9 on: 26 / October / 2021, 15:41:48 »
your mail was a -spoiler alert- Yes, confirmed: "connection error" at 9261 images on disk
Does it happen when you connect the cable, or when you connect chdkptp?

I'd like to know if the error still happens when you are in rec mode, but thinking about it, unless you shoot with PTP already connected, getting to rec and having it not switch to play when you connect may be difficult.

Quote
i guess i have to go for a pc that saves the images..so the next question i'll have to post is in the chdkptp tread?
I'd say keep going in this, IMO keeping discussion around one project together seems to work better than having the parts in a bunch of different ones.
Don't forget what the H stands for.

 

Related Topics