is it possible to download images while a script is running? - page 4 - General Discussion and Assistance - CHDK Forum

is it possible to download images while a script is running?

  • 63 Replies
  • 11316 Views
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #30 on: 26 / November / 2021, 12:17:24 »
Advertisements
Code: [Select]
while #chdk.list_usb_devices() == 0 do
 sys.sleep(100)
end
-- camera has appeared, connect to it
cli:execute('connect')
-- tell the camera script the download is done and it can reset the remote bit and resume shooting
Every tenth of a second is better than per second so that's a keeper :)
 
Download done is a bit more tricky i guess and led me to create the -not working- 10 sec window:

if we download a file after shot taken and reset the remote bit after download. it will probably not interfere with even the shortest interval. However, pc has to be active 24/7 with no flaws.
 
if we download in bulk and wait to finish, this will stop the interval timer for quite some time. ~12Gb a day for the S110 at USB2 speed....max 30Mb/s.....7 minutes plus erase and reboot:  minimum 16 frames lost.
Scale that to what would happen if we could remove the limit and i could do a weekly or longer download before erasing and rebooting...i would loose close to an hour per download.

As far as i can see bulk would thus entail downloading in groups of a fixed number or as much as possible images between each shot.

f. i. If we could do groups of only 3 it would download 24 hours of pictures in 8 hours. the pc could possibly be scripted to power down after that.

Quote
A script running does not prevent the OS or chdkptp from seeing the camera. If you connect in the GUI while a script is running, you'll see an error like 'update_mode_list failed a script is already running' but this does not prevent connecting or using other functions which do not use camera script.
From CHDKs POV, script is "running" regardless of whether it's in sleep or actually doing something.
Thanks, i think i understand the working of chdkptp and the buttons in the gui much better by now.
i assume we can not download while a shot is being taken and written to the sd card.
Haven't been able to replicate the error you wrote while the cam had switched to play and script was sleeping.
That may have been poor timing on my side though.
I've only seen 'a script is already running' when i tried the download button while cam was running script, with use of: set_config_value(121,1)
Quote
It looks like the SUIx log you posted contains both the folder and image name. So you should be able to do something like

download A/SUIx.csv
parse SUIx into a list of full paths
download the files

Of course, you could arrange your script to write just the file names to a different file, to make the parsing easier. It just has to be a predictable name the chdkptp side can download.

As long as the downloads use download -nolua or the equivalent chdkptp Lua, your camera side script doesn't need to exit.

I can provide more specific code suggestions, but it will depend on what approach you want to try
i'm happy with the folder/img structure canon uses copied on the pc if they are dropped in a folder that spans all folders/images between reboot with the csv of that period.
When i started i used weeknumbers since i had 64Gb cards, now i use month names to keep the folders of different downloads seperated.
However, with the file limit it will be a daily download, that's why i wrote a download-dated folder.

 Working with a 24/7 powered pc that would download every interval would be my second choice...and probably how i would set it up for testing.
It would be nice because you have the shots direcly available for processing, not after a day or more.
But if  24/7 is a necessity to make it work it would probably lead to more downtime wouldn't you think?

I would favor some form of bulk copying giving the ability to power down and up the pc (or unplug usb), to me that sounds like the most stable solution.
This would lead to a bit more steps than you suggested, maybe something in the line of:
Code: [Select]
#!/bin/bash
# Purpose: Copy files to pc without interrupting interval on cam
# ------------------------------------------
logname=SUIx.csv
cam_id_p=0x325b

-- wait until connection is available
while #chdk.list_usb_devices() == 0 do
 sys.sleep(100)
end
-- camera has appeared, connect to it
cli:execute('connect')
-- get the list of files to download or use the last folder name/image number downloaded for your next batch.
-- download the next batch of images or download images for X seconds to the same name folder/file on the pc.
-- tell the camera script the download is done and it can reset the remote bit and resume to next shot, after it sets remote bit again.
-- repeat download batch sequence after reconnect until no new images available/found
-- reboot time <= current time
-- download final csv for this series
-- erase folder DCIM and SUIx.csv from cam: rm A:\DCIM A:\SUIx.csv
-- set_clock to current time
-- reboot cam
-- set next reboot moment (file limit and interval dependant, for now that's one day)
-- create new download-dated folder: mkdir -p "%Y/%m%d"
-- repeat the script.

This way the erase will only be triggered once per run, and the on-camera reboot sequence i use already moves the camera to play before reboot to prevent crashing that feature on some models.
I think the same reboot sequence and minimum timing should be used in this setup, since it took a lot of testing and refining to get it to work stable on most cams.
If we were the erase to be triggered there, just before or after set_clock it should prevent cams that can't handle erase in record to crash.
Code: [Select]
-- routine to reboot the camera and restart the cam script
function camera_reboot()
    activate_display(20)
    switch_mode(PLAYBACK)
    local ts=11                         -- allow 11 seconds in case camera not setup to retract immediately
    if (debug_mode ) then ts=10 end
    printf(" Scheduled reboot ")
    printf(" lens retraction wait")
    repeat
        show_msg_box(string.format("   rebooting in %d", ts))
        ts=ts-1
        sleep(1000)
    until ( ts == 0)
        f:flush()
        f:close()
    end

    -- time to restart ?
    if (elapsed_days ~= maximum_days) then
        set_autostart(2)               -- autostart once
        printf(" rebooting now\n\n")
        sleep(1000)
        t=os.date('*t', os.time())  --change to pc os time
        set_clock(t.year,t.month,t.day,t.hour, t.min, t.sec)  -- set_clock(year, month, day, hour, minute, second)
        printf(" time set %d sec\n\n", reboot_tcorr)
--        rm A:\DCIM A:\SUIx.csv – remove old files and folders
        sleep(1000)
        reboot()
    end

end


The reason to trigger a reboot from the script after the erase is to make sure it reboots when no images/folders/log are on the card creating a fresh SUIx.csv and 100* folder with img_0001.jpg to start with for the next sequence.
It would help to keep things organised.

Since we started somewhere in this thread with that need, i adapted the SUIx log to contain folder names. But later thought that we might not need it since the folder and files are numbered predictable.

If you have time for some more specific code suggestions i would really appreciate that :)
Making less progress by turning down the offer for help and guide from a specialist? I'm not that hard on myself nor experienced enough to decline ;)
« Last Edit: 27 / November / 2021, 04:17:49 by Mlapse »
frustration is a key ingredient in progress

*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #31 on: 27 / November / 2021, 04:33:38 »
Code: [Select]
sudo sh /chdkptp-r964/chdkptp.sh -c"-p=0x325b" -e"set_lcd_display(1)" -e"switch_mode(PLAYBACK)" -e"exec sys.sleep(11000)" -e"set_autostart(2)" -e"rm A:/DCIM A:/SUIx.csv" -e"exec sys.sleep(1000)" -e"set_clock(%Y,%m,%d,%H,%M,%S)" -e"exec sys.sleep(1000)" -e"reboot()"

an attempt to translate the reboot sequence to chdkptp.

-cam id
-the first 3 params are to prevent crashes on various models before or during reboot and should also cover cams that can't handle delete in record mode.
-autostart flag set once
-deletes the files and folders, so it can start fresh after the reboot.
-sleep, just a guess
-set_clock to solve unstable results from difference between RTC and OS clock after reboot, using PC os time to sync clocks
-sleep to give the cam a moment
-reboot
« Last Edit: 27 / November / 2021, 13:33:39 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 14125
Re: is it possible to download images while a script is running?
« Reply #32 on: 27 / November / 2021, 16:18:04 »
Code: [Select]
sudo sh /chdkptp-r964/chdkptp.sh -c"-p=0x325b" -e"set_lcd_display(1)" -e"switch_mode(PLAYBACK)" -e"exec sys.sleep(11000)" -e"set_autostart(2)" -e"rm A:/DCIM A:/SUIx.csv" -e"exec sys.sleep(1000)" -e"set_clock(%Y,%m,%d,%H,%M,%S)" -e"exec sys.sleep(1000)" -e"reboot()"
Note -e runs a chdkptp CLI command (commands listed when you enter "help" in the CLI), not CHDK script functions (listed in https://chdk.fandom.com/wiki/CHDK_Scripting_Cross_Reference_Page). If you want to execute CHDK Lua script functions like set_lcd_display, you need to use one of the chdkptp commands that executes Lua on the camera, like
Code: [Select]
-e"luar set_lcd_display(0)"
(luar as the same as '=' in the CLI, but when using -e, you should use the long version because = also has meaning to the option parser  :-[)

Code: [Select]
switch_mode(PLAYBACK)
would be switch_mode_usb(0)

Code: [Select]
-e"set_clock(%Y,%m,%d,%H,%M,%S)"
If you want to synchronize with the pc clock, you can use
Code: [Select]
-e"clock -sync"
In any case, the % stuff is not correct syntax. (edit: I guess this was just meant as pseudocode, in which case, never mind :))

Code: [Select]
-e"reboot()"
Again, you should either use chdkptp reboot command -e"reboot" or use 'lua' to send the camera side function (not luar, because luar waits for a return value and that won't happen if the camrea reboots). I suggest the using the CLI command, since reboot requires some additional logic to ensure chdkptp has closed its side of the ptp connection before the reboot actually happens.

Once you start passing more than a few commands with -e, it's generally easier to put them in a file and run it like
Code: [Select]
chdkptp.sh -e"source somefile.chdkptp"
'source' executes a series of chdkptp CLI commands, as if you had typed them at the prompted or executed them with -e.

Or if you prefer you top-level code in chdkptp-side Lua, you can run it like
Code: [Select]
chdkptp.sh -e"exec dofile('somefile.lua')"
« Last Edit: 27 / November / 2021, 19:39:43 by reyalp »
Don't forget what the H stands for.

*

Offline reyalp

  • ******
  • 14125
Re: is it possible to download images while a script is running?
« Reply #33 on: 27 / November / 2021, 18:32:57 »
Download done is a bit more tricky i guess and led me to create the -not working- 10 sec window:
In your camera side script, you can wait for a message, like
Code: [Select]
read_usb_msg(10000)

and in the chdkptp side code, you'd do something like
Code: [Select]
putm done
after the download.

read_usb_msg waits for up to the timeout value (10 seconds above, but if 10 seconds the nominal time, you might want to wait longer and then do something different if it times out) for a message to be available, and returns nil at the if there wasn't one. The content of the message is irrelevant in the above case, but if you use different messages you can check the value like
Code: [Select]
local m=read_usb_msg(10000)
if m == nil then
... handle timeout
elseif msg == 'done' then
... do something else
elseif msg == ...

Note you can also communicate the other way, using write_usb_msg. For example, if you were downloading after each shot, your camera script could do
Code: [Select]
write_usb_msg(string.format('%s/IMG_%04d.JPG',get_image_dir(),get_exp_count()))
and then on the PC side
Code: [Select]
exec local m=con:wait_msg{mtype='user'} some_download_function(m.value)
Of course in real code you would want to think about error conditions and timeouts and so on.

I'm not suggesting you necessarily want to use the message system this way, just noting tools that are available.

Quote
f. i. If we could do groups of only 3 it would download 24 hours of pictures in 8 hours. the pc could possibly be scripted to power down after that.
Having the PC still running 1/3 of the time and needing to constantly interact with the script seems like a lot of complexity for maybe not much gain to me.

Quote
i assume we can not download while a shot is being taken and written to the sd card.
In my experience you actually can download files while a shot is in progress, but I haven't tested extensively and it wouldn't be at all surprising if it were unreliable. So I wouldn't recommend doing it this way, at least without some stress testing.

Quote
Haven't been able to replicate the error you wrote while the cam had switched to play and script was sleeping.
I don't understand what this refers to.

Quote
I've only seen 'a script is already running' when i tried the download button while cam was running script, with use of: set_config_value(121,1)
If you try to use any chdkptp functionality that requires running a script on the camera (which is most chdkptp functionality), you will get that error when a script is running. It doesn't really have anything to do with the remote setting.

Quote
i'm happy with the folder/img structure canon uses copied on the pc if they are dropped in a folder that spans all folders/images between reboot with the csv of that period.
My suggestion was only related to knowing the exact filenames on the camera, so you can download without stopping your script.

Quote
Working with a 24/7 powered pc that would download every interval would be my second choice...and probably how i would set it up for testing.
This certainly seems more straightforward to implement.

Quote
But if  24/7 is a necessity to make it work it would probably lead to more downtime wouldn't you think?
Possible, but it depends where the unreliability is. With stable power or sufficient backup, a Linux PC may not be the weakest link. It also depends on the failure modes, if the USB connection goes away, does the camera just keep shooting and saving to the card until you fix it, or do you lose a bunch of shots.

This post is already too long, I'll try to respond to the rest later ;)
Don't forget what the H stands for.

*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #34 on: 28 / November / 2021, 03:01:24 »
Possible, but it depends where the unreliability is. With stable power or sufficient backup, a Linux PC may not be the weakest link. It also depends on the failure modes, if the USB connection goes away, does the camera just keep shooting and saving to the card until you fix it, or do you lose a bunch of shots.
at this moment i can connect and disconnect without loosing shots, but only with the gui and live view while set_config_value(121,1) is enabled.
i can't even manage a connect in the terminal version.
and yes, you are right the posts are getting too long, i'll do my best to keep it shorter.

if I understand your suggestion of a seperate file for calls, it would lead to something like this:
Code: [Select]
-- reboot.chdkptp
luar set_lcd_display(1)
luar set_record(0)
exec sys.sleep(11000)
luar set_autostart(2)
rm A:/DCIM A:/SUIx.csv
exec sys.sleep(1000)
clock -sync
exec sys.sleep(1000)
reboot
« Last Edit: 28 / November / 2021, 03:21:32 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 14125
Re: is it possible to download images while a script is running?
« Reply #35 on: 28 / November / 2021, 03:25:09 »
at this moment i can connect and disconnect without loosing shots, but only with the gui and live view.
i can't even manage a connect in the terminal version.
What command are you using with the terminal version, and what happens? It really shouldn't make a difference for connecting.

Quote
and yes, you are right the posts are getting too long, i'll do my best to keep it shorter.
Don't worry about it too much, I also have trouble writing short posts. It's easy to want to cover all the bases :)

Quote
if I understand your suggestion of a seperate file and calls, it would lead to something like this:
Yes. It just makes it easier to read compared to a long line of -e commands, and makes it so you don't need to worry about quoting/escaping things for the shell.
Don't forget what the H stands for.

*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #36 on: 28 / November / 2021, 03:42:58 »
I use: connect
and that will work if the cam is not running a script yet.

the same with the gui, i can connect before i start SUIx and get live view, as soon as SUIx is running i cannot change anything anymore, but live view stayes active. if i disconnect the cam goes on.

script does set_config_value(121,1) to prevent it going to play on connect, so that's a fair reason why i have no control.
i forgot setting read_usb_msg(10000) after each shot.
i also have trouble sifting through those long posts and quotes :)
so i will test again......ERROR: no matching device found...no salvation there.
the cam now stops completely and after a minute it goes to play. disabling the script is not possible, but won't take shots since in play.
unplugging the usb will not give me back control.
the play(selected script disappears and appears) and menu button(CF menu) are working, nothing else.
i had to pull the battery to get the cam functional again.

« Last Edit: 28 / November / 2021, 04:45:13 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 14125
Re: is it possible to download images while a script is running?
« Reply #37 on: 28 / November / 2021, 15:02:42 »
This is odd. It sounds like maybe the camera USB stack is getting in a confused state, but there's so much going on it's a bit hard to follow.

I'd suggest reducing it to the simplest pieces you can test alone, and only trying to put it together once they are working.

I tested the following, on both windows and raspberry pi, using elph180
* power on camera, remote not enabled, switch type = none, control mode = none
* connect USB cable
* chdkptp list command shows the camera, connect  / disconnect works
* with chdkptp disconnected, enable remote in CHDK menu
* chdkptp list does not show the camera, nothing to connect to
* disable remote in CHDK menu
* chdkptp list command shows the camera, connect / disconnect work

Switching to rec and back to play while the remote is enabled seems fine, chdkptp is able to connect once the remote is disabled.

Note it's quite possible that different camera models or PC hardware/software respond differently to manipulating the the remote setting.

Quote
the same with the gui, i can connect before i start SUIx and get live view, as soon as SUIx is running i cannot change anything anymore, but live view stayes active. if i disconnect the cam goes on.
There are several separate issues which shouldn't be confused:
1) Whether chdkptp (and your OS) see the USB device and can connect to it. This depends on the remote setting, and does not depend on whether a script is running. The 'list' command is an easy way to see which devices your OS is aware of.

2) Whether you can use chdkptp commands which use script. Obviously, you don't get to this if #1 fails, but otherwise, it's completely unrelated, and only depends on whether a script is running. Live view does not use script. The chdkptp keyboard buttons do, so they will not work while another script is running.

3) Whether the Canon UI responds to some key presses when USB is connected. Normally, when you connect USB, the Canon UI ignores most key presses, until you switch to rec or send one of the unlock events like post_levent_to_ui(4484).

None of the above depend on whether you are using the GUI or CLI.

Quote
the cam now stops completely and after a minute it goes to play. disabling the script is not possible, but won't take shots since in play.
The camera going to play after a minute sounds like Canon power saving settings, unless it was triggered by the script.
Don't forget what the H stands for.

*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #38 on: 28 / November / 2021, 16:17:24 »
I tested the following, on both windows and raspberry pi, using elph180
* power on camera, remote not enabled, switch type = none, control mode = none
* connect USB cable
cam turns to play
Quote
* chdkptp list command shows the camera, connect  / disconnect works
* with chdkptp disconnected, enable remote in CHDK menu
cam stays in shoot or can be set back in shoot with shoot button
Quote
* chdkptp list does not show the camera, nothing to connect to
* disable remote in CHDK menu
cam turns to play
Quote
* chdkptp list command shows the camera, connect / disconnect work
same here, tested with S95 & S110 on ubuntu 20.04

and yes, that was the canon power savings.
« Last Edit: 28 / November / 2021, 16:20:05 by Mlapse »
frustration is a key ingredient in progress

*

Offline reyalp

  • ******
  • 14125
Re: is it possible to download images while a script is running?
« Reply #39 on: 28 / November / 2021, 16:57:27 »
cam turns to play
OK, this is a difference from my test: I started the camera in play. I would strongly recommend only connecting the USB cable (or turning off the remote) when the camera is already in play or powered off (edit for clarity: when using CHDK script / ptp / mode switching).
« Last Edit: 28 / November / 2021, 17:17:31 by reyalp »
Don't forget what the H stands for.

 

Related Topics


SimplePortal © 2008-2014, SimplePortal