is it possible to download images while a script is running? - page 3 - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: 1 2 [3] 4 5 6 7   Go Down

is it possible to download images while a script is running?

  • 63 Replies
  • 10541 Views
*

Offline reyalp

  • ******
  • 14110
Re: is it possible to download images while a script is running?
« Reply #20 on: 01 / November / 2021, 00:09:35 »
Advertisements
I think I've identified the check that triggers the error. elph130 100a addresses

ff077188 - logs "PTPRspnd.StartUpPTPFrameworkClient"
-> first call after log ff171fcc
-> fourth call ff07a934
checks if result of third call (ff165080, passed by pointer in first arg) is < 3089, and fails if not.

On elph180, which doesn't have the limit mentioned in the manual, the equivalent check (in elph180 100c ff854430) is 10006, suggesting that these cameras may actually have a ~10k limit, which could still be relevant to CHDK users, though I haven't verified it.

I haven't yet tried to bypass the check, and the logic that gets the values is fairly non-obvious. It clearly involves the DCIM/MISC stuff. It's not clear if bypassing the limit check (with cache hacks, for exmple) would result in some kind of buffer overflow.
« Last Edit: 01 / November / 2021, 00:37:57 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #21 on: 18 / November / 2021, 13:18:02 »
although 10k is a better number, it would only give me 3 days between downloads.
so it appeares that a dedicated computer to save the images is unavoidable.
unless bypassing the check works naturally.

In case bypassing does not work, what would be a good setup to make this as reliable as possible?

at this moment the cam uses SUIx and takes one picture per 30 seconds, reboots every 21 days.
it now has a battery pack that keeps the cam running even when power fails.

I could use that same pack to power a laptop so they both keep running....although a lot shorter.
that might not be needed if it is possible that the cam resorts to saving on the sd card when the usb fails.
then i could change the sd card and reboot when there has been a connection problem.


 
« Last Edit: 18 / November / 2021, 13:25:15 by Mlapse »
Logged
frustration is a key ingredient in progress
*

Offline reyalp

  • ******
  • 14110
Re: is it possible to download images while a script is running?
« Reply #22 on: 18 / November / 2021, 15:45:16 »
Quote from: Mlapse on 18 / November / 2021, 13:18:02
although 10k is a better number, it would only give me 3 days between downloads.
so it appeares that a dedicated computer to save the images is unavoidable.
unless bypassing the check works naturally.
Note I haven't verified that it actually fails at 10k. The part of code where the check is looks like it should, but it's possible the Canon logic changed elsewhere so the error no longer occurs.

Quote
I could use that same pack to power a laptop so they both keep running....although a lot shorter.
A raspberry pi would be considerably lower power consumption that most laptops. See https://www.pidramble.com/wiki/benchmarks/power-consumption for example.

If the host only booted or woke up from sleep periodically to download, that would obviously be a lot more efficient.
Quote
that might not be needed if it is possible that the cam resorts to saving on the sd card when the usb fails.
IMO trying to fail over from remoteshoot to saving locally would be complicated and fragile.

Shooting normally and having the host do periodic downloads should be doable, and would naturally leave the images on the card until you intervened.
Logged
Don't forget what the H stands for.
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #23 on: 18 / November / 2021, 17:20:56 »
it does not really matter to me if it fails at 3k or 10k...and i don't see it as big enough a difference to go looking for a different camera.

recently i did away with a rpi4b because i was a bit disappointed about the quality.
as for power consumption, too bad there isn't a sd to sata converter on the market  :) i'm almost hoping for a revival of the cf card.
a celeron, possibly running ubuntu, might triple the energy consumption it is also capable of a lot better transfer speeds and has usually more stable hardware.

was afraid remote shoot would be more fragile than storing locally. that triggered my question in the first place
if you think doing periodic downloads would be the best chance of uninterupted interval, this would be my first choice, so i agree completely.
but since the s110 will make 2880 shots a day periodic still means it will have to happen every day.

Whipped up an old laptop to test and after following the instruction https://app.assembla.com/spaces/chdkptp/wiki/Install i got stuck close to the finish  :(
Just tried sudo bash chdkptp.sh on a fresh ubuntu install, but at this moment i get
error while loading shared libraries libusb-0.1.so.4...cannot open/no such file or directory
....it has been a while for me on a linux machine....hopefully in a day or so i've got it working.
advice is appreciated.

btw when the cam is plugged in it goes to play mode and screen of cam stayes active....so i'm getting close  :)
when it goes in play mode the script just counts down and then stops/locks up without taking a shot.

However waterwingz already build in 'pause on usb', so when i activate that the cam will resume shooting as soon as i unplug the usb cable.

I assume that i cannot prevent it from going to play.
should I get the script to go back to shoot after the usb cable is connected instead of pausing when it shifts to play or is there a smarter solution?
something like, after each shot
Code: [Select]
set_record(0)
sleep(10000)
set_record(1)
while not get_mode() do
  sleep(10)
end
giving me a moment when pictures can be downloaded?
« Last Edit: 21 / November / 2021, 13:40:19 by Mlapse »
Logged
frustration is a key ingredient in progress
*

Offline reyalp

  • ******
  • 14110
Re: is it possible to download images while a script is running?
« Reply #24 on: 21 / November / 2021, 14:04:05 »
Quote from: Mlapse on 18 / November / 2021, 17:20:56
Whipped up an old laptop to test and after following the instruction https://app.assembla.com/spaces/chdkptp/wiki/Install i got stuck close to the finish  :(
Just tried sudo bash chdkptp.sh on a fresh ubuntu install, but at this moment i get
error while loading shared libraries libusb-0.1.so.4...cannot open/no such file or directory
....it has been a while for me on a linux machine....hopefully in a day or so i've got it working.
advice is appreciated.
You need probably need to install the libusb-0.1 compatibility library, like

sudo apt-get install libusb-0.1

Quote
I assume that i cannot prevent it from going to play.
You would have to hide the presence of USB from the original firmware, which can be done using the USB remote settings: Enable the remote and set the control mode to none (you probably want to do this with set_config_value in script). CHDK script can still detect USB power and switch to play under its own control, and then make USB visible to the canon firmware by using set_config_value to disable the remote.

Quote
should I get the script to go back to shoot after the usb cable is connected instead of pausing when it shifts to play or is there a smarter solution?
Generally, the Canon firmware causing a switch to play mode tends to break subsequent switches under CHDK control, though it may be OK after USB is disconnected.

You could use the approach described above, or keep the USB present all the time, in which case no switch to play will be triggered. It's possible the latter would work with usb power (or usb_force_active https://chdk.fandom.com/wiki/Lua/PTP_Scripting#usb_force_active) but I'm not certain.
Logged
Don't forget what the H stands for.
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #25 on: 21 / November / 2021, 16:36:48 »
great, now the script stays active with the usb connected.

i've adapted the script as a test and this works, it switches to play mode after each shot...maybe 10 seconds is a bit long, but this is just to try it.

solved, interval is working, switch to ptp after each shot.
since switch_mode was not needed, I commented it out, so you still have an idea what will happen

Code: [Select]
                if (get_usb_power(1)==1) then
--                    switch_mode(PLAYBACK)
                    set_config_value(121,0)           -- USB PTP enable
                    sleep(10000)                      --  for 10 sec.
                    set_config_value(121,1)           -- USB PTP disable
                    while not get_mode() do
                        sleep(100)
                    end
--                    switch_mode(SHOOTING)
                    sleep(1000)
                end


This will give me a window of a bit less than 10 seconds after each shot....i hope the click i hear when switching between play and record is a very sturdy switch.
because most non-industrial relais i know won't go into the millions.


and now for the really tricky part  :)
what's next?  :)
to automatically download the log and some pictures in a timeframe of 6-8 seconds after cam is detected by chdkptp?

i tried to find the cam id: list did the trick :) right, onwards.

well, haven't scripted anything yet, but thought the gui could help in testing.
using the gui i could download some images when not running a script, take remote shots and store them locally.

however when i then run the script that sets ptp open for 10 seconds after each shot i get a connection error and no images will be downloaded (you see the timer next to the connect button disappear and reappear for 10sec)
if i time a connect and download when it turns to play i get this ERROR: no matching device no camera connected.
pausing the script does not work. at this moment it seems the only way is to stop and start the script to get it to download.
i can code the script to end when usb is detected, could i use chdk autostart for that in some way. f.i. start after 10 seconds in play or should that be triggered by chdkptp?
or do we have a better method?


If I run the script without changing to play for download i can have live view while the script records to the sd card, pushing download just states it does not work because a script is already running. This only works if you start record in chdkptp first and then press shoot to activate the script, so if a live feed is established before the script, it stays active.

a small step for mankind, but a big one for me :)

« Last Edit: 25 / November / 2021, 03:31:32 by Mlapse »
Logged
frustration is a key ingredient in progress
*

Offline reyalp

  • ******
  • 14110
Re: is it possible to download images while a script is running?
« Reply #26 on: 25 / November / 2021, 15:48:59 »
FIWW, if you edit a post, it's quite hard to follow what's new. So if you have a specific question, I'd suggest making a new post, even if it's a reply to your own previous post. Replying to your own post is absolutely fine when it's a new question or new information.

Quote
however when i then run the script that sets ptp open for 10 seconds after each shot i get a connection error and no images will be downloaded (you see the timer next to the connect button disappear and reappear for 10sec)
By "set ptp open" you mean disabling USB remote? Trying to synchronize this 10 second window with a process on the PC seems quite difficult to me.
Quote
(you see the timer next to the connect button disappear and reappear for 10sec)
I'm not clear what this refers to, there should not be anything like a timer next to the connect button in the chdkptp UI. The only thing next to connect button should be a dropdown, which contains a list of available devices.

Quote
if i time a connect and download when it turns to play i get this ERROR: no matching device no camera connected.
This error means just what it says, the PC for whatever reason doesn't see any compatible devices. It won't see it while the USB remote setting is active, and it takes some small amount of time to notice after a camera is connected.
Quote
pausing the script does not work.
I don't understand what "pausing the script" means?

Quote
i can code the script to end when usb is detected, could i use chdk autostart for that in some way. f.i. start after 10 seconds in play or should that be triggered by chdkptp?
You can run a camera-side script from chdkptp, as described in https://chdk.fandom.com/wiki/Lua/PTP_Scripting
However, you have to set all the variables corresponding to the menu settings yourself. In the latest SVN of chdkptp, there is a module which can parse the script header and saved settings files to generate the appropriate glue.

Quote
or do we have a better method?
Not necessarily better, but your camera side script can communicate with a chdkptp script using  read_usb_msg and write_usb_msg. In particular, this means the chdkptp side could indicate when it's done. The camera side could provide information about files to be downloaded.

If you know the exact filenames you need to download (from your log file, or reported from your script using usb msg), then you can download while the script is running, using the -nolua option in the CLI download command, or using con:download or con:imglist_download in chdkptp Lua.

Quote
If I run the script without changing to play for download i can have live view while the script records to the sd card, pushing download just states it does not work because a script is already running.
If you're talking about the GUI download button, that requires camera-side script to identify files to download.
Logged
Don't forget what the H stands for.
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #27 on: 25 / November / 2021, 16:54:54 »
i'm considering that possibly i'm busy with an x=y question by focussing on what i saw as a solution to me wanting to upgrade my setup...sorry.

Quote from: reyalp on 25 / November / 2021, 15:48:59
By "set ptp open" you mean disabling USB remote? Trying to synchronize this 10 second window with a process on the PC seems quite difficult to me.
Yes, i set the script to disable usb remote after each shot assuming that i could only download when the camera is in play mode.
If that is a wrong assumption, correct me, because i could do without all that switching between play and shoot.
it would then sleep for 10 seconds before enabling usb remote again and go back to taking a shot.
And yes, it was hard to time, since that 10 seconds only are 6 or 7 after the cam is connected. But as i understand now it would never work because the script is paused and the download button in the gui request a file lookup on the cam.
Quote
I'm not clear what this refers to, there should not be anything like a timer next to the connect button in the chdkptp UI. The only thing next to connect button should be a dropdown, which contains a list of available devices.
sorry, you are right, it is that dropdown box that appears when the usb remote is disabled, the cam goes to play and the pc connects...and disappears when the remote is enabled again

Quote
This error means just what it says, the PC for whatever reason doesn't see any compatible devices. It won't see it while the USB remote setting is active, and it takes some small amount of time to notice after a camera is connected.
I don't understand what "pausing the script" means?
it does not see any compatible device because the script is still running on the camera,
it has only disabled usb remote and is now sleep(10000) until it will enable usb remote and go back to shooting.

Quote
You can run a camera-side script from chdkptp, as described in https://chdk.fandom.com/wiki/Lua/PTP_Scripting
However, you have to set all the variables corresponding to the menu settings yourself. In the latest SVN of chdkptp, there is a module which can parse the script header and saved settings files to generate the appropriate glue.
Not necessarily better, but your camera side script can communicate with a chdkptp script using  read_usb_msg and write_usb_msg. In particular, this means the chdkptp side could indicate when it's done. The camera side could provide information about files to be downloaded.
If you know the exact filenames you need to download (from your log file, or reported from your script using usb msg), then you can download while the script is running, using the -nolua option in the CLI download command, or using con:download or con:imglist_download in chdkptp Lua.
this wil take some time for me to read up and make something that works.
-i know the file names for each folder: starting with: IMG_0001.JPG until: IMG_XXXX.JPG (this can be anywhere from 2 to 2000, if there is no reboot or other disruption:2 folders per day one of 2000 the other 880),
-i could blindly guess the complete folder names since they are dates in this setup, but as far as i know it's 100* 101* etc on all canons...so that seems obvious enough to me.
-but how to time the moment of download best? as you were wondering and i experienced it is something of a delicate short moment for a human....for a pc....maybe just trying every second until connect?
i sure will make the next post below this one :)
Quote
If you're talking about the GUI download button, that requires camera-side script to identify files to download.
sounds logical.
« Last Edit: 25 / November / 2021, 17:11:44 by Mlapse »
Logged
frustration is a key ingredient in progress
*

Offline Mlapse

  • *****
  • 584
  • S95 S110
Re: is it possible to download images while a script is running?
« Reply #28 on: 25 / November / 2021, 17:06:02 »
the x=y question would be that i am looking for a solution to use a higher resolution cam while reducing the times i have to touch the cam and still use my long exposure calculation.
i might introduce a second cam if this works out great that should take shots simultaniously...but that's a long way away at this moment.

I won't go for a higher resolution cam, because i can hardly manage the data that the 10MP S95 saves on the sd card, going to the 12MP S110 would mean going from ~200GB to ~375GB per month. and that would mean at least: 2x 512GB sd cards, a lot longer reboot times and not reaching my goal of less times touching my cam.....or I should go for 1TB sd cards....and then even a new intel nuc or laptop is cheaper and could give me the advantage of being able to upgrade to a modern, even more mp cam, not having to swap sd cards and earlyer access to the shots taken and more chance of discovering disruptions....the list probably goes on. 

i only use a 30 second interval, let P handle anything above 1 second exposure.
iso, zoom, WB etc are all set in the custom setting of the camera.
i actually have no use for my script in this setup except for the long exposure calculation and the 30 sec interval, maybe focus at infinity.
it would however undermine me working on it if i don't use it anymore on my almost 4 years running setup :)
but that's a small price to pay for moving forward.
« Last Edit: 25 / November / 2021, 18:13:07 by Mlapse »
Logged
frustration is a key ingredient in progress
*

Offline reyalp

  • ******
  • 14110
Re: is it possible to download images while a script is running?
« Reply #29 on: 25 / November / 2021, 18:44:39 »
Quote from: Mlapse on 25 / November / 2021, 16:54:54
Yes, i set the script to disable usb remote after each shot assuming that i could only download when the camera is in play mode.
If that is a wrong assumption, correct me, because i could do without all that switching between play and shoot.
With chdkptp, you do not have to be in play mode to download files. However, depending on how you do things, you might still want or need to switch to play or use the USB remote setting like you described above.

1) If the Canon firmware detects a new USB connection (+5v goes from 0 to +5) it switches to play mode. Having this happen at a random point while your script was shooting would likely be bad. Additionally, after the camera has done this automatic switch, switching under CHDK control is likely to be broken until you reboot or maybe disconnect USB.

So if your USB connection is only powered intermittently when you want to download files, then you probably want to use the USB remote setting to block the Canon firmware from detecting it, and use script to switch to playback before allowing the Canon firmware to notice the USB connection. If the USB connection is powered all the time then hiding the USB connection should not be necessary. You can either download in rec, or use CHDK script functions to switch between play and rec.

2) On some cameras, if you delete image files while in rec, the camera crashes when you attempt to switch to playback or shut down. Since you need to limit the number of files to avoid the "COMMUNICATION ERROR" issue we discussed earlier, you may need to switch to playback to delete. However, this does not require that you use the USB remote setting, you can just switch between play and rec using CHDK script functions if USB is always present.

Quote
And yes, it was hard to time, since that 10 seconds only are 6 or 7 after the cam is connected. But as i understand now it would never work because the script is paused and the download button in the gui request a file lookup on the cam.
For actual use, you will surely want to use some chdkptp side script, not the GUI button. Conceptually, if you wanted the PC side to wait for the camera to enable the USB connection and then download, you could do something like this in chdkptp Lua
Code: [Select]
-- wait for a device to appear
while #chdk.list_usb_devices() == 0 do
 sys.sleep(100)
end
-- camera has appeared, connect to it
cli:execute('connect')
-- get the list of files to download
-- download the files
-- tell the camera script the download is done and it can reset the remote bit and resume shooting
Note I'd generally use the chdkptp CLI rather than the GUI for this kind of automation. chdkptp automatically starts in CLI mode if given command line arguments other than -g.

However, if you want to download frequently (every shot or few shots), then IMO it would make more sense to just have USB connected all the time. In that case, your camera side script could send the filename(s) in message with write_usb_msg and your chdkptp side script could use con:wait_msg or con:wait_status

Quote
it does not see any compatible device because the script is still running on the camera,
it has only disabled usb remote and is now sleep(10000) until it will enable usb remote and go back to shooting.
A script running does not prevent the OS or chdkptp from seeing the camera. If you connect in the GUI while a script is running, you'll see an error like 'update_mode_list failed a script is already running' but this does not prevent connecting or using other functions which do not use camera script.

From CHDKs POV, script is "running" regardless of whether it's in sleep or actually doing something.

Quote
-i know the file names for each folder: starting with: IMG_0001.JPG until: IMG_XXXX.JPG (this can be anywhere from 2 to 2000, if there is no reboot or other disruption:2 folders per day one of 2000 the other 880),
It looks like the SUIx log you posted contains both the folder and image name. So you should be able to do something like

download A/SUIx.csv
parse SUIx into a list of full paths
download the files

Of course, you could arrange your script to write just the file names to a different file, to make the parsing easier. It just has to be a predictable name the chdkptp side can download.

As long as the downloads use download -nolua or the equivalent chdkptp Lua, your camera side script doesn't need to exit.

I can provide more specific code suggestions, but it will depend on what approach you want to try.
Logged
Don't forget what the H stands for.
Pages: 1 2 [3] 4 5 6 7   Go Up
« previous next »
 

Related Topics
SimplePortal 2.3.6 © 2008-2014, SimplePortal