Lapser Script Development

lapser

1093

Re: Lapser Script Development

« Reply #10 on: 07 / July / 2017, 21:43:08 »

Advertisements

Quote from: c_joerg on 07 / July / 2017, 15:28:00

If have just seen your last video. It’s great to see the comparing videos.
How do you made the split window (or zooming)?

I process the raw photos with Lightroom, then use VirtualDub to combine the jpg images into an AVI file. I then use TMPGEnc Video Mastering Works 6 to make the final video (including zoom, pan, split screen etc.

Quote

ETTR (Exposure To The Right) means the photo is exposed as brightly as possible without overexposing any part of the image.
Quote
Why do you think you need to have no overexposing? I think the core of the sun can be overexposed.

The Sun is overexposed because of the maximum shutter speed, which I set to 1/500. I also set the histogram meter so it can only decrease the exposure 2.5 ev (set by input parameter) below the averaging meter. With the Moon, this means it is overexposed, but other stars remain visible.

Quote

Can it be that the shutter starts to flicker? I notice some in the previous video.
How much shoots have your G1x right now?

It may flicker a bit at 1/500 exposure time. I used to use 1/600 but changed to 1/500 because of flickering. The accuracy goes down as the shutter wears out.
Parameter 1, the shutter count, has a value of:
1,316,805
The camera was refurbished when the shutter failed at around 500,000. Since then, I've kept the shutter at 1/600 or 1/500, instead of 1/4000.
===
Even though there are no overexposed pixels with ETTR, there are a lot of very bright pixels where the colors are lost. They come back if you lower the brightness, or even better, lower the highlights.

Lowered brightness with Video Mastering Works 6:

Lowered highlights with Photoshop Lightroom (more work!)

Histogram 2.5 ev limit with Moon in frame:

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

c_joerg

1248

Re: Lapser Script Development

« Reply #11 on: 08 / July / 2017, 03:11:39 »

Quote from: lapser on 07 / July / 2017, 21:43:08

use VirtualDub to combine the jpg images into an AVI file

Exactly as I do it. Have you ever tried a filter in VirtualDub?
I did some test with deflicker filters.

Quote from: lapser on 07 / July / 2017, 21:43:08

I then use TMPGEnc Video Mastering Works 6 to make the final video (including zoom, pan,

Does the following works with TMPGEnc?
Having a AVI video with 6000*4000 (3:2) and doing a zoom to a 1920*1080(16:9) Video.
I’m using Adobe Premiere Element and this work only with max 4K Video input.
So now I’m using a MATLAB Script which crop and resize the JPG images continuously, without preview

Quote from: lapser on 07 / July / 2017, 21:43:08

Parameter 1, the shutter count, has a value of:
1,316,805
The camera was refurbished when the shutter failed at around 500,000.

I below 200000… so far away from that…

Quote from: lapser on 07 / July / 2017, 21:43:08

I've kept the shutter at 1/600 or 1/500, instead of 1/4000.

Do you think that saves the shutter?

Quote from: lapser on 07 / July / 2017, 21:43:08

Lowered highlights with Photoshop Lightroom

I do this on most of my videos highlights down and shadows a little bit up….

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

lapser

1093

Re: Lapser Script Development

« Reply #12 on: 09 / July / 2017, 02:00:41 »

Quote from: c_joerg on 08 / July / 2017, 03:11:39

Have you ever tried a filter in VirtualDub?

I've used a lot of filters in the past, but now I try to do everything in photoshop lightroom. But those deflicker filters look really useful. Are any of them available in 64 bit?

Quote

Quote from: lapser on 07 / July / 2017, 21:43:08
I then use TMPGEnc Video Mastering Works 6 to make the final video (including zoom, pan,
Does the following works with TMPGEnc?
Having a AVI video with 6000*4000 (3:2) and doing a zoom to a 1920*1080(16:9) Video.

Yes, it works great with EOS M3 full resolution avi files out of VirtualDub. The program can create lower resolution proxy editing files, which are necessary with that much resolution.

I zoomed into full resolution EOS M3 photos at the end of this video using TMPGEnc:

Quote

Quote from: lapser on 07 / July / 2017, 21:43:08
I've kept the shutter at 1/600 or 1/500, instead of 1/4000.
Do you think that saves the shutter?

I'm pretty sure it saved the shutter, because I've taken almost a million more photos with the refurbished camera than with the original camera, by limiting the shutter speed.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Lapser Script Development

« Reply #13 on: 11 / July / 2017, 13:34:16 »

Here's what I came up with after looking at reyalp's hook code at:
http://subversion.assembla.com/svn/chdk/trunk/modules/script_shoot_hook.c

I need 3 new lua functions to be able to adapt my Tlapser.lua script to use reyalp's fantastic new shoot hook functions. They should not be called while in the hook.

hook.set_tick_continue(tick_count)
--sets tick count to exit the hook for next shot

hook.get_tick_ready()
-- gets tick count when hook.ready() became true, for last shot

hook.get_tick_countinue()
-- gets tick count of call to hook.continue(), or hook timed out, for last shot
-- hook_shoot.get_tick_continue() returns the last shutter open time

The way reyalp programmed the hooks (very clever), these functions will apply to all the hooks. Here's the changes I'm proposing to script_shoot_hook.c:

[see corrected file attached IN NEXT MESSAGE]

Here's a simple interval timer with the new code:

Code: [Select]

hook_shoot.set(10000000)
hook_raw.set(10000000)
n=0
interval=1000
press("shoot_full")
repeat
   while not hook_raw.ready() do sleep(10) end
   tnext=hook_shoot.get_tick_continue()+interval
   hook_shoot.set_tick_continue(tnext)
   n=n+1
   hook_raw_continue()
until n==5
release("shoot_full")

I'm hoping someone will compile and test (debug) these changes for me, since I've sworn off compiling (waterwingz?)

I'll be out of computer range for about a week starting tomorrow.

[EDIT]I added corrections to the attached file. Download the file from the next message.

HOOK_Changes.txt (3.04 kB - downloaded 35 times.)

« Last Edit: 11 / July / 2017, 22:46:39 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Lapser Script Development

« Reply #14 on: 11 / July / 2017, 22:43:17 »

Attached is a corrected version of the C code in the last post. Hopefully, you get the idea what I'm trying to do. Thanks for the help.

HOOK_Changes.txt (3.15 kB - downloaded 32 times.)

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

reyalp

14080

Re: Lapser Script Development

« Reply #15 on: 12 / July / 2017, 00:51:04 »

Again, before I go adding stuff, I need to be convinced that it solves an actual problem or provides worthwhile functionality.

As I mentioned earlier, I'm not convinced by the diagnosis of the problems this was proposed to solve https://chdk.setepontos.com/index.php?topic=13194.msg133807#msg133807

I do see that having a hook automatically continue a given tick or interval might be useful in it's own right. I worry about making the interface and possible behaviors more complex than it needs to be though, and the boilerplate code require to maintain an interval in lua doesn't really seem excessive.

Logged

Don't forget what the H stands for.

lapser

1093

Re: Lapser Script Development

« Reply #16 on: 12 / July / 2017, 02:12:35 »

Quote from: reyalp on 12 / July / 2017, 00:51:04

Again, before I go adding stuff, I need to be convinced that it solves an actual problem or provides worthwhile functionality.

I'm trying to adapt my existing script, Tlapser.lua, to use your functions. I use an interval timer in Tlapser, but I'd rather have the script set the tick timeout for each shot so I can vary the interval. It would require major changes to the basic structure of Tlapser to require the script to do all the precise shot timing.

For one thing, I found that the interval is accurate enough to keep two cameras in sync. I've started two cameras at the same time and they stay in sync. That brings up the possibility of panoramic time lapses with multiple cameras.

I want to experiment with lua histogram processing using your new histogram object. If the script is freed from the precise timing of the next shot, it can process histogram data during the entire time outside the raw hook. It opens up a whole new realm of possibilities.

This is a relatively simple thing to add, and I'm sure you can do it correctly with your advanced programming skills. It's very important to me to have the capability. If I was the inspiration for the incredible programming you've done, maybe you can do one thing for me because I think it's really important? Please?

Quote

As I mentioned earlier, I'm not convinced by the diagnosis of the problems this was proposed to solve https://chdk.setepontos.com/index.php?topic=13194.msg133807#msg133807

I shouldn't have framed it as task priority. When the script writes to a file when the file system is busy, especially writing raw files, it can hang for a significant amount of time. The script sometimes does things, like file writing and garbage collection, that take a significant amount of time. The capt.seq task doesn't do garbage collection.

Bottom line, it is much better to have the shot timing dependent on one task than two. The way it is now, anything that delays either task will delay the shot.

I have seen this in action with Tlapser. I save the tick time before and after printing a line of log. A single line can take more time than the interval without affecting the next shot time. This happens fairly often in my time lapses.

Quote

I do see that having a hook automatically continue a given tick or interval might be useful in it's own right.

YES !!!!! Continue at a given tick is more important than interval, although both would be useful.

Quote

I worry about making the interface and possible behaviors more complex than it needs to be though, and the boilerplate code require to maintain an interval in lua doesn't really seem excessive.

I agree completely with keeping it simple. Hook_shoot is the only hook that needs to continue at a given tick. I see no use for that capability in the other hooks, do you?

Don't forget the other functions I need: get_tick_ready() and get_tick_continue(). You can use them to tell if a hook timed out, which is a "ToDo" in your comments.

if get_tick_continue - get_tick_ready() >= timeout then [hook timed out]

I'll be gone for about a week, but I hope you'll reconsider and please add the three Lua functions that I need. Thanks!

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

waterwingz

12138

Re: Lapser Script Development

« Reply #17 on: 12 / July / 2017, 07:44:22 »

Quote from: lapser on 12 / July / 2017, 02:12:35

Quote
As I mentioned earlier, I'm not convinced by the diagnosis of the problems this was proposed to solve https://chdk.setepontos.com/index.php?topic=13194.msg133807#msg133807
I shouldn't have framed it as task priority. When the script writes to a file when the file system is busy, especially writing raw files, it can hang for a significant amount of time. The script sometimes does things, like file writing and garbage collection, that take a significant amount of time. The capt.seq task doesn't do garbage collection.

Looking back at the possible SD card "wear leveling" problem that might be causing at least part of what lapser is seeing, I found this post from reyalp that asked if the results were seen across different card types. I don't think I ever got an answer to that question from MarkB.

reyalp : have you seen that issue in any of your trials?

Logged

Ported : A1200 SD940 G10 Powershot N G16

lapser

1093

Re: Lapser Script Development

« Reply #18 on: 12 / July / 2017, 08:16:45 »

I've noticed that the problem of long script print() delays is worse when saving Canon raw files at high shot rates to my older SD cards.

One idea I have is to only do metering on every other shot at higher shot rates. With reyalp's histogram object, which doesn't disappear after each shot, I can calculate the metering data in Lua for the entire two shots.

It's possible to calculate everything you need, and more, from the histogram object in Lua. You can find the mean, and brightest pixels in one pass. It might also be useful to use the median value of the histogram for exposure calculations. All this is impossible if the script is required to control all the shot timing as well.

Because reyalp had the idea to give as much data to the script as possible, it opens up a whole new range of possibilities for Lua scripting. Requiring the script to do the timing, though, is like locking me up in a tiny half bathroom of a 20,000 square foot mansion. Let me out!

Speaking of out, I'm outta here. See you next week.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

reyalp

14080

Re: Lapser Script Development

« Reply #19 on: 12 / July / 2017, 16:59:17 »

Quote from: lapser on 12 / July / 2017, 02:12:35

I'm trying to adapt my existing script, Tlapser.lua, to use your functions.

I certainly understand not wanting to re-structure your script, but for me, that's not a compelling argument for to spend my time making significant changes to CHDK core code.

In developing these features, I provided working examples (such as rawopint, fixedint, hooktest..) and extensive documentation (http://chdk.wikia.com/wiki/Script_Shooting_Hooks http://chdk.wikia.com/wiki/Lua/Raw_Hook_Operations )

Quote

For one thing, I found that the interval is accurate enough to keep two cameras in sync. I've started two cameras at the same time and they stay in sync.

The interval accuracy available with the existing hook structure should only be maybe 10ms worse than what your suggestion would produce. As I explained before, anything other than the hardware remote has >10 ms uncertainty anyway.

If you want really precise sync (<10ms), you need to use hardware.

If you can show a case where the current method is actually insufficient, and your method is not, I might change my mind.

Quote

I want to experiment with lua histogram processing using your new histogram object. If the script is freed from the precise timing of the next shot, it can process histogram data during the entire time outside the raw hook. It opens up a whole new realm of possibilities.

Unless you are doing something really weird, all the significant processing time should be in generating the histogram data, which must be done inside the raw hook.

Do you have a real world case where the time required to do the analysis is actually significant?

Quote

When the script writes to a file when the file system is busy, especially writing raw files, it can hang for a significant amount of time. The script sometimes does things, like file writing and garbage collection, that take a significant amount of time. The capt.seq task doesn't do garbage collection.

Bottom line, it is much better to have the shot timing dependent on one task than two. The way it is now, anything that delays either task will delay the shot.

Again, I have not seen this, over the course many hundreds of thousands of frames on 5 different cameras. My suspicion is the issues you saw were specific to some combination of how your script logging was implemented and possibly issues with the specific CHDK ports or SD cards.

Note that using "print" / "print_screen" is significantly different from using Lua io like rawopint does. The log code in rawopint is modular, you can re-use it by just copying the code between "-- log module" and "-- end log module"

For garbage collection, rawopint avoids potential big Lua GC stalls and excessive garbage memory usage by calling collectgarbage('step') every frame. I have never seen any Lua GC related problems, and memory usage seems to stabilize pretty quickly.

If you are talking about SD card garbage collection, that's totally out of control of CHDK and not tied to a particular task (except to the extent it might make file IO slow), but again, I haven't seen logging cause a problem with my scripts.

Logged

Don't forget what the H stands for.