proposal - script shooting hooks - page 19 - General Discussion and Assistance - CHDK Forum supplierdeeply
« previous next »
Pages: 1 ... 14 15 16 17 18 [19] 20 21 22 23 24 ... 30   Go Down

proposal - script shooting hooks

  • 290 Replies
  • 106729 Views
*

Offline c_joerg

  • *****
  • 1248
Re: proposal - script shooting hooks
« Reply #180 on: 01 / July / 2015, 04:09:31 »
Advertisements
Just for completeness ..
Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline reyalp

  • ******
  • 14079
Re: proposal - script shooting hooks
« Reply #181 on: 01 / July / 2015, 22:35:31 »
Quote from: c_joerg on 01 / July / 2015, 04:07:34
I wondering, that the parameter of interval is not in the log, or I have not seen
It's not, I should add it.

Quote
(start(n)-exp_start(n))-(start(n-1)-exp_start(n-1))
If this is what you are using, it's not correct.

start is the tick counter (10ms resolution clock, counted from camera boot) value at which the shooting loop iteration started.
exp_start is how many ms after that the script signaled the exposure should start.

So the tick time of a given exposure started at is start + exp_start.

If you want the number of milliseconds between two exposures, then you should use
 (start(n)+exp_start(n))-(start(n-1)+exp_start(n-1))

but as I said before, this should always be exactly equal to the interval as long as sleep is >= 0, so you might as well just look at sleep.

In your log, there is one single instance where sleep is <0. As you say, this might be an SD hiccup due to garbage collection or something like that.

The rest of the sleep times are all over 300ms, so you could probably get away with a 200ms shorter interval.

It's possible that setting "disable script yield" would reduce jitter, but this would not be detectable in any of the log values. You'd need to take pictures of an millisecond resolution wall clock or something.
edit: also, it might have unspecified side effect.
Quote
tick(n)-tick(n-1)
the "tick" column just records when the log line was written.
« Last Edit: 01 / July / 2015, 22:44:47 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline c_joerg

  • *****
  • 1248
Re: proposal - script shooting hooks
« Reply #182 on: 02 / July / 2015, 15:05:56 »
Yes, sorry, I mixed the sign, now it looks good.

With the tick I was only wondering, that is still very accurate.


Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline c_joerg

  • *****
  • 1248
Re: proposal - script shooting hooks
« Reply #183 on: 03 / July / 2015, 04:07:25 »
If have seen your video from the moon, yes it is full moon, great ;)

I tried it as well with G1X, but I made a couple of things wrong. My moon is overexposed :(

I forgot to switch of the ND, so cam goes to maximum exposure time and ISO. I set max ISO to 400 (I thought I had set ISO = 300). And I set overexposure to 0.1% which was probably too high. May be I get another changes tonight…

http://youtu.be/F5z95312A0g
Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline reyalp

  • ******
  • 14079
Re: proposal - script shooting hooks
« Reply #184 on: 03 / July / 2015, 22:45:06 »
Quote from: c_joerg on 03 / July / 2015, 04:07:25
If have seen your video from the moon, yes it is full moon, great ;)

I tried it as well with G1X, but I made a couple of things wrong. My moon is overexposed :(
G1x is going to be difficult as long as the ISO override issues aren't resolved. The over exposure logic in the version of the script you have is also not very good for the moon.

Here's an updated version of the script. I'm not really satisfied with it, but I've been sitting on it for too long and it does some things better than before.

Histogram now works in parts per million, and over / under thresh are specified in parts per 100k. However, to use low values you must set your histogram step appropriately. Very small values are also more likely to suffer oscillation, though it will depend on the scene.

Added options:

Minimum shutter speed. This should be set to shortest shutter speed your camera can physically achieve, or longer. This is especially important if you prioritize over exposure protection. Setting it close to the Canon factory limit may help avoid flicker.

Meter high and low max weight. These control how the influence of the meter area changes between the thresh and limit values. The default 200 is the same as previous versions of the script. If set to 100, the corresponding thresh and limit are effectively disabled, and the meter influence does not increase (the limits still affect Bv Ev shift.)

Overexp and underexp max weight. These control the maximum strength of histogram based limits. These don't work in quite the same way as the meter weight limits: Histogram limits always reach weight 100 at the "thresh" value. If the thresh is exceeded, they continue to increase up to the max weight. The default 200 behaves the same as previous versions.

Overexp and underep prio (=priority) values. The default 0 behaves the same as previous versions. Greater values reduce the weight of opposing meter or histogram values, proportional to the fraction of thresh used. So overexp prio 50 will reduce the weight of meter by 50 when thresh is reached, if and only if the meter would drive the exposure in the other direction. This ramps linearly with the fraction of the thresh value used. This also applies to opposing histogram limits, so if over prio is set, and there is under exposure in the scene, the influence of the under-exposure will be reduced.

Add some information to the log
* histo_time is the time required to measure the histogram
* on startup, the chdk build info, interval and some zoom + focus related values are logged

The new settings are probably as clear as mud, so here's some hints:
* The default settings will generally keep the scene within the meter limits, as long as the shutter / iso limits permit. Over and under exposure protection will balance against the meter at the limits. This is good for situations where you don't want things like the sun, moon, or areas of strong shadow to completely throw off the overall exposure of the scene.
* Reducing the meter max weights allows over / under protection to push beyond the meter limits. This is useful if you want to ensure the over or under limit is not exceeded. If meter low max weight is set to 100, then over exposure will balance the meter when the over exposure fraction is reached. However, it will still respond slowly because it will be opposing the standard 100 weight of the meter.
* Adding "prio" reduces the influence of opposing meter or histogram values. As the name suggests, this lets you prioritize the over or under inputs. However, high values are likely to lead to oscillation.
* The over / under max weights only influence the behavior when the threshold is exceeded. They control how far the weight is allowed to increase.

I'll do another post for the last moon video, which gives an example of these.
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 14079
Re: proposal - script shooting hooks
« Reply #185 on: 03 / July / 2015, 23:00:56 »
Example:
http://youtu.be/WBM7o9dmhpg

This was done with a slightly different version of the script, so some of the log values are the same as the current script would produce. In particular, this was before I changed the histogram to parts per million.

Relevant settings:
Meter width & height 90%
Meter step 15
Bv Ev shift % 30
Bv Ev shift base Bv 9
Meter low thesh 5 (only relevant to Bv / Ev shift)
Meter low limit 6 (only relevant to Bv / Ev shift)
Meter low limit weight 100 (it's 0 in the log due to script differences)
Overexp ev range 1/2
Overexp thresh  0.04%
Over prio 50
Underexp thresh 0 (histogram under limit disabled)
histogram step 11 (130k pixels)

over_frac is on the right axis, other values on the left.

edit:
Focal length was ~200 35 mm equivalent.
« Last Edit: 04 / July / 2015, 18:10:22 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline reyalp

  • ******
  • 14079
Re: proposal - script shooting hooks
« Reply #186 on: 05 / July / 2015, 19:11:44 »
Thinking about what needs to be done to wrap up the C code changes for 1.4 release. This is a long post and is partly just to get my own thoughts in order, but any feedback would be appreciated.

Code organization
Currently, the code is built into the Lua script module. It depends on the scripts hooks, which are also built into the same module.
Originally, I was going to put the raw manipulation code in it's own module, but currently a large fraction of code is Lua interface "glue".
The glue can't be in a separate .flt, because there's no convenient way to export the Lua API from the main Lua module to another. This is something I'd like to address in the future, but not for 1.4

The rawop code adds a few kb to the Lua module, which I  think is acceptable. If significantly more non-glue code is added, it should probably be split out. There are also possible uses for raw manipulation outside of script, which would also argue for splitting it out.

Error handling
Currently, the code doesn't check if you are in the raw hook, or if a valid raw buffer is available in the current mode (IE cameras that don't have raw in auto, or binned modes with low res raw).
* IMO it should, since writing outside of these conditions could be fatal
* OTOH the hook could time out while a long running call is in progress so it will never be completely safe.
* If a function is called when raw is not valid, should it error(), or just ignore the call? I'm inclined to use error. This means scripts will terminate if the hook timeout is too short, but script writers should use generous timeouts.
* The overhead of error checking itself could be a concern for functions like get_pixel and set_pixel. My current plan is to set / clear a flag on hook entry / exit
* a related question is how to handle invalid coordinates etc. The current code clamps/ignores calls that would fall outside the frame buffer, and I'm inclined to leave it this way.

G1x blacklevel issue
I don't really want to change the code a lot to deal with quirks in the one camera, especially since the ISO issues aren't really resolved. However, we will probably run into other cameras with quirks later.
* values associated raw "neutral" needs to be re-calculated when black level changes.
* a callback on entry into the hook could be used to re-calculate the values.
* scripts will no longer be able to pre-calculate values before shooting, which is inconvenient
* fb_info is not well suited to values changing on the fly. It's possible that other values could change at runtime in the future so this should probably be converted to individual calls. This may make it less convenient to write efficient lua code, but C calls shouldn't be worse than a table lookup.

meter interface
The valid meter step and count values are limited by overflow, and the limits depend on sensor bit depth. It would be possible to make a higher level function that makes multiple meter calls and averages them, but in practice I haven't found it to be a problem.

shot_histogram vs rawop histogram
Since shot_histogram is now a module, and is also available in ubasic, I'm inclined to leave it alone.

combined meter and histogram
If you want both a histogram and average value from a given area of the screen, getting the average from the histogram could be much more efficient, since reading large numbers of pixels is slow. However, I haven't come up with a good way to do this without either risking integer overflow, or using FP or 64 bit math.

Additional drawing functions
Some things I originally had in mind were
* arbitrary lines
* ellipses
* text
I'm inclined to skip the first two for 1.4, since I haven't found myself needing them. If someone *really* wants they could implement them in lua with set_pixel, and if there is demand, they can be added for 1.5

Text seems potentially more useful.
* People have requested things like date stamps before, and putting text on the image could be convenient for debugging.
* Adding date stamp functionality implies that some of the code must be outside of the script module
* Reusing the CHDK font infrastructure looks like it would be complicated, especially if you don't use the current OSD font.
* For numbers, you could a 7 segment display style output pretty easily with the current code.
* It would be possible to do simple text entirely from Lua too.

Other functions
Some other stuff I've thought about, but am not planning to implement for 1.4
* some kind of color matrix interface, to allow scripts to work in roughly correct RGB
* allow script to efficiently load/save chunks of the raw buffer

rawops in binned modes
A long time ago, srsa_4c investigated the the raw buffer for some half-res low light modes: http://chdk.setepontos.com/index.php?topic=10648.10
At the time, we concluded that it wasn't worth pursing since the quality was low, but having it available for rawops could be quite useful. Metering and histogram don't depend on quality, and these modes typically shoot very fast, so people making timelapse for find the quality vs speed trade acceptable. If your final  output is significantly smaller than the binned resolution, the quality loss is probably not a big deal. I'm not planning to pursue this for 1.4, but it argues for structuring the code to support varying raw size in the future.
« Last Edit: 11 / July / 2015, 19:45:32 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline c_joerg

  • *****
  • 1248
Re: proposal - script shooting hooks
« Reply #187 on: 07 / July / 2015, 02:40:41 »
I made the boot ride with G1X. I did not look as I expected. It is probably a lot of post processing necessary, to stabilize the pictures.

! No longer available

I have not update rawopint until now. The run was made with version rawopscripts-2015-05-17. I still have some other runs with this version but I have not analyzed until now.

On the run itself, I couldn’t see anything on the display with the sun. It is really hard to stabilizing the boot. If you drive to slow, the boot gets instable. If you drive to fast, the interval from 0.7s is still too long.

The video was made in two parts. The first part was made with approximately 10km/h and the second part (starting around picture 1040) with 15km/h. I don’t use all pictures for the video.

May be it is not worse to think about:
There are a lot of exceptions in sleep. I’m not sure, with effect this has on the video. I have some other runs with 3s interval, where sleep goes close to 0. I have seen this on three different SD cards.

Another interesting think about the run:
The Battery has still more than 8,4V after 1500 shoots. The specification of G1X says 700 pictures with display off. 

For the rest of the stuff you wrote, it takes time for me to understand… ;)



« Last Edit: 07 / July / 2015, 08:28:32 by c_joerg »
Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline c_joerg

  • *****
  • 1248
Re: proposal - script shooting hooks
« Reply #188 on: 07 / July / 2015, 08:25:19 »
This is another sunset run with G1X. The run was made with version rawopscripts-2015-05-17. It starts to get flicker on the end of the video. Not as much as on the last video in Reply #144
May be your newest version will reduce it…

http://youtu.be/RiKxICDgYJw


Quote
G1x is going to be difficult as long as the ISO override issues aren't resolved.
The over exposure logic in the version of the script you have is also not very good for the moon.

My understanding was, as long I stay with ISO under 300, it should be OK or?

Quote
G1x blacklevel issue
I don't really want to change the code a lot to deal with quirks in the one camera

I can understand you…


« Last Edit: 07 / July / 2015, 11:57:55 by c_joerg »
Logged
M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd
*

Offline reyalp

  • ******
  • 14079
Re: proposal - script shooting hooks
« Reply #189 on: 07 / July / 2015, 16:58:18 »
Quote from: c_joerg on 07 / July / 2015, 02:40:41
There are a lot of exceptions in sleep. I’m not sure, with effect this has on the video. I have some other runs with 3s interval, where sleep goes close to 0. I have seen this on three different SD cards.
I'll take a closer look at how mine are behaving. My impression is that it's been pretty stable, I usually run 1.2 sec interval on D10, and see few if any negative sleep until the exposure gets long. D10 continuous shooting rate is around 1 FPS.

SD cards have their own CPU and do wear leveling and garbage collection so their performance can be quite complicated, especially if they are used in a different way than the designers expected. See this page for some background https://wiki.linaro.org/WorkingGroups/KernelArchived/Projects/FlashCardSurvey

The script logging might fall into this category. It writes one log line every shot, and the camera firmware probably buffers it up to 32 k, but exactly how isn't clear. SD cards are mostly designed for continuous write speed of large files, so this might have an impact. I could make the log buffer to RAM for the whole run, but this would probably cause errors on long runs. You could also turn off logging, but then you wouldn't know how it's performing ;)

If you haven't tried it already, formatting the card might also give more stable performance. I haven't tried this, and only reformat my cards very rarely.

Quote
The Battery has still more than 8,4V after 1500 shoots. The specification of G1X says 700 pictures with display off. 
On the cameras I've used, it seems to depend more on how long the camera is running rather than how many shots, so a longer interval gives you a bit more run time, but many less shots. On D10 I've had over 7000 shots, which isn't bad for a CIPA spec of 220 (screen on)

Quote
My understanding was, as long I stay with ISO under 300, it should be OK or?
I think so, unless there are other quirks we don't know about.
Logged
Don't forget what the H stands for.
Pages: 1 ... 14 15 16 17 18 [19] 20 21 22 23 24 ... 30   Go Up
« previous next »
 

Related Topics