Shot Histogram Request

philmoz

3450

Re: Shot Histogram Request

« Reply #100 on: 01 / February / 2013, 16:47:55 »

Advertisements

Quote from: lapser on 27 / January / 2013, 14:51:56

The sx260 halted with an unspecified Lua error at about 1500 shots, giving only the "SCRIPT TERMINATED" message. This has happened before on both cameras while writing a log file, but this time I wasn't writing a log, so it must be related to something else.

There isn't anything in CHDK that would print "SCRIPT TERMINATED" - did you mean the "*** TERMINATED ***" message when there is an error in the script?

If so it should have displayed an error message as well - what did that say?

Phil.

Logged

CHDK ports:
sx30is (1.00c, 1.00h, 1.00l, 1.00n & 1.00p)
g12 (1.00c, 1.00e, 1.00f & 1.00g)
sx130is (1.01d & 1.01f)
ixus310hs (1.00a & 1.01a)
sx40hs (1.00d, 1.00g & 1.00i)
g1x (1.00e, 1.00f & 1.00g)
g5x (1.00c, 1.01a, 1.01b)
g7x2 (1.01a, 1.01b, 1.10b)

lapser

1093

Re: Shot Histogram Request

« Reply #101 on: 01 / February / 2013, 17:41:27 »

Quote from: philmoz on 01 / February / 2013, 16:47:55

There isn't anything in CHDK that would print "SCRIPT TERMINATED" - did you mean the "*** TERMINATED ***" message when there is an error in the script?

If so it should have displayed an error message as well - what did that say?

Yes, it said "*** TERMINATED ***", with a large number like, 65094782 or something like that. Unfortunately, I lost the log file when I restarted the script. I'll have to use the print_screen(-1) option next time to append the log. I'll see if I can reproduce it and let you know the exact number if I can.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Shot Histogram Request

« Reply #102 on: 03 / February / 2013, 01:29:43 »

I made a new time lapse tonight with over 4,600 shots, without the script halting with an error like the last time. I added a busy flag to build_shot_histogram that I think may have worked. get_shot_meter now waits until the busy flag is clear, so it doesn't read values that are being calculated at the same time in a different thread. Hopefully, this will fix the problem.

Another interesting thing I tested tonight was the ND filter. I started with the ND filter IN, so the sun wouldn't be over-exposed. It worked pretty well. When it got darker, I stopped the script, set the ND filter to OUT, and restarted the script. I deleted the first shot in the new sequence. The shot meter exposed the 2nd shot perfectly, matching the shot with the ND filter IN at the end of the first sequence. The exposure change was about 2.75 EV when the ND filter went out. You can see the glitch at 1:11 in the video. There's a noticeable color change, so I guess the ND filter isn't quite ND. Here's the video:
http://www.youtube.com/watch?v=cpKlTpvoSWk#ws

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

New Discoveries

« Reply #103 on: 04 / February / 2013, 23:59:00 »

I've been experimenting with Continuous Mode with some interesting results. First, I want to match up the exposure of the shot with the result from the shot_meter. The script sets the exposure, but triggering of the shots is done automatically by the camera. So the exposure the script sets may not end up as the exposure of the next shot_meter.

So I set up a buffer to save the exposure values Just before shutter open, I read the propcases for sv, tv, av, and nd_filter and save them in a buffer array that corresponds to the shutter shot count. I also have a raw shot count, which I use to read the exposure values out of the buffer. That way, these values correspond to the actual exposure of the shot as saved just before shutter open.

To avoid having the script set new exposure values at the wrong time, I added a new lua function:

set_exp96(tv[,sv,av,nd_filter])

The function saves the exposure values, which are stored into the propcases just before shutter open. These values usually become valid for the shot, but it looks like it may need a 10 msec delay after storing them to be totally reliable.

I decided to triple buffer the values. The third buffer is used for the script to print out the values without slowing the shot rate. The sequence is as follows:

1. Exposure values written to buffer using shutter_count index%3 (triple buffer)

2. Values are read from buffer in raw_save_file() using raw_count. These values are passed to the shot meter, which I'll use to computer Bv.

3. When the shot meters are ready, after build_shot_histogram(), the raw_shot_counter is incremented. This signals get_shot_ready() to return true. The script is waiting for this, and reads the shot meters. It computes new exposure values and saves them with: set_exp96(tv,sv). During this time, raw_save_file() waits for the next call to get_shot_ready() from the script.

4. After the script calls set_exp96, it then calls get_shot_ready(-1), which triggers raw_savefile to stop waiting for the script and move on to the next shot. However, the exposure values remain in the buffer, and can still be read by the script using the wait_count. get_shot_ready() sets wait_count=raw_count whenever it's called when the shot isn't ready. So the wait_count stays valid until the script finishes writing the data.

It sounds a little complicated, but the triple buffer keeps the shots coming as fast as possible without having to wait for the script to write log data. The camera can be exposing a new shot into one raw picture buffer while build_histogram is computing the shot meters for the last shot from the other raw buffer, and the script is printing out the log data out of the data buffer from the shot before that.

I tried a new time lapse today with a shot interval of 800 msec, or 1.25 shots per second. Everything went smoothly for 3,235 shots, without a glitch.
Skinner Butte 130204 CHDK G1X

« Last Edit: 05 / February / 2013, 00:20:13 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Shot Histogram Request

« Reply #104 on: 06 / February / 2013, 02:03:52 »

I finally figured out how to return bv96 from the shot meters. Before calculating the meters in build_shot_histogram(), I save bvbase=tv96+av96-sv96. The meter value + bvbase is the bv96 for that meter.

One of the reasons to use a shot meter rather than the built in camera meter is that get_bv96(), which returns the camera meter value, loses accuracy around -700 or so, which is a dark, starry night. The time lapse flashes at these low light levels, every time bv96 changes. By -750, it's changing by 70 ev96 at a time.

I tested my new shot meter bv96 with the lens cap on in a dark closet. The value was around -1500, varying by 50 maximum, probably from pixel noise. I set the exposure at 10 seconds and ISO 800. I'm hoping this means the meter will be accurate in the -750 range.

The other thing you can do with bv is gradually underexpose as it gets darker. I found that underexposing 1.5 ev at -500 starting from 0 at +500 looked good. I'll program that into my script for the next time lapse.

I have it set now for get_shot_meter(0) to return the bvbase value. So you can find the bv for any meter by adding it's value to bvbase. For example:

bvbase=get_shot_meter(0)
bv1=get_shot_meter(1)+bvbase

That's useful, but a little confusing. It may be better to add a get_shot_bv96() function.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

More Continuous Mode findings

« Reply #105 on: 08 / February / 2013, 14:15:53 »

I've been doing a lot of testing in continuous drive mode. I got all those buffers working, but I discovered I may not need them.

I have 2 hooks where I delay the shooting process. First is in capt_seq_hook_nr(), which I call pre_shot. It happens right before the shutter opens. Next is in raw_save_file(), after the exposure is finished and the raw picture buffer is valid. I call this post_shot.

pre_shot: This is where I delay to get the desired shot interval. As long as the new exposure values are set before the end of the delay, it applies to the shot. If not, it takes effect on the next shot, which means you're 2 shots behind.

post_shot: Here I call build_shot_histogram() which calculates the shot meters. After they are ready, the Lua routine, get_shot_ready() returns true (once). The script then calculates and stores the new exposure values and signals the post_shot routine, which is now waiting for the script, to go ahead.

To faciliate this, I added a new Lua function: set_exp96(tv,sv,av,nd). This sets all the possible exposure values at one time, and then signals the post_shot_routine. The next call to get_shot_ready() also signals the post shot routine to proceed, in case the script isn't setting exposure.

A major discovery is that the pre shot and post shot routines never execute at the same time (at least on the G1X). As long as the post shot routine is waiting for the script signal, the pre shot routine isn't called. This means that the next shot always gets the exposure from the last shot, without missing shots. The key is to delay post shot until the exposure is set. It actually works (on the G1X). I discovered this when I added a pre shot wait for script, and stored the delay time. It was always 0. That is, the script was always ready before the pre shot routine was called. I confirmed this by writing values to the screen at the beginning and end of the pre and post shot routines.

So this is good news. I don't have to buffer all the exposure values for the shot meters. I just need to store one value, I call bvbase=tv+av-sv. I then return this value with the next get_shot_meter(0), and add it to the value returned by a meter to get bv96 for that meter.

The buffers did work really well for printing the log file, though. The script sets the exposure first, as soon as possible, and the next pre shot starts immediately afterwards. But the script is still reading the values from the last shot out of the buffer. Without the buffer, they disappear as soon as the next pre shot starts. The script could print the exposure values it just calculated as well as the shot meter values from the last shot that it used for those calculations. That should be enough, as long as the pre shot and post shot routines are always executed in sequence. I'll have to test this on my other cameras, the sx260 and sx50 (when I get it working).

I think the new handshake timing, along with the busy flag in build_shot_histogram() has fixed the script error problem. The new timing prevents the script from calling get_shot_meter() during the histogram build, but it's also possible that the script won't follow the handshake, so the busy flag is the backup.

The handshake only occurs in continuous mode, since it's not needed in single shot. Activating the handshake also requires a call to set_shot_interval(interval, nshots). If nshots>0, then the handshake is activated. nshots is the number of shots per interval. 0 is the same as 1, except without the handshake. Everything is disabled when a script is not active.

I'll leave the buffers in for now, for testing, since they also contain a lot of shot timing data useful for debugging.

I also modified my script to adjust for Bv as it gets darker, to avoid overexposure. Here's the exposure portion of the script, for those who might be interested:

Code: (php) [Select]

set_shot_interval(m,n)
if(not single)then press("shoot_full_only") end
repeat
  if(single)then press("shoot_full_only") end
  repeat
    sleeper(10) -- sleeps at least once, so it will yield each loop
  until get_shot_ready() or is_pressed("menu") -- waits for build_histogram --
  if(is_pressed("menu"))then break end
  if(single)then release("shoot_full_only") end
  --find meter with lowest exposure (brightest)
  tvd=-100000
  for i=1,nmeters do
    tx=get_shot_meter(i)
    if(tvd<tx)then
      tvd=tx
      im=i
    end
  end
  --adjust bv as it gets darker
  bv=tvd+get_shot_meter(0) --bv96
  if(bv>=bv1)then bvd=0
  elseif(bv<=bv2)then bvd=bvdmax
  else bvd=((bv-bv1)*bvdmax)/(bv2-bv1) end

  tv=tv+tvd+bvd+svmin-sv
  sv=svmin
  if(tv<tvs)then
    sv=sv+tvs-tv
    tv=tvs
    if(sv>svmax)then
      tv=tv-(sv-svmax)
      sv=svmax
    end
  end

  if(tv<tvmin)then tv=tvmin
  elseif(tv>tvmax)then tv=tvmax end

  set_exp96(tv,sv) -- aborts script wait too

And finally, here's the time lapse from last night:
http://www.youtube.com/watch?v=eOtKYe11pIk#ws

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

More testing

« Reply #106 on: 10 / February / 2013, 18:54:44 »

I found a spot near a pioneer cemetery with a view to the west, and set up the G1X at 1 shot per second. It took 16 seconds to take the 3rd picture, but after that it took the next 4,000 or so pictures without any delays. I have to work on the initialization of the handshake part. But at least it always got the exposure change applied on the next shot, and never missed the 1000 msec interval until the shutter open time was over 0.6 seconds. The shutter open to raw ready time stayed relatively constant at 300 msec for the whole time.

I started right before sunset, and stayed until it got as dark as it was going to get. As usual, the clouds rolled in and then reflected the city lights. The Bv bottomed out at around -900 because of the lit clouds. Bv reached -949 on the G1X with the camera meter when it was totally dark. It stepped from -908 to -949 with nothing in between, resulting in a flash in the time lapse. The shot meter should have better resolution at these levels. It didn't get that dark, but there was no flash with the shot meters at low light levels. I'll be sure to point the camera away from city lights next time, and see if I can get down to -949 or below.

I got the new SX260 1.01a that I just got from Canon repair working with CHDK. I installed it first on a 4GB class 6 card and tested the time lapse. The slower card makes a big difference in the shot rate. The class 6 card could only keep up with 1 shot per second for around 20 shots, and then started to fall behind and become irregular. It will be a good way to test my handshake routines to see if I can get them to handle this. But the slowdown happens even without CHDK installed, just by holding the shutter down in continuous mode. Fortunately, after I switched to a Class 10 card, everything kept up at 1 shot per second, and faster.

I did explore the pioneer cemetery after getting the camera going. It was really sad to see how many very young children were buried there. Child and infant mortality was very high in the 1800's before antibiotics and vaccinations.

For this test, I set the Bv to underexpose by 1 2/3 EV at -500. Next time, I'll try 2 EV. The shutter time reached about 12 seconds at the end. I lowered the frame rate at the end to compensate somewhat, at the expense of a jerky video. The plan is to fade between shots to get a smoother video at lower frame rates, in the future.

http://www.youtube.com/watch?v=1gxImATXFJo#ws

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Shot Histogram Request

« Reply #107 on: 13 / February / 2013, 01:01:35 »

After much testing, I've confirmed that raw_save_file() and capt_seq_hook_nr() are never active at the same time. This simplifies things, eliminating the need for data buffers. I was able to re-write and simplify the continuous mode time lapse code and get it working well.

The pre-shot function sets the interval exactly. Then the post shot function calls build_shot_histogram() to calculate the shot meters, and signals the script that they're ready by returning get_shot_ready() true. Then the post shot function waits until the next call to get_shot_ready(). That is, it waits until the script has processed the shot. Then it returns, and the camera then converts the shot to jpg and saves it. This happens AFTER the script has written it's log data, and doesn't hang the script writing. The time waiting for the script is about 50 or 60 msec if the post shot function waits for it. Without the wait, the script time can be 1000 msec since it's competing with the camera jpg save. So waiting for the script in the post shot function solves a lot of problems.
======
I did a time lapse test with the G1X and the SX260 in a darker location. The Bv values went down below -1000 smoothly, so it looks like this will work well with low light, below the range of the camera pre-shot meter. I did have one picture that was exposed almost 1 ev brighter than it should have been, but the next picture was exposed correctly. This happened while adjusting sv96, so it may be related to that, or it could be the shot meter timing. That is, it might be that I'm reading the shot meter in one thread before the other thread is finished.

http://www.youtube.com/watch?v=aVzhkdzdpZg#ws

====
I also did a time lapse with the sx260 in the car at 1.33 shots per second (750 msec interval). The script stopped once with that "Interrupted" message and a large number. It took over 7,000 pictures successfully, so it's probably related to the shot meter calculation being interrupted. The script stopped again when the battery ran down, so there are 2 breaks in the video.

http://www.youtube.com/watch?v=b_nVZbV-MR4#ws

« Last Edit: 15 / February / 2013, 13:44:12 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

* TERMINATED *

« Reply #108 on: 13 / February / 2013, 13:50:53 »

I finally got the script termination message error while printing a log file. This time it said:

set
*** TERMINATED ***

So I guess the error string is "set". Do you know what that means?

I triggered the error by setting the shot interval to 1 msec (no pre shot delay). With the post shot wait for script delay enabled, I got a measured interval of around 550 msec. Without waiting for the script, the interval was about 430 msec, and this triggered the terminated error. If I can trigger it repeatedly this way, I can test different things to prevent it, and figure out the cause.

Without the wait for script, the script is calling get_shot_meter() and set_exp96(), and writing log data, without regard to the camera state, so the error is probably triggered by a thread conflict.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

reyalp

14082

Re: Shot Histogram Request

« Reply #109 on: 14 / February / 2013, 21:16:31 »

It's very unlikely this is a deliberate error message, actual lua errors aren't single words or guru mediation numbers. It sound a bit like the issue discussed in http://chdk.setepontos.com/index.php?topic=8273.15 and http://chdk.setepontos.com/index.php/topic,4131.msg39296.html#msg39296 where some random data would be treated as the error. Current versions should tell if the error was null or empty, but not if they are garbage.

Logged

Don't forget what the H stands for.