Night-time time-lapse - page 5 - Completed and Working Scripts - CHDK Forum

Night-time time-lapse

  • 80 Replies
  • 56150 Views
Re: Night-time time-lapse
« Reply #40 on: 27 / September / 2008, 19:13:09 »
Advertisements
I did forget about the A/D converter. So yes, you are right. Sensor output is clipped, apparently to 10 bits. As I see it, the conclusion is the same though: Even with the clipped input the camera is able to resolve signal from noise all the way down to meter values of ~-800. The real floor would be where the signal becomes comparable  to the standard deviation in the noise for the sensor as a whole, and the result would be jumping meter values - which doesn't happen. I have no idea why the meter spits out specific values in uneven steps though, but then the camera also uses uneven steps in shutter speed/aperture which doesn't make sense to me either.
SX100, S3

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Night-time time-lapse
« Reply #41 on: 20 / October / 2008, 16:14:51 »
I'm modifying this script and I think what I have done works, but I have bit of a problem with both the orignial set_b_14.lua and my modified one.

The problem is that when it's getting dark or bright, exposure does awkward jumps. I plotted some curves and it looks like the script doesn't ramp up ISO smoothly when Tv is already cranked up to max, possibly because of noise in bv measurement (I think I'm hitting the noise floor of the CCD before the script is running at max ISO + max tv and that my camera is noisier than yours). But before I go into the trouble of posting data, I'd like to verify one thing that caught my eye:

From set_b_14.lua:
Code: [Select]
--Current_tv gets throught the smoothing mechanism
--feed_smoother(current_tv)
--smoothed_tv=smoothed_value()
smoothed_tv=current_tv

The smoother is disabled? Is that's how it's supposed to be? And how does exposure stay smooth when Sv is the only thing that can be altered (when Tv is at maximum)?

*

Offline fbonomi

  • ****
  • 469
  • A570IS SD1100/Ixus80
    • Francesco Bonomi
Re: Night-time time-lapse
« Reply #42 on: 20 / October / 2008, 18:06:09 »
Yes, the smoother is disabled because it seemed unnecessary after some tries.

If you think about it, this line:
current_tv=current_tv + (correct/50)
already functions a little bit like a smoother: the previous calculations decide what the correction should be, and we apply it divided by a large-ish number.
You will have noticed that if for example you start the script, close the camera in a box, leave it there for a while and then point it to a sunny landscape it will take several shots slowly adapting up to the correct exposure.
That's the "smothing" effect created by the line above.
The other smoother (the one that is commented out) is mathematically more robust but (maybe) less effective for what we need.

Quote
The problem is that when it's getting dark or bright, exposure does awkward jumps. I plotted some curves and it looks like the script doesn't ramp up ISO smoothly when Tv is already cranked up to max,
when you see jumps in ISO, do you also see the same jumps in current_tv?

Quote
possibly because of noise in bv measurement (I think I'm hitting the noise floor of the CCD before the script is running at max ISO + max tv and that my camera is noisier than yours).

The CCD noise should be avoided by the day/night correction mechanisms.

If it's day, then we use the CCD values, but noise should be minimal
If it's night, then we use the previous photo, so the CCD noise is also muche washed out.
If it's dusk we mix the two in proportion with the "dusk factor" that tells when we switch from night to day, so it should stay noise-free anyway.

Unless you have drastically changed the dusk start and dusk duration parameters.....

What does dusk_factor say when you have those jumps?

You could post the PR_SCREEN.TXT file, so I could give a look

in case you didn't find out already, line 259 of the script:
is made like that so that you can easily import logged data into excel
print("##", bv, count_frames,  ....
Open PR_SCREEN.TXT, select all, copy, paste in excel, sort the worksheet by column A, delete all lines that do not start with ## and you have all data you want, and doing comparative graphs between all the values is trivial

Quote
And how does exposure stay smooth when Sv is the only thing that can be altered (when Tv is at maximum)?
? I don't understand ...
shoot_tv is smooth (or should be) but if the shutter time correspinding to this tv is too lng then we shorten shutter time and compensate with a higher ISO.

one last thing: default_sv should not be higher than 2100. for reasons explained in fb-lib.ua, line 109
but anyway, a higher default_sv should just get truncated, and should not cause the problem you talk about..

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Night-time time-lapse
« Reply #43 on: 20 / October / 2008, 19:01:15 »
when you see jumps in ISO, do you also see the same jumps in current_tv?
Yes, current_tv does, but the problem is that when shoot_sv jumps up and down, shoot_tv doesn't change since it's maxed out. So, when I plot the differential of ev96 (shoot_sv change from one frame to the next summed with change of shoot_tv at the same frame), I get an oscillating waveform, worst case almost 100 ev96 peak-to-peak.

Unless you have drastically changed the dusk start and dusk duration parameters.....

What does dusk_factor say when you have those jumps?

I've only changed max Tv (around 10 s), max ISO (usually 400) and sample rate. During ISO oscillation, dusk_factor is usually exactly 1000 and it doesn't go higher than that, but during some of these bursts it's a bit less (like varying close to 950).

You could post the PR_SCREEN.TXT file, so I could give a look
I'll have to find a good example... the two long ones I inspected and plotted in oocalc were not perfect (one is with my new version and other one suffered from black frames). But attached is a plot from one of the transitions.

? I don't understand ...
shoot_tv is smooth (or should be) but if the shutter time correspinding to this tv is too lng then we shorten shutter time and compensate with a higher ISO.
Well to me it looks like the script keeps rapidly changing ISO when Tv is maxed out, as described above. It's an oscillating behavior and happens on each evening-->night and night->morning transition and during nightly showers when urban lights reflect from the water causing an increase in scene luminance.

All in all the script appears to mostly underexpose during the day and has a desire to overexpose during the night (but I don't let it, by specifying insufficient ISO and Tv to meet its will).


*

Offline fbonomi

  • ****
  • 469
  • A570IS SD1100/Ixus80
    • Francesco Bonomi
Re: Night-time time-lapse
« Reply #44 on: 22 / October / 2008, 05:54:48 »
(sorry for slow reply)
I think it's problem of oscillations caused by feedback

for example, in a situation where ideal exposure would be 100, this is what is happenning

Code: [Select]
current exposure is 50
:loop
camera shoots at 50
camera examines frame, says "it's too dark!" and decides to expose more
current_exposure is increased by 100 and goes to 150
camera shoots at 150
camera examines frame, says "it's too bright!" and decides to expose less
current_exposure is decreased by 100 and goes to 50
goto loop

the problem is, this should not happen, as the line:
   current_tv=current_tv + (correct/50)
supposedly avoids too strong variations, as the correction factor is divided by 50

the simplest way could be changing the line to
current_tv=current_tv + (correct/100)
and making the script less responsive to changes (slower in transitions)

more advanced solution would be to make the smoothing more clever, and making it stronger for higher values of correct. something like

current_tv=current_tv + sqrt(correct) * N

for a series of reasons, this is VERY hypothetical
- correct can be negative, so it should be current_tv=current_tv + sgn(correct)*sqrt(abs(correct)) * N
- all these functions are not available il CHDK Lua
anyway, we would need something along these lines.

Another alternative version, also very primitive (pseudocode):

we see what' is the value of correct that causes this obscillations

if correct>N or correct<-N then
 current_tv=current_tv + (correct/100)
else
 current_tv=current_tv + (correct/50)
endif

yet another option is to give a maximum to the correction factor:
if correct>N then
 correct=N
endif
if correct<-N then
 correct=-N
endif

Anyway, to better diagnose the problem I would need the PR_SCREEN.TXT file, so that we can follow the actual calculations made by the program

hope this helps

*

Offline fbonomi

  • ****
  • 469
  • A570IS SD1100/Ixus80
    • Francesco Bonomi
Re: Night-time time-lapse
« Reply #45 on: 22 / October / 2008, 06:02:13 »
actually, looking at your graph (especially the line for delta_shoot_EV96) iit's even worst, as the oscillation increase in amplitude, i.e. we have an amplifying oscillator:

(ideal exposure is 100=

shoots at 80
"it's dark, I need to correct a bit"
increases by 50
shoots at 130
"it's VERY bright, i need to correct a lot"
decreases by 60
shoots at 70
"it's VERY VERY dark, I need to correct a HUGE LOT"
increases by 70
shoots at 140
"it's VERY VERY VERY bright, i need to correct a HUGE DAMN LOT"
decreases by 80
shoots at 60

again, the PR_SCREEN data will give us the gory details

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Night-time time-lapse
« Reply #46 on: 22 / October / 2008, 17:53:27 »
Thanks for your trouble fbonomi! Your reply made me look into the right things more quickly ;)

I looked at the log a bit more and went through all my timelapses. It seems only I only experienced this twice, once in a timelapse that suffered from black frames and once running with my modified script, which clearly needs more work.

The black framing timelapse got wildly oscillating values for night_correct=histo_measure() every now and then (-4800 right after -200), but I suppose this is to be expected with the black frame problem so there's nothing to be worried there. The image above appears to be from this timelapse.

The other one (running my script) has by coincidence(!) the same wildly oscillating shape in night_correct. It doesn't have black frames but there's something completely different screwing it up: the timelapse has 7000 JPG frames but in the log I have 10000.  When it starts to become darker and exposure time + dark frame reduction exceeds the desired minimum sample rate, I get images with double the minimum sample rate.

But I think I already know what the problem is, I forgot to add a get_shooting() wait loop after the shot, so the script must have started histogram measurement etc while doing the dark frame etc etc...

I'll fix that and test some more.  :D

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Night-time time-lapse
« Reply #47 on: 26 / October / 2008, 19:40:22 »
SunsetF15.lua documentation i.e. changes between set_b_14.lua and sunsetF15.lua

fudgey 2008/10/27

This is a modified (and improvement, I hope) version of Francesco Bonomi's
wonderful set_b_14.lua sunset timelapse script. This text file mainly documents
the various changes I've made. I have not touched the core of the script, i.e.
the exposure decision part. In that sense this version should work exactly as
set_b_14.lua.

If you are new to the script, you should see fbonomi's thread to find the
original version and especially its documentation:

Night-time time-lapse

Be aware that I haven't tested modes other than M, and it's likely that
this version behaves differently if you are not using M mode (many cameras
have no M mode, I know). If you don't have M mode, Av is your best bet.
But then again, cameras lacking M mode often lack an iris as well.

Also, I'm using file I/O functionality recently introduced by reyalp. So
far it hasn't been tested on all cameras, and it's possible that horrible
things could happen. See install instructions below for precautions.



Now, about the changes:

I fixed the dark/bright frame toggling problem by optimizing the required
delays for each step of the process. As a side effect the script is of course
significantly faster, especially for cameras like mine that needed those
additional delays to prevent said problem.

I replaced the single shoot command with a series of commands to
shoot half -> override exposure -> shoot full -> release shutter, because
that gives me the chance to use propcases directly to adjust Tv and Sv for
the coming shot, instead of using the less reliable way thru set_tv* and set_sv*.
I'm using set_prop, but it's possible that set_tv96_direct and set_sv96_direct
could be used as direct replacements; I just didn't try it because I knew
set_prop would work (I'd have to check CHDK source code to see what those
commands do and/or test it). It shouldn't matter much that I used propcases as
the new propcase.lua makes it transparent across camera models.

Another speed optimization involves moving day exposure metering to be done
during shooting instead of its own half shoot, which was also an issue
during the night.

I also added and modified a big bunch of smaller things:
  - Print remaining disk space in terms of frames and time on console. This
    does not use Canon's approximations, which I have found to be very
    pessimistic in my timelapses. Created f-disk.lua for this.
  - Use new io commands for logging instead of print to keep console nice
    and clean for better uses and to keep non-verbose log clean of unnecessary
    console status messages for direct spreadsheet import. Created f-log.lua
    for this.
  - Moved frame count print to be done after frame instead of before sleep
    (it seemed off by one to me).
  - Added datestamp to complement timestamp so that a spreadsheet could better
    calculate & plot things without wrapping around at midnight or requiring
    plotting against frame counter. Created f-date.lua for this, and moved
    timestamping functions from fb-lib.lua there.
  - Moved datestamp to beginning of each log line (after ## which I didn't
    remove as it's still useful for grepping those lines in verbose mode).
  - Added temperature logging (all three sensors that may be available).
  - For treaceability/repeatability, always log script version, parameters,
    camera model and CHDK version information.
  - Append log file instead of overwriting it (may not be a good idea for
    users who never use card readers, though... but it can be easily changed
    in f-log.lua by replacing '..name,"a")' with '..name,"w"))'.
  - Disable IS at script start (I don't know if this works).
  - Set manual focus mode at script start (this doesn't seem to work on my
    a570is but maybe it works on someone else's camera?).
  - Set focus during half-shoot for each shot (don't know if it works either,
    especially if you aren't in manual focus mode).
    It's set on each shot because there's been some problems to make it
    stick previously. Related to this, I added a new parameter:
    "Set focus to Inf". If "yes", focuses to infinity, if "no", script will use
    the focus distance that was effective when the script started. It's
    probably for the best to always use manual focus (and if you want to set
    focus distance using auto focus, use focus lock..automatic focus will fail
    during the night).
  - Disable flash at script start (this one definitely works on my a570is)
  - Changed Sv/Tv/period defaults a bit.
  - Added two new parameters for JPEG settings (quality and resolution).
    This is because I normally shoot full resolution "super fine" JPEG images,
    but to conserve space I shoot my timelapses in normal or fine quality
    wide angle (3072x1728 downscales nicely to 1080p 16:9 HD video).
   
    Default is -1, which means the script will not alter these settings at all.
   
    I don't know what the proper values are for all cameras, but from our wiki
    I can tell that for probably all cameras JPG quality:
   
    0=super fine, 1=fine, 2=normal

    and JPG resolution (we can assume those numbers are always the same,
    but availability may vary):
   
    For a570is Digic III: 0,1,2,3,4,6,8 = L,M1,M2,M3,S,Postcard,W
    For s3is   Digic  II: 0,1,2,  4,  8 = L,M1,M2,   S,         W   
   
  - Added new option: startup delay. Script will wait the specified number
    of minutes before starting capture. It's useful when you have limited
    card space and will not be around close enough to the sunset. You can
    set up the camera in the morning, leave for work and not worry about
    getting home in time to start the script.
  - Start exposure at Tv equal to current user Tv (this is the value user
    has set if in Tv or M mode) instead of a fixed value to provide quicker
    startup.
  - Use the brand new propcase.lua library for propcases.
  - Fixed a tiny bug (I think) in fb-lib.lua: "if verbose" didn't work,
    changed it to "if verbose==1". If it should have worked it's probably
    an user error on my part or a CHDK bug.
  - Additions to dummy.lua (the PC version, of course) to make new version
    run on a PC (tested on Ubuntu's lua5.1)
  - Named this version "Sunset F15"
 
Note that the script is evil and does not return settings it changes to values
from before calling the script. It uses some overrides which will not be visible
on the Canon OSD. So, the camera may be in a funny state after stopping the
script. Restart the camera after the script to be sure all OSD is up to date.
The script isn't ment to be stopped until the card is full or the battery
drains and then you'll be forced to shut the camera down anyway...

Before starting the script, I recommend you to:

-Disable IS (image stabilizer causes blur in long exposures)
-Disable focus beam
-Disable manual focus assist features
-Disable shot review
-Disable red eye removal
-Disable flash (until you verify that the script is able to disable your flash).
-Set camera to M mode (I haven't tested any other mode)
-Select a white balance preset (anything but AUTO is good, custom is best)
-Set manual focus (and focus to inf, and remember the new focus parameter
 in the script and that it may or may not work for you).


Install instructions:

1) Get a new CHDK from CHDK Download - build 0.6.6, revision #544 install it on an
empty card and run CHDK/SCRIPTS/TEST/LLIBTST.LUA to check how the new IO
library functions work on your camera. Report your findings to the thread here:

preliminary lua iolib and oslib port

I suggested using an empty card because if that test fails miserably,
it's possible that it will corrupt the filesystem on your card, which
could lead to the need to re-format the card and loss of photos!

2) Place sunsetF15.lua to CHDK/SCRIPTS

3) Place everything in lualib/ to CHDK/LUALIB. You should already have
propcase.lua and GEN/*, so I didn't include them there.

If you need to use an older build, you can find the required
CHDK/LUALIB/propcase.lua and the entire CHDK/LUALIB/GEN directory from
the "pc" subdirectory (which you don't need on the camera, it's for
debugging the script on a PC outside the camera) or in a "complete"
zip archive of a new build.
 
4) In CHDK, load and start script sunsetF15.lua.

After each shot the script will update remaining time and remaining
number of frames counters on the script console. Keep in mind that
they are approximations calculated during the script run. They will
be very inaccurate at start. If your frame rate slows down at dark,
know that early in the morning the "remaining time" approximation
will be very optimistic, but "remaining frames" will be more accurate.

The script writes a log file to CHDK/LOGS/timelaps.log. It's always
appended, so you should remove it every now and then to prevent it
growing too big. The script logs various interesting things including
the script version and parameters and CHDK version you used, and
information of each shot (such as perceived brightness, Tv and Sv
used in the shot etc, and camera temperature sensor data).

The log is tab separated for instant import to spreadsheet apps.
If the iolib commands are not available for your camera, traditional
print() will be used instead. The log file will not be quite as
clean in that case, extra spaces will appear.

--------------------------------------------------------------------

TODO:

- Test & modify the script to work outside M mode if possible.
- End script on "menu" keypress + return propcases to startup values.
- There's no reason why flash with manual power shouldn't be allowed. Flash
  will light up things close to the camera as well as traffic signs and
  someone may find the effect useful. Don't know how smart it is to shoot
  thousands of flash shots a few seconds apart, though...

--------------------------------------------------------------------

The script is attached below in a zip file. If you don't see it, you are not logged in the forum.


*

Offline fbonomi

  • ****
  • 469
  • A570IS SD1100/Ixus80
    • Francesco Bonomi
Re: Night-time time-lapse
« Reply #48 on: 27 / October / 2008, 02:11:05 »
wow, fudgey, thanks!
will study your stuff with attention and pleasure!

*

Offline fudgey

  • *****
  • 1705
  • a570is
Re: Night-time time-lapse
« Reply #49 on: 28 / October / 2008, 13:54:03 »
I've split some in-depth Lua coding discussion of more general interest than just this script to a new topic: http://chdk.setepontos.com/index.php/topic,2511.0.html by request.

 

Related Topics