Shot Histogram Request

lapser

1093

Re: Shot Histogram Request

« Reply #370 on: 22 / August / 2013, 16:34:35 »

Advertisements

I found the Canon D20 at Costco for $249, which is $40 less than the current Amazon price, so I couldn't resist. It seems to be a pretty good camera. My time lapse mods worked on nafraf's new port, so I was able to take my first time lapse with the D20 last night on Spencer's Butte.

Something was off in the first few pictures. The exposure was flashing up and down almost 1 fstop until my smoothing routine kicked in after 5 pictures. I'll have to check that out. But the smoothing routine in the script is designed to smooth that kind of variation out, and it worked great.

One thing I like about the camera for sunsets is that the lens is very simple, contained entirely within the camera to make it waterproof. It's only 5X zoom, but I'm usually doing wide angle with time lapse anyway. The short, simple lens didn't show any lens reflections from the sun like all the other cameras. In the video, there's a small reflection right under the sun, but that's a building with a shiny roof, not a lens reflection.

Here's the D20 time lapse video, complete with a group of people walking in front of the camera:

http://www.youtube.com/watch?v=3dGj4CRDeqw#

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

ahull

Re: Shot Histogram Request

« Reply #371 on: 23 / August / 2013, 17:42:07 »

The first thing that has made me smile after a particularly hard day.
By the way lapser, what is the music?

Logged

lapser

1093

Re: Shot Histogram Request

« Reply #372 on: 23 / August / 2013, 18:53:37 »

Quote from: ahull on 23 / August / 2013, 17:42:07

By the way lapser, what is the music?

Here's the music link:
http://www.jamendo.com/en/track/709234/demande-au-ciel

I just used the first minute of the song. The rest of it is very pretty too.

I'm pretty excited about using the D20 for time lapses. Since it's waterproof, I should be able to use it in storms. We had some really nasty thunderstorms with hail and lightning on the Hand Lake backpacking trip, but I was too chicken to leave the G1X out in it, even wrapped in plastic film.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Continuous vs Single Shot mode discoveries

« Reply #373 on: 24 / August / 2013, 12:36:17 »

I tried the D20 with an overnight time lapse powered by an external, A/C supply (Chinese cheapo). The power kept cutting off for some reason. The connectors all seemed OK, so maybe the voltage regulation or filtering in the power supply is bad, or the D20 is more sensitive to power fluctuations. Anyway, I found out some interesting things about the camera.

In continuous mode, all the pictures taken by the time lapse script are LOST if the external power suddenly shuts off while the script is running. In single shot mode, only the last picture is lost. This happened on the D20 and the SX260. I haven't tested it on the G1X, but I'm sure glad my G1X external battery lasted the whole night on my backpack trips!

With fast shutter speeds, you can take pictures at almost 2 shots per second in continuous on the SX260. In single shot, it drops to a little slower than 1 shot per second. So I set up my overnight time lapse with a 2 second interval, and put it in single shot to avoid losing all the pictures if the power failed.

The power did fail after about an hour, but I had all the pictures until that point. However, single shot isn't as good for two reasons:

1: In single shot, the screen doesn't display the pictures as they come in. The screen is blank during a long exposure, so you can't see the log data, except briefly in between pictures. In continuous mode, the last picture, and log data is on the screen until it is replaced by the next picture. This makes it easier to see what's happening.

2. In single shot, the time in between pictures is about 1000 msec PLUS the shutter time. At night, the shutter time frequently exceeds the interval time, so the camera is taking pictures as fast as possible. So in single shot, there is a 1000 msec gap between exposures at night.

In continuous mode, the time between pictures is about 300 msec PLUS the shutter time. So there is only a 300 msec gap between pictures. This should make for smoother video or star trails, and you'll catch more meteors.

But if you're using continuous mode, it's important to make sure the battery doesn't shut down or you'll lose all the pictures. I think the script could do a release shoot_full_only and then hold it down again every so often. That should save all the pictures taken up until that point. Doing this every 30 minutes or hour would insure against the total catastrophe of losing all the pictures (that happened to me with the SX260 during the Perseid meteor shower).

I'm a little disappointed that the D20 didn't run all night on A/C external power. I'll try it with one of my big, external batteries after I solder up a cable for it. The D20 and the SX260 use nominal 3.7 volt batteries, but they run fine on USB power at 5 volts. A fully charged battery puts out about a volt more than its nominal voltage anyway, so the camera can easily handle this variation.

I set up the D20 behind my house last night pointed at the sky. I let the ISO go to 1600, and the shutter time up to 30 seconds. The shutter time only reached 25 seconds, so the exposure is pretty bright. I processed the pictures somewhat with Videomach, but Lightroom would do better. I tacked the orginal video on at the end.

http://www.youtube.com/watch?v=BjuEt1jnUng#

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

New display off function

« Reply #374 on: 25 / August / 2013, 13:46:01 »

Reyalp figured out a way to keep the screen display off without flashing. I added it to the script and tried it out last night. It worked great, and it was so nice not to have the camera screen on or flashing after it got dark. For more info, see this post:

http://chdk.setepontos.com/index.php?topic=10551.msg104446#msg104446

Here's the two time lapses I made from last night's sunset and ISS flyover, and a cropped picture of the ISS:

http://www.youtube.com/watch?v=LstH_s4DBPk#

http://www.youtube.com/watch?v=dDqJpYiC4IU#

ForestRidge130824_5782ISS.jpg (65.89 kB, 1054x793 - viewed 34 times.)

« Last Edit: 25 / August / 2013, 13:53:05 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Shot Histogram Request

« Reply #375 on: 26 / August / 2013, 03:34:09 »

I tried out the Canon D20 for a rainy sunset on Skinner Butte, overlooking Eugene, OR. The camera takes pretty good pictures, and the time lapse looks smooth.

The D20 only has an ND filter, which is used as aperture. There's no aperture priority mode, only P mode, so I'll need to set an aperture with the script to make sure I don't have the ND filter in. I'll have to experiment with the camera a little here.

Philmoz recently improved the set_av96(n) function so it picks the nearest valid Canon aperture, rather than ignoring the call if you don't happen to pick a valid value. I wonder if this means I could set an aperture value of 1 (av96 equivalent) and end up with the widest possible (i.e. ND filter out). It's probably best to set aperture on this camera rather than try to control the ND filter directly.

I also discovered that the D20 doesn't work with my focusing routine. The other cameras have to be in manual focus mode for set_focus(d) to work. But the D20 crashes when you try to do this. I'll have to check this out more too. Fortunately, my script has an option to let the camera do its own focusing. So I can manually focus the camera before starting the script, and the script will hold this focus point. It worked for this time lapse.

Here's the time lapse:

http://www.youtube.com/watch?v=gUSApNkYRFA#

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

reyalp

13882

Re: Shot Histogram Request

« Reply #376 on: 26 / August / 2013, 16:29:08 »

Quote from: lapser on 26 / August / 2013, 03:34:09

Philmoz recently improved the set_av96(n) function so it picks the nearest valid Canon aperture, rather than ignoring the call if you don't happen to pick a valid value. I wonder if this means I could set an aperture value of 1 (av96 equivalent) and end up with the widest possible (i.e. ND filter out). It's probably best to set aperture on this camera rather than try to control the ND filter directly.

In general, the aperture propcases do not control the ND filter (there may be some exceptions). You should use set_nd_filter for that.

Since d20 has CAM_HAS_IRIS_DIAPHRAGM is undefined, the set_av* functions should be NOOP.

Some further discussion of ND control http://chdk.setepontos.com/index.php?topic=10432.0

Logged

Don't forget what the H stands for.

straessi

Re: Shot Histogram Request

« Reply #377 on: 28 / August / 2013, 14:07:18 »

I finally was able to get my G1X (101a) running with the TLapse script. Great work, thanks a lot! I compiled the latest trunk 3034 (CHDK v1.3) after applying "shot_histogram.patch" and "backlight_off.patch", which I found as attachments in this forum thread. Not quite sure if that is the way to get to the most recent version. Let me know if there is a better way.

It works quite smooth, but there is one thing that I couldn't figure out so far. When it gets dark the exposure changes up and down quite fast, which results to massive flickering. I tried different metering placements but it doesn't really help.
I'm still trying to figure out what is causing that. My first guess is that it happens when max. shutter time is reached (after ISO is also at the maximum). But I have to confirm that... What happens in that case? It just stays there until it gets brighter again?

I have a few questions for debugging:
I have read the previous posts about glitch and smoothing parameters, but I'm not quite sure if I got it correctly. So let me try and explain what they are doing and I would be happy to know if that is correct.

Glitch: The glitch parameter is the maximum exposure difference to the previous picture in order to change exposure normally for the next picture. If that limit is overstepped it takes two shots. The second one with the same exposure as the picture before the glitch happend. So the higher this parameter is set, the more exposure difference is needed to detect a glitch and take a second picture.
What happens if this Parameter is set to 0? Is this function then deactivated?

Smoothing: Defines the maximum exposure difference allowed for the next picture. So the smaller the number, the slower the exposure will adjust, if it for example gets bright suddenly and it should be smoother.

Last question is about the log file. Everything is clear until the shots are taken. For every shot there are 9 parameters written on a line. This must be the code line that forms this string:

sp=string.format("%d %d %d %d %2d %2d %d %s %-6d",
get_shot_data(2),get_shot_data(3),get_shot_data(6),
sv96_to_iso(sv+svdiff),tvd0,tvd,get_vbatt(),seconds(tv),ptime)

I don't understand these parameters (well except the obvious ones... like the battery voltage). Could you briefly tell me what they represent?

Thanks for the help. I know that these questions can be answered by looking at the code, but it takes me ages and I want to get some time lapses

Cheers Daniel

Logged

lapser

1093

Re: Shot Histogram Request

« Reply #378 on: 28 / August / 2013, 14:40:15 »

I added a new Lua function to my CHDK mods that turns off all CHDK screen displays at once, including the OSD, console, and also the <ALT> line at the bottom. The new function sets a flag that is read in main.c, which then skips gui_redraw(). Also, the current main.c (core_spy_task) skips gui_redraw() while taking a picture, unless the picture review is active. This assumes the screen is dark, but this is only true in single shot mode. In continuous mode, the last picture is displayed while taking the next picture. So I added a test to always do gui_redraw() except in single shot mode.

It works really well on all my cameras. The nice thing is that as soon as you reset the flag, the OSD and console appear fully updated. That is, whatever you wrote to the console while the OSD was off comes back when you turn it on again. I suppose you could also use the flag outside a script to get rid of the screen clutter temporarily. I can post a separate .diff of the changes if anyone is interested.
=====
I also added reyalp's new method of turning off the display using eventprocs to my script. It worked great last night on 3 of my cameras. On the SX260, it turns off the display, but the camera turns it back on when it displays the next picture, just like set_backlight(0).

Finally, I added the <ALT> button script exit mod, which frees up the shoot_full button for script use. It works really well on all my cameras, is very easy to use, and only takes a few lines of code. I think it's a major improvement to CHDK, but there's a lot of inertia to keep things working the way they always have, so it doesn't look like it will make it into the trunk. Again, I can post a .diff if anyone is interested in using it in their own, custom version of CHDK.

I have two good tripods with really solid heads, and a 3rd cheapo tripod. I wanted to test the D20 zoom, so I set it up on the cheapo. I was playing with the new display on/off and OSD on/off in my script, which requires pressing camera buttons. The D20 shook a little every time I did this, because of the 5X (maximum) zoom, and cheap tripod. I didn't see any shake with the other two cameras, but they were zoomed out. I'll have to use the display on/off feature sparingly. I tried the anti-shake thing in YouTube, which zoomed the picture a little, and didn't work well after it got dark because of the motion blur in the original. It's pretty distracting in a time lapse, but might be useful as a last resort.

http://www.youtube.com/watch?v=beKZsxYiI8A#

http://www.youtube.com/watch?v=aAPbRlsh3ys#

http://www.youtube.com/watch?v=be_T_DyWWLg#

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

lapser

1093

Re: Shot Histogram Request

« Reply #379 on: 28 / August / 2013, 16:04:28 »

Quote from: straessi on 28 / August / 2013, 14:07:18

Not quite sure if that is the way to get to the most recent version. Let me know if there is a better way.

I'll attach the custom build and latest script so you know you've got exactly what I have.

Quote

When it gets dark the exposure changes up and down quite fast, which results to massive flickering.

That sounds like a problem with your build. It happens when the meter calculation is off. If you still have the problem with the attached build, it could be that the port for the G1X 101a isn't switching raw buffers properly. To check this, you could try saving two CHDK raw (dng) files, and see if they are different.
-------
Here's how smoothing and glitch work:

I spent months testing and improving the smoothing routine. I think it's about as perfect as it can be. To understand how it works, I'll use an analogy.

Imagine you are driving down a road. The smoothing parameter is the width of the road (+-32 ev96). If you are within 2 ev96 of the center of the road, the script doesn't change exposure. If you are on the road, but more than 2 from the center, the script will move you 1 ev96 towards the center every few shots (faster if you're moving away from the center). But if you start to go off the edge of the road (32ev96), the script always moves you back to the edge, so you never go off the road. So you will turn very slowly until you get to the edge of the road, then yank the wheel as hard as necessary to stay on the road.

I've found this technique eliminates small oscillations in exposure caused by small inaccuracies in the desired shutter time. You say increase the shutter by 1 ev96, but it actually increases by 1.5 ev96, for example. The next shot is off the other direction and tells you to decrease by 1 ev96. It shows up as "fluttering" in the dark areas of the picture. The smoothing technique makes sure you only change exposure in 1 direction, so it eliminates the problem nicely.

The glitch parameter is much simpler than you're describing. A value of 0 is disabled (script converts it to 999999). This should probably be the default. The first measurement that is more than the glitch value is ignored. The next 5 values are assumed not to be glitches.

I take a lot of my time lapses near a road, with cars going by at night. Glitch prevents the bright, headlight flash in 1 picture from messing up the exposure of the next picture. A lightning flash would do the same thing. If you know you're not going to have sudden exposure changes, set glitch=32 (or try other values). If you're expecting sudden changes, like in a moving car, set glitch=0.

Here's is the code in shot_histogram.c that defines the get_shot_data(n) results depending on n:

Code: [Select]

enum
{
  SHOT_COUNT=0, //1
  EXP_COUNT,    //2
  PRE_DELAY,    //3
  INTERVAL,     //4
  
  HISTO,        //5
  SCRIPT_DELAY, //6
  SHUTTER_TO_RAW, //7
  
  TICK_SHUTTER_READY, //8 tick at start of pre shot delay
  TICK_SHUTTER_OPEN, //9 tick at end of pre shot delay
  
  SHOT_TV96,    //10
  SHOT_AV96,    //11
  SHOT_SV96,    //12
  SHOT_ND,      //13
  
  SHOT_METERS,  //14 meter 1
  //4 METERS HERE //15 16 17 meter 2,3,4
  NUM_SHOT_DATA=SHOT_METERS+NMETERS, 
};

Quote

Last question is about the log file.

Here's a log file I posted somewhere back in this thread that shows what the numbers mean. I should probably print them in every log.
Exposure # is the number of the image file, i.e. IMG_nnnn

pre-shot delay is the time waiting to take the next shot, which is always < the interval time. If it's negative, it shows that you missed the interval, and the camera isn't keeping up with the desired shot rate. This is usually because the shutter time is getting longer than the desired interval, but occasionally writing the log file is slow. I can handle 1 slow log write, but two in a row sometimes overwhelms the buffers and you miss the shot interval for 1 shot. This was another several months of testing to get the timing to work.

Code: [Select]

Sat Mar 30 19:50:19 2013
platform:  g1x 100g
version:  CHDK 1.2.0-2619  built on  Mar 28 2013 18:03:42
30 =t tvmin= -470 10 =m n= 1 ev= 0
1 0 0 100 57
Focus 2787
Result 2762
false =single  Drive mode= 1
Press <MENU> to exit

[EDIT]
1 shot per second, ISO 100 to 1600, 30 sec max shutter time
Meter set to top 57% of raw image, focus at hyperfocal distance

exposure #, pre-shot delay, script time to set exposure, ISO Market,
   unsmoothed EV96 change, smoothed change, battery voltage, shutter time,
   log print time

769 0 0 100 106 106 8103 1/3.41 0
770 0 30 100 -5 -1 8123 1/3.38 10
771 360 20 100 -6 -1 8103 1/3.36 30
772 370 20 100 -7 -1 8123 1/3.33 10
773 370 20 100 -6 -1 8103 1/3.31 10
774 370 20 100 -6 -1 8113 1/3.29 10
775 360 30 100 -7 -1 8103 1/3.26 20
776 370 30 100 -4 -1 8093 1/3.24 20
777 370 20 100 -5 -1 8103 1/3.22 10
778 360 20 100 -5 -1 8103 1/3.19 10
779 370 20 100 -4 -1 8093 1/3.17 10
780 360 20 100 -3 -1 8064 1/3.15 340
781 370 20 100 -4 -1 8093 1/3.12 290
782 350 20 100 -1  0 8103 1/3.12 10
783 360 20 100 -2  0 8113 1/3.12 10
784 360 20 100 -3 -1 8093 1/3.10 10
785 360 20 100 -3 -1 8093 1/3.08 10
786 360 20 100 -3 -1 8103 1/3.06 10
787 360 20 100 -1  0 8093 1/3.06 10
788 350 20 100 -1  0 8084 1/3.06 200
789 350 20 100 -1  0 8103 1/3.06 260
790 350 20 100 -2  0 8093 1/3.06 10
791 350 20 100 -2  0 8093 1/3.06 10
792 350 20 100 -2  0 8103 1/3.06 10
793 350 20 100 -2  0 8093 1/3.06 10
794 350 20 100 -2  0 8093 1/3.06 10
795 350 20 100 -3 -1 8084 1/3.04 20
796 350 20 100 -3 -1 8084 1/3.01 430
797 350 20 100 -4 -1 8093 1/2.99 200
798 350 20 100 -1  0 8093 1/2.99 10
799 340 30 100 -2  0 8093 1/2.99 10
800 340 20 100 -3 -1 8103 1/2.97 10

... snipped to fit in post...

2454 -28680 20 1602 -33 -17 7663 29.7 10
2455 -28680 20 1602 -38 -22 7683 29.7 10
2456 -28680 20 1602 -43 -27 7653 29.7 200
Menu Key Pressed
2457 -28670 20 1602 -47 -31 7653 29.7 10
*** FINISHED ***

When you reach the maximum shutter time and ISO, the exposure setting holds there. You can see this at the end of the log above, where the ISO is stuck at 1602, and the shutter is 29.7, which corresponds to the input parameters of 1600 ISO and 30 seconds max shutter.

Incidentally, on the G1X, using a shutter time of 1/4000 doesn't give consistent exposure between pictures, even without CHDK. It's best to limit it to 1/3200 or less. Using the ND filter is on my to do list, which should help with this problem.

The new script has added functions for the <set> <left>, <right>, and <up> keys. Try them while the script is taking pictures to see what they do. Also, you should exit with <menu>, not abort with <ALT>.

g1x-101a-1.3.0-r3038.zip (282.18 kB - downloaded 15 times.)

TLapser.lua (11.07 kB - downloaded 17 times.)

« Last Edit: 28 / August / 2013, 16:08:20 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos