int shot_histogram_get_range(int histo_from, int histo_to)
// Examines the histogram, and returns the percentage of pixels that
// have luminance between histo_from and histo_to
{
int x, tot, rng ;
tot=0;
rng=0;
//New Code 1
int frng;
frng=histo_to; //<0 flags full rng return
if(histo_to<0)
{
histo_to=-histo_to;
}
//End New Code 1
if(!shot_histogram_isenabled()) // TODO we could return an error somehow
return 0;
for (x = 0 ; x < SHOT_HISTOGRAM_SAMPLES; x ++ )
{
tot += shot_histogram[x];
if (x>=histo_from && x <= histo_to)
{
rng += shot_histogram[x];
}
}
//New Code 2
if(frng<0)
{
return rng;
}
//End New Code 2
return (rng*100)/tot;
}
Or just add a new function - a small code size increase to ensure backward compatability ?That would probably be better. Maybe an optional parameter
Edit: A backwards compatible solution would be to add a new function called 'set_histo_params(x1,y1,x2,y2,precision)'. This would set the histo rectangle and the return value precision for 'get_histo_range'. The defaults if the function is not called would be full image & 100 so existing scripts would not be affected.That sounds perfect. Since you don't know the full range of x and y, they would need to be fractions. 1000 is 1.000, 500 is 0.500 of the full range, for example.
Edit: A backwards compatible solution would be to add a new function called 'set_histo_params(x1,y1,x2,y2,precision)'. This would set the histo rectangle and the return value precision for 'get_histo_range'. The defaults if the function is not called would be full image & 100 so existing scripts would not be affected.That sounds perfect. Since you don't know the full range of x and y, they would need to be fractions. 1000 is 1.000, 500 is 0.500 of the full range, for example.
Will you make it so no precision parameter returns the actual values? The function could also return the total number.
Another thing that would be extremely useful would be a function that returns the average brightness of all the pixels used for the histogram, within the area, in Apex96 units. I think it would just be the average pixel value normalized to bit depth, converted to Apex96 units. It might be called: get_histo_exp96() or get_shot_exp96(). It would be the amount of over or underexposure. That is, if the pixel average is 50% of maximum, then the exp96() value would be 0.
[edit] Would it be possible for set_histo_params to rebuild the histogram (and brightness calculation) each time it's called, so you could get histograms and brightness from different parts of the same image?
CHDK is a collaborative effort - since you have some development experience you can implement something like this if you think it would be useful, and submit a patch for inclusion in the core code.OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge. I really appreciate your input and in depth knowledge of CHDK.
OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge.Its a pretty small group of developers from CHDK beginning to current state when you think about how many Canon P&S cameras are out there in people's purses, pockets and desk drawers and even about how many people have found some pleasure / amusement from using CHDK.
OK, I figured out how to compile with the windows shell. Can you explain the patch and diff files? If I just change the source files, how do I create the patch file to submit for inclusion in CHDK? If you have a link, that would be great, too.OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge.Its a pretty small group of developers from CHDK beginning to current state when you think about how many Canon P&S cameras are out there in people's purses, pockets and desk drawers and even about how many people have found some pleasure / amusement from using CHDK.
I think philmoz was suggesting all help is appreciated .. although debate about coding style, quality, value of features and functions, testing and documentation come with the territory.
OK, I figured out how to compile with the windows shell. Can you explain the patch and diff files? If I just change the source files, how do I create the patch file to submit for inclusion in CHDK? If you have a link, that would be great, too.OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge.Its a pretty small group of developers from CHDK beginning to current state when you think about how many Canon P&S cameras are out there in people's purses, pockets and desk drawers and even about how many people have found some pleasure / amusement from using CHDK.
I think philmoz was suggesting all help is appreciated .. although debate about coding style, quality, value of features and functions, testing and documentation come with the territory.
If you're using Windows then TortoiseSVN is the easiest thing to use.You can also get the 1.1.0 branch using a similiar process.
Create a local repository from the CHDK SVN server.
Install TortoiseSVN then right click somewhere in Windows Explorer and select 'Repo borwser'.
Enter https://tools.assembla.com/svn/chdk as the URL.
When it finishes loading right click on the 'trunk' folder and select 'Checkout' - enter a local directory to store your copy.
The shot histogram is calculated from the RAW sensor data (scaled to 10 bits), and ignores the RGB BAYER pattern of the sensor.I'm trying to use the shot_histogram data to adjust the exposure when taking multiple shots with "shoot_full_only." I know the eV of the first picture, and want to know the change in ev compared to the first image. I'm hoping to be able to compute that from the histogram.
The average value of this data on its own is not a measure of brightness or exposure. The sensor data is linear, you would first need to demosaic the BAYER sensor data then apply a tone curve, and then you would have to factor in the shutter speed, aperture and ISO used to take the image.
Thanks for the info. I noticed that CHDK shell has a button for "source code tools" which opens a window with diff and patch options. Do you have any information on what that does?
The shot histogram is calculated from the RAW sensor data (scaled to 10 bits), and ignores the RGB BAYER pattern of the sensor.I'm trying to use the shot_histogram data to adjust the exposure when taking multiple shots with "shoot_full_only." I know the eV of the first picture, and want to know the change in ev compared to the first image. I'm hoping to be able to compute that from the histogram.
The average value of this data on its own is not a measure of brightness or exposure. The sensor data is linear, you would first need to demosaic the BAYER sensor data then apply a tone curve, and then you would have to factor in the shutter speed, aperture and ISO used to take the image.
Now that I can get accurate values for all 1024 histogram points, I can experiment in Lua to see how that relates to eV. I'm thinking I can choose an area, say the upper left, that contains only sky. As the sun goes down, the sky gets darker and the average of the histogram data decreases. I'll try to relate that to eV. I'm not trying to set camera exposure from scratch, just sense slow lighting changes and slowly change exposure to compensate.
I noticed that the histogram ignores RGB and steps by 31. I assume it's sampling R, G, and B pixels alternately that way? Wouldn't averaging a large enough area be related to luminosity, especially if it was clear sky?
I never did figure out a way to use CHDK-Shell and Tortoise SVN together - CHDK-Shell likes to do things its own way.It's not a big deal. You need only observe the subdir structures of the CHDK shell.
I noticed that CHDK shell has a button for "source code tools" which opens a window with diff and patch options. Do you have any information on what that does?In the shell you can see a big number of the revision. Click left beside this number and you can create a copy of the current source code. This is your working copy. In 'source code tools' you can make a patch file with the option 'diff'. You can choose a reference. With the option 'patch' you can insert patch files in the current source code.
If you use the "DOS box" icon in CHDK-Shell, the banner text tells you what to type in to build a particular camera & firmware version. Using that, you can change to wherever your SVN trunk is locate and compile.I never did figure out a way to use CHDK-Shell and Tortoise SVN together - CHDK-Shell likes to do things its own way.It's not a big deal. You need only observe the subdir structures of the CHDK shell.
The diff function works but using Tortoise SVN allows you to create patch files that are 100% usable by the devs. I only played with diff for a bit but was never able to really get the same compatibility.I noticed that CHDK shell has a button for "source code tools" which opens a window with diff and patch options. Do you have any information on what that does?In the shell you can see a big number of the revision. Click left beside this number and you can create a copy of the current source code. This is your working copy. In 'source code tools' you can make a patch file with the option 'diff'. You can choose a reference. With the option 'patch' you can insert patch files in the current source code.
If you use the "DOS box" icon in CHDK-Shell, the banner text tells you what to type in to build a particular camera & firmware version. Using that, you can change to wherever your SVN trunk is locate and compileThat's another way but not my way. I'm not the command line junkie. ;)
Have a look at the 'get_bv96' Lua function - it may give you what you need.get_bv96() will give me a valid value for the first shot, but the rest of the shots are done with "shoot_full_only" without releasing "half_shoot". If I also compute the average of the histogram data for the first shot and save it, I have an eV reference point. If the average pixel value of subsequent shots goes above this reference, I know I'm over exposed, and can reduce exposure in the next shot.
static int luaCB_shot_histo_enable( lua_State* L )
{
shot_histogram_set(luaL_checknumber( L, 1 ));
return 0;
}
I want to change shot_histo_enable(1) to shot_histo_enable(1,x1,y1,x2,y2) where the new parameters are optional. How do I tell if the number of parameters is 1 or 5?
It looks like the total number of pixel samples can be returned by shot_histogram_set, and this value returned by Lua without breaking any old scripts. Is this correct?
lua_pushnumber( L, total )
coord x1 =luaL_optnumber(L,2,0);
coord y1 =luaL_optnumber(L,3,0);
coord x2 =luaL_optnumber(L,4,100);
coord y2 =luaL_optnumber(L,5,100);
Do you know if build_shot_histogram() is called right after each picture, regardless of whether or not shot_histo is enabled? Thanks.
build_shot_histogram is only called if a script is running and shot_histogram_isenabled returns true.Thanks. I was able to add shot_ready() and test it, and it didn't work right anyway. It was a good exercise in learning to add Lua functions though.
Use luaL_optnumber, third parameter is default value if the parameter is not included.Thanks again Phil. You're a great resource. I'll give it a try.
step=64
for i=0,1023,step do
hrange=get_histo_range(i,i+step)
hfile:write(string.format("%5d",hrange))
end
hfile:write("\n")
for i=0,1023,step do
hrange=get_histo_range(i,i+step,1000)
hfile:write(string.format("%5d",hrange))
end
hfile:write("\n")
for i=0,1023,step do
hrange=get_histo_range(i,i+step,0)
hfile:write(string.format("%5d",hrange))
end
It produces this output:From looking at the Bayer filter pictures, it looks likes if I use an odd number step size when sampling the raw pixels, I'll alternate between 2 colors and get a better estimate of luminace.
I'm planning to add up all the pixel values and return the average as a 2nd parameter from get_histo_range, possibly converted to an APEX96 value if I can figure it out. A brightness of 50% of full white will return an APEX96 value of 0.
I'm hoping to be able to use the average to set the shutter speed for the next shot in a time lapse. Since I need to enter an Apex96 value for shutter speed (and ISO), it would be best if the value I return is also Apex96. The absolute value of the average won't mean much, but how it changes between shots will tell me how to adjust the exposure for the next shot. It will take some experimenting to figure it out. I added a new function called:QuoteI'm planning to add up all the pixel values and return the average as a 2nd parameter from get_histo_range, possibly converted to an APEX96 value if I can figure it out. A brightness of 50% of full white will return an APEX96 value of 0.What does the average of the raw sensor pixels tell you?
Why would an APEX number be any better than a value from 0 -1023?
Phil.
ev+- | deltaHSUM96
-192 0
-160 -33
-128 -29
-96 -29
-64 -29
-32 -27
0 -27
32 -26
64 -23
96 -23
128 -22
160 -18
192 -19
-192 304
-160 -30
-128 -31
-96 -28
-64 -29
-32 -29
0 -25
32 -26
64 -25
96 -21
128 -21
160 -20
192 -16
-192 301
-160 -30
-128 -26
-96 -31
-64 -30
-32 -26
0 -27
32 -26
64 -23
96 -24
128 -22
160 -18
192 -18
-192 300
-160 -28
-128 -31
-96 -30
-64 -27
-32 -29
0 -27
32 -24
64 -25
96 -23
128 -21
160 -20
192 -17
-192 303
-160 -30
-128 -31
-96 -27
-64 -29
-32 -28
0 -24
32 -27
64 -25
96 -21
128 -22
160 -20
192 -17
Fri Nov 30 18:09:04 2012
-192 0
-160 -32
-128 -31
-96 -27
-64 -30
-32 -28
0 -25
32 -26
64 -25
96 -20
128 -22
160 -19
192 -17
Fri Nov 30 18:09:29 2012
-192 0
-160 -28
-128 -30
-96 -31
-64 -26
-32 -28
0 -27
32 -25
64 -25
96 -23
128 -19
160 -20
192 -19
-192 301
-160 -28
-128 -31
-96 -30
-64 -27
-32 -27
0 -28
32 -23
64 -25
96 -23
128 -20
160 -20
192 -19
ev+- | deltaHSUM96
-192 0
-160 -29
-128 -29
-96 -30
-64 -32
-32 -29
0 -32
32 -32
64 -29
96 -32
128 -32
160 -30
192 -32
-192 366
-160 -32
-128 -29
-96 -31
-64 -32
-32 -30
0 -32
32 -32
64 -30
96 -32
128 -33
160 -30
192 -32
-192 375
-160 -32
-128 -30
-96 -32
-64 -33
-32 -29
0 -32
32 -33
64 -30
96 -32
128 -32
160 -29
192 -33
-192 378
-160 -32
-128 -33
-96 -29
-64 -33
-32 -33
0 -29
32 -32
64 -32
96 -30
128 -32
160 -32
192 -30
ev+- | deltaHSUM96
-192 0
-160 -29
-128 -28
-96 -32
-64 -32
-32 -29
0 -32
32 -33
64 -30
96 -31
128 -32
160 -29
192 -33 (series average -30.8 )
-192 368
-160 -33
-128 -29
-96 -32
-64 -33
-32 -28
0 -33
32 -33
64 -29
96 -32
128 -33
160 -30
192 -32 (series average -31.4)
-192 376
-160 -32
-128 -29
-96 -32
-64 -33
-32 -29
0 -33
32 -32
64 -30
96 -32
128 -33
160 -30
192 -31 (series average -31.3)
-192 376
-160 -30
-128 -32
-96 -33
-64 -30
-32 -32
0 -33
32 -29
64 -32
96 -33
128 -30
160 -33
192 -32 (series average -31.6)
-192 379
-160 -33
-128 -33
-96 -36
-64 -32
-32 -28
0 -37
32 -32
64 -29
96 -32
128 -28
160 -28
192 -33 (series average -32.6)
(average of all -31.5)
ev+- | deltaHSUM96
-192 0
-160 -31
-128 -34
-96 -33
-64 -29
-32 -34
0 -33
32 -30
64 -34
96 -32
128 -30
160 -33
192 -33
average=32.166
-192 384
-160 -32
-128 -34
-96 -29
-64 -34
-32 -33
0 -30
32 -32
64 -34
96 -30
128 -32
160 -33
192 -31
average=32.00
-192 382
-160 -30
-128 -33
-96 -33
-64 -30
-32 -34
0 -32
32 -30
64 -33
96 -34
128 -30
160 -32
192 -34
average=32.08
-192 385
-160 -30
-128 -32
-96 -34
-64 -29
-32 -34
0 -33
32 -30
64 -33
96 -33
128 -30
160 -33
192 -33
average=32.00
-192 385
-160 -33
-128 -34
-96 -30
-64 -26
-32 -35
0 -31
32 -33
64 -32
96 -30
128 -34
160 -32
192 -31
average=31.75
overall average=31.99
I'm also assuming your using the existing logic in build_shot_histogram that only uses the top 10 bits of each raw value.I'm adding the sum of the pixel values before the bit shift, as I said in the other topic (I see what you mean, Waterwingz. You're right as usual.)
So your logic will only work if the dynamic range of the sensor is close to 10 f/stops.
--[[
@title New Time Lapse
@param m = interval
@default 3
--]]
function restore()
release("shoot_half")
hfile:close()
shot_histo_enable(0)
end
function press_half()
press("shoot_half")
repeat sleep(50) until get_shooting()
end
function release_half()
release("shoot_half")
while(get_shooting())do sleep(50) end
end
nkey=6
keys={"set","menu","left","right","up","down"}
function keypress(k)
repeat
for i=1,nkey do
if(is_pressed(keys[i])) then
if(k==nil or k==keys[i]) then return keys[i] end
end
end
sleep(50)
until false
end
function nopress()
repeat
none=true
for i=1,nkey do
if(is_pressed(keys[i])) then
none=false
sleep(50)
end
end
until none
end
fname="A/CHDK/LOGS/H_"..os.date("%y%m%d%H%M")..".LOG"
props=require("propcase")
pTV=props.TV
pTV2=props.TV2
props=nil
shot_histo_enable(1,3,3,50,25) --upper left
hfile=io.open(fname,"wb")
hfile:write(string.format("%s\n",os.date()))
print("press set / menu exits")
done=keypress()=="menu"
press_half()
hfile:write(string.format("tv0=%d\n pic tv tvd\n",tv))
while(not done)do
get_histo_ev96rel(2) --reset building histo flag
press("shoot_full_only")
repeat
sleep(100)
until get_histo_ev96rel(3)==0 -- building histo flag set to 0 when building is done
release("shoot_full_only")
sleep(500)
tvd=get_histo_ev96rel() -- get exposure difference of this shot and reference shot
tv=tv+tvd -- sets exposure of next shot based on the reference shot (first shot after histo_enable)
if(tv<0)then tv=0 end
set_prop(pTV,tv)
set_prop(pTV2,tv)
sleep(500)
nxp=get_exp_count()
hfile:write(string.format("%d %d %d\n",nxp,tv,tvd))
done=is_pressed("menu")
end
restore()
Note that the call to: shot_histo_enable(1,3,3,50,25) sets the measurement area to the upper left corner. It also sets a flag to make the next shot the reference shot. The output of get_ev96rel() is relative to the reference shot. The first value for the first picture is always 0, since that IS the reference shot.
But I don't think a histogram of the previous shot is useful.WRONG! I realized that I can find a relative exposure change from the full histogram based on a criteria like:
shot_meter_enable(1,0,0,25,25)
shot_meter_enable(2,75,0,100,25)
shot_meter_enable(3,0,75,25,100)
shot_meter_enable(4,75,75,100,100)
shoot()
But you don't want to let Lua just set a delay. If you set a delay of 10 minutes by mistake, the camera would hang for 10 minutes. Not good.There are lots of ways to hang the camera while debugging a script. This would just be one more that needed to be debugged and fixed. I wouldn't waste any time trying to protect the programmer from this sort of mistake.
There are lots of ways to hang the camera while debugging a script. This would just be one more that needed to be debugged and fixed. I wouldn't waste any time trying to protect the programmer from this sort of mistake.That's a good point. Thanks for the input. The idea is also to make it easy to program.
interval=2000 -- for 2 seconds
set_shot_interval(interval)
press("shoot_full") -- in continuous mode
repeat
nshot=get_shot_number()
repeat sleep(10) until get_shot_number~=nshot
-- set new exposure levels here using get_shot_meter()
until false
When the script is finished, CHDK would call set_shot_interval(0), like it does for shot_histogram. This would prevent a script from disabling the entire camera, and reset everything back to normal if you interrupt a script. The above script would work this way.The interval delay should happen right before triggering a new shot (not in raw_save_file(). It would be good if the timing accuracy could be better than the 10 or 20 msec you get from get_tick_count(). I'll see if I can figure out where to put the new code, and how to do the timing from looking through the source files. If someone wants to help point me in the right direction, that would be nice. Anyone? . . . Anyone? . . . Philmoz?
10 msec is the best accuracy you can get - there aren't any higher resolution timers that I know and that's the time interval for task switching.Thanks Phil. I found something that looks interesting:
10 msec is the best accuracy you can get - there aren't any higher resolution timers that I know and that's the time interval for task switching.Thanks Phil. I found something that looks interesting:
extern volatile long shutter_open_tick_count; // defined in platform/generic/capt_seq.c
I just tested it, and it does indeed contain a value that appears to be the time the shutter opened.
However, I can probably get away with simply delaying in raw_save_file based on the the time of the last call to raw_save_file, and not bother with anything else. That's probably the best solution, and plenty accurate. The per shot time should remain accurate, which is the main thing. Correcting for the shutter open time probably won't be necessary, although I'll have to think about that a little more.
As for the accuracy issue, take a look at line #321 in main.c. It calculates a value for:
camera_info.tick_count_offset
Can you explain what that is? It's used in dng.c at lines #567 and #576
And on a related issue, I thought I read that "long" was just a 32 bit "int" in CHDK. Is this correct? I hope not.
Thanks again.
The camera_info.tick_count_offset value is used to set the msec part of the image capture time in the DNG header. It's only accurate to the same 10msec range as everything else.Thanks! I like to explore all the options, and pick the simplest one. I appreciate your patience in answering so many questions.
The size of long and int are both 32 bits.
...in raw_save_file()
if (state_kbd_script_run)
{
if(shot_histogram_isenabled())build_shot_histogram();
while((shot_interval>0)&&(get_tick_count()<shot_next))msleep(10); //set_shot_interval(0) aborts
shot_next=get_tick_count()+shot_interval;
}
shot_number++; //polled by Lua get_shot_count()
...
int raw_get_shot_number() // called from Lua - increments after shot histo is ready
{
return shot_number; //incremented in raw_save_file() ... different thread
}
void raw_set_shot_interval(int interval) //called from Lua ... sets minimum interval
{
shot_interval=interval-5; //subtract 5 for rounding to 10msec sleep time
}
...
static int shot_number=0; //incremented after build_shot_histogram()
static int shot_interval=0; //msec .. minimum time between shots .. works in continuous mode
static int shot_next=0; //minimum tick time for next shot . . delay until this time
I guess I'll have to take a few thousand more pictures of my blank wall!
Also I think a blank wall is an unrealistically low dynamic range to test with.You probably need to spend several hundred hours of your time working on CHDK development to understand what lapser meant. Reyalp's keyboard, the oak tree outside my window and a lone koala bear might be the most photographed things in history.
Also I think a blank wall is an unrealistically low dynamic range to test with. I've also graphed a sunrise with a light meter, and it obviously goes over a huge range very quickly.Actually, you're correct that a blank wall wouldn't work for the problem I stated, which is, too many pixels blown out to get an accurate exposure correction for the next shot. The solution isn't all that complicated, though, as you suggested. When you have a lot of blown pixels, give up on getting the right exposure in the next shot. Just decrease by 6 or 8 EV instead of 3. Once you find a spot where the pixels are in the linear range, not too many totally dark or blown pixels, you can hit the right exposure on the next shot. I got to it eventually (in 3 to 5 shots) with 3 EV steps.
OK, first of all, I changed my mind (again) on the size of the histogram. It's now 2K, would go up to 8K for 12 bit cameras, and 32K for the 14-bit G1X. I just can't bring myself to bit shift away all the extra precision of 12 and 14 bit cameras for a measly 8K or 32K memory savings. Activating a histogram is optional, but if you want it, you want it all the way.What application actually needs this precision ? 32K is actually quite a lot in CHDK land...
OK, first of all, I changed my mind (again) on the size of the histogram. It's now 2K, would go up to 8K for 12 bit cameras, and 32K for the 14-bit G1X. I just can't bring myself to bit shift away all the extra precision of 12 and 14 bit cameras for a measly 8K or 32K memory savings. Activating a histogram is optional, but if you want it, you want it all the way.What application actually needs this precision ? 32K is actually quite a lot in CHDK land...
If you are calculating exposures, it seems to me the discarded precision is going to be a tiny fraction of a stop, utterly meaningless from a photographic perspective.
What application actually needs this precision ? 32K is actually quite a lot in CHDK land...That's what I was thinking when I decided to stay with 10 bits and 2K. But it gets more significant at low light levels. One of the reasons I wanted to try metering the shot in the raw buffer was that the standard meter started jumping by 1/2 Fstop when it got real dark. My time lapses were flashing noticeably.
If you are calculating exposures, it seems to me the discarded precision is going to be a tiny fraction of a stop, utterly meaningless from a photographic perspective.
The G1X has 2.5MB free memory without using exmem so should be no problem using 32K extra.Thanks Phil. I appreciate you putting in your two bits worth. :) And I promise to bit shift the 16 bit cameras. Or maybe . . . ??
void __attribute__((naked,noinline)) capt_seq_hook_set_nr()
{
asm volatile("STMFD SP!, {R0-R12,LR}\n");
switch (core_get_noise_reduction_value()){
case NOISE_REDUCTION_AUTO_CANON:
// leave it alone
...
raw_shot_delay(); //wait delay interval, if any, before continuing shot ******************
shutter_open_time=_time((void*)0);
shutter_open_tick_count = get_tick_count();
asm volatile("LDMFD SP!, {R0-R12,PC}\n");
}I also added a double shot option to use with shot metering in continuous mode. It's activated with set_shot_interval(-2). After this, every other shot will have no delay.Making too many changes now. See the documentation in the next post for more info on what's coming soon.Lapser !
I'll be back in two weeks.Good one! Have a nice trip, assuming you're tripping!
Let me know how this all worked out ? Do the drugs wear off after a while ?
set_shot_meter(1) -- enable meter #1 with full sensor measurement area
set_shot_interval(0,500) -- no pre-shot delay, 1/2 second (max) post shot delay
-- if you want your shots at 1 per second: set_shot_interval(1000,500)
ishot=0 -- get_shot_number() always starts at 0 now
press("shoot_full")
repeat
nshot=ishot
repeat
sleep(10)
ishot=get_shot_number()
until ishot~=nshot --signals new shot is ready
-- post shot delay starts here
if(nshot==0)then tv=get_tv96_direct() end-- metered by camera on first shot
tv=tv+get_shot_meter(1) -- adjust exposure time
-- more processing here if needed, i.e. write log file
set_tv96_direct(tv) -- happens immediately now instead of next half shoot
set_shot_interval(-1) -- abort post shot delay. Camera takes next shot as fast as possible now
until false
An unexpected bonus from using this method is that the pictures are displayed on the screen (even with review off). In other shooting modes, the live view is displayed, and the screen goes blank during the exposure. At night, with long exposures, all you see is the screen flash between shots. But with this method, the screen shows the last shot for the entire duration of the long exposure. This is a big improvement.
Rather than releasing new diff files, it might be easier to send them to philmoz to put in his branch?@lapser : I've scanned as many of your threads as I could find looking for patch files. I have a few moments and I thought I'd like to try out some of the tricks you've learned. But the only one I could find is the recent one in the "Improvements to press, click, release for shooting keys" tread and that's not what I'm looking for.
Did you ever pull together anything on your histogram metering & continuous shooting sync / timing stuff ?I did get a little side tracked on the key function changes. I finally found that all day bug, and have it working.
lapser, I absolutely love your work mate, I would like to go thank everyone in this forum/community but not to be annoying...I won't.Thanks for the comments. I'm looking forward to your input. Let me know if there's anything I can do to help, and don't be afraid to ask stupid questions. Hopefully, I'll have something ready for you to try out when you get your CHDK camera going. Good luck.
I have followed your work now for a while at the same time as trying to get my head around using and working with CHDK. The things you are working on will come in very handy for me one day, that is when I get a few P&S's(already ordered) and see what I can't blow up...I mean figure out! :) Going to be a steep learning curve for me but one day I hope I can help out some and give back something!
Great timelapse work. I see you use LR. I also use LR for my timelapse work, but with LRTimelase. I find LRT to be the ideal adjunct to LR. Have you looked at LRT?Thanks! The higher speed shot interval really helps.
Also how do you use your ETTR calls?
@lapser : Its been a couple of week, time for a "bump" on this one? Unless I missed it in one of your (many) other threads ;)Did you ever pull together anything on your histogram metering & continuous shooting sync / timing stuff ?I did get a little side tracked on the key function changes. I finally found that all day bug, and have it working. I'll make it the top priority to get a new diff file out, and an understandable demo Lua script. I'll post it here as soon as possible. Thanks for your interest, and your help.
@lapser : Its been a couple of week, time for a "bump" on this one? Unless I missed it in one of your (many) other threads ;)Point taken! It would be easier for me to compile a version for your camera and tidy up the documentation for the functions that are working so far so you can start playing with them. Would that be acceptable?
Anyway, let me know your camera (or cameras) model and firmware revision, and I'll post something for you today.Thanks, but I tend to rebuild daily (to test my changes and those posted by others) so having a custom build is not a lot of use. A patch file would help - even if it won't apply immediately I can probaly tweak the parts I need for the current build and then just carry those changes forward myself.
I've updated and attached the documentation for the time lapse modifications as they exist at the current moment.Thanks again but without a patch file to work with having documentation for functions but no source code doesn't do me a lot of good.
Thanks, but I tend to rebuild daily (to test my changes and those posted by others) so having a custom build is not a lot of use. A patch file would help - even if it won't apply immediately I can probaly tweak the parts I need for the current build and then just carry those changes forward myself.OK, when I have a more stable patch file ready, I'll post it and notify you. Thanks for offering to test it.
OK, here's the current diff file.Thanks - wasn't really expecting you to take time out to do this. I'll play with it this week after I get over the shock of doing the first pass through my 2012 taxes.
I assume you did no post processing deflickering? In other words your script ensures a sufficiently smooth transition during the day to night shooting? Do you do any ISO changes, or do you restrict yourself to only Tv changes?I made the video with the free program, Videomach, without processing the pictures. In fact, I left all the photos on the SD card, and output only the video file to the computer. Each picture is a frame. Using the shot meters to adjust exposure in 1/96 ev increments, combined with the high shot rate makes a very smooth time lapse, plus a very easy post production process. I think I tried auto white balance this time. That's where LRTimelapse would come in handy, because CHDK can't adjust white balance (yet).
Also, would you be prepared to share your script? I find looking at other scripts is a great way of learning more about CHDL/LUA scripting. I am still a novice 'scripter', but wishing to learn more.My creative process is pretty messy, but I'll see if I can clean up the script a little and post it for you soon.
Finally, are all the functions you use in the version you previously sent me for my S95 and G11? In other words the exposure adjustment calls?I've been changing things pretty fast, so it might be better to start with new builds. I'll post those with the script for you when I get something ready.
--[[
@title Clapse
--]]
--MUST BE IN CONTINUOUS MODE
press("shoot_half")
repeat sleep(10) until get_shooting()
print("Holding shoot_full")
print("Press <menu> to exit")
press("shoot_full_only") -- continuous mode
repeat
sleep(100)
until is_pressed("menu")
The second time I run it in continuous mode, it does NOT auto focus, although it does shoot continuously. Also, it seems to only focus with shoot_full_only, and not with just shoot_full.The G1X has a continuous drive mode with auto-focus that returns a drive mode of 2. Using a script, though, it doesn't always auto-focus. The first time I run this script, it does auto-focus:Code: (lua) [Select]--[[The second time I run it in continuous mode, it does NOT auto focus, although it does shoot continuously.
@title Clapse
--]]
--MUST BE IN CONTINUOUS MODE
press("shoot_half")
repeat sleep(10) until get_shooting()
print("Holding shoot_full")
print("Press <menu> to exit")
press("shoot_full_only") -- continuous mode
repeat
sleep(100)
until is_pressed("menu")
Works fine on my G1X.Thanks Phil. I reformatted my SD card and started from scratch with the full version. The script worked before and after I applied my time lapse patches, so you were right (as usual).
Possibly some other camera or CHDK setting might be interfering.
Try deleting or renaming your CCHDK2.CFG or CCHDK3.CFG file (depends on version you are running) and using the unmodified trunk or release version.
The sx260 halted with an unspecified Lua error at about 1500 shots, giving only the "SCRIPT TERMINATED" message. This has happened before on both cameras while writing a log file, but this time I wasn't writing a log, so it must be related to something else.
There isn't anything in CHDK that would print "SCRIPT TERMINATED" - did you mean the "*** TERMINATED ***" message when there is an error in the script?Yes, it said "*** TERMINATED ***", with a large number like, 65094782 or something like that. Unfortunately, I lost the log file when I restarted the script. I'll have to use the print_screen(-1) option next time to append the log. I'll see if I can reproduce it and let you know the exact number if I can.
If so it should have displayed an error message as well - what did that say?
set_shot_interval(m,n)
if(not single)then press("shoot_full_only") end
repeat
if(single)then press("shoot_full_only") end
repeat
sleeper(10) -- sleeps at least once, so it will yield each loop
until get_shot_ready() or is_pressed("menu") -- waits for build_histogram --
if(is_pressed("menu"))then break end
if(single)then release("shoot_full_only") end
--find meter with lowest exposure (brightest)
tvd=-100000
for i=1,nmeters do
tx=get_shot_meter(i)
if(tvd<tx)then
tvd=tx
im=i
end
end
--adjust bv as it gets darker
bv=tvd+get_shot_meter(0) --bv96
if(bv>=bv1)then bvd=0
elseif(bv<=bv2)then bvd=bvdmax
else bvd=((bv-bv1)*bvdmax)/(bv2-bv1) end
tv=tv+tvd+bvd+svmin-sv
sv=svmin
if(tv<tvs)then
sv=sv+tvs-tv
tv=tvs
if(sv>svmax)then
tv=tv-(sv-svmax)
sv=svmax
end
end
if(tv<tvmin)then tv=tvmin
elseif(tv>tvmax)then tv=tvmax end
set_exp96(tv,sv) -- aborts script wait tooIt's very unlikely this is a deliberate error messageI'm pretty sure it's related to the script accessing variables as they're being changed by build_shot_histogram() in a different thread. I added a busy flag to build_shot_histogram(), so the script thread waits until it's finished. I don't have a busy flag for the script call to get_shot_meter(), so that could be the problem. I'll have to be absolutely certain both busy flags are never true at the same time, or that would hang both threads.
I'm pretty sure it's related to the script accessing variables as they're being changed by build_shot_histogram() in a different thread.Having not looked at the code, I don't know. However, the only way I could see this causing the behavior you describe is if your code got the values and did something with them that corrupted the Lua state or other memory. Lua isn't going to care that C code in one task is reading variables that are "out of date" with respect to some other task.
I'm thinking it would be best to make both flags "volatile" so I know the values are always current. Does that sound like a good idea?I prefer something like
static int some_busy_flag;
int get_some_busy_flag() {return some_busy_flag;}
Because So the problem isn't reading the busy flag, it's making sure it is stored immediately in build_shot_histogram().Code: [Select]static int some_busy_flag;
int get_some_busy_flag() {return some_busy_flag;}
//this is what I have right now
static int hbusy; //change to volatile?
void build_shot_histogram()
{
hbusy=1;
//calculations that take about 30msec
hbusy=0;
}
int get_shot_meter()
{
while(hbusy)msleep(10);
//calculations, with call to log10(x)
}
result= //calculations
if(result>0)result--;
if(result<0)result++;
return result;Anyway, I'm getting a little closer to an answer, I hope. If anyone has an idea how a multi-threading problem could cause a script interrupted error at this point in the shooting cycle, please let me know.I don't really see how you concluded this is a "multi-threading" problem. The script stuff all runs in one task. Other threads doing stuff won't cause it to randomly die, unless they are corrupting memory or something like that.
The error message is "set" the last 3 times it happened. Here's the end of the last log file:As I said before, this isn't an error message in any normal sense. It's garbage, presumably left around due to the is_pressed("set") in your code.
I think core_spytask() is calling is_pressed() too, so it might be possible that is_pressed() is being called from different threads at the same time.This would not be a problem.
As I said before, this isn't an error message in any normal sense. It's garbage, presumably left around due to the is_pressed("set") in your code.That's a help, thanks.
I switched to the new wait_click(10) in my loop waiting for get_shot_ready() instead of using 3 is_press() calls with "set", "menu", and "display". I was getting script interrupted errors once in awhile, but did not have one with the new wait_click until last night. Instead of a "set" error, I got a "NULL" error with the sx260.Was the error "ERROR:NULL error message" ?
Was the error "ERROR:NULL error message" ?Yes, that's the one. I've been able to reproduce the bug once a night on the sx260, and I'm tracking it down slowly but surely. The bug happens when lua calls wait_click(10) or sleep(10). I set a variable to a different number in the C code for each function that my script calls, and sleep(10) is the culprit with this in the script:
The bug occurs when lua is in a yield / resume loop. Something about the timing causes an error with resume.You know this how?
I set a variable, "lapser1", to a different value in every luascript.c function that my script calls. The error only occurs in the sleep() function that is called repeatedly waiting for get_shot_ready() to signal that the shot histogram is done.The bug occurs when lua is in a yield / resume loop. Something about the timing causes an error with resume.You know this how?
The bizarre thing is that luaD_rawrunprotected doesn't look like it is creating an error status.Could this happen with a hardware interrupt inside an internal lua function that isn't thread safe?
Could this happen with a hardware interrupt inside an internal lua function that isn't thread safe?A hardware interrupt should be completely transparent to Lua, just like any other vanilla C code. I don't see how thread safety wouldn't come into it, the Lua code is single threaded since it executes entirely in kbd_task.
Do you know how the shutter open/close is timed?No. Interrupts are probably involved, see http://chdk.setepontos.com/index.php?topic=8312.msg87949#msg87949 (http://chdk.setepontos.com/index.php?topic=8312.msg87949#msg87949)
Maybe there's an extra interrupt timer that generates an extra interrupt when its time to close the shutter?I don't still don't see how interrupts would be involved. But then, I don't have a good theory of how this happens... or a bad one, for that matter :-[
If I trigger the bug again, can you describe how to do a memory dump?If you want to dump the entire camera RAM, see main.c dump_memory() (used by the debug menu memory dump option)
status = luaD_rawrunprotected(L, resume, L->top - nargs);
if (status != 0) { /* error? */
+ if(status != LUA_YIELD) {
+ dump_memory();
+ }
L->status = cast_byte(status); /* mark thread as `dead' */
in ldo.c lua_resumeThe combination of the debug hook yield and sleep yield also seems like it might be worth further investigation. One way to do this might be to make the debug hook yield more frequently and see if the problem happens more often.Except the bug still happened with:
If this problem is in fact the problem encountered earlier, it is not strictly tied to shooting. The fact that you get the NULL error message error suggests that it is the same, or at least closely related.Actually, the null error message was when I was using wait_click(10) for my delay. With sleep(10), the error message is the last string passed to is_key().
If the bug happens again, assuming I can restrain the urge to throw my camera against the wall, I'll look more at the gui_handler stuff. Is that called from the keyboard task?Yes, it should be.
I'm wondering if script_gui_handler could be generating a spurious shoot_full click that's interrupting the script when it's not in the right state or something. If get_clicked_key() or similar is called outside the keyboard task, that could cause trouble.If the script were somehow being interrupted by a spurious shoot click, then you wouldn't get the bogus or null error messages. It would just act like you had pressed the shutter. When the shutter is pressed, it triggers kbd_task to kill the Lua environment *outside* of the Lua code (meaning lua has already returned from lua_resume, and probably waited for the next iteration of kbd_task)
I think it's using what was on the stack before the yield as the error. It sounds like a stack error of some kind.The error comes form the lua state stack, not the C stack. Lua is dying in some way that returns an error (edit: I mean, a code path that would normally be an error), but fails to put an error on the (edit: Lua) stack.
The next step is to see if the bug happens with the yield hook enabled. For a good test, I'll try set_yield(1,10), for a yield every 200 lua operations.You could also lower the multiplier in the code, YIELD_CHECK_COUNT in luascript.c
Could I end wait_shot_ready() with:I have no idea where wait_shot_ready is or what it does.
return lua_yield( L, 1 ); ??
Then, I would use lua_pushnumber() from the action stack routine when it finishes?I think you are supposed to be able to do this, but if you haven't already you should take a look at the lua documentation for yield and resume. http://www.lua.org/manual/5.1/manual.html#lua_yield (http://www.lua.org/manual/5.1/manual.html#lua_yield)
Speaking of action stacks, maybe that could be causing this lua problem. There's a lot of pushing and popping going on with a function like sleep(10). I haven't gone through all the action stack logic, but it might be worth it to follow the code for a sleep(10) call from start to finish.I don't understand how this could be connected.
You might want to look at read_usb_msg() which does something similar.That's exactly what I was looking for. Thanks!
I got all 3 cameras to take around 5,000 pictures each through an entire sunset, without a script interrupt. Hurrah!
This is with wait_shot_ready() and set_yield(-1,-1). The script actually works better with the yield hook disabled. I haven't found the cause of the bug, but at least I have a way to work around it now, and make sure I get a complete time lapse.
The next step is to see if the bug happens with the yield hook enabled. For a good test, I'll try set_yield(1,10), for a yield every 200 lua operations. My guess is that the yield hook won't trigger the bug, and that it comes from the transition from Lua to a C subroutine getting the stack wrong or something. My wait_shot_ready() has no parameters. Maybe I'll try it with a dummy parameter and see if that causes trouble.
--[[
@title Yield Test
@param y yield hook?
@default y 0
--]]
--set_console_layout(0,0,45,14) -- full screen
if(y==0)then set_yield(-1,-1) end
tick0=get_tick_count()
n=1
repeat
coroutine.yield()
tick=get_tick_count()
print(n,tick-tick0)
tick0=tick
n=n+1
until is_pressed("menu")
set_yield()
It looks like coroutine.yield() is the same as sleep(10), but without the overhead and bug risk of a C yield. repeat
wait_click(1) -- just checks for key, no delay or yield
if(is_key("set"))then
bl=bitxor(bl,1) -- 0 or 1
set_backlight(bl)
elseif(is_key("display")) then
nd=bitxor(nd,3) -- 1 or 2 for nd filter
end
done=is_key("menu")
coroutine.yield() -- same as sleep(10) but without C yield bug (hopefully)
until get_shot_ready() or done
if(done) then break end -- script over
This is something exciting that I just discovered after looking at the Lua docs for yield (thanks reyalp). I always assumed that sleep(10) was the only way to yield, but a better way is:What do you think coroutine.yield does under the hood?
coroutine.yield()
This eliminates the call to a C function from Lua, which yields from C, and may be the cause of the bug. It seems much safer to yield from Lua.
What do you think coroutine.yield does under the hood?Well, I guess everything is C under the hood. I meant it yields from code written by Lua programmers instead of us. At least it doesn't start a new action stack with a delay. It just yields. I think it will work, but we'll C.
static int luaB_yield (lua_State *L) {
return lua_yield(L, lua_gettop(L));
}
Well, I guess everything is C under the hood. I meant it yields from code written by Lua programmers instead of us.It's a C function just like ours, which calls exactly the same lua API function that sleep() etc call. (luaB_yield is registered in lua as coroutine.yield)
At least it doesn't start a new action stack with a delay. It just yields. I think it will work, but we'll C.Huh?
If it doesn't put something on the action stack, then you will just yield for one iteration of kdb_task, which is what sleep(0) does now in the trunk, because AS_SCRIPT_RUN will still be on top of the stack. Unless you've butchered the script action stack handling in some other way...Shouldn't sleep(<=10) also just yield() without action stacking?
//kbd.c
long kbd_process()
{
static int key_pressed;
if(camera_info.state.backlight_off)
{
if (camera_info.state.state_kbd_script_run)TurnOffBackLight();
else
{
camera_info.state.backlight_off=0;
TurnOnBackLight();
}
}
//luascript.c
static int luaCB_set_backlight( lua_State* L )
{
camera_info.state.backlight_off = (luaL_checknumber(L,1)==0);
if (!camera_info.state.backlight_off) TurnOnBackLight(); //turns off in kbd.c
return 0;
}
//in lua_script_run()
if (Lres == LUA_YIELD)return SCRIPT_RUN_RUNNING; //yielded
camera_info.state.backlight_off=0; //error or finished script
TurnOnBackLight(); //makes error visible
--[[
@title Shoot Test
--]]
bl=1 -- backlight on
print("<set> to toggle backlight")
print("<menu> to exit")
print("using shoot()")
repeat
shoot()
wait_click(1)
if(is_key("set"))then
bl=bitxor(bl,1) -- 0 or 1
set_backlight(bl)
end
until is_key("menu")
bl=1
set_backlight(1)
print("using shoot_full_only")
press("shoot_half")
repeat sleep(10) until get_shooting()
repeat
sleep(2000)
click("shoot_full_only")
wait_click(1)
if(is_key("set"))then
bl=bitxor(bl,1) -- 0 or 1
set_backlight(bl)
end
until is_key("menu")
I'll do some new power tests, but I think this will save a significant amount of power over trying to keep the backlight off with Lua or Basic alone.Based on my testing here,
you gain about 20% on you battery life by turning the backlight off. I would expect results from your mods to be about the same.You're probably right, but this way doesn't even flash the backlight in half_shoot. I'm shooting at 1 shot per second with no focusing or pre-shot metering, so maybe the backlight flashing will be a larger part of the power drain.
I want to extend drivelapse.bas to turn off the LCD display to save battery and avoid distraction while the script is running (and then turn it back on when the script ends). Is this possible?So the code is ready to fix this problem if it's needed.
If you post a patch, then it it would be available to anyone who want it, regardless of whether I think it should be in the trunk or not.
So the code is ready to fix this problem if it's needed.
If you post a patch, then it it would be available to anyone who want it, regardless of whether I think it should be in the trunk or not.That's a good idea. But now that I've figured out that the script interrupt bug is related to Lua yield(), and a way around it, my time lapse / shot meter code is working reliably. I think I'll work on cleaning up the code and improving my Lua time lapse script, and post the whole thing as one big patch and a finished script.
I could capture a comet for the first time but I really struggled.That makes me feel better. I never would have found the comet even if the horizon was clear, so at least I had a pretty cloud to catch in the time lapse.
you would call this with:I don't like this at all. It's ugly and pcall imposes severe limits on what chdk functions the called code can use.
wait_for("func")
it could be modified to work like this:
wait_for("get_shooting",false)
As far as I know, pcall() doesn't do lua_yield() so it should work around the bug.
A secondary benefit would be to make the lua script a little more concise when writing wait loops.This is not a benefit, it's the opposite. Making scripts use less characters is not a goal of the script inteferace. Wait loops are a clear description of what is going on, and are far more flexible than your approach.
I don't like this at all. It's ugly...Ha, I knew you'd say that. My job is to throw up new ideas. Your job is to tell me to throw away what I throw up.
Fixing the bug would be a much better solution. If the bug is in fact caused by repeated yields, that should be a significant step toward nailing down the root cause.Agreed.
Working around the Lua yield() bug
... it seems to be a bug in lua yield(). I'll see if I can try Phils lua updated branch when I get the time.I hope you don't mind my question, but have you tried that branch already?
Wrong handling of nCcalls in coroutines.
reported by Alexander Gavrilov on 18 Apr 2012. existed since 5.2.0. fixed in 5.2.1.
I hope you don't mind my question, but have you tried that branch already?Phil's branch is still Lua 5.1.something. Apparently, updating to 5.2 would require a lot more changes.
http://www.lua.org/bugs.html#5.2.0-4 (http://www.lua.org/bugs.html#5.2.0-4)I don't think that bug applies to the 5.1.x code (hence the "existed since 5.2.0" comment), but I could be wrong.
I don't think that bug applies to the 5.1.x code (hence the "existed since 5.2.0" comment), but I could be wrong.Thanks, I didn't notice that. I think you're right.
4. Updated Lua to version 5.1.5 (Feb 2012 release) from 5.1.3 (Jan 2008 release).
I also looked at 5.2; but the changes are much more substantial so will take a lot more effort to update.
How soon do you think it will be until Phil merges the 5.1.5 branch into the trunk?
Not for a while I'm afraid. Work getting in the way at the moment.OK, I'll test that branch as soon as I finish what I'm working on (setting the metering area using draw_rect)
I'm planning to save the shot meter area data in:I think you can create a new directory in A/CHDK. Don't forget that your cameras can't handle long filenames, and that there's a limit (32 characters?) on path + filename length.
A/CHDK/DATA/Shot_Meters.dat
Is that the right place?
I think you can create a new directory in A/CHDK. Don't forget that your cameras can't handle long filenames, and that there's a limit (32 characters?) on path + filename length.Thanks. I just finished the lua function that will set the shot meter areas. You can look through the script to see what the keys do, but this is basically how it works.
--[[
@title Draw Test
--]]
function get_meters()
local move=262 --green
local grow=259 --red
local nmeters
local imeter -- current
local meters=nil
local i
local mode
local step
repeat
if(meters==nil)then
meters={30,30,70,70, 30,30,70,70, 30,30,70,70, 30,30,70,70}
nmeters=1
imeter=1
mode=move
step=10
end
local xd=0
local yd=0
local step_string=string.format("%2d",step)
draw_clear()
repeat
draw_string(0,0,step_string,mode,257)
i=1
local cmeter
for cmeter=1,nmeters do
local ax1=(meters[i]*359+50)/100
local ay1=(meters[i+1]*239+50)/100
local ax2=(meters[i+2]*359+50)/100
local ay2=(meters[i+3]*239+50)/100
local color=271 -- yellow
local bcolor=256 -- clear
if(imeter==cmeter)then
color=mode
bcolor=257 -- black
end
draw_rect(ax1-1,ay1-1,ax2+1,ay2+1,257,3)
draw_rect(ax1,ay1,ax2,ay2,color)
draw_string((ax1-5+(ax2-ax1)/2),(ay1-6+(ay2-ay1)/2),cmeter,color,bcolor)
i=i+4
end
wait_click(200)
until not is_key("no_key")
if(is_key("menu"))then break end
if(is_key("set"))then
if(mode==move)then mode=grow else mode=move end
elseif(is_key("left"))then
xd=-step
elseif(is_key("right"))then
xd=step
elseif(is_key("up"))then
yd=-step
elseif(is_key("down"))then
yd=step
elseif(is_key("zoom_in"))then
step=10
elseif (is_key("zoom_out"))then
step=1
elseif (is_key("shoot_half"))then
meters=nil -- start over
elseif (is_key("display"))then
imeter=imeter+1
if(nmeters<4)then nmeters=nmeters+1 end
if(imeter>4)then imeter=1 end
step=10
mode=move
end
if((xd~=0)or(yd~=0))then
i=(imeter-1)*4+1
local x1=meters[i]
local y1=meters[i+1]
local x2=meters[i+2]
local y2=meters[i+3]
x2=x2+xd
y2=y2+yd
if(mode==move)then
x1=x1+xd
if(x2>100)then
x1=x1+100-x2
x2=100
elseif(x1<0)then
x2=x2-x1
x1=0
end
y1=y1+yd
if(y2>100)then
y1=y1+100-y2
y2=100
elseif(y1<0)then
y2=y2-y1
y1=0
end
else -- mode==grow
if(x2>100)then x2=100 end
if(y2>100)then y2=100 end
end
if(x2<=x1)then x2=x1+1 end
if(y2<=y1)then y2=y1+1 end
meters[i]=x1
meters[i+1]=y1
meters[i+2]=x2
meters[i+3]=y2
end
until false
for imeter=1,nmeters do
i=(imeter-1)*4+1
print(meters[i],meters[i+1],meters[i+2],meters[i+3])
end
end
get_meters()
To do HDR, for example, you could place one meter on the brightest area, and one meter on the darkest. Then you would take 2 pictures, one based on each meter.My iPod Touch 4g has a few apps that do what you describe, dob the touch screen and the exposure points can be moved around and the shot exp comp is previewed.
My iPod Touch 4g has a few apps that do what you describe, dob the touch screen and the exposure points can be moved around and the shot exp comp is previewed.That's interesting. I think of CHDK as an interface for Lua "apps". It's a good idea to be able to move the metering areas around and see the result. My metering is done after the shot, and applied to the next shot. But you can see the pictures as they're taken when in continuous mode. So I could move the meters during the time lapse and see the result. It might be useful.
Would you mind summarising your work so far on shot histogram and say how it will fit into the general usage of CHDK, for simpletons like me?So far, I've just increased the resolution of the histogram. There's only one way to get the results of the histogram, at the moment:
If the bug is in fact caused by repeated yields, that should be a significant step toward nailing down the root cause.FWIW...
I finished adding the shot meter area drawing function posted above to my main time lapse program. It works well, and lets you set the meter areas based on what the camera is seeing.Making this work on a touch screen camera would replicate app behaviour on iDevices/Android 4.2 devices, where HDR exposures are not made by using preset Ev ranges, but are metered from relevant areas of the shot, multi spot readings if you like, which are identified as High or Low (exposure). I think @philmoz has got CHDK working on a touch screen model, Ixus 310 HS I think. Your shot metering thingy seems to have a large potential for constructive uses
For sunsets, I'd like to try putting a relatively small metering area over the sun before it sets...
When I added the capability to do multiple meters, I didn't know how I would use them. But now that I can adjust the location and size of each meter, there are a lot of possibilities. A small meter on the brightest part of the picture would be like histogram based exposure compensation. Averaging the meters and putting one inside the other would give a kind of center weighted metering.
To do HDR, for example, you could place one meter on the brightest area, and one meter on the darkest. Then you would take 2 pictures, one based on each meter.
I'm looking forward to exploring the possibilities opened up by being able to set the size and location of 4 different exposure meters. Try the test script in the post above to get an idea of how the meter sizes and postions are set.
...
Making this work on a touch screen camera would replicate app behaviour on iDevices/Android 4.2 devices, where HDR exposures are not made by using preset Ev ranges, but are metered from relevant areas of the shot, multi spot readings if you like, which are identified as High or Low (exposure).I didn't realize there was "an app for that." Thanks for the info. I love when other people use my ideas before I think of them. :)
To test the hypothesis that there could be some conflict between debug hook yield and explicit yield with sleep() etc,I'm pretty sure this isn't the problem because the bug happened with set_yield(-1,-1). It happened with sleep(10) or coroutine.yield(). It's prevented by using the action stack to wait for the shot to be ready, rather than repeated sleep(10) or coroutine.yield()
As an aside, there is a lua assert if the script is interrupted in the shutter button, presumably due to trying to yield in the restore() function.I assume that's one of the "ugly" problems with pcall() you were talking about. Does this only happen with a yield in the yield hook? It seems like doing a sleep() in restore works.
lapser :D - I could watch that all day (don't think I'd get paid though).Thanks! It is pretty fascinating to see things with the help of technology in a way that was previously impossible. I was hoping for clear skies, but now I'm glad I got the clouds. It made the video more interesting.
--tvd is the change in exposure computed by the shot meter
--you add tvd to the current exposure and use it for the next shot
tx=0
if(tvd>0)then
if(tvd>2)then tx=1 end
tvd=tvd-smooth
if(tvd<0)then tvd=tx end
elseif(tvd<0)then
if(tvd<-2)then tx=-1 end
tvd=tvd+smooth
if(tvd>0)then tvd=tx end
end
I'll try to test this tonight, weather permitting. tx=0
if(tvd>0)then
if(tvd>2)then tx=1 end
tvd=tvd-smooth
if(tvd<=0)then tvd=tx end -- changed to <=
elseif(tvd<0)then
if(tvd<-2)then tx=-1 end
tvd=tvd+smooth
if(tvd>=0)then tvd=tx end -- changed to >=
end
Here's one of the pictures from the G1X:Would your script work on my ancient SD300 (Ixus 40) and the latest SVN build ? If so, any chance of posting the full script, the results are pretty impressive.It requires a custom build I've been developing that adds a lot of functions to CHDK. I'd be interested in seeing if it does work on old cameras, so if you're game, let me know your firmware version and I'll post a build for you to try.
ixus40_sd300-100k-1.2.0-2653-fullOK, see if the attached CHDK build runs your current scripts.
Bad news, RawTherapee cant make sense of the .DNG file (I'll investigate this later, maybe try DNG 1.1).This is fixed in the ixus30 port. Does it help if you set image size to L (I mean if it's currently set to a smaller size)?
EDIT: Nope, and .CRW looks like nonsense as well. Neither RawTherapee nor UFRAW can make any sense of them, they look like a noise with a narrow horizontal band of dark colour about 1/4 of the way down from the top.
Image size is set to Superfine and L (2272x1704)It's worse then. I'll look into backporting the ixus30 version in the near future...
Good news is that that build loads OK on my camera (confused me slightly because it doesn't show the CHDK splash screen at startup, the first time, so I thought it wasn't loading).I think the splash screen is a configuration issue. I'm not sure if it works for older cameras, but I would try resetting by deleting the file:
Image size is set to Superfine and L (2272x1704)It's worse then. I'll look into backporting the ixus30 version in the near future...
The raw hook is at the wrong place.
Good news is that that build loads OK on my camera (confused me slightly because it doesn't show the CHDK splash screen at startup, the first time, so I thought it wasn't loading).I think the splash screen is a configuration issue. I'm not sure if it works for older cameras, but I would try resetting by deleting the file:
CHDK/CCHDK3.CFG
I'm still back on 2619, so anything that's been fixed on the camera since then isn't there.
Also, are you sure you got ALL the files in the zip onto your card, including the modules? You have to overwrite all the modules, since a lot of my new code is in the lua module.
Forgetting about the raw files for now, the next thing to try would be to run the time lapse script. It's still pretty rough, and not documented, but see if you can figure it out. You can try using all the defaults, and just press <menu> to get it started.
If it works, you can try changing the various parameters, and setting the metering areas before you press <menu> and see what happens. I predict that the camera may crash if you set the focus to 1 (hyperfocal distance) or -1 (infinity). Good luck.
The script runs, I can move the metering square about, and takes a picture (just the one), I'll have a play with this tomorrow. Looks like it may well work.That doesn't sound good. It should start taking 1 picture every 2 seconds.
It's worse then. I'll look into backporting the ixus30 version in the near future...That might hang my script without crashing, if raw_savefile() is never called. I also need capt_seq_hook_set_nr() to be working.
The raw hook is at the wrong place.
The script runs, I can move the metering square about, and takes a picture (just the one), I'll have a play with this tomorrow. Looks like it may well work.That doesn't sound good. It should start taking 1 picture every 2 seconds.
What happens after it takes 1 picture? The next thing I do is sum up pixel samples in the raw buffer to compute the shot meter, so if the raw buffer address is wrong, it would crash or something. We may need to move on to a camera that can save raw correctly. Does yours work with the standard CHDK build for saving raw?
I think the problem with RAW was actually due to low battery causing SD card corruption during the long save process. I am now running on a fresh battery and have saved a couple of raw files which RawTherapee *does* like. I wont get a chance to do any further testing till tomorrow. This camera takes longer than 2 seconds (more like 5 or 6 seconds) to save a raw picture, not sure if that will be a problem or not.OK, so if raw save is working, then the raw hook is ok. When you test tomorrow, try it first with raw save off. Also, see if you can demonstrate that you can turn dark frames off and on in the raw menu, and that it works. It should take twice as long to take a picture with dark frames on (use a long exposure with the camera only).
If that is a show stopper, I may abandon the idea and use some other camera. It would be nice to know if lapser's enhancements work on older cameras though, so I am prepared to keep trying till we run out of options.
Also, see if you can demonstrate that you can turn dark frames off and on in the raw menu, and that it works.That does not currently work, but soon will.
The important thing is that capt_seq_hook_set_nr() is called right before the shot. I added a pre-shot delay here so I can set a precise shot interval that also works in continuous mode. This routine also saves the precise shutter open time so it can be saved in the CHDK raw file, if needed.Also, see if you can demonstrate that you can turn dark frames off and on in the raw menu, and that it works.That does not currently work, but soon will.
Another issue with this camera is that CHDK overrides are only applied before the actual shot, but not when half-shooting. I guess it's not your concern, since you're working with the raw data.
Improved capt_seq.c for ixus40 100k attached. It has all needed hooks, DNG seems alright with it.THANKS!
Actually what the camera says briefly after taking the picture isThe log says
***Terminated***
6 2 0 -2
Does the script rely on continuous drive mode? 'cos this camera doesn't have that.It does. Press "down" in shooting mode...
There's an error in the tv96 to shutter speed conversion.Actually what the camera says briefly after taking the picture isThe log says
***Terminated***
6 2 0 -2
:353: bad argument #6 to 'format' (number expected, got nil)
XLapser.lua from the previous page.QuoteDoes the script rely on continuous drive mode? 'cos this camera doesn't have that.It does. Press "down" in shooting mode...
There's an error in the tv96 to shutter speed conversion.There's a list of function pointers in this firmware, in their names' alphabetical order, it should be correct.
This relies on the log10 function. There was an error in the address of the log10 function that we discovered with the SX50. Can you check to make sure that address is correct?
print(shutter_to_tv96(500))96
Line 326 of the script says:Better :)
tvd0=tdv -- for log
It should say tvd0=tvd
Better :)Great!
It now appears to work (shoots continuously, exposition is adjusted).
Meanwhile, here is yesterdays effort with a different script. (Quite a few exposure issues as you can see).You have a beautiful spot for time lapsing. Is there much light pollution at night? Where are you located?
??? Hmmm now we get one shot, then the camera crashes. I will check everything later. srsa_4c I presume it worked for you on an Ixus 40, if so, must be something odd on my setup.There's a known way to crash the cam, by setting a too low ISO override (the source mentions crash below ISO 58).
(I assumed P mode was Portrait, as this cam has an Auto and Manual mode, but nothing specifically called P If this is not the correct mode srsa_4, let me know)... and it works :D. It seems a little flakey if I use any other settings.P is "Program" auto exposure that sets aperture and shutter speed. You could try it in Manual, setting the aperture as wide as possible (lowest F stop). Portrait mode probably isn't right for a sunset.
I just sat it on the fence post, no waterproof housing, slightly risky as there is a chance of frost. I notice the script doesn't switch off the display, not sure if this is possible, but it would help preserve the battery life a bit if it is.You switch the display on and off by pressing <set> while the shots are being taken. Be careful not to jiggle the camera, though. <up> and <down> move the log line. <right> and <left> turns the metering area display on or off. <display> controls the ND filter (if any). That may crash the camera, so don't try it unless you're ready for that possibility.
Canon is only using "P" mode on cameras with manual modes. On Ixus cameras, it's displayed as "M" mode (but it doesn't provide full manual control, the name is misleading).I seemed to have a conflict in Manual Exposure mode on the SX50 when overriding exposure in half shoot. I haven't tracked it down, but it may have been related to setting the Av propcase. I heard clicking, and it slowed down the shot rate until I switched the mode dial to Aperture Priority mode. It was also in continuous shooting mode, so that might be it too. There's a continuous mode with auto focus that may have been triggered for some reason.
Since I couldn't make it load in Auto or Manual mode for some reason (one shot, then camera crash, as I described previously...) I decided to see what mode other than Portrait it would work in. So I put the camera in night snapshot mode, set it up and left it to do its thing.It's still a very experimental script, and CHDK modifications, so it's not something you did. For testing, you don't need to wait for sunset. Just point the camera at the screen and hold it there, and start the script.
When I returned it had taken a bunch of pictures, *but* every other picture was exposed differently.
Good suggestions. I'll give them a try when I can.OK, be sure to try continuous mode and infinity focus. I think you press the <left> for focus and <down> for continuous drive mode (burst mode).
--smoothing
if(tvd>1)then
tvd=tvd-smooth
if(tvd<=0)then
if(nztvd>3)then tvd=1 else tvd=0 end
end
elseif(tvd<-1)then
tvd=tvd+smooth
if(tvd>=0)then
if(nztvd>3)then tvd=-1 else tvd=0 end
end
else
tvd=0
nztvd=-1
end
if(tvd==0)then nztvd=nztvd+1 else nztvd=0 end
int iround(double x)
{
return (int)(x+((x>0)?0.5:-0.5))
}#!/bin/bash
# The Ixus 40 is set up to create a new folder for every 10 images
# I copy these folders to ~/Pictures/Timelpase
# Then the script grabs all of them and moves them to a folder called ./Pics
# it then builds te movie with mencoder.
#
for file in $(find . |grep JPG)
do
mv $file ./Pics/.
done
PIC_DIR="./Pics"
# For lower res, use something like..
#WIDTH="640"
#HEIGHT="480"
# 1920x1080 gives full HD
WIDTH="1920"
HEIGHT="1080"
FPS=12
BITRATE="1500"
DEST_VIDEO="timelapse6.mpg"
mencoder mf://$PIC_DIR/\*.JPG -mf w=$WIDTH:h=$HEIGHT:fps=$FPS:type=jpg -ovc lavc -lavcopts vcodec=mpeg4 -oac copy -o $DEST_VIDEO
I'm quite pleased with the results I got tonight, I put the 2361 frames shot on a single battery together at 6 fps, but I might redo at 12 fpsVery nice! That's a really beautiful place.
Thanks, it can be a very beautiful spot, particularly in the spring when the sunsets are long (we are relatively far north).I'm quite pleased with the results I got tonight, I put the 2361 frames shot on a single battery together at 6 fps, but I might redo at 12 fpsVery nice! That's a really beautiful place.
I usually shoot at 1 or 2 frames per second, and play back at 25 or 30 FPS. 2 or 3 minutes is a good target time for a sunset video.
From your previous posts, I was worried that the exposure adjustments weren't working right. Did you find a mode that worked? How did you set the focus? The tree at the left was in sharp focus, so I wonder what the focal length was.
If you set the log file parameter to 1, it will save a log that shows the focal distance, and a lot more.
Glad to see your having some success with the script, and the camera.
EDIT: Logging is enabled, log file attached from last night.Thanks for posting the log file. It's interesting to see the comparison with an older camera.
If you have merged back in to the "bleeding edge" build, does that mean I can try you script out on a couple of other models now?
how about rolling a version for the Ixus 115 HS and the Powershot A2200I need the firmware versions.
how about rolling a version for the Ixus 115 HS and the Powershot A2200I need the firmware versions.
My time lapse mods are too experimental for the "experimental" trunk, so you'll have to get the build from me for each camera.
OK lapser I'll get the versions for you soon.Start by installing the "full" version 1.2 from the main download page for each camera. My builds install over the full version, since they don't include all the files. They only work with the card locked (auto boot).
// Wait for a button to be pressed and released (or the timeout to expire)
static int action_stack_AS_LUA_WAIT_CLICK()
{
lapser1=-9;
// Check key pressed or timeout
if ((get_tick_count() >= action_top(2)) || camera_info.state.kbd_last_clicked)
{
// If timed out set key state to "no_key", otherwise key pressed so set last checked time
if (!camera_info.state.kbd_last_clicked)
camera_info.state.kbd_last_clicked=0xFFFF;
else
camera_info.state.kbd_last_checked_time = camera_info.state.kbd_last_clicked_time;
lapser1=9;
action_pop_func(1);
return 1;
}
return 0;
}
So it looks like action_stack_AS_LUA_WAIT_CLICK() didn't return, or lapser1 would be 9, not -9. Interesting!long get_tick_count()
{
long t;
#if !CAM_DRYOS
_GetSystemTime(&t);
return t;
#else
return (int)_GetSystemTime(&t);
#endif
}
tick=get_tick_count();
while(tick<shot_next)
{
msleep(10);
tick=get_tick_count();
}
My theory is that there's a hard interrupt during the call to get_tick_count() above, and then get_tick_count() is called in wait_click(10) and messes up the stack. tick=get_tick_count();
if(tick<shot_next)
{
msleep(shot_next-tick);
tick=shot_next;
}
I was using msleep(10) so I cold abort the wait by setting shot_next to 0 when I exit the program, but I can minimize the calls to get_tick_count() if that's will fix the bug.int lua_script_run(void)
{
//snip
Lres = lua_resume( Lt, top );
lapser2=Lres;
Here's are the values for lapser2:unless I'm mistaken, lapser1 would stay -9 if the return 0 (timeout) case were hit.lapser1=9;
Looking at the implementation of _GetSystemTime (on D10) I don't see how it would not be thread safe. It's a very simple function, it basically just disables interrupts, reads a variable, restores interrupts and returns.That's good to know that _GetSystemTime is threadsafe (disables interrupts). However, I don't think the wrapper is threadsafe, since it could be interrupted and called again from another thread.
There is an oddity in luaCB_wait_click though, it returns 1 even though it pushes nothing on the Lua stack. That is wrong, although I suspect it is harmless.I think Phil changed the meaning of the 1 return when he re-worked action stack.
/* thread status; 0 is OK */
#define LUA_YIELD 1
#define LUA_ERRRUN 2
#define LUA_ERRSYNTAX 3
#define LUA_ERRMEM 4
#define LUA_ERRERR 5
You might check whether the values you got are within the CHDK binary or heap space.The script should have been suspended until action_stack_AS_LUA_WAIT_CLICK finished (after the action_pop_func call).Yes, that's what caught my attention. Here's where I print lapser1:
So I want to double check that the '-9' being displayed is not a result of the simply not updating the display/log after the script has ended.
int lua_script_error(lua_State *Lt,int runtime)
{
//snip
script_console_add_line(LANG_CONSOLE_TEXT_TERMINATED);
char osd_buf[50];
sprintf(osd_buf, "Lapser: %6d%6d%6d%6d",lapser1,lapser2,lapser3,lapser4);
script_console_add_line((long)osd_buf);
script_print_screen_statement(0); //close print file
shot_histogram_set(0); //disables meters and interval too
return SCRIPT_RUN_ERROR;
}
That's good to know that _GetSystemTime is threadsafe (disables interrupts).It's not thread safe because it disables interrupts. It's thread safe because it only reads globals and uses the stack. Even if some other thread was changing the globals underneath it, that wouldn't cause the code to malfunction.
However, I don't think the wrapper is threadsafe, since it could be interrupted and called again from another thread.This is nonsense. Each task has it's own stack, and the OS handles switching them. t is a stack variable, so every thread gets it's own, although if you look at the dryos version of _GetSystemTime, it doesn't even use t ;) (edit: well, the D10 version doesn't, some others might...)
long t;
return (int)_GetSystemTime(&t);
is actually 2 returns. If an interrupt occurs between the returns, that could be a problem. If the stack pointer were in the wrong place because of it, it's possible that it would effectively do the action stack pop thinking it was popping variable "t".
I think Phil changed the meaning of the 1 return when he re-worked action stack.No, phil did not change the semantics of lua C functions. In a lua C function, (one that is called from lua, meaning all the luaCB functions in the CHDK API), the return value is the number of values that function returns in Lua. The actual values are pushed on the Lua stack.
The only way I can see that lapser1 could be displayed as -9 is if the error occurred calling luaCB_wait_click.Yes, that makes sense. I'm calling wait_click(10), so it's probably just yielding right away? Anyway, the bug also occurs with sleep(10) and coroutine.yield(), so I guess we haven't learned much new.
The luaCB_wait_click function calls action_stack_AS_LUA_WAIT_CLICK which will set lapser1 to -9.
Since action_stack_AS_LUA_WAIT_CLICK would have returned 0, then luaCB_wait_click must have called return lua_yield(L,0).
The Lua error must have then been triggered from this return - otherwise action_stack_AS_LUA_WAIT_CLICK would have taken over the action stack and the script would have been suspended.
Phil.
Do you think it's possible that resume() could occur in the same action stack cycle after the yield()?Off the top of my head, I can't think of a way that would occur in the current chdk code. Each kbd_task iteration should only call lua_script_run once.
Would that cause a problem if it did?It shouldn't matter at all. Lua yield and resume are really just regular C function, they don't know anything about OS level tasks.
static int action_stack_AS_LUA_WAIT_CLICK()
{
lapser1=-9;
// Check key pressed or timeout
if ((get_tick_count() >= action_top(2)) || camera_info.state.kbd_last_clicked)
{
// If timed out set key state to "no_key", otherwise key pressed so set last checked time
if (!camera_info.state.kbd_last_clicked)
camera_info.state.kbd_last_clicked=0xFFFF;
else
camera_info.state.kbd_last_checked_time = camera_info.state.kbd_last_clicked_time;
lapser1=9;
action_pop_func(1);
return 1;
}
return 0;
}
static int luaCB_wait_click( lua_State* L )
{
lapser1=8;
int delay = luaL_optnumber( L, 1, 0 );
//if (delay == 0) delay = -1;
if(delay<2)delay--; //delay 1 for no delay or 0 for no timeout
delay = sleep_delay(delay);
// Reset 'clicked' key if it has not changed since last time
if (camera_info.state.kbd_last_clicked_time <= camera_info.state.kbd_last_checked_time)
{
camera_info.state.kbd_last_clicked = 0;
}
// Set up for wait or click testing
action_push(delay);
action_push_func(action_stack_AS_LUA_WAIT_CLICK);
// Check for short delay or key already pressed by calling action stack routine once now
if (action_stack_AS_LUA_WAIT_CLICK() == 0)
{
return lua_yield( L, 0 );
}
//return 1;
return 0;
}
static int luaCB_is_key( lua_State* L )
{
lapser1=7;
lua_pushboolean( L, script_key_is_clicked(lua_get_key_arg( L, 1 )));
return 1;
}
repeat wait_click(10)
if(is_key("set"))then
[snip series of is_key calls]
until get_shot_ready()
[Lua error was exactly the same as yesterday on the same camera, sx50)
ERROR: NULL error message
*** TERMINATED ***
Lapser: -9 3209878 0 0
Sure. Give me the models and firmware versions and I'll post builds for you.
I've attached the latest script and ixus40 buildIf you have merged back in to the "bleeding edge" build, does that mean I can try you script out on a couple of other models now?
I suspect I have failed to copy over some files.. I'll check.Maybe. Be sure to copy all the files in the zip. Lua is in a module, which changes with each update.
First of all, I think the Lua error was triggered by CHDK calling resume() without a yield().This should not be possible. It would mean either that multiple tasks were running the action stack, or that something within a lua C function called lua_script_run.
I think this happened because the action stack cleared abnormally before action_stack_AS_LUA_WAIT_CLICK() was finished.I don't understand how this could happen. kbd_task isn't going to run another iteration until the call returns, which means that lua has yielded. The entire kbd_task including all the lua code and action stack functions run sequentially. Once lua has been entered, action stack processing does not happen again until lua has yielded or exited. The only way to get back into lua is if action_stack_AS_SCRIPT_RUN is on top of the stack.
So do you think this could be happening?No.
Is action stack called from a different task?Not unless something is very badly wrong, and if it did, it would lead to near instant corruption of the lua state, which would very quickly cause crashes and other bad behavior. The exact nature of the bad behavior would be highly dependent on timing and memory organization, so it would be exceedingly unlikely to show the exact same symptom across different builds, cameras and scripts.
However, you could test what happens if you call resume without a yield by adding code to force that.I was trying to do that through action stack, but just crashed the camera.
Looking at your debug code:Yes, I know it's calling lua_yield() with lapser1==-9. But that also means that AS_LUA_WAIT_CLICK() is still on the action stack. It should continue executing that function until AS_LUA_WAIT_CLICK() times out (next cycle), and calls POP_FUNC. But it also sets lapser1=9 when it does that.
If action_stack_AS_LUA_WAIT_CLICK() returns 0, then you can yield with labser1 == -9
lua_yield will eventually make lua_resume return, so it's quite possible that the error occurs after that yield.
So the question is, how does it get to lua_resume() without lapser1 being set to 9? Forget all my other theories. This is the problem I can't figure out.
It doesn't 'get' to lua_resume - the call to 'luaCB_wait_click' was made from within a previous call to 'lua_resume' from the last time Lua regained control.OH, I see it now (finally). When wait_click() calls lua_yield(), lua then RETURNS from resume() (with lapser1==-9), and then CHDK enters action stack. When it finishes action stack, it then calls resume() (with lapser1==9).
Given the random nature of the errors I wonder if it's a problem with the Lua garbage collector.
http://chdk.setepontos.com/index.php?topic=9607.msg99095#msg99095 (http://chdk.setepontos.com/index.php?topic=9607.msg99095#msg99095)Since we've already done so much discussion of the bug here, it might be clearer to keep all the code related stuff in one thread.
I've attached an updated version of my debug dump patch.OK, I got it working, including the module logging. I added a test error to my script by calling an unknown function (with an obscene name in German) :) when you press <display>. I ended up with 4 .DMP files, and MODULES.LOG
I also modified luascript.c to use one Lua thread, as Phil described. It works, except for ptpcamgui fails to init after the change. I guess it needs the 2 threads?I doubt it. It shouldn't be much different from regular script, but ptp specific code might reference Lt somewhere.
It shouldn't be much different from regular script, but ptp specific code might reference Lt somewhere.First I just replaced the new thread call with Lt=L; That worked, but ptpCamGui wouldn't init. Then I replaced all "L" with "LT" and removed the declaration for L and made Lt static, and removed L and Lt from luascript.h. So there must can't be any references to L because it's no longer there. But I got the same result.
I'm also including the romlog from the SX260, but the crash may not be significant. The time in the romlog is 2013:04:10 09:40:45. I have the camera clock in AM/PM mode, so if the time was 9 p.m. that would make sense.I would be surprised if the romlog respected the camera UI 12/24 hour setting, but in any case that romlog looks like an assert that is triggered you try to have too many files open simultaneously. This could happen if you try to log too soon after a shot.
If you have the main.bin.dump (or main.bin) from the CHDK build that was running when this was taken, that will help. The module elfs would also help.Where do I find those files?
that romlog looks like an assert that is triggered you try to have too many files openThat's probably from earlier then. I crashed the camera that morning when I intentionally test triggered your patch to write its logs, without the proper compile options.
Where do I find those files?main.bin and main.bin.dump are in the core directory after you build.
status = luaD_rawrunprotected(L, resume, L->top - nargs);
if (status != 0) { /* error? */
dbg_dump_write("A/LUARES1.DMP",0,sizeof(lua_State),(void *)L);
L->status = cast_byte(status); /* mark thread as `dead' */
luaD_seterrorobj(L, status, L->top);
L->ci->top = L->top;
}
This is the only place LUARES1.DMP can be created. It will be hit if luaD_rawrunprotected returns non-zero. int r = setjmp(lj.b);
if(r == 0) {
(*f)(L, ud);
} else {
dbg_dump_write("LUALJ1.DMP",0,r,NULL);
}
if(lj.status) {
dbg_dump_write("A/LUALJ2.DMP",0,sizeof(lj),(char *)&lj);
}
L->errorJmp = lj.previous; /* restore old error handler */
return lj.status;
If lj.status is non-zero, A/LUALJ2.DMP should be written.My gut want's blame setjmp/longjmp, especially since it is a custom implementation lifted from web, but I can't really see how.
The setjmp.h / setjmp.s code is difficult to follow with all the #ifdef clutter.It wouldn't hurt, but you can get a de-cluttered version from the disassembler
Is it worth stripping these files back to just what is needed for CHDK?
00010c20 <setjmp>:
10c20: 4778 bx pc
10c22: 46c0 nop ; (mov r8, r8)
00010c24 <.arm_start_of.setjmp>:
10c24: e8a07ff0 stmia r0!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, sp, lr}
10c28: e3a00000 mov r0, #0
10c2c: e31e0001 tst lr, #1
10c30: 01a0f00e moveq pc, lr
10c34: e12fff1e .word 0xe12fff1e
00010c38 <longjmp>:
10c38: 4778 bx pc
10c3a: 46c0 nop ; (mov r8, r8)
00010c3c <.arm_start_of.longjmp>:
10c3c: e8b07ff0 ldm r0!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, sp, lr}
10c40: e1b00001 movs r0, r1
10c44: 03a00001 moveq r0, #1
10c48: e31e0001 tst lr, #1
10c4c: 01a0f00e moveq pc, lr
10c50: e12fff1e .word 0xe12fff1e
Note the last .word is a bx lr
The setjmp.h / setjmp.s code is difficult to follow with all the #ifdef clutter.It wouldn't hurt, but you can get a de-cluttered version from the disassembler
Is it worth stripping these files back to just what is needed for CHDK?Code: [Select]00010c20 <setjmp>:Note the last .word is a bx lr
10c20: 4778 bx pc
10c22: 46c0 nop ; (mov r8, r8)
00010c24 <.arm_start_of.setjmp>:
10c24: e8a07ff0 stmia r0!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, sp, lr}
10c28: e3a00000 mov r0, #0
10c2c: e31e0001 tst lr, #1
10c30: 01a0f00e moveq pc, lr
10c34: e12fff1e .word 0xe12fff1e
00010c38 <longjmp>:
10c38: 4778 bx pc
10c3a: 46c0 nop ; (mov r8, r8)
00010c3c <.arm_start_of.longjmp>:
10c3c: e8b07ff0 ldm r0!, {r4, r5, r6, r7, r8, r9, sl, fp, ip, sp, lr}
10c40: e1b00001 movs r0, r1
10c44: 03a00001 moveq r0, #1
10c48: e31e0001 tst lr, #1
10c4c: 01a0f00e moveq pc, lr
10c50: e12fff1e .word 0xe12fff1e
How is the first 'bx pc' instruction in each function switching to ARM mode?From: "ARM Architecture Reference Manual" A6.1 About the Thumb instruction set
The PC value should still be odd at this point so why would this cause a mode switch?
When R15 is read, bit[0] is zero and bits[31:1] contain the PC.
Also I thought using the PC register in any BX instructions was deprecated.gcc interworking does it the same way:
00175db8 <__mkdir_from_thumb>:
175db8: 4778 bx pc
175dba: 46c0 nop ; (mov r8, r8)
175dbc: eaff847a b 156fac <mkdir>
The kicker is that for any defect in setjmp to be causing this, it would have to succeed 99.999% or more of the time, every call to lua resume goes through the setjmp code, and every lua error goes through longjmp. More likely would be the jumpbuf on the stack getting corrupted, but the check on lj.status is done after it has returned. //Lt = lua_newthread( L );
Lt=L;
I left the SX50 with the new build running at the house, and took the G1X and SX260 to a spot with a better view of more sky so I could catch a ISS flyby. I managed to see the ISS in both cameras. The SX260 was using the old build (with 2 Lua threads), and it triggered the bug before the ISS appeared. So I restarted the script, and the bug happened again after a few seconds. I started it again, and it ran for about an hour with no bug, and caught the ISS.Here's quick and dirty patch to log some calls to a buffer, maybe we can see how it gets to that "impossible" return value.I'll install the new patch and give it a try tomorrow when I test all 3 cameras with the single thread. Or I can go back to 2 threads and try to trigger the bug again so you'll have the new log?
so why not :-[
Or I can go back to 2 threads and try to trigger the bug again so you'll have the new log?It's up to you, but for me the closer it is to stock CHDK the less chance there is of confounding factors.
I wonder what the second thread is doing?As I said in the other thread (:P), nothing. Everything is running the Lt thread, nothing is executed in the "main" thread at all. Lua "threads" aren't really like OS threads, a better name would be "coroutine state" or something like that. They don't execute unless they are explicitly executed.
And why does ptpCamGui fail to init when there's only one thread.I have no idea, and I don't have time to dig into that code right now.
And why does ptpCamGui fail to init when there's only one thread. Do you think you could test this on your cameras with the Lt=L change and see if you can get ptpCam to init? I would really like to solve that problem because 2 of my SD cards don't have write protect tabs now and ptpCamGui is the only way to load new builds (other than Scotch tape). Thanks.
I very much doubt the L vs Lt thing is the cause, but then again, I don't have any good explanation so why not :-[Ha. I have a thousand explanations. Unfortunately all of them are wrong (so far).
I don't use ptpCamGui; but chdkptp works fine with this change.Thanks for that info. This may be my incentive to learn chdkptp batch mode, which I'm sure works much better once you learn it.
My theory (one of them) is that the bug is triggered by an interrupt that occurs while the Lua code is executing a function that isn't thread safe, possibly garbage collection.I don't understand this. Lua code is just regular C code. It's the OS's responsibility to ensure interrupts don't affect the state of a running task, there's no special "thread safety" required for this. If there was some defect in this area, the odds of it it causing exactly the same symptom in one specific place over many different builds appear small.
I still wonder why ptpCamGui won't init, though.So why not debug it? ptpCamGui uses regular PTP calls and lua. Some specific thing must be failing...
Also, did anyone look at the LUAASRT.DMP file I posted here?It means that that an assert was hit in Lua code. The specific assertion is
ASSERT ldo.c:401 ci == L->base_ci && firstArg > L->base
I haven't looked into what this means yet.
I tested ptpcam/ptpcamGUI and chdkptp cli with change "Lt = L". All applications will not return luar results.And why does ptpCamGui fail to init when there's only one thread. Do you think you could test this on your cameras with the Lt=L change and see if you can get ptpCam to init? I would really like to solve that problem because 2 of my SD cards don't have write protect tabs now and ptpCamGui is the only way to load new builds (other than Scotch tape). Thanks.
I don't use ptpCamGui; but chdkptp works fine with this change.
You can also use chdkptp in console mode (with batch files) to simplify downloading updates to the camera.
Phil.
I tested ptpcam/ptpcamGUI and chdkptp cli with change "Lt = L". All applications will not return luar results.This suggests the PTP message interface is broken. I don't see why offhand, but it does use Lt explicitly in a few places. chkdptp relies on this heavily, so if chdkptp works correctly for Phil, maybe he has implemented the change differently.
Notes for tests:
- changes are in lua modul not in bin file
- luar is different
ptpcam: luar get_buildinfo()
chdkptp: luar return get_buildinfo()
I tested ptpcam/ptpcamGUI and chdkptp cli with change "Lt = L". All applications will not return luar results.This suggests the PTP message interface is broken. I don't see why offhand, but it does use Lt explicitly in a few places. chkdptp relies on this heavily, so if chdkptp works correctly for Phil, maybe he has implemented the change differently.
Notes for tests:
- changes are in lua modul not in bin file
- luar is different
ptpcam: luar get_buildinfo()
chdkptp: luar return get_buildinfo()
Comment out the line after 'Lt = L;' that contains:Thanks Phil. I was typing a question about that line when your post appeared. I'll give it a try.
lua_setfield( L, LUA_REGISTRYINDEX, "Lt" );
This will fix the return results from luar.
(If you don't call lua_newthread to create Lt then there is nothing on the lua stack for lua_setfield to use).
So the Lres value is in the middle of the Lua code @ 0x7AA4 bytes from the start of the module.Can you see what's at that address?
So the Lres value is in the middle of the Lua code @ 0x7AA4 bytes from the start of the module.Can you see what's at that address?
To be accurate the source code needs to match what you used when you created the dump files.Hmm, I'm not confident I can do that. I've also added reyalp's new patch since then. However, I do have some new data.
I did another test this morning with the sx260 (2 lua threads) and sx50 (1 lua thread). The sx260 worked normally, but the sx50 crashed (power off, lens extended).One thing that might be useful is to keep a detailed list of how long each build ran and whether it hit the bug.
One thing interrupts do is use extra stack space. I added a function call in capt_seq_hook_set_nr() that use stack, as well as a function call in raw_savefile(). If an interrupt happened at a spot when CHDK code was deeper into the stack for that task than the camera ever gets without CHDK, that might cause a stack overflow and trash whatever memory is next to the stack for that task. I don't know the internals, so maybe you can think of a better idea.This is possible, but the we modify the kbd_task to have a much larger stack. In general I would say if you are running simple lua code it's very unlikely you would use the whole 8K of stack. An interrupt would basically just push the registers on the stack, which is roughly ~ 128 bytes. If this were the cause, it seems quite unlikely it would cause exactly the same symptom across many different builds, scripts and cameras. There should be enough variation that stack corruption would cause all kinds of different crashes, hangs etc. One extra stack variable or function call in the entire stack should change the behavior significantly.
I also compiled with your modules makefile and added lua.elf.dumpobj to the zip. That made it too big to attach, so it's posted here:Can you clarify whether this objdump is from the same elf that was actually running on the camera? If it wasn't built from exactly the same source code, then the value is limited.
To be accurate the source code needs to match what you used when you created the dump files.Hmm, I'm not confident I can do that. I've also added reyalp's new patch since then. However, I do have some new data.
I did another test this morning with the sx260 (2 lua threads) and sx50 (1 lua thread). The sx260 worked normally, but the sx50 crashed (power off, lens extended).
One thing interrupts do is use extra stack space. I added a function call in capt_seq_hook_set_nr() that use stack, as well as a function call in raw_savefile(). If an interrupt happened at a spot when CHDK code was deeper into the stack for that task than the camera ever gets without CHDK, that might cause a stack overflow and trash whatever memory is next to the stack for that task. I don't know the internals, so maybe you can think of a better idea.
I created a zip with the romlog from the sx50 and the TLUARES1.DMP file. The time stamp on this file is the same as the last jpg picture before the crash. I had the backlight off with my backlight patch, which put a lot of turnbacklightoff calls into the romlog, so that's not the problem.
I also compiled with your modules makefile and added lua.elf.dumpobj to the zip. That made it too big to attach, so it's posted here:
http://www.adrive.com/public/b7bfgJ/ReyalpLogs_130413.zip (http://www.adrive.com/public/b7bfgJ/ReyalpLogs_130413.zip)
Also, I should have requested core/main.elf instead of main.bin.Thanks for all your help. Yes, the files come from the same SX50 build that was running when the crash happened. Here's the main.elf file from that build:
where did the "TLUARES1.DMP" name come from?
So I captured reyalp's new lua log file for the first time. I've also included all the relevant files you requested before. Hopefully, this will provide some useful info this time.This captures the trace log. It's doesn't solve the mystery, but it certainly confirms the weirdness
>lua_resume
>luaD_rawrunprotected
>resume
resume yield
>lua_yield
<lua_yield
<resume
<luaD_rawrunprotected
lua_resume 310252
0x310252 is the Lres value (confirmed in the script log). "resume yield" means resume is resuming from a yield (rather than starting the script).000070d8 <luaD_reallocstack>:
70d8: b5f0 push {r4, r5, r6, r7, lr}
70da: 6a03 ldr r3, [r0, #32]
This it doesn't seem like 70da would end up on the stack as a return address (and it should be thumb if it was) or as a pointer in the code. Note reallocstack refers the Lua stack, not the C stack.Taking the lua load address as 0x309178, this makes the lres 0x70da in the lua module, which is not the same as before, but not far off.I'm using a new trunk 2698 in this build. I think Phil made some changes in the Lua module in this trunk.
I may have some spare time later on today if you would like me to test on the Ixus 40Sound good. Attached is the latest CHDK build, and the latest script.
Before the Lua update to 5.5, the Lres value was apparently a pointer to a string on the Lua stack, because the error message would be "display" or "set", which came from the script calling is_key("display") or is_key("set") before the lua yield that triggered the bug.The lua error message is the value on the top of the stack. The lres value is not the value on the top of the stack. It *should* be the return status of resume which would be one of a handful of predefined constants mentioned earlier. In all the cases where we've had sufficient information to tell, it's an address inside the lua module. It doesn't have the low bit set, so it's not a return address.
This does suggest a really horrible workaround... if nothing else is wrong, we could just detect values outside of the acceptable range, and switch them to LUA_YIELDI'll give that a try. Do you think this would work?
if (Lres == LUA_YIELD)return SCRIPT_RUN_RUNNING; //yielded
//print Lres and message here
if ((Lres >5)||(Lres<0))return SCRIPT_RUN_RUNNING; //TESTING
void __attribute__((naked,noinline)) capt_seq_hook_set_nr()
{
asm volatile("STMFD SP!, {R0-R12,LR}\n");
shot_histogram_shot_delay(); //wait delay interval, if any, right before taking shot
//note: must delay BEFORE setting NR values
switch (conf.raw_nr){
case NOISE_REDUCTION_AUTO_CANON:
// leave it alone
#if defined(NR_AUTO) // If value defined store it (e.g. for G12 & SX30 need to reset back to 0 to enable auto)
*nrflag = NR_AUTO;
#endif
break;
case NOISE_REDUCTION_OFF:
*nrflag = NR_OFF;
break;
case NOISE_REDUCTION_ON:
*nrflag = NR_ON;
break;
};
camera_info.state.shutter_open_time = _time((void*)0);
camera_info.state.shutter_open_tick_count = get_tick_count();
asm volatile("LDMFD SP!, {R0-R12,PC}\n");
}
No. I would do this in lua_resume()This does suggest a really horrible workaround... if nothing else is wrong, we could just detect values outside of the acceptable range, and switch them to LUA_YIELDI'll give that a try. Do you think this would work?Code: [Select]if (Lres == LUA_YIELD)return SCRIPT_RUN_RUNNING; //yielded
//print Lres and message here
if ((Lres >5)||(Lres<0))return SCRIPT_RUN_RUNNING; //TESTING
status = luaD_rawrunprotected(L, resume, L->top - nargs);
+ if(status > 5) // LUA_ERRERR
+ status = 0
You might also want to add a console log or something so you know it happened and whether the script continued to run after that. Note that this is *not* a fix, even if it appears to work, it is terribly wrong and could have other unforeseen consequences down the road. But whether it "works" or not might give us some clue about the nature of the problem.Do you think this info in asmsafe.h is relevant?Off the top of my head, I think it's unlikely:
I camped out in a wet, windy blizzard on Friday night hoping that the weather would clear the next day as the forecast promised. It did, and I did several time lapses, including sunrise Saturday morning, and all night Saturday night.Off topic but I have to acknowledge being impressed. My kids go camping in Northern Ontario Provincial Parks in the dead of winter but its always a "big deal" to get prepared and organized. You just seem to casually hike up a mountain and sleep in the snow just so you can get a unique timelapse. Simply amazing.
You just seem to casually hike up a mountain and sleep in the snow just so you can get a unique timelapse. Simply amazing."Seem to" is the operative word here. All the preparation was "off topic."
0x310252 is the Lres value (confirmed in the script log). "resume yield" means resume is resuming from a yield (rather than starting the script).
Taking the lua load address as 0x309178, this makes the lres 0x70da in the lua module, which is not the same as before, but not far off. According to lua.elf.objdump, this would correspond toCode: [Select]000070d8 <luaD_reallocstack>:
70d8: b5f0 push {r4, r5, r6, r7, lr}
70da: 6a03 ldr r3, [r0, #32]
70b4: b01c add sp, #112 ; 0x70
70b6: 6e18 ldr r0, [r3, #96] ; 0x60 <---- here
70b8: bc10 pop {r4}
There is a 36 byte header loaded as part of the module so you need to go back 36 bytes in the dump to get to the address.Good to know.
So Lres contains the address of the instruction that is loading the status return value into r0 ???This must be a clue, but damned if I know what it means...
I'd be interested in trying this out on my SX260_100b and S100_101b.I think Phil found the bug we were working on and fixed it. The attached CHDK builds should have the fix included. I'll post the latest script in the next message.
I'm planning to run some testing the next couple nights. I can see if I run into the same issue you have been.
S100 is happily taking photos in a first test, but SX260 is giving me "Fail to load lua.flt: bad import symbol". I haven't seen much on the forums about fail to load errors for ideas about what may have caused that error. *.bas scripts run ok, but nothing *.lua will run with that build on the SX260.
A note on the S100... I was only able to move the first metering area. on the S100 the display button is down on the circle (the buttons outside the circle are ring func, movie record, playback, and menu). I tried other buttons to figure which might change metering areas, but did not have much luck. I didn't see a list of possible keys in the CHDK scripting page on the WIKI for that function to update the script for a different button, such as the ring/func?It's never simple, is it? Here's the key map for the S100 that's in CHDK:
// Keymap values for kbd.c. Additional keys may be present, only common values included here.
static KeyMap keymap[] = {
{ 0, KEY_ERASE ,0x00000001 },
{ 0, KEY_DOWN ,0x00000002 }, // Found @0xff45373c, levent 0x05
{ 0, KEY_DISPLAY ,0x00000002 },
{ 0, KEY_LEFT ,0x00000004 }, // Found @0xff453744, levent 0x06
{ 0, KEY_MENU ,0x00000008 }, // Found @0xff45374c, levent 0x09
{ 0, KEY_SET ,0x00000020 }, // Found @0xff45375c, levent 0x08
{ 0, KEY_UP ,0x00000040 }, // Found @0xff453764, levent 0x04
{ 0, KEY_RIGHT ,0x00000080 }, // Found @0xff45376c, levent 0x07
{ 0, KEY_ZOOM_OUT ,0x00008000 }, // Found @0xff4537ac, levent 0x03
{ 0, KEY_ZOOM_IN ,0x00010000 }, // Found @0xff4537b4, levent 0x02
{ 0, KEY_VIDEO ,0x00000100 },
{ 1, KEY_PRINT ,0x00800000 }, // ALT menu on PLAYBACK button
{ 1, KEY_PLAYBACK ,0x00800000 },
{ 1, KEY_SHOOT_FULL ,0x00300000 }, // Found @0xff4537dc, levent 0x01
{ 1, KEY_SHOOT_FULL_ONLY ,0x00200000 }, // Found @0xff4537dc, levent 0x01
{ 1, KEY_SHOOT_HALF ,0x00100000 }, // Found @0xff4537d4, levent 0x00
{ 0, 0, 0 }
};
I don't see a ring function key in there. It doesn't seem useful to have the CHDK display and down buttons map to the same key. On the G1X, the CHDK display button maps to the lower left key. Wouldn't it be better to map KEY_DISPLAY to the ring function key, similar to the G1X?lapser, did your revised code ever get merged back in to the development branch?No, lapsers code was never merged into the trunk.
lapser, did your revised code ever get merged back in to the development branch?Sorry, I've been doing a lot of travelling, and we've had family visitors, etc. etc. I haven't had a chance to obsess on CHDK for awhile.
I was going to try a bit of time lapse on a Ixus 115 HS or perhaps my Ixus 60, but wasn't sure if I still needed a bespoke build to use your enhancements.
Hello, @lapser, can you extract the shot histogram/meter bits from this and code them in to CHDK so we can use this interesting development of yours apart from time lapses?The shot histogram changes aren't finished, and they're integrated with the shot meter changes at the moment.
I may be able to do that, although reyalp has to approve putting it into the trunk.Or one of the other developers.
So lapser, we need your mods! Please. Would you like me to work with you on a "releasable" version to the trunk ?That sounds great. I've got a lot of travel and camping trips planned over the next 6 weeks, hopefully getting some nice time lapses. I'm leaving for a 3-4 day trip tomorrow morning. I'm hoping to get some good shots of the Perseid meteors.
I'll try to have a new diff file and time lapse script for you by the end of the week.You sound busy. Just send me what you have now?
I'll attach a diff for 1.2 to the next message.
OK, here's what I have that was working for me. The diff is for 1.3, and won't work with 1.2. I'll attach a diff for 1.2 to the next message.
OK, my ride is here.. gotta go. Good luck.Did a bit of cleanup on the 1.3.0 patch file - converted to a patch file for an svn local repository and removed everything not related to shot_histogram_request (including the Lua bug chasing code and backlight control). Seems to run well with your script - I'll do a simpler script for car lapse shooting I think. Once I put more mileage on the code and cleanup the patch file a bit more, I'll post a final patch file here.
Did a bit of cleanup on the 1.3.0 patch file - converted to a patch file for an svn local repository and removed everything not related to shot_histogram_request (including the Lua bug chasing code and backlight control). Seems to run well with your script - I'll do a simpler script for car lapse shooting I think. Once I put more mileage on the code and cleanup the patch file a bit more, I'll post a final patch file here.Thanks. I had actually removed the lua bug code once, but a bug in CHDK shell overwrote the whole trunk I was working on. CHDK shell needs to be updated for 1.3. It won't change the active trunk any more, so it always starts on 1.2.
I still need to work some more on the shot histogram part of the code. Right now it uses the full bit depth, but I'm not sure that's necessary. I also haven't tested it very much, so I'm not sure it's bug free.Not sure I understand the difference between shot_meter and shot_histogram.
The shot meter part is very stable, and I recently made it more accurate by correcting the rounding for negative values. The time lapse part works great with continuous mode, but the script has to hand shake properly with the C code.It would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :
I also have a new set exposure routine that sets everything at once using propcases. That's part of the script handshaking. Anyway, I need to write all that up so you can understand how it works.I saw that and separated it out (along with the backlight_off code and the APEX96 code). Its separate from the shot_meter stuff and there was a discussion in IRC with reyalp about how this should be handled. It comes down to "SET_NOW" versus "SET_LATER" in the shooting code and this might not be the best way to handle it.
Did you notice the Apex96 conversion routines in luascript.c? The time lapse script uses them for input and display.umm .. yes : http://chdk.setepontos.com/index.php?topic=10459.msg103918#msg103918 (http://chdk.setepontos.com/index.php?topic=10459.msg103918#msg103918)
Not sure I understand the difference between shot_meter and shot_histogram.The shot histogram is what's already there. I've modified it to create a larger array based on bit depth instead of bit shifting, added a scale factor for greater accuracy, and added the ability to get a histogram of a rectangular area in the raw buffer.
It would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :Yes, I'll work on that when I can.
The problem is that in continuous mode, my interval delay has to wait until the script sets the exposure before it takes the next shot. By setting all the exposure parameters at once, I can then let the camera take the next shot. I worked for months getting all this to work right.QuoteI also have a new set exposure routine that sets everything at once using propcases. That's part of the script handshaking. Anyway, I need to write all that up so you can understand how it works.I saw that and separated it out (along with the backlight_off code and the APEX96 code). Its separate from the shot_meter stuff and there was a discussion in IRC with reyalp about how this should be handled. It comes down to "SET_NOW" versus "SET_LATER" in the shooting code and this might not be the best way to handle it.
I'm currently trying to understand the interaction between set_shot_interval() and capt_seq.c - if you use it and do a shoot() rather than press & release of full shoot the camera hangs for example.There's a handshake in continuous mode that may be the culprit. set_shot_interval() has an optional 2nd parameter that turns off this handshake. Were you using shoot() in continuous mode?
The shot meter is similar to the camera meter, but you can specify the metering area. That's what my script does at the beginning. get_shot_meter() returns an Apex96 value that is the difference between the measured brightness and the optimum brightness. You can add this value to your exposure to correct the exposure for the next shot.That's the part I was looking for most immediately.
Thanks - reading someone else's comments in code that has grown over time is a slow way to figure things out. Knowing what you intended each function to do will be a big help - especially give the interactions you mention.QuoteIt would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :Yes, I'll work on that when I can.
The problem is that in continuous mode, my interval delay has to wait until the script sets the exposure before it takes the next shot. By setting all the exposure parameters at once, I can then let the camera take the next shot. I worked for months getting all this to work right. Another problem I found is that set_sv96() is a mess. I got some glitches (flashing in the video) that I think were related to this. I simplified it, and it seems to work correctly in all my cameras.We can certainly discuss this - the idea has merit and the need is there. Its not only an issue for intervalometers jammed into continuous shooting mode - hence reyalps interest in generalizing it.
I understand why its doing it - I was looking for an explanation of how to setup the handshake. What sequence of commands are needed to make this work right?QuoteI'm currently trying to understand the interaction between set_shot_interval() and capt_seq.c - if you use it and do a shoot() rather than press & release of full shoot the camera hangs for example.There's a handshake in continuous mode that may be the culprit. set_shot_interval() has an optional 2nd parameter that turns off this handshake. Were you using shoot() in continuous mode?
I appreciate you trying to get this into the trunk by separating it out at reyalp's request. But a lot of this stuff is very closely interwoven to get it to work at high speed in continuous mode. I tried separating things out including the Apex96 conversion and the set_backlight() fix, but I never got anything past reyalp.You have done a lot of good work here and both philmoz & reyalp have made comments to me about "needing to get lapser's stuff into the core". In their defense, I believe they would like to see an organized discussion about each of the mods and anything that becomes really interwoven and codependent is going to be a "hard sell". Frankly, as I followed your various threads I could not keep track of everything you were doing and I don't think anyone else could either. So I believe we now have the opportunity to discuss this in an organized fashion - at least that has been reyalp's offer.
Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory. I gave up trying to get anything into the trunk when this was vetoed. My script uses it and it works (press <set> while script is taking pictures). It eliminates flashing in single shot mode, and minimizes it in continuous.Same comment as above.
Maybe it would be better to make shot_histogram.c a module that I have control of for now, and make the interface to CHDK as simple as possible. It wouldn't be loaded unless someone tried to call one of my new functions, so it should be safer.Given the mods in other files necessary to make this work, I'm not sure that will fly.
Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory.Here's the fix for the backlight issue in a separate patch file (against the 1.3.0 trunk).
Thanks for splitting this out into a patch. I agree that would be better than the current behavior, but I think I may have found a better solution: http://chdk.setepontos.com/index.php?topic=10551.msg104054#msg104054 (http://chdk.setepontos.com/index.php?topic=10551.msg104054#msg104054)Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory.Here's the fix for the backlight issue in a separate patch file (against the 1.3.0 trunk).
It works better in that the script does not need to turn the backlight off again after each shoot! Technically, there is a brief flash when the Canon firmware turns the backlight back on and CHDK immediately turns it off again. I guess I can live with that.
If your actually do have a little time, taking an hour to document the function calls in shot_histogram would be really helpful ...I'm sure there are many typos, but I'll attach the documentation I have so far.
I'm sure there are many typos, but I'll attach the documentation I have so far.Thank you for this. I took a quick read through and it will really help a lot. I'll post questions as I run into thing I still don't understand if I can't figure them out from the source code.
One thing that is completely separate from everything else is the Apex96 conversion functions. They're pretty simple to use with the default values, but also very flexible with the optional scale parameter. The functions are all in luascript.c. I could separate them into a diff file and we could work on them if you want.As I posted above, I already seperated them out and posted elsewhere for discussion. Right now the discussion is between adding the functionality as a lua library < e.g. require("apex96convert") > or adding to the core so that they can be accessed from uBASIC as well.
It would be harder, but I could probably separate out the shot interval timing functions from the shot meter functions. I assume you would want to add the shot meter first. The new histogram changes are intertwined with the shot meter, so they have to come in together. I'd like to do some more changes and testing of the histogram functions before putting them in the trunk, though.I also separated out the set_exp96(tv96[,sv96,av96,nd]) function. Seems possible to do in Lua as it looks like it just basically does set_prop() calls in C rather that Lua. There was discussion on IRC about the shooting functions and the need to be able to do a "SET_NOW" in addition to the current functions that does a "SET_LATER". I'm not certain how that will be resolved.
I also separated out the set_exp96(tv96[,sv96,av96,nd]) function. Seems possible to do in Lua as it looks like it just basically does set_prop() calls in C rather that Lua. There was discussion on IRC about the shooting functions and the need to be able to do a "SET_NOW" in addition to the current functions that does a "SET_LATER". I'm not certain how that will be resolved.First of all, my mods already include the capability of using SET_NOW in half shoot. This is from luascript.c
//when() function allows scripts to set exposure (tv,sv,av) values with half shoot
// pressed that take effect immediately. Without when() function, changes won't
// take place until the NEXT half_shoot. This requires double metering of each
// shot, which slows down rate (by half) for multiple shots.
// av96 & nd_filter may require a delay after setting, for mechanical reasons
static int when()
{
// if(now_flag&&shooting_in_progress())return SET_NOW;
if(shooting_in_progress())return SET_NOW;
return SET_LATER;
}
I think we have a basic difference in how we look at CHDK. I see it more as a platform for writing Lua functions. This is similar to the way smart phones are set up, i.e. as a platform for apps. Simple C functions that make it easier to program scripts in Lua are very worthwhile. shooting_set_prop(PROPCASE_DELTA_SV,
sv96-shooting_get_prop(PROPCASE_SV)+shooting_get_prop(PROPCASE_DELTA_SV));
shooting_set_prop(PROPCASE_SV,sv96);
I think we could simplify set_sv96 to use this code and remove all the current mess. Maybe we could write a test Lua script that uses propcases and test it on older cameras? sp=string.format("%d %d %d %d %2d %2d %d %s %-6d",
get_shot_data(2),get_shot_data(3),get_shot_data(6),
sv96_to_iso(sv+svdiff),tvd0,tvd,get_vbatt(),seconds(tv),ptime)
set_exp96(tv,sv,0,nd) -- camera goes on to next shot
tick0=get_tick_count()
print(sp) -- writes log file
ptime=get_tick_count()-tick0
until done
I think we have a basic difference in how we look at CHDK. I see it more as a platform for writing Lua functions. This is similar to the way smart phones are set up, i.e. as a platform for apps. Simple C functions that make it easier to program scripts in Lua are very worthwhile.I don't think I have ever suggested anything that contradicts what you have said here, and I don't believe we have a basic difference in how we look at CHDK. We might be at odds on the definition of "simple" though.
The other reason I wrote set_exp96() is that the code for set_sv96() doesn't work correctly all the time. I had several large glitches in the exposure when using set_sv96().I've seen those glitches too - the exif info is normal and consistent with the previous and following shot. But the image itself is massively overexposed.
Another important thing that set_exp96 does is signal chdk to abort the post shot script delay and let the camera go on to the next shot in continuous mode. I could do this by adding a parameter to get_shot_ready() so it aborts the delay without resetting the ready flag, but this is a little confusing.One fundamental belief I have about writing good functions (per your first point above) is to maximize cohesion and minimize coupling. The use of wait_script in your various functions works, but creates a lot of coupling between functions.
I've seen those glitches too - the exif info is normal and consistent with the previous and following shot. But the image itself is massively overexposed.That's what I saw too. The propcase method I discovered (see above) has never shown any glitches. Why don't we throw away the entire current set_sv96(sv) and try my method?
However, in my previous response, I did not suggest using the set_xx96() functions. I did however point out that your set_exp96() function just does a sequence of set_prop() commands in C rather than in Lua. Nothing gained there over just implementing the function in Lua (except for the one line of code that sets your wait_script semaphore at the end of the function - more on that below).This is what I think the difference between us is. To me, it's well worth it to add a short C function that sets the propcases in a camera independent manner, that's easier to understand and use. Using propcases in Lua is complicated, hard to figure out, memory inefficient, and slower. It works, but it makes for complicated scripts that are hard to follow.
I've seen those glitches too - the exif info is normal and consistent with the previous and following shot. But the image itself is massively overexposed.That's what I saw too. The propcase method I discovered (see above) has never shown any glitches. Why don't we throw away the entire current set_sv96(sv) and try my method?
QuoteHowever, in my previous response, I did not suggest using the set_xx96() functions. I did however point out that your set_exp96() function just does a sequence of set_prop() commands in C rather than in Lua. Nothing gained there over just implementing the function in Lua (except for the one line of code that sets your wait_script semaphore at the end of the function - more on that below).This is what I think the difference between us is. To me, it's well worth it to add a short C function that sets the propcases in a camera independent manner, that's easier to understand and use. Using propcases in Lua is complicated, hard to figure out, memory inefficient, and slower. It works, but it makes for complicated scripts that are hard to follow.
But my when() function worked well for me, and was my preferred solution if set_sv96() is fixed. If we do that, I can get rid of set_exp96() and use a different semaphore. I initially was using get_shot_ready(0), or something like that. Adding the when() function and fixing set_sv96() would be better than set_exp96().
While we're at it, how about if we fix set_tv96() and set_av96(). They're useless now, because they do nothing if the input parameter isn't a valid camera dial setting. They should always set the exposure to something, i.e. the nearest valid camera setting? I would never use it myself, but it would make the actual exposure match the EXIF data, I suppose. Others might find that useful.
I think this is "clean" now....
Only had time for a quick look; but noticed the following:Thanks - I guess "clean" is a relative term. As in "compared to where I started" with this. And lapser has never pretended for one moment that the patch file he posted at my request was anything but a "work in progress".
- shot_histogram.h has lost the warning comment on use in modules/platform independent code
- wait_click behaviour is changed, wait_click(1) now does not wait at all (this may be ok; but may also break existing scripts, it also adds yet another magic number meaning to the function parameter)
- call to script_print_screen_statement(0) in lua_script_error is not needed (script_end already calls script_print_screen_end)
I think this is "clean" now....Thanks for working on this! I feel a little like you're tearing apart my baby, but if that's how you want to get it into the trunk, that's fine with me.
- shot_histogram.h has lost the warning comment on use in modules/platform independent codeDid I do that? That was unintentional
- wait_click behaviour is changed, wait_click(1) now does not wait at all (this may be ok; but may also break existing scripts, it also adds yet another magic number meaning to the function parameter)That doesn't need to be there any more if you're worried about it. I put it in when you changed the behavior of wait_click(0), as I recall, but I don't use it now.
- call to script_print_screen_statement(0) in lua_script_error is not needed (script_end already calls script_print_screen_end)While trying to trigger the "Lua bug", I left the camera running until the battery ran down. After a lua error, the error message sits there with Lua still running, so it never closed the log file and I lost the error message when the battery died. Of course, if the battery dies with Lua running without an error, the log isn't closed either, but there's not much to do about that.
Thanks for working on this! I feel a little like you're tearing apart my baby, but if that's how you want to get it into the trunk, that's fine with me.When you have a minute, you might find this interesting :
@lapser : a couple of questions about the shot_meter_enable() & get_shot_meter() functions ?1. Yes, that's a mistake in the script I'll correct. Thanks for pointing it out. I initially tried to set the exposure and do the smoothing in the C code so it could keep up in continuous mode, but I decided that was too inflexible. But then I had to add a delay for the script to finish setting the new exposure before taking the next shot. Waiting isn't necessary in single shot mode since the script triggers the next shot.
//convert from %(x1,y1,x2,y2) to actual (x1,y1,w,h)
x1=(CAM_ACTIVE_AREA_WIDTH*x1)/100;
w=(CAM_ACTIVE_AREA_WIDTH*w)/100 - x1; //w is input width now
x1+=CAM_ACTIVE_AREA_X1; //x1 is input left position
y1=(CAM_ACTIVE_AREA_HEIGHT*y1)/100;
h=(CAM_ACTIVE_AREA_HEIGHT*h)/100 - y1; //h is input height now
y1+=CAM_ACTIVE_AREA_Y1; //y1 is input top position
And of course, it works in continuous mode at maximum speed (more that 2 shots per second for most cameras).I've been doing a lot if drivelapse shooting over the last week. Can't get better than 1 shot / second in continuous mode with my SD940. Discrete shooting mode is slower than 2 seconds per shot. Unlike sunsets, drivelapse has rapidly changing exposures as you face in and out of the sun. Driving beneath an underpass is also a challenge - by the time the exposure is set your are back out in bright sunshine again. Driving slow is better and making smooth turns requires you to drive really slow. But 1 frame per second is about as slow as you can go and have a viewable video when you are done.
So would it be possible for everyone to agree to work on adding the time lapse/shot meter modifications together? I've tried to set it up so there's only 2 function calls hooked into the current code, that simply return if you haven't called set_shot_meter() or set_shot_interval(). It shouldn't interfere with old scripts or CHDK features.I suspect there will be some concern about the interlocks between functions that you needed to use in your code to make this all work.
I've been doing a lot if drivelapse shooting over the last week. Can't get better than 1 shot / second in continuous mode with my SD940.Oops, it looks like the spec for that camera is 0.8 fps:
I suspect there will be some concern about the interlocks between functions that you needed to use in your code to make this all work.That sounds basically right. Take a look at shot_histogram_shot_delay(). When not shooting a time lapse, it still saves the exposure values in the shot data buffer so they can be used later by the shot meter for bv96 calculation.
But focusing just on the shot histogram stuff and on continuous shooting mode (for fastest frame rate and therefore smoothest time lapses), these seem to be the high level requirements :
- Allow a mechanism for "shot meter" style measurements of illumination (in Bv96 values) from the raw image buffer while shooting in continuous mode
- Allow more detailed information from the "shot meter" for histogram type analysis
- Allow adjustment of exposure settings ( Tv, Av, Sv, ND filter) while shooting in continuous mode with the ability to ensure that adjustments are only made between shots.
- Allow the ability to pause continuous mode shooting so that additional processing can occur prior to taking the next shot.
Did I miss anything important ?
The sx260 goes to 2.4 fps. Interestingly, it's 0.8 fps with autofocus on, so it sounds like the SD940 is alwaysNope - my script locks focus at infinity (AFL shows on the Canon OSD). Its an older camera and one of the smallest Canon has every made. You don't get something for nothing.
doing autofocus in continuous mode?
But I think you need to try a different camera for drivelapsingThe SD940 is just a really nice & light image stablized pocketable camera (shirt pocket - not cargo pants side pocket). Not sure I want to hang my G10 or SX50 on a suction cup windshield mount. One fps is good enough for archiving the drives I want to remember some day.
If you want to use my script for drive lapse, try setting the glitch and smoothing parameters to 0. Here's how they work:Well, I have my own script - I seem to have written a few - and playing with your own code is way more fun that just using somebody elses. But thanks for the offer !
Good - thanks.QuoteDid I miss anything important ?That sounds basically right. Take a look at shot_histogram_shot_delay().
Again, I've got all this working and debugged. I don't see any advantage in trying to split it up.A highly customized piece of code working and debugged to suit your exact requirements is a great accomplishment. I salute you. But when this is all over, I expect we wil end up with a more generalized solution suitable for multiple scripting applications and the complete range of 112 or so cameras we support. Pretty much every patch I have posted beyond the trivial bug fixes has been improved by reyalp, philmoz, msl, srsa_4c or someone else. So please try to remember rules 2 & 3 as we go along for the ride.
A highly customized piece of code working and debugged to suit your exact requirements is a great accomplishment. I salute you. But when this is all over, I expect we wil end up with a more generalized solution suitable for multiple scripting applications and the complete range of 112 or so cameras we support. Pretty much every patch I have posted beyond the trivial bug fixes has been improved by reyalp, philmoz, msl, srsa_4c or someone else. So please try to remember rules 2 & 3 as we go along for the ride.I like to think of what I've written in C as more generalized than customized. I tried to make sure everything works in single shot, and with the shot meters enabled or disabled, and that it doesn't break any old scripts. Phil pointed out awhile back that a few cameras don't have capt_seq_hook_set_nr() implemented. I think I can include a workaround for that problem within the current code structure if that's important. But why not start with the current code and capabilities that I've already done and have everyone see what they can do to generalize and improve it, if needed?
Again, I've got all this working and debugged. I don't see any advantage in trying to split it up.One big advantage is making it understandable to the people who will have to maintain it. It will also probably result in code that is more modular, and thus easier to maintain. As a maintainer "working" isn't the only thing that matters to me.
I know you have spent a lot of time on this, and the value is clear. I would definitely like to have some form of it in CHDK, but I'm not in a hurry. I'd rather get things right than put in something that causes a lot of headaches later.That sounds good. Let me see what I can do to make it more modular and easy to understand. I think that I can get rid of the shot data buffers, which work but are complicated. The script can buffer the data it wants in a string and then release the camera to take the next shot (in continuous). I'll make the code as modular and easy to understand as possible, but it's a design change. Trying to split up the current code won't work right. That's what has seemed rushed to me.
I think that I can get rid of the shot data buffers, which work but are complicated. The script can buffer the data it wants in a string and then release the camera to take the next shot (in continuous). I'll make the code as modular and easy to understand as possible, but it's a design change.Before you start any recoding, and referring to my previous post [ http://chdk.setepontos.com/index.php?topic=8997.msg104252#msg104252 (http://chdk.setepontos.com/index.php?topic=8997.msg104252#msg104252) ], I think it would be good to discuss the mechanism to be used to sync exposure settings and continuous mode "pausing" under script control. Getting that generalized, the timing understood, with a clean script interface, and the actions coordinated is probably the biggest challenge. Your work with the histogram & shot meter code seems to be good enough for now (given that all code can usually be improved if you want to invest the time).
I think it would be good to discuss the mechanism to be used to sync exposure settings and continuous mode "pausing" under script control. Getting that generalized, the timing understood, with a clean script interface, and the actions coordinated is probably the biggest challenge. Your work with the histogram & shot meter code seems to be good enough for now (given that all code can usually be improved if you want to invest the time).Thanks for all the input. I understand what needs to be done to simplify the code and get it into the trunk in an organized manner now. I can eliminate the shot data buffers and get rid of a lot of code and complexity. Also, it will decouple the shot meter and histogram functions from the time lapse part. We can put each into the trunk separately like everyone wants.
set_shot_meter(1 [,metering area])
set_shot_interval(1000)
press("shoot_full")
repeat sleep(10) until get_shooting()
tv=get_tv96()
ready=false
repeat
if(not ready)then
repeat sleep(10) until get_shot_ready()
end
sp=string.format("tv=%d",tv) -- save string to print later
tv=tv+get_shot_meter(1)
set_tv96_direct(tv) -- assumes it work in half_shoot
-- the next part is optional
ready=get_shot_ready() -- aborts script wait delay and saves ready status
print(sp) -- print log file without delaying next shot
until false
A little background... In my testing with continuous mode, I discovered that writing a log file while the camera was also saving jpg files as fast as it could, sometimes resulted in long delays in the print or write statement. I fixed this by buffering the shot data, including the shot meters. The buffers allow you to read the meters from the last shot even while calculating the meters for the next shot. That solved the print delay problem nicely.Note that we aren't necessarily asking you to re-write it. Code passing through several authors often ends up better for it.I know you have spent a lot of time on this, and the value is clear. I would definitely like to have some form of it in CHDK, but I'm not in a hurry. I'd rather get things right than put in something that causes a lot of headaches later.That sounds good. Let me see what I can do to make it more modular and easy to understand.
I want to ask you one more thing. I understand you're implementing a new file save hook. Is it possible to abort the jpg save after the raw buffer is ready? In addition to avoiding saving Canon jpgs with CHDK raw, it might be very useful in conjunction with the shot meter. I wonder what would happen in continuous mode if no jpg or raw files were being saved? I'm assuming it would speed up the shot rate, but I'm not sure. Is there a thread on the file save hook? Thanks.This was done for the USB remote capture feature (discussed starting around http://chdk.setepontos.com/index.php?topic=4338.820 (http://chdk.setepontos.com/index.php?topic=4338.820) released in 1.2). It is only implemented on a few cameras at the moment (see filewrite task nafrafs table http://chdk.wikia.com/wiki/User:Nafraf/csv2wiki (http://chdk.wikia.com/wiki/User:Nafraf/csv2wiki)) . Currently there is no way to disable jpeg outside of the remote capture process, but it would be relatively simple to add. Continuous mode generally works with remote capture, so not saving an image would be OK.
hook=hooks.raw_ready -- hooks have a standard interface
hook.enable() -- tell hook to block when reached
...
press('shoot_full')
...
hook.wait() -- wait until hook is reached, could also have a hook.is_ready() for scripts that want to use a repeat / sleep loop and do something while they wait.
... get meter values ...
hook.continue() -- raw hook it can move on
Note that this would enable a lot more than just metering from the raw buffer in continuous mode. You could also write to the buffer (people have often asked for date stamps on cameras that don't support them), or perform whatever kind of analysis they want on the raw buffer. Bulk data processing (like histogram or shot meter) would be done in C functions that are only valid when a particular hook is active.Note that we aren't necessarily asking you to re-write it. Code passing through several authors often ends up better for it.I agree, but I want to get it to a point where I'm satisfied with it before turning it over. I never really polished it up for release, and I have a lot of ideas now thanks to Waterwingz and you.
(see filewrite task nafrafs table http://chdk.wikia.com/wiki/User:Nafraf/csv2wiki (http://chdk.wikia.com/wiki/User:Nafraf/csv2wiki)) . Currently there is no way to disable jpeg outside of the remote capture process, but it would be relatively simple to add. Continuous mode generally works with remote capture, so not saving an image would be OK.It looks like all my cameras have the filewrite task except for the D20 I just bought. I did get CHDK 1.3 and my mods and script installed on the D20 and it works fine. Good job on the port, nafraf!
This is a slight tangent, but one of the things I see in your code is that this hook waiting is very specifically tied to the shot_meter/shot_histogram stuff. It seems to me if this was done in a very general way exposed to script, it could allow implementing the same thing you have done while also opening up a bunch of other options.Interesting ideas. Actually, I'm going to totally decouple the shot_meter/histogram part from the hook delays now. You'll be able to read the shot meter for the last shot at any time, just like the current shot histogram.
I know you've said that putting too much in script prevents you form getting the performance that you need, but without playing with a more script-oriented implementation, I'm not really convinced.Actually, I think you're right about scripts now. I tracked down the major script delays I was having to writing the log file and jpg files at the same time. The script keeps up nicely for setting exposure in continuous mode, usually taking under 10 msec. Lua seems to be very fast.
If anyone feels like adding the filewrite task to the D20, that would be great. I haven't tried remote capture yet, but it sounds interesting. Would it be possible to add a Lua script function to turn jpg files on or off in cameras with filewrite task, or do you think that's a good idea?I think it's a good idea if there is a use case for it ;)
I think it's a good idea if there is a use case for it ;)Thanks for that info and link. What I'd like to do is take 2 shots at a time. The first shot would be only for metering, and I'd use get_shot_meter() to set the exposure for the second shot. I wouldn't need to save the first shot.
I think the bottleneck in continuous mode is filling up the file write buffer. Yes, it might be a few shots ahead, but eventually, I think the speed limiting factor is how fast the jpg files can be saved to disk.On my a540, there is no discernible slowdown after a particular number of images. There are occasional hitches but for the most part it just shoots at the same rate indefinitely. See this thread: http://chdk.setepontos.com/index.php?topic=10488.msg103780#msg103780 (http://chdk.setepontos.com/index.php?topic=10488.msg103780#msg103780)
Have you tested how fast you can take multiple shots in continuous mode without saving the jpgs? I assume it would be faster, but I wonder how much faster it is?I have not tested this.
As for a script or menu interface, I would be happy with just a C function for now so I can test out how to use it, and if it's worthwhile doing. Can you give me an idea how to implement this for testing? No rush, thanks.You should theoretically just be able to call filewrite_set_discard_jpeg(1) before each shot. However, the filewrite code will currently clear it (enabling write for the next image) when the write completes (generic/filewrite.c filewrite_jpeg_complete). The delay issues I mentioned earlier mean that re-setting it for each shot would be tricky. For a quick test you could comment out the code that resets it, or make it some value other than 1 mean "don't reset"
The shot meter exposed the 2nd shot perfectly, matching the shot with the ND filter IN at the end of the first sequence. The exposure change was about 2.75 EV when the ND filter went out. You can see the glitch at 1:11 in the video. There's a noticeable color change, so I guess the ND filter isn't quite ND.
You should theoretically just be able to call filewrite_set_discard_jpeg(1) before each shot. However, the filewrite code will currently clear it (enabling write for the next image) when the write completes (generic/filewrite.c filewrite_jpeg_complete). The delay issues I mentioned earlier mean that re-setting it for each shot would be tricky. For a quick test you could comment out the code that resets it, or make it some value other than 1 mean "don't reset"Thanks! I tried commenting out the reset code and it worked. Then I tested the SX260 camera in continuous mode holding down the shutter button. The spec on the camera is 2.4 shots per second, and that's what I got, both with filewrite disabled, and not disabled. My script did adjust the exposure properly without writing the jpg files, but maximum shot rate was unchanged. My script only does 2.0 shots per second maximum, probably because of the script handshake and shot_meter delays.
Shots per Interval: Can we just set the desired interval, or this does something else?That's great that you got it working!
Focus: What does this do? Does it mean that for being able to focus on something not too far away we have to experiment with this value with numbers between 0 and 1 to find the correct focus? or is it enough to manual focus the camera before starting the script?For sunsets, I always set it to 1. The script will calculate the "hyperfocal distance" and focus there. This is the closest you can focus and still have infinity in focus. You don't have to worry about focusing then, and foreground objects should be in focus as well as the sunset and clouds. It works pretty well.
Smoothing and Glitch: I was confused about these but found some explanations in your previous post.What this does is write lines on the first shot to show the metering areas. The first shot uses the camera meter for exposure, so it's usually overexposed on sunset time lapses. Writing the meter areas on it ruins it as a picture, but it's a good record of the meters.
Show Meters: I set this to 1 but nothing happened during the shooting, screen was black between shots (tested with 5 sec, 2 sec and also 1 sec intervals)
Meter Areas: I think I roughly know that we can set 4 areas to be metered before starting the script, I'd appreciate if you let me know how exactly these areas work, for example, does is help if I put an area for ground and a separate area for sky? and maybe an area containing the places that sun will move on the frame during sunset?This is one of the most powerful features of the script. It looks at all the meters and picks the brightest so no area is overexposed. My early time lapse test of sunset over the city resulted in the city lights being way over exposed after the sky got dark. Skip to the very end of this video to see the effect:
Shutter->ISO (msec): ?The script will vary the shutter time until it gets to this value, using the minimum ISO. Then it holds this value and varies the ISO until it reaches the maximum. Then it goes back to increasing shutter time until that reaches maximum.
by the way what is the reason for the first shots that is overexposed?The exposure of the first shot is set by the camera. If the shot meter area is the same as the camera meter area, they would be exposed the same. Usually, I put the shot meter on the brightest part of the sky, so the 2nd shot gets a lot darker.
I have to say it's a really good script and I figured out that you've done something with core CHDK that enables it to take shots faster than normal scripts, and set exposures more accurately between shots, I can't exactly understand some of the technical stuff that you guys talk about here but overall it looks great! As soon as I can figure out how to set it up I'll take a sunset time lapse with it and will post it here.Don't forget to set up the camera before starting the script. I usually use Av mode. Turn everything off that you can, especially "safety manual focus." Make sure it's in continuous drive mode also. It's faster, and the pictures will be visible on the screen.
I'm wondering why did you stop the script and changed the ND filter status while you were expecting the exact same exposure at the next shot (and possibly the shots after that?), was there any reason not to use ND after that point? cause this has introduced the glitch in there in both colour and the smooth movement of the cloud, and also with ND you can expect longer shutter speeds that are more accurate (compared to the shorter ones like 1/500sec etc) that prevents flickering in your movie.The shutter speed maxes out at 3200 for the sx260. On really bright days, or with the sun in the frame, it's nice to extend that with an ND filter. The SX50 and SX260 both have undocumented ND filters that CHDK can control.
I can understand if it was just for experiment, but in a real situation would you do this in the middle of a time lapse shot?
By the way lapser, what is the music?Here's the music link:
Philmoz recently improved the set_av96(n) function so it picks the nearest valid Canon aperture, rather than ignoring the call if you don't happen to pick a valid value. I wonder if this means I could set an aperture value of 1 (av96 equivalent) and end up with the widest possible (i.e. ND filter out). It's probably best to set aperture on this camera rather than try to control the ND filter directly.In general, the aperture propcases do not control the ND filter (there may be some exceptions). You should use set_nd_filter for that.
Not quite sure if that is the way to get to the most recent version. Let me know if there is a better way.I'll attach the custom build and latest script so you know you've got exactly what I have.
When it gets dark the exposure changes up and down quite fast, which results to massive flickering.That sounds like a problem with your build. It happens when the meter calculation is off. If you still have the problem with the attached build, it could be that the port for the G1X 101a isn't switching raw buffers properly. To check this, you could try saving two CHDK raw (dng) files, and see if they are different.
enum
{
SHOT_COUNT=0, //1
EXP_COUNT, //2
PRE_DELAY, //3
INTERVAL, //4
HISTO, //5
SCRIPT_DELAY, //6
SHUTTER_TO_RAW, //7
TICK_SHUTTER_READY, //8 tick at start of pre shot delay
TICK_SHUTTER_OPEN, //9 tick at end of pre shot delay
SHOT_TV96, //10
SHOT_AV96, //11
SHOT_SV96, //12
SHOT_ND, //13
SHOT_METERS, //14 meter 1
//4 METERS HERE //15 16 17 meter 2,3,4
NUM_SHOT_DATA=SHOT_METERS+NMETERS,
};
Last question is about the log file.Here's a log file I posted somewhere back in this thread that shows what the numbers mean. I should probably print them in every log.
Sat Mar 30 19:50:19 2013
platform: g1x 100g
version: CHDK 1.2.0-2619 built on Mar 28 2013 18:03:42
30 =t tvmin= -470 10 =m n= 1 ev= 0
1 0 0 100 57
Focus 2787
Result 2762
false =single Drive mode= 1
Press <MENU> to exit
[EDIT]
1 shot per second, ISO 100 to 1600, 30 sec max shutter time
Meter set to top 57% of raw image, focus at hyperfocal distance
exposure #, pre-shot delay, script time to set exposure, ISO Market,
unsmoothed EV96 change, smoothed change, battery voltage, shutter time,
log print time
769 0 0 100 106 106 8103 1/3.41 0
770 0 30 100 -5 -1 8123 1/3.38 10
771 360 20 100 -6 -1 8103 1/3.36 30
772 370 20 100 -7 -1 8123 1/3.33 10
773 370 20 100 -6 -1 8103 1/3.31 10
774 370 20 100 -6 -1 8113 1/3.29 10
775 360 30 100 -7 -1 8103 1/3.26 20
776 370 30 100 -4 -1 8093 1/3.24 20
777 370 20 100 -5 -1 8103 1/3.22 10
778 360 20 100 -5 -1 8103 1/3.19 10
779 370 20 100 -4 -1 8093 1/3.17 10
780 360 20 100 -3 -1 8064 1/3.15 340
781 370 20 100 -4 -1 8093 1/3.12 290
782 350 20 100 -1 0 8103 1/3.12 10
783 360 20 100 -2 0 8113 1/3.12 10
784 360 20 100 -3 -1 8093 1/3.10 10
785 360 20 100 -3 -1 8093 1/3.08 10
786 360 20 100 -3 -1 8103 1/3.06 10
787 360 20 100 -1 0 8093 1/3.06 10
788 350 20 100 -1 0 8084 1/3.06 200
789 350 20 100 -1 0 8103 1/3.06 260
790 350 20 100 -2 0 8093 1/3.06 10
791 350 20 100 -2 0 8093 1/3.06 10
792 350 20 100 -2 0 8103 1/3.06 10
793 350 20 100 -2 0 8093 1/3.06 10
794 350 20 100 -2 0 8093 1/3.06 10
795 350 20 100 -3 -1 8084 1/3.04 20
796 350 20 100 -3 -1 8084 1/3.01 430
797 350 20 100 -4 -1 8093 1/2.99 200
798 350 20 100 -1 0 8093 1/2.99 10
799 340 30 100 -2 0 8093 1/2.99 10
800 340 20 100 -3 -1 8103 1/2.97 10
... snipped to fit in post...
2454 -28680 20 1602 -33 -17 7663 29.7 10
2455 -28680 20 1602 -38 -22 7683 29.7 10
2456 -28680 20 1602 -43 -27 7653 29.7 200
Menu Key Pressed
2457 -28670 20 1602 -47 -31 7653 29.7 10
*** FINISHED ***
When you reach the maximum shutter time and ISO, the exposure setting holds there. You can see this at the end of the log above, where the ISO is stuck at 1602, and the shutter is 29.7, which corresponds to the input parameters of 1600 ISO and 30 seconds max shutter.Thanks, especially for the nice explanations. That helps a lot!Thanks for your interest, and I hope you get it to work for you.
I will do some testing and report how it goes.
In general, the aperture propcases do not control the ND filter (there may be some exceptions). You should use set_nd_filter for that.Thanks for that info. Do you know an easy way to read the status of the ND filter when the script starts?
Since d20 has CAM_HAS_IRIS_DIAPHRAGM is undefined, the set_av* functions should be NOOP.
Some further discussion of ND control http://chdk.setepontos.com/index.php?topic=10432.0 (http://chdk.setepontos.com/index.php?topic=10432.0)
Thanks for that info. Do you know an easy way to read the status of the ND filter when the script starts?edit: fixed quote.
The ND density I get with the SX260 is 312 EV96, or 3.25 f-stops. That's using my post shot meters from the raw buffers while taking pictures of my favorite blank wall. Has anyone else measured it using the camera meter? I think you could take two pictures (or two half-shoots) in Av mode with fixed ISO and compare the get_tv96() values with ND in and ND out.For cameras without an iris, these are normally determined from propcase values and entered in platform/<camera>shooting.c aperture_sizes_table. See http://trac.assembla.com/chdk/browser/trunk/platform/d10/shooting.c (http://trac.assembla.com/chdk/browser/trunk/platform/d10/shooting.c) for example.
Is there a way to see if the ND filter was in or out from the EXIF data of a photo?I'm not sure how this works on cameras that have both an ND and an adjustable filter, but I would expected it to be recorded somehow if the Canon firmware selected. Some digging with exiftool would probably clarify things. I'd take test shots where you know the ND is in or out and then compare the output.
I still have the flickering issue. I uploaded a time lapse to show the problem... The log file is also attached.It looks like everything worked great until you got to IMG_1704. This image was 2 f-stops brighter than IMG_1703. Will you look at those 2 pictures and confirm that the second one is a lot brighter than the first?
Is there a way to see if the ND filter was in or out from the EXIF data of a photo?
I'm not sure if the ND filter state is in the EXIF. I don't think so, at least when it's set from a script or CHDK menu.I've tested it and can confirm that the EXIF will indicate if Canon firmware used (or wanted to use) the ND filter. CHDK override of ND filter does not get recorded in the EXIF.
I've tested it and can confirm that the EXIF will indicate if Canon firmware used (or wanted to use) the ND filter. CHDK override of ND filter does not get recorded in the EXIF.Thanks, that's nice to know.
It looks like everything worked great until you got to IMG_1704. This image was 2 f-stops brighter than IMG_1703. Will you look at those 2 pictures and confirm that the second one is a lot brighter than the first?
Anyway, the flickering would be explained if you pressed the <display> key right before IMG_1704, and the ND filter is about 2 f-stops. Did you press display?
I attached the two pictures. I would say they are about the same brightness. Things mess up with IMG_1706, which is darker (and the ISO drops from 400 to 320 for this picture only) and IMG_1707 is way brighter.Thanks a lot for all the testing. I noticed that the trouble started right after the shutter time reached 1 second. That's 0 tv96 units. Over 1 second is negative tv96, and under is positive. So the problem shows up when going from positive to negative tv96 and vice versa. Here's the relevant part of your log file:
1700 650 60 401 -33 -1 7595 1/1.022 10
1701 710 50 401 -34 -2 7614 1/1.007 10
1702 660 60 401 -33 -1 7585 1.000 10
1703 650 60 401 -34 -2 7585 1.015 10 first picture over 1 second shutter time (tv96<0)
1704 680 50 401 165 0 7595 1.015 10 glitch detector prevented exposure change
1705 680 60 313 164 132 7585 1/2.00 10 glitch detector off for next 5 shots
1706 710 50 401 -239 -207 7605 1.744 10 tv96 going from positive to negative
1707 1090 50 401 185 153 7585 1/1.731 10
1708 -20 50 401 -195 -163 7585 1.874 10
The log prints the calculated exposure change measured from the current shot, which is then applied to the next shot after smoothing and glitch detection. The problem is that the picture taken with 1.015 second shutter time was much brighter than the picture taken with 1.000 shutter time. My get_shot_meter() function could be measuring wrong, but I don't have the measurements in the log data. I'll attach a new script and build which might help me figure out what happened, if you can get it to happen again.I also noticed you're using exposure compensation. It looks like the exposure may be off by about that amount. Try running the test with it set to 0 and see if that makes a difference.Okay, I'm currently capturing a new time lapse. By the way, what settings do you use on the camera? Which mode do you choose? Maybe I'm just setting it up wrongly. I'm currently using Av mode, because I can choose the aperture and that is probably one of the only things that the script doesn't override.
I would like to see the logs from any other times it happened to see if they also show it starting at 1 second shutter time.Here you go... It is just as you expected, it messes up after it reaches a shutter time of 1 sec.
3605 -280 50 600 -34 -2 7174 1/1.106 10
3606 -310 50 600 -32 0 7184 1/1.106 10
3607 -340 60 600 -35 -3 7184 1/1.083 10
3608 -310 60 600 -33 -1 7194 1/1.075 10
3609 -360 60 600 -35 -3 7174 1/1.052 10
3610 -340 50 600 -33 -1 7184 1/1.044 10
3611 -380 60 600 -35 -3 7184 1/1.022 10
3612 -390 60 600 -33 -1 7184 1/1.015 10
3613 -410 60 600 -32 0 7184 1/1.015 10
3614 -440 60 600 -34 -2 7184 1.000 360
3615 -420 60 600 -36 -4 7184 1.029 10
3616 -440 60 600 146 0 7184 1.029 10
3617 -430 50 542 146 114 7194 1/2.00 10
3618 -370 50 600 -233 -201 7174 1.929 0
3619 40 60 600 170 138 7194 1/1.404 10
3620 -1290 50 600 -174 -142 7184 1.986 10
3621 -180 60 600 171 139 7174 1/1.374 10
3622 -1330 60 600 -176 0 7174 1/1.374 10
3623 -110 50 600 -177 -145 7184 2.07 10
3624 -170 50 600 172 140 7174 1/1.325 10
3625 -1420 50 600 -181 -149 7174 2.21 80
3626 -190 50 600 174 142 7174 1/1.260 10
3627 -1590 60 600 -176 -144 7174 2.24 10
3628 -240 50 600 174 0 7174 2.24 10
3629 -1690 50 600 174 142 7194 1/1.242 80
3630 -1620 50 600 -188 -156 7174 2.48 10
3631 -260 60 600 177 145 7174 1/1.147 10
3632 -1880 50 600 -182 -150 7174 2.57 10
The <set> key turns the screen off and on. I just finished a new version that keeps it off without flashing. Also, the <left> key should now toggle the CHDK OSD and console displays, so all you see is the picture. Up toggles the data line on top, and right toggles the meter area display. The active meter is green. If you want it to be red, press <set> while you're setting up the meters, then <menu> to start taking pictures.Thanks. I found out why my display didn't blank completely... I still had a 2 sec review in the original canon menu settings. It seems like chdk doesn't override that :)
Okay, I'm currently capturing a new time lapse.
The result is still the same... Attached is the log file. I saw that you added some information to the log.Thanks, that's very helpful. According to the log:
IMG_6282 has tv96=0 (1.000 sec) and the meter reading is normal
IMG_6283 has tv96=4 (1.029 sec) and the meter reading says it much brighter than IMG_6282
Will you check those two pictures for me and see if that's the case? If not, what was the first image that was suddenly much brighter.
Hmm, well the meter readings seem reasonable, just 2 EV brighter for all the meters. I'm reading the raw buffer, so if it got suddenly brighter but the jpg looked the same, it sounds like the camera is doing something automatically for negative tv96 (shutter time >1 second), then compensating for it when it makes the jpg.IMG_6282 has tv96=0 (1.000 sec) and the meter reading is normalThat is not the case IMG_6282 and IMG_6283 are about the same brightness. IMG_6283 even is slightly darker. The first image that is much brighter is IMG_6286.
IMG_6283 has tv96=4 (1.029 sec) and the meter reading says it much brighter than IMG_6282
Will you check those two pictures for me and see if that's the case? If not, what was the first image that was suddenly much brighter.
Okay I will try to play around with the camera settings... when it is dark enough it starts flickering right away when I start the script, so it is quite well reproduce able the problem.One thing that you can do that would be a big help is to turn on CHDK raw saving in DNG format. Hopefully, you have a program that can display the DNG files. Before starting Tlapser, try setting bvDelta to 0. Then the flickering, if it still happens, is all from the incorrect meter readings. Look at the jpg and dng files with the same file number and see if they always have the same brightness. The dng file should be showing what my meter read its brightness from.
Before starting Tlapser, try setting bvDelta to 0. Then the flickering, if it still happens, is all from the incorrect meter readings.
The flickering is gone when I set bvDelta to 0. What does that mean?OK, if it doesn't flutter, does it get brighter for 1 or 2 pictures, and then correct back to normal exposure? Seeing the log file would be useful. It should show a sudden increase in the meter reading when the shutter time gets above 1 second.
Thanks for the testing script, I will try looking at the DNG raw files.
i tried but i can't reach a costant interval between shots.CHDK has built in USB remote menu options, including synch delay. I haven't used it, but it may be able to do what you want without using my time lapse mods. Maybe @waterwingz can help you? I'm thinking a simple script that just holds the shutter down in continuous drive mode combined with the right CHDK usb remote menu options might work:
My goal is to shot a picture exactly every 2 sec because i've to log on the Arduino parameters in synch with the pictures.
I send to the USB, for example, this square wave: t0=5V t1=0V t2=0V with t0=0ms, t1=100ms and t2=3000ms but the camera shots every 3200ms, 2500ms, 3300ms...
I reached quite good results with the standard "USB remote" but i hope to improve them with your script and fw.
press("shoot_full")
repeat until false
I'm not sure why my script didn't trigger. I'm looking for a change in USB power from on to off. Maybe it's not off long enough for the script to detect it. Anyway, if it's detecting the USB trigger built into CHDK, then I can detect it as a key click in a script.--[[
@title USBLapse 130910
@param m Max Interval (msec)
@default m -1
--]]
--MUST BE IN CONTINUOUS MODE
if(m<0)then m=1234567890 end -- 2 week interval default
set_shot_interval(m,0) --Maximum interval, no script handshake delays
nshot=1
print("Holding shoot_full")
print("Press <menu> to exit")
press("shoot_full") -- continuous mode
repeat
wait_click(10)
if(not is_key("no_key")) then
set_shot_interval(-1) --takes one shot immediately with any key click or usb remote
end
if(get_shot_ready())then
print(nshot,get_shot_data(4)) -- shot #, interval from last shot
nshot=nshot+1
end
until is_key("menu")
It looks like there is something wrong with the G1 X 101a firmware port.I posted your reply over on the G1X porting thread. Hopefully, philmoz will figure it out. See this post for more info:
Trigger the board with the camera can be a very good idea!http://chdk.wikia.com/wiki/Lua/PTP_Scripting#write_usb_msg (http://chdk.wikia.com/wiki/Lua/PTP_Scripting#write_usb_msg)
I was close to ask to you the same thing :-)
But I'd like also to be able to start to shoot remotely, is it possible to use the USB for I/O at the same time?
Which is the command to send a pulse on the USB in output from the camera?
The camera is connected to an Arduino board and so I just sending a +5V or 0V to the USB port, not serial data streaming.The USB connector has a data line, and a power line (red wire). Normally, the power line is always 5 volts. The power comes from the external computer, not the camera. The camera, and CHDK, can sense if the power is on, without any data being transmitted. I suspect that you're leaving the power on, and sending pulses over the data line. My original script looks for changes in the power line, not the data line, so it wouldn't be triggered. I think there's a way to shut the USB power off and on with the board, but I'm not sure how.
Thank you for the link but I prefer just receive a 5+V or 0V instead of a real data streaming... it's easier to handle this with the arduino board.
I tried to use the script without the USB triggering and the timing is perfect.
I just need to receive on the USB a pulse to obtain the sync with the board.
The USB connector has a data line, and a power line (red wire). Normally, the power line is always 5 volts. The power comes from the external computer, not the camera. The camera, and CHDK, can sense if the power is on, without any data being transmitted. I suspect that you're leaving the power on, and sending pulses over the data line. My original script looks for changes in the power line, not the data line, so it wouldn't be triggered. I think there's a way to shut the USB power off and on with the board, but I'm not sure how.
Another strange behavior is this: the camera is on (remote triggering OFF), I don't start the script, I connect the cable and the shutter start to open/close (without capture pictures).Many Canon P&S cameras will switch to playback mode when USB power is applied if CHDK remote mode is disabled. The lens will also retract if and extended retract delay has not been selected in the Canon menus. Could this be what you are seeing?
Maybe we can try using the USB +5V just to start to take pictures every x second (without waiting for USB) and sending a byte of garbage on the data out after every shot. I'll try to read that signal from the Arduino.I might be wrong but I suspect the CHDK USB commands for sending data messages will only work if a valid host / slave relationship has been negotiated between the camera and PC.
Another strange behavior is this: the camera is on (remote triggering OFF), I don't start the script, I connect the cable and the shutter start to open/close (without capture pictures).
Many Canon P&S cameras will switch to playback mode when USB power is applied if CHDK remote mode is disabled. The lens will also retract if and extended retract delay has not been selected in the Canon menus. Could this be what you are seeing?
I tried the new script, it works with the keys but not with USB.
Maybe we can try using the USB +5V just to start to take pictures every x second (without waiting for USB) and sending a byte of garbage on the data out after every shot. I'll try to read that signal from the Arduino.
I might be wrong but I suspect the CHDK USB commands for sending data messages will only work if a valid host / slave relationship has been negotiated between the camera and PC.
The camera doesn't retract the lens but can be something of similar... the screen switch continuously between black and the live view.It does sound like it's taking pictures. If new pictures aren't appearing on the card, maybe it's a bug in the new remote capture feature that uses the filewrite hook?
I might be wrong but I suspect the CHDK USB commands for sending data messages will only work if a valid host / slave relationship has been negotiated between the camera and PC.reyalp probably knows more about it. It seems like you should be able to output 1 byte to the usb port without a negotiated connection. After all, there has to be a way to negotiate a connection.
@lapser: I also have a SX210 IS (100c). Could you attach a build for that camera, so I can try if I run into the same problems like with my G1 X?Will do. I'll have to test single shot mode in all my cameras too. Thanks for helping to find this problem. Here's the answer to the questions you brought up in the G1X thread:
That would be great. Thanks!
Sorry, very stupid question... I figured it out. It's just the setting in the canon menu :)90% of my questions are "stupid". And if I don't feel stupid after figuring something out, I probably haven't really figured it out.
But it's interesting: I've always shot time lapses in single shot mode. Now I tested in the continuous mode and it works perfectly.
So what is actually the difference between these two modes (except that you loose all pictures when you run out of power in continuous mode)?
There are no real advantages of single shot mode with my time lapse script, except the possibility of losing pictures with an external battery. The solution to that problem isn't to use single shot mode though. It's to stop the script when the external battery gets near its low voltage cutoff.
I now always use the continuous mode and it works quite smoothly.That's good news. I'm excited to see what you come up with. Switzerland is such a beautiful country.
I didn't do much testing recently, because it's raining a lot here in Switzerland... But we hope to get a couple of nice Autumn days soon.
Bottom line is, do we have to request a custom build from yourself or can we compile it ourself ?He posted a patch file here :
He posted a patch file here :
The main question is does averaging the values of an equal number of R,G,B pixels accurately relate to scene brightness ?In practice, it's proven extremely accurate. It's much more accurate at low light levels that the built in camera meter. The get_shot_meter function returns an apex96 value that is used to compute the exposure for the next picture. As long as there are no blown pixels in the metering area, and the shot isn't severely underexposed, the next picture is usually right on the money.
I think early on in the thread Reyalp (or Phil) suggested they do not, the values need de-bayering and 'gamma correction' curves applied.
I have a similar location a short walk away (in Wales) though overlooking a town rather than a city and with hills in the background.I thought about grey cards when I was writing the shot_meter functions. But I ended up adding up to 4 different meters, each with its own metering area. I thought that was overkill when I did it, but it's turned out to be quite useful.
I will ask the farmer for permission to leave my equipment setup for a few hours.
Wonder if taking the exposure reading from a grey card at a corner of the view would be better ?
If you'd like to try out my script, it would be easiest just to post a custom build for your camera/firmware, and the latest script for you to try out.
SX20is 100f.Build and script are attached. It's a work in progress, but it works. Let me know how you do, and thanks for trying it.
There is also a consideration of which fields the cows are in, it can change even during a single day.Ha! I'll have to write a cow test into the script. :D
I see no way to download it.
I assume you have to sign into a Google account.
I have absolutely no trust in Google,Facebook,Skype, Microsoft or any American file-hosting companies.
Your Information and surfing habits will be added to the NSA haystack and shared with GCHQ and the so-called state of Israel.
It is not ideal but how about DropBox or similar ?
I see no way to download it.No, you don't need to sign in. Find your camera firmware and click on it. The download button is a tiny down arrow in the extreme lower right corner. Pretty stupid, eh? Will you try again? Also, see if you can download Tlapser.lua too. Thanks.
I assume you have to sign into a Google account.
Uploading is great. I just finished uploading all the cameras.So we now have a second official CHDK spin-off. Good job!
And don't worry. NSA only breaks into your house and takes you away to an undisclosed location if you type something stupid, like "Osama bin Laden" ....... uh ... oops?Worrying about the NSA by avoiding "American" web sites is kind of pointless. What would make someone think the NSA restricts its activities to "American" web sites? If you are online anywhere in the world, they have you.
The download button is a tiny down arrow in the extreme lower right corner.
So we now have a second official CHDK spin-off. Good job - do you have a name for your project? LHDK?I keep getting requests from my YouTube videos from people who want to run the time lapse script on their cameras. This way, I can post a link with instructions on how to do it, instead of custom compiling for each request.
LDM ? The mind boggles ...
While I asked that question somewhat "tongue in cheek", its actually something I've thought about too.So we now have a second official CHDK spin-off. Good job - do you have a name for your project? LHDK?I keep getting requests from my YouTube videos from people who want to run the time lapse script on their cameras. This way, I can post a link with instructions on how to do it, instead of custom compiling for each request.
LDM ? The mind boggles ...
This is just a way to post my patches in compiled form for testing. It's not a spin-off like SDM.
While I asked that question somewhat "tongue in cheek", its actually something I've thought about too.I thought I would have to set up a complicated web site, but it actually ended up being incredibly simple. I had it working in about 30 minutes.
How does one provide a "customized" version of CHDK without going back to the early days of "Allbest" vs "juiciphox" vs "cail" vs ...?
I thought I would have to set up a complicated web site, but it actually ended up being incredibly simple.
The idea of building and maintaining a custom repository does not appeal to me much.
At some point, manually rebuilding against the latest trunk and uploading will get tiring. Not that I have a better solution though ....With CHDK Shell, it's really not any harder than what I'm doing now for one camera. I build one camera, upload the zip to that camera, and test it on that camera.
To update the web site, I check two boxes in CHDK shell, build all the cameras with one click, and drag all the zip files to the Google drive. It's so simple, it's hard to comprehend!Do you plan to do that every night after the trunk updates?
Do you plan to do that every night after the trunk updates?I think it will only be necessary when I have something new to test, or there is something new in the trunk that is important to include, like a new camera. It's nice to have because it makes it possible for anyone with CHDK to use a new patch, instead of just people who can compile from a diff file. I think it will prove useful. Of course, you need to compile with CHDK shell.
I was just thinking about that too and came to the same conclusion. Especially if you build your repository against the 1.2.0 stable rather than 1.3.0 dev trunks.Do you plan to do that every night after the trunk updates?I think it will only be necessary when I have something new to test, or there is something new in the trunk that is important to include, like a new camera.
Of course, you need to compile with CHDK shell.Or you need to be able to write simple bash scripts :P
ifndef OPT_DE_VERSION
VER=CHDK
ifndef OPT_DEFAULT_LANG
OPT_DEFAULT_LANG=english
endif
else
VER=CHDK_DE
ifndef OPT_DEFAULT_LANG
OPT_DEFAULT_LANG=german
endif
endif
toVER=LAPSER
ifndef OPT_DEFAULT_LANG
OPT_DEFAULT_LANG=english
endif
Thanks, that's a good idea. The splash screen and misc/build info say "LAPSER" now, so you can blame everything on me.Code: [Select]VER=LAPSER
ifndef OPT_DEFAULT_LANG
OPT_DEFAULT_LANG=english
endif
One of the biggest headaches I've had with my script is writing the log. I have a timer for the print statement that writes the log, and most of the time it takes 10 msec or less. However, sometimes it takes much longer, especially at the start of the time lapse. In this time lapse, the print statement took over 3 seconds one time! I'm pretty sure this happens because the camera is saving pictures as fast as it can in continuous mode. Sometimes, the log file has to wait its turn.Since most or all your cameras have the file write hook, you could start experimenting with writing the log file before or after a jpeg file is written (place a hook into fwt_open or fwt_close, before the open() or after the close() call). Of course you'll need to buffer the log output for this.
Since most or all your cameras have the file write hook, you could start experimenting with writing the log file before or after a jpeg file is written (place a hook into fwt_open or fwt_close, before the open() or after the close() call). Of course you'll need to buffer the log output for this.I thought about using the file write hook but couldn't figure out a good way to do it. That's a good suggestion.
void script_console_add_line(long str_id)
{
const char* str = lang_str(str_id);
console_add_line(str);
if (print_screen_p && (print_screen_d >= 0)) {
char nl = '\n';
// TODO this should be uncached memory
//set flag to delay fwt_open
//wait until fwt_close (if fwt_open was called)
write(print_screen_d, str, strlen(str) );
write(print_screen_d, &nl, 1);
//clear flag to delay fwt_open
}
}
I don't understand the comment about uncached memory. Would this cause a problem with using this idea?Every once in awhile, I create a diff file with my current patches, and apply it to the latest trunk. Then all I need to do is batch compile all the cameras for that trunk and drag the bin/zip files to the google drive on my computer. That updates them on the web download page too.If you distribute CHDK binaries, you also need to provide the source that was used to build them. Including the patch in the download directory would meet this requirement.
I don't understand the comment about uncached memory. Would this cause a problem with using this idea?This refers CPU cache. The camera RAM is accessible through two different address ranges, one that goes through the CPU cache and write buffer, and one that does not. Low level functions like write() only work 100% correctly using uncached memory, presumably because the actual writing is done with DMA, which doesn't know about the CPU cache. In practice, you can often get away with ignoring this.
I will have to return Monday to discuss it.
Anything that I need to know about camera settings and using the script ?Try testing the script inside to see how it works. Most of the default settings should be OK. I set the interval to 1000 msec, but that's up to you.
Lapser must be Dr.Lapser from youtube (I searched youtube for G1X, timelapse) and this confirms it!:
Now should I add the G1X?CHDK scripts can go a lot further than bulb ramping. In addition to directly setting the shutter time, a script can set the ISO, aperture, and control the G1X built in ND filter.
I'm hoping to create a lightweight backpackable astro kit...the G1x + Ioptron Skytracker + Berlebach Mini.
CHDK removes the need for the external intervalometer and I'm thinking once I can really dive into the CHDK threads I'll find a script what honors bulb ramping (or something close).
All that is left is to figure out how you did the external battery (or did you yet).I used this battery: http://www.bixnet.com/mumiexbapaki1.html (http://www.bixnet.com/mumiexbapaki1.html)
I see the ongoing quest to defeat the manual focus lost to autofocus once the screen is turned off BUT with the lightweight nature of lithium batteries why not just give in and carry the extra larger external battery and be done with it?reyalp and philmoz figured out how to turn the display off from a script without losing autofocus. My script turns the display on and off when you press <set> on the G1X. It would be useful to trigger this with a shortcut key outside of a script, but this hasn't been implemented yet.
p.s. the plastic bag trick for weather proofing was brilliant!
.. and the fear of amassing so many shutter clicks BUT you don't seem worried with multiple thousands of clicks on your G1X.I tend to believe that shutter clicks - even by hard core CHDK users - do not kill many P&S camera. Dust infiltration onto the sensor, lens errors from jammed gears, and the release of new&better models every year probably relegate more cameras to the back of the closet that actual wear and tear on the shutter. And I do have a little experience (http://chdk.setepontos.com/index.php?topic=9049) to back up that statement. So my advice is to enjoy these little machines as much as you can now and push them to their limits like lapser does - they are not a life-time investment.
-Did you ever get the histogram/metering features into trunk? If not, where can I get a build for SX50? Also your timelapse script.Thanks for posting. Sorry, I've been pretty busy time lapsing and haven't posted much to the CHDK forum. If you'd like to catch up on what I've been doing lately, look at my YouTube channel, including the description section for the videos:
-I'm interested in exposing the moon properly as it moves across the entire frame, so what I was orginally searching for is a way to get a histogram for the entire frame (based on the RAW buffer), and in fact I have no stuck pixels that I can detect so I want *every* pixel to be less than 3968 (the measured saturation on my camera)I haven't gotten to the histogram processing functions in my CHDK build yet, but I'm planning to do that soon. But my current script has the ability to adjust exposure bias on the fly. What I do for SX50 zoomed moon shots is set a single metering area that covers the entire frame, so the moon is always in the metering area as long as it's in the frame. Then I adjust the exposure with the up and down arrow keys until it looks right on the screen. A histogram would be more accurate, but this way works pretty well, especially with raw files.
Many thanks for those ideas. I've been handling white balance by fading between two tracks in the final video. Here's a good example where I faded from Milky Way white balance to daytime so the RV is white at the end:
-Your problems with whitebalance are easy to solve in an Avisynth script, I could make that for you. Interpolation between white balances between start and end frames can be done with the animate command.
-You may also be interested in what I do further with my moon shots, it's called stacking, a common technique in astrophotography. I find the simplest program is Astrostakkert!. Basically drag the 25 shots or so of the moon moving across frame, and it lines them up, including pixel-by-pixel as it waves due to atmospheric disturbence. The stacking accomplishes sharpening and noise reduction. The result is quite amazing!
-There is also a planetcrop.avs script in Avisynth that centers each moon image, for some other stacking programs that have trouble locating the moon anywhere across the frame.
-Finally I'm interested in the full frame histogram to do extreme HDR, to make sure there's a shot with no blown highlights and some lower limit for the shadows. At night I've had to exposure up to 8 minutes. (and I have no hot pixels).I think I can do this once I get the histogram part done. I need to shift into obsessive programming mode for awhile. Right now, I'm in obsessive time lapsing mode, especially in the summer when the skies are clear. There's another total lunar eclipse on October 8-9 that I'm gearing up for, so I might try to do something before then. It would be nice to have histogram based metering of the moon with raw files on the SX50 for that eclipse. Lately, though, I've been working on getting the tracking right.
-I'd also like a real normal daytime exposure mode that makes sure no highlights are missing. As long as the sky is not in the shot :) In fact in some cases I really hate the sky looking all white instead of blue.