Shot Histogram Request

lapser

1093

Re: Shot Histogram Request

« Reply #330 on: 11 / August / 2013, 19:31:23 »

Advertisements

Quote from: waterwingz on 09 / August / 2013, 21:04:51

Did a bit of cleanup on the 1.3.0 patch file - converted to a patch file for an svn local repository and removed everything not related to shot_histogram_request (including the Lua bug chasing code and backlight control). Seems to run well with your script - I'll do a simpler script for car lapse shooting I think. Once I put more mileage on the code and cleanup the patch file a bit more, I'll post a final patch file here.

Thanks. I had actually removed the lua bug code once, but a bug in CHDK shell overwrote the whole trunk I was working on. CHDK shell needs to be updated for 1.3. It won't change the active trunk any more, so it always starts on 1.2.

I still need to work some more on the shot histogram part of the code. Right now it uses the full bit depth, but I'm not sure that's necessary. I also haven't tested it very much, so I'm not sure it's bug free.

The shot meter part is very stable, and I recently made it more accurate by correcting the rounding for negative values. The time lapse part works great with continuous mode, but the script has to hand shake properly with the C code. I also have a new set exposure routine that sets everything at once using propcases. That's part of the script handshaking. Anyway, I need to write all that up so you can understand how it works.

Did you notice the Apex96 conversion routines in luascript.c? The time lapse script uses them for input and display.

There's also a small print_screen modification that sets a different output file name for input >9999.

I just got back from a 2 night backpack trip in the mountains with thousands of new time lapse pics to process. There was a lot of rain, lightning and thunder both afternoons but I didn't want to risk the cameras, so I don't have pics of that. Lightning hit pretty close to my tent, and a really loud thunderclap arrived less than a second later. Great fun!

Also 2 very bright meteors streaked by quite low, which I think were early Perseids. I missed those with the cameras too, I think. I was hoping to get some meteor pictures tonight, but I'm pretty wiped out. I'll definitely have cameras going all night tomorrow, the peak of the meteor shower. The forecast is for clears skies here, to if I can find a dark spot, I should get some good pics. After that, I'll have some time for CHDK programming.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

waterwingz

12139

Re: Shot Histogram Request

« Reply #331 on: 11 / August / 2013, 19:44:57 »

Quote from: lapser on 11 / August / 2013, 19:31:23

I still need to work some more on the shot histogram part of the code. Right now it uses the full bit depth, but I'm not sure that's necessary. I also haven't tested it very much, so I'm not sure it's bug free.

Not sure I understand the difference between shot_meter and shot_histogram.

Quote

The shot meter part is very stable, and I recently made it more accurate by correcting the rounding for negative values. The time lapse part works great with continuous mode, but the script has to hand shake properly with the C code.

It would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :

set_shot_interval()
shot_meter_enable()
get_shot_meter()
get_shot_ready()
get_shot_data()

Quote

I also have a new set exposure routine that sets everything at once using propcases. That's part of the script handshaking. Anyway, I need to write all that up so you can understand how it works.

I saw that and separated it out (along with the backlight_off code and the APEX96 code). Its separate from the shot_meter stuff and there was a discussion in IRC with reyalp about how this should be handled. It comes down to "SET_NOW" versus "SET_LATER" in the shooting code and this might not be the best way to handle it.

Quote

Did you notice the Apex96 conversion routines in luascript.c? The time lapse script uses them for input and display.

umm .. yes : http://chdk.setepontos.com/index.php?topic=10459.msg103918#msg103918

Attached is a patch file that just includes the shot_meter / shot_histogram stuff. I'm currently trying to understand the interaction between set_shot_interval() and capt_seq.c - if you use it and do a shoot() rather than press & release of full shoot the camera hangs for example.

shot_histo.patch (32.23 kB - downloaded 21 times.)

Logged

Ported : A1200 SD940 G10 Powershot N G16

lapser

1093

Re: Shot Histogram Request

« Reply #332 on: 12 / August / 2013, 02:04:21 »

Quote from: waterwingz on 11 / August / 2013, 19:44:57

Not sure I understand the difference between shot_meter and shot_histogram.

The shot histogram is what's already there. I've modified it to create a larger array based on bit depth instead of bit shifting, added a scale factor for greater accuracy, and added the ability to get a histogram of a rectangular area in the raw buffer.

The shot meter is similar to the camera meter, but you can specify the metering area. That's what my script does at the beginning. get_shot_meter() returns an Apex96 value that is the difference between the measured brightness and the optimum brightness. You can add this value to your exposure to correct the exposure for the next shot.

Quote

It would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :

Yes, I'll work on that when I can.

Quote

Quote
I also have a new set exposure routine that sets everything at once using propcases. That's part of the script handshaking. Anyway, I need to write all that up so you can understand how it works.
I saw that and separated it out (along with the backlight_off code and the APEX96 code). Its separate from the shot_meter stuff and there was a discussion in IRC with reyalp about how this should be handled. It comes down to "SET_NOW" versus "SET_LATER" in the shooting code and this might not be the best way to handle it.

The problem is that in continuous mode, my interval delay has to wait until the script sets the exposure before it takes the next shot. By setting all the exposure parameters at once, I can then let the camera take the next shot. I worked for months getting all this to work right.

Another problem I found is that set_sv96() is a mess. I got some glitches (flashing in the video) that I think were related to this. I simplified it, and it seems to work correctly in all my cameras.

Quote

I'm currently trying to understand the interaction between set_shot_interval() and capt_seq.c - if you use it and do a shoot() rather than press & release of full shoot the camera hangs for example.

There's a handshake in continuous mode that may be the culprit. set_shot_interval() has an optional 2nd parameter that turns off this handshake. Were you using shoot() in continuous mode?

I've got some documentation in the comments in the C code before most of the functions you're talking about.

I appreciate you trying to get this into the trunk by separating it out at reyalp's request. But a lot of this stuff is very closely interwoven to get it to work at high speed in continuous mode. I tried separating things out including the Apex96 conversion and the set_backlight() fix, but I never got anything past reyalp.

Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory. I gave up trying to get anything into the trunk when this was vetoed. My script uses it and it works (press <set> while script is taking pictures). It eliminates flashing in single shot mode, and minimizes it in continuous.

Maybe it would be better to make shot_histogram.c a module that I have control of for now, and make the interface to CHDK as simple as possible. It wouldn't be loaded unless someone tried to call one of my new functions, so it should be safer.

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

waterwingz

12139

Re: Shot Histogram Request

« Reply #333 on: 12 / August / 2013, 10:43:57 »

Quote from: lapser on 12 / August / 2013, 02:04:21

The shot meter is similar to the camera meter, but you can specify the metering area. That's what my script does at the beginning. get_shot_meter() returns an Apex96 value that is the difference between the measured brightness and the optimum brightness. You can add this value to your exposure to correct the exposure for the next shot.

That's the part I was looking for most immediately.

Quote

Quote
It would be good to get a definition of what each of these functions does, what parameters it expects, what it returns and maybe how they are used (they seem to interlock) :
Yes, I'll work on that when I can.

Thanks - reading someone else's comments in code that has grown over time is a slow way to figure things out. Knowing what you intended each function to do will be a big help - especially give the interactions you mention.

Quote

The problem is that in continuous mode, my interval delay has to wait until the script sets the exposure before it takes the next shot. By setting all the exposure parameters at once, I can then let the camera take the next shot. I worked for months getting all this to work right. Another problem I found is that set_sv96() is a mess. I got some glitches (flashing in the video) that I think were related to this. I simplified it, and it seems to work correctly in all my cameras.

We can certainly discuss this - the idea has merit and the need is there. Its not only an issue for intervalometers jammed into continuous shooting mode - hence reyalps interest in generalizing it.

Quote

Quote
I'm currently trying to understand the interaction between set_shot_interval() and capt_seq.c - if you use it and do a shoot() rather than press & release of full shoot the camera hangs for example.
There's a handshake in continuous mode that may be the culprit. set_shot_interval() has an optional 2nd parameter that turns off this handshake. Were you using shoot() in continuous mode?

I understand why its doing it - I was looking for an explanation of how to setup the handshake. What sequence of commands are needed to make this work right?

Quote

I appreciate you trying to get this into the trunk by separating it out at reyalp's request. But a lot of this stuff is very closely interwoven to get it to work at high speed in continuous mode. I tried separating things out including the Apex96 conversion and the set_backlight() fix, but I never got anything past reyalp.

You have done a lot of good work here and both philmoz & reyalp have made comments to me about "needing to get lapser's stuff into the core". In their defense, I believe they would like to see an organized discussion about each of the mods and anything that becomes really interwoven and codependent is going to be a "hard sell". Frankly, as I followed your various threads I could not keep track of everything you were doing and I don't think anyone else could either. So I believe we now have the opportunity to discuss this in an organized fashion - at least that has been reyalp's offer.

Quote

Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory. I gave up trying to get anything into the trunk when this was vetoed. My script uses it and it works (press <set> while script is taking pictures). It eliminates flashing in single shot mode, and minimizes it in continuous.

Same comment as above.

Quote

Maybe it would be better to make shot_histogram.c a module that I have control of for now, and make the interface to CHDK as simple as possible. It wouldn't be loaded unless someone tried to call one of my new functions, so it should be safer.

Given the mods in other files necessary to make this work, I'm not sure that will fly.

Let's take this in order as I suggested and if we can't work it out then you & I will both be using custom builds going forward.

Logged

Ported : A1200 SD940 G10 Powershot N G16

lapser

1093

Re: Shot Histogram Request

« Reply #334 on: 12 / August / 2013, 13:43:16 »

There's only a month or two when summer backpacking in the Oregon cascades is bearable. Most of the summer, the mosquitoes will drive you nuts, and winter snows close all the mountain roads in November or December. Tonight, I want to catch some meteors, but I promise to work on CHDK while I'm scratching mosquito bites, and when I get the time.

Here are some of the time lapses I got last weekend:

This is my favorite. You can see smoke from us starting the campfire, and our headlamps moving around after it gets dark:
http://www.youtube.com/watch?v=Sl3Sl1gELTU#ws

This is an all night time lapse starting Saturday around 1800. The last part is just fog, but it's kind of interesting. I should have let it run a little longer to watch the fog burn off.
http://www.youtube.com/watch?v=P6jrJuI8xZs#ws

This one was taken with the SX50 starting at the same time as the first video:
http://www.youtube.com/watch?v=5WzzfBEd_Zw#ws

And a foggy sunrise on the SX260
http://www.youtube.com/watch?v=1Ya474jgUJg#ws

« Last Edit: 12 / August / 2013, 16:35:04 by lapser »

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

waterwingz

12139

Re: Shot Histogram Request

« Reply #335 on: 13 / August / 2013, 22:15:56 »

Quote from: lapser on 12 / August / 2013, 02:04:21

Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory.

Here's the fix for the backlight issue in a separate patch file (against the 1.3.0 trunk).

It works better in that the script does not need to turn the backlight off again after each shoot! Technically, there is a brief flash when the Canon firmware turns the backlight back on and CHDK immediately turns it off again. I guess I can live with that.

backlight_off.patch (3.07 kB - downloaded 30 times.)

Logged

Ported : A1200 SD940 G10 Powershot N G16

reyalp

14082

Re: Shot Histogram Request

« Reply #336 on: 14 / August / 2013, 01:10:22 »

Quote from: waterwingz on 13 / August / 2013, 22:15:56

Quote from: lapser on 12 / August / 2013, 02:04:21
Fixing set_backlight() is very important to me and a lot of other posters. My fix works, is simple, and doesn't take significant time or memory.
Here's the fix for the backlight issue in a separate patch file (against the 1.3.0 trunk).

It works better in that the script does not need to turn the backlight off again after each shoot! Technically, there is a brief flash when the Canon firmware turns the backlight back on and CHDK immediately turns it off again. I guess I can live with that.

Thanks for splitting this out into a patch. I agree that would be better than the current behavior, but I think I may have found a better solution: http://chdk.setepontos.com/index.php?topic=10551.msg104054#msg104054

Logged

Don't forget what the H stands for.

lapser

1093

Re: Shot Histogram Request

« Reply #337 on: 16 / August / 2013, 02:02:24 »

Quote from: waterwingz on 15 / August / 2013, 19:44:26

If your actually do have a little time, taking an hour to document the function calls in shot_histogram would be really helpful ...

I'm sure there are many typos, but I'll attach the documentation I have so far.

One thing that is completely separate from everything else is the Apex96 conversion functions. They're pretty simple to use with the default values, but also very flexible with the optional scale parameter. The functions are all in luascript.c. I could separate them into a diff file and we could work on them if you want.

It would be harder, but I could probably separate out the shot interval timing functions from the shot meter functions. I assume you would want to add the shot meter first. The new histogram changes are intertwined with the shot meter, so they have to come in together. I'd like to do some more changes and testing of the histogram functions before putting them in the trunk, though.

Anyway, I hope the documentation helps you figure out how to use the functions, and what Tlapser.lua is doing.

Lapser_TimeLapseMods_130815.txt (12.35 kB - downloaded 28 times.)

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos

waterwingz

12139

Re: Shot Histogram Request

« Reply #338 on: 16 / August / 2013, 09:37:27 »

Quote from: lapser on 16 / August / 2013, 02:02:24

I'm sure there are many typos, but I'll attach the documentation I have so far.

Thank you for this. I took a quick read through and it will really help a lot. I'll post questions as I run into thing I still don't understand if I can't figure them out from the source code.

Quote

One thing that is completely separate from everything else is the Apex96 conversion functions. They're pretty simple to use with the default values, but also very flexible with the optional scale parameter. The functions are all in luascript.c. I could separate them into a diff file and we could work on them if you want.

As I posted above, I already seperated them out and posted elsewhere for discussion. Right now the discussion is between adding the functionality as a lua library < e.g. require("apex96convert") > or adding to the core so that they can be accessed from uBASIC as well.

Quote

It would be harder, but I could probably separate out the shot interval timing functions from the shot meter functions. I assume you would want to add the shot meter first. The new histogram changes are intertwined with the shot meter, so they have to come in together. I'd like to do some more changes and testing of the histogram functions before putting them in the trunk, though.

I also separated out the set_exp96(tv96[,sv96,av96,nd]) function. Seems possible to do in Lua as it looks like it just basically does set_prop() calls in C rather that Lua. There was discussion on IRC about the shooting functions and the need to be able to do a "SET_NOW" in addition to the current functions that does a "SET_LATER". I'm not certain how that will be resolved.

Logged

Ported : A1200 SD940 G10 Powershot N G16

lapser

1093

Re: Shot Histogram Request

« Reply #339 on: 16 / August / 2013, 14:34:56 »

Quote from: waterwingz on 16 / August / 2013, 09:37:27

I also separated out the set_exp96(tv96[,sv96,av96,nd]) function. Seems possible to do in Lua as it looks like it just basically does set_prop() calls in C rather that Lua. There was discussion on IRC about the shooting functions and the need to be able to do a "SET_NOW" in addition to the current functions that does a "SET_LATER". I'm not certain how that will be resolved.

First of all, my mods already include the capability of using SET_NOW in half shoot. This is from luascript.c

Code: [Select]

//when() function allows scripts to set exposure (tv,sv,av) values with half shoot
//  pressed that take effect immediately. Without when() function, changes won't
//  take place until the NEXT half_shoot. This requires double metering of each
//  shot, which slows down rate (by half) for multiple shots.
//  av96 & nd_filter may require a delay after setting, for mechanical reasons
static int when()
{
//  if(now_flag&&shooting_in_progress())return SET_NOW;
  if(shooting_in_progress())return SET_NOW;
  return SET_LATER;
}

I think we have a basic difference in how we look at CHDK. I see it more as a platform for writing Lua functions. This is similar to the way smart phones are set up, i.e. as a platform for apps. Simple C functions that make it easier to program scripts in Lua are very worthwhile.

I started out using propcases from Lua, and it took quite awhile to figure it out. Because you CAN do something in Lua, doesn't mean it's not worthwhile to greatly simplify doing it with a C function. I think we need to optimize the Lua programming environment as a priority over the C environment (within reason).

The other reason I wrote set_exp96() is that the code for set_sv96() doesn't work correctly all the time. I had several large glitches in the exposure when using set_sv96(). The code for set_sv96() is very complicated, computing a base sv96 that isn't necessary. This is the simple way to do it, that works well with all my cameras:

Code: [Select]

    shooting_set_prop(PROPCASE_DELTA_SV,
      sv96-shooting_get_prop(PROPCASE_SV)+shooting_get_prop(PROPCASE_DELTA_SV));
    shooting_set_prop(PROPCASE_SV,sv96);

I think we could simplify set_sv96 to use this code and remove all the current mess. Maybe we could write a test Lua script that uses propcases and test it on older cameras?

Another important thing that set_exp96 does is signal chdk to abort the post shot script delay and let the camera go on to the next shot in continuous mode. I could do this by adding a parameter to get_shot_ready() so it aborts the delay without resetting the ready flag, but this is a little confusing.

I spent a lot of time getting the maximum speed out of continuous mode, while writing the log file. The shot data is buffered so you can be writing the log of the last shot while taking a new shot. This is how I do it in my script:

Code: [Select]

  sp=string.format("%d %d %d %d %2d %2d %d %s %-6d",
    get_shot_data(2),get_shot_data(3),get_shot_data(6),
    sv96_to_iso(sv+svdiff),tvd0,tvd,get_vbatt(),seconds(tv),ptime)
  set_exp96(tv,sv,0,nd) -- camera goes on to next shot
  tick0=get_tick_count()
  print(sp) -- writes log file
  ptime=get_tick_count()-tick0
until done

Logged

EOS-M3_120f / SX50_100b / SX260_101a / G1X_100g / D20_100b
https://www.youtube.com/user/DrLapser/videos