Shot Histogram Request

I've been experimenting with using the get_histo_range(from,to) function to set exposure in a time lapse script. The main problem I'm having is the resolution of the histogram. The function returns the percent of the total samples in the range instead of just the total samples. The offending instruction is the last line in shot_histogram.c

 return (rng*100)/tot;

It should just be

  return rng;

You could then remove the "tot" variable completely, which makes the routine more efficient

You can get the total (tot) easily in Lua with:

tot=get_histo_range(0,1023)

Of course, this will break all current scripts that use get_histo_range. Considering the improvement, I think that it would be worth it. Old scripts could be modified to take advantage of the increased resolution, or be changed to:

x=(get_histo_range(from,to)*100)/get_histo_range(0,1023)

If you want to avoid breaking old scripts, one way would be to test for "histo_to" less than zero as a flag to return rng instead of return rng*100/tot. Here's what could be done without breaking old scripts, but my vote is for just changing to : return rng
int shot_histogram_get_range(int histo_from, int histo_to)
// Examines the histogram, and returns the percentage of pixels that
// have luminance between histo_from and histo_to
{
 int x, tot, rng ;
 tot=0;
 rng=0;
 
//New Code 1
 int frng;
 frng=histo_to; //<0 flags full rng return
 if(histo_to<0)
 {
   histo_to=-histo_to;
 }
//End New Code 1

 if(!shot_histogram_isenabled()) // TODO we could return an error somehow
  return 0;
 for (x = 0 ; x < SHOT_HISTOGRAM_SAMPLES; x ++ ) 
 {
tot += shot_histogram[x];
if (x>=histo_from  && x <= histo_to) 
   {
  rng += shot_histogram[x];
   }
 }

//New Code 2
 if(frng<0)
 {
   return rng;
 }
//End New Code 2

 return (rng*100)/tot;
}


Or just add a new function - a small code size increase to ensure backward compatability ?

Or just add a new function - a small code size increase to ensure backward compatability ?

That would probably be better. Maybe an optional parameter

function get_histo_range(from,to,flag)

flag: 0 or nil=old way, 1 new way

I broke down and compiled a custom version with "return rng;". What an improvement!

My test program takes a series of shots from -2 ev to +2 ev by varying the shutter speed. After each picture, I wrote out the histogram with the new values, and then truncated to the old values in a line underneath. The step size for the histogram is 64, out of 1024 total.

The attached file shows the Tv96 starting value for each series of shots. I wrote out 3 lines for each shot. The first line shows get_exp_count(), relative ev from -192 to +192, and then the total points from get_histo_range(0,1023), which doesn't vary.

The 2nd line is the modified histogram with the actual count. The 3rd line is the truncated histogram that the function currently produces. You can see how much more accurate the new function is.

If we're going to modify the function, it would be useful to have 4 optional input parameters  that set the rectangular area to sample from (like spot metering). Whenever at least one of these parameters is present, the function would return the new, full resolution value.

Returning the value alone doesn't really tell you much unless you also know the total as well.

The current code attempts to give you the percentage of the histogram in the range; but loses too much precision.

Changing the return to (rng*10000)/tot would give you 2 extra digits of precision; while still keeping the value camera independent. This is probably accurate enough; but (rng*100000)/tot would give 3 extra digits precision if you want.

As you say, you could calculate the total separately and use it in your script; but then your just moving the overhead from the C code to Lua code. Calculating the total in the C code is not much overhead (but should be done in 'build_shot_histogram' not 'shot_histogram_get_range' as it is now).

Adding 4 parameters to 'get_histo_range' to set the histo sample rectangle won't work as the histogram has already been calculated by the time you call this. It would need a new function to be called before you take the shot (the histogram is calculated in raw_savefile).

Phil.

Edit: A backwards compatible solution would be to add a new function called 'set_histo_params(x1,y1,x2,y2,precision)'. This would set the histo rectangle and the return value precision for 'get_histo_range'. The defaults if the function is not called would be full image & 100 so existing scripts would not be affected.

Edit: A backwards compatible solution would be to add a new function called 'set_histo_params(x1,y1,x2,y2,precision)'. This would set the histo rectangle and the return value precision for 'get_histo_range'. The defaults if the function is not called would be full image & 100 so existing scripts would not be affected.

That sounds perfect. Since you don't know the full range of x and y, they would need to be fractions. 1000 is 1.000, 500 is 0.500 of the full range, for example.

Will you make it so no precision parameter returns the actual values? The function could also return the total number.

Another thing that would be extremely useful would be a function that returns the average brightness of all the pixels used for the histogram, within the area, in Apex96 units. I think it would just be the average pixel value normalized to bit depth, converted to Apex96 units. It might be called:  get_histo_exp96() or get_shot_exp96(). It would be the amount of over or underexposure. That is, if the pixel average is 50% of maximum, then the exp96() value would be 0.

[edit] Would it be possible for set_histo_params to rebuild the histogram (and brightness calculation) each time it's called, so you could get histograms and brightness from different parts of the same image?

Quote from: philmoz on 23 / November / 2012, 03:05:41

Edit: A backwards compatible solution would be to add a new function called 'set_histo_params(x1,y1,x2,y2,precision)'. This would set the histo rectangle and the return value precision for 'get_histo_range'. The defaults if the function is not called would be full image & 100 so existing scripts would not be affected.

That sounds perfect. Since you don't know the full range of x and y, they would need to be fractions. 1000 is 1.000, 500 is 0.500 of the full range, for example.

Will you make it so no precision parameter returns the actual values? The function could also return the total number.

I was offering a suggestion for doing this in a way that would not break existing scripts, plus offer better functionality.

CHDK is a collaborative effort - since you have some development experience you can implement something like this if you think it would be useful, and submit a patch for inclusion in the core code.

Another thing that would be extremely useful would be a function that returns the average brightness of all the pixels used for the histogram, within the area, in Apex96 units. I think it would just be the average pixel value normalized to bit depth, converted to Apex96 units. It might be called:  get_histo_exp96() or get_shot_exp96(). It would be the amount of over or underexposure. That is, if the pixel average is 50% of maximum, then the exp96() value would be 0.

The shot histogram is calculated from the RAW sensor data (scaled to 10 bits), and ignores the RGB BAYER pattern of the sensor.

The average value of this data on its own is not a measure of brightness or exposure. The sensor data is linear, you would first need to demosaic the BAYER sensor data then apply a tone curve, and then you would have to factor in the shutter speed, aperture and ISO used to take the image.

I can't see how this would tell you anything about under or over exposure of the image.

[edit] Would it be possible for set_histo_params to rebuild the histogram (and brightness calculation) each time it's called, so you could get histograms and brightness from different parts of the same image?

No, the shot histogram is calculated from the RAW sensor data at the time the image is captured. On most cameras this data is destroyed during the JPEG conversion process so the shot histogram can't be recalculated later.

Phil.

CHDK is a collaborative effort - since you have some development experience you can implement something like this if you think it would be useful, and submit a patch for inclusion in the core code.

OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge. I really appreciate your input and in depth knowledge of CHDK.

OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge.

Its a pretty small group of developers from CHDK beginning to current state when you think about how many Canon P&S cameras are out there in people's purses, pockets and desk drawers and even about how many people have found some pleasure / amusement from using CHDK.

I think philmoz was suggesting all help is appreciated .. although debate about coding style, quality, value of  features and functions, testing and documentation come with the territory.

Quote from: lapser on 23 / November / 2012, 22:09:09

OK, I've been avoiding getting into the C programming side of CHDK, but I guess it's time to cross that bridge.

Its a pretty small group of developers from CHDK beginning to current state when you think about how many Canon P&S cameras are out there in people's purses, pockets and desk drawers and even about how many people have found some pleasure / amusement from using CHDK.

I think philmoz was suggesting all help is appreciated .. although debate about coding style, quality, value of  features and functions, testing and documentation come with the territory.

OK, I figured out how to compile with the windows shell. Can you explain the patch and diff files? If I just change the source files, how do I create the patch file to submit for inclusion in CHDK? If you have a link, that would be great, too.