Thanks for all that info. Here's what I came up with:
const int xwidth=CAM_ACTIVE_AREA_X2 - CAM_ACTIVE_AREA_X1;
const int ywidth=CAM_ACTIVE_AREA_Y2 - CAM_ACTIVE_AREA_Y1;
int set_histogram_area(int xp1,int yp1,int xp2, int yp2)
{
hx1=(xwidth*xp1)/1000 + CAM_ACTIVE_AREA_X1;
hx2=(xwidth*xp2)/1000 + CAM_ACTIVE_AREA_X1;
hy1=(ywidth*yp1)/1000 + CAM_ACTIVE_AREA_Y1;
hy2=(ywidth*yp2)/1000 + CAM_ACTIVE_AREA_Y1;
I wrote a hack in get_raw_pixel(x,y), which is called to build the histogram, so that it stores 0xFF bytes in the image buffer after reading the value. The points that were sampled end up in the image and you can see where the active area is. On the sx260, it just fills up the screen when viewing the image in play mode.