Fasthdr script using continuous mode + shooting speed observations

After this post http://chdk.setepontos.com/index.php?topic=11506.msg112937#msg112937 I decided to convert http://chdk.wikia.com/wiki/HDR_Fast_Shooter to use continuous mode with CHDK 1.3 "shoot hooks", in the hopes of making it even faster and avoiding the DNG exif issue mentioned in that post.

To my surprise, it was slower, by about 250ms for 5 shots 

My first theory was the overhead from the sleep() loops used for hook waiting. However, waiting for hook_raw.count() without setting the hook made no noticeable difference. Instead, it appears there is some fixed overhead associated with continuous mode.

#shots	fasthdr	conthdr
2	1.8 s	2.28 s
5	4.75 s	5.0 s
10	9.70 s	9.6 s
20	19.4 s	18.5 s

Notice that clicking shoot_full_only repeatedly is faster than continuous mode up to about 10 shots, but clearly falls behind by 20.

For testing, I made the initial tv value fixed, used identical scenes, and ran multiple runs to make sure there wasn't significant variation between runs. I also modified fasthdr.lua to print ms precision time.

All testing was done on D10, shooting jpeg only.

Both scripts are attached.

To my surprise, it was slower, by about 250ms for 5 shots 

Nice work.

I wonder if it  changes with different camera models.  Continuous shooting on some recent cams is lot faster than older ones and IIRC also depends somewhat on available memory?.

All testing was done on D10, shooting jpeg only.

The OP was saving DNG files.  Did you try it that way.

When I get a minute,  I'll run some tests on some of my cameras I guess.

Works well, but still has the same issue with the first 2 images having the same exposure info

To further delay things, I used a larger number of shots to try and get some slow shutter speeds

I wonder if it has anything to do with it saving one less raw file than jpeg.

e.g., in this case, it took 16 jpegs and 15 raws (first 2 have the same exposure info, and the last image is skipped all together

PS on the SX230HS this new script is just as fast as the old one, and both can make for some quick HDR's

https://www.dropbox.com/s/4pwngza6f18t0z0/rawsandjpegs.zip

Here is one i took the other day, camera took about 20 seconds to capture the 17 images, (script saved 16 dng's (since it does 1 less  raw than jpeg I use 17)

I then loaded the 16 dng's into photomatix (specified a custom white balance since the auto white balance still functions while taking the images, (parts are blurred due to the wind moving the leaves and causing ghosting), the rest of the image was then processed using the adobe camera raw tool for minor tweaks to contrast and other basic stuff) (multiple exposures covering smaller steps allows for HDR's with much less noise, than 2 or 3 exposures covering a large number of stops, since data from each file is not being pushed as far)

https://dl.dropboxusercontent.com/s/crf4i65k2rcaw2p/fasthdrtest.jpg

e.g., in this case, it took 16 jpegs and 15 raws (first 2 have the same exposure info, and the last image is skipped all together

Something is definitely broken in that case, there should be one raw for each JPEG.

Can you look at the sequence of image numbers to see which raw is missing?

One thing that can cause raws to go missing is if the camera needs "pause for file counter" but don't have it (or don't wait long enough). In that case, raws can get saved before the file counter is updated, while will have the effect of one raw being missing. Since the exposure adjustment in these scripts depend on the file counter updating, this could also cause the exposure information to be wrong.

PS on the SX230HS this new script is just as fast as the old one, and both can make for some quick HDR's

To see the differences I measured you'll need to time the shots with sub-second accuracy, and ensure that the exposure times are equal.

The OP was saving DNG files.  Did you try it that way.

Not yet, I got distracted by the speed differences and my 10 minute project turned into several hours 

Will try later.

When it seems that the last file is missing, e.g., iif I set it to take 17 images, it will have jpegs 1-17 but only raw 1-16

(lower number files are higher exposed, (image 1 is the highest exposed, and image 17 is the lowest exposed and the last image to be taken.

Here is a complete download of every jpeg and raw file that the camera took when set to take 17 images

https://www.dropbox.com/s/1lr6z0tdshvsx28/brackettedcomplete.zip

When it seems that the last file is missing, e.g., iif I set it to take 17 images, it will have jpegs 1-17 but only raw 1-16

I downloaded your zip, but it's a bit hard to tell what's going on. The first and last are so under and over exposed, I can't tell which jpeg they correspond to.

I see the number is one less, but as I mentioned earlier the problem could be that the number isn't updated when the raw is created, so the raws would actually overwrite the previous file. You can't tell from the exif either, since that might end up getting the value from the wrong shot too.

If you set it to 3 shots, do you only get two raws? 3 shots with a tv step of 96 should make it a lot easier to be sure which one is missing.

FWIW, I ran the conthdr script on my d10 with 10 shots. It saved 10 raws and 10 jpegs and the exif appear to be correct.

sx230 already has "pause for file counter" set to 100ms. I've attached a test build with it set to 500 ms (this is for 101b, if that isn't your canon firmware, let me know)

Raws and jpegs using a larger step size and only 3 exposrtues set. (saved 3 jpegs and 2 raws)

jpeg 180 matches exposure with raw 179

https://www.dropbox.com/s/orpd5jtds4upca9/rawsplusjpegs.zip

And here is a set of files with the new build you dropped fixed the issue of it saving 1 less raw file, but still has the issue with 2 of the raw files having the same exposure info

https://www.dropbox.com/s/7p1torz8l18p5o6/rawsandjpegs%20500ms.zip

https://www.dropbox.com/s/7p1torz8l18p5o6/rawsandjpegs%20500ms.zip

Thanks for testing. It looks like this camera needs the longer pause in some cases. It would be unfortunate if this caused a 500ms delay in normal shooting, so I've made a new build and version of the fasthdr script to test this.

This version of fasthdr will record a log file called fasthdr.log in the root of your SD card. If you can run it and post the result log, that will be very helpful.

Additionally, if you go to "miscellaneous stuff" -> "Debug parameters" and enable "misc debug values" in the chdk menu, it will show the last wait time after FP: and the max wait time after MP:

If you can shoot a few shots manually with raw enabled and report the values, that would be helpful. Note max value is only reset on reboot, so this should be done before running the script, or after a fresh boot.


Regarding the DNG exif issue:
The "fasthdr" script above and the filecounter change are not expected to resolve this. The conthdr.lua script above probably would, but it requires CHDK 1.3, and I haven't checked in the filecounter change yet, so you probably get the missing DNGs again.

You can probably also avoid it in the old fasthdr by adding a small wait after the exposure count changes. I've added an option for this to the version of fasthdr attached. Set the "exif wait" parameter to something like 10 or 20 or 30. I suspect 10 will do the trick.

Thanks again for bringing this up and testing. Since we don't have every camera, it's the only way we can get things like this fixed.

with the debug enabled, simply taking a raw file after turning the camera on (DNG 1.1)
MEM: 0x53369555
FP: 500
MP: 500

I ran the CHDK build, in addition to the script, and ended up with the following log file

exp 3 pause 0
exp 4 pause 10
exp 5 pause 10
exp 6 pause 10
exp 7 pause 10
exp 8 pause 10
exp 9 pause 20
exp 10 pause 10
exp 11 pause 20
exp 12 pause 120
exp 13 pause 150
exp 14 pause 280
exp 15 pause 270

That is with the delay set to 10

Here is another with the delay in the script set to 0

For this, FP: 210
MP: 500

running it a second time after another reboot, came back with

FP: 190
MP: 500

exp 45 pause 0
exp 46 pause 20
exp 47 pause 20
exp 48 pause 80
exp 49 pause 130
exp 50 pause 240
exp 51 pause 250
exp 52 pause 280
exp 53 pause 280
exp 54 pause 200
exp 55 pause 190
exp 56 pause 220
exp 57 pause 190

This is with the script set to take 13 images