HDR without clipping

SX720

43

« on: 18 / May / 2020, 20:29:39 »

Advertisements

With traditional exposure bracketing it's likely that some parts of the image may still be clipped.

With this script you will have details on the entire image, no clipped regions. Prior to writing this script I've taken many HDR images but have been disappointed because quite often the image still has lots of clipped shadows and highlights because I did not use a wide enough range when taking the images.

This script solves this problem by first taking a normal exposure then taking brighter pictures until all pixels in the image meets a minimum threshold. The script then does the opposite for highlights.

Also it takes a dark frame after each picture which is very important for long exposures.

Also this script has the option to turn off the camera when it's done.

Testing results:

Camera Model	Results
A1100 IS	Works great even for a 2048 second exposure. This camera has higher dark current than my A2200 but the final result seems a-bit cleaner.
A2200 IS	It failed after taking a 2048 second exposure. The associated dark-frame for some reason had more time in the metadata and the RAW data was black but the JPEG looked correct.

SX 150 IS	If the exposure is too long it will not take the dark frame, instead it will time out on the raw hook.
SX 230 HS	It correctly did a 512 second exposure and the associated dark frame then after that it took the next exposure in the sequence but for some reason the dark frame after it had a longer exposure time then it failed after that. Also I used this on a bright day and it got stuck in the highlight loop because the camera did not support a higher shutter speed.

There is still some area for improvement in this script. The main issue is that it fails on very long exposures.

Also while making this script I've used the wiki quite extensively. I've tried to cite where I got all my information from but if I missed anything I apologize.

auto_hdr_v4.lua (9.21 kB - downloaded 56 times.)

« Last Edit: 04 / September / 2020, 11:24:13 by SX720 »

Logged

reyalp

14079

Re: HDR without clipping

« Reply #1 on: 18 / May / 2020, 21:22:56 »

Nice work, and nice to see someone using the raw hook stuff

Quote from: SX720 on 18 / May / 2020, 20:29:39

The main issue is it does not take into account the camera's limitations so the script cannot break out of the shadow and highlight loop as result of attempting to exceed the minimum or maximum shutter speed.

Two ways you could deal with this
Add a hard limit. Could be a menu option, or just pick something like 1/32000 which should be close to the useful limit on most cams. If some cams stop before that, you'd just get a few shots at the same level. Min and Max shutter options would probably be useful anyway, some people might not want to wait for it to get to 2000 seconds

CHDK 1.4 style script params would let you put them in a more user friendly format https://chdk.fandom.com/wiki/CHDK_Script_Header#Lua_Shorthand_Parameter_Syntax

Use the histogram or rawop meter to decide when the exposure stops changing by the expected amount. This could get tricky in extreme situations, but from my experience with rawopint I suspect it wouldn't be too bad in practice.

Logged

Don't forget what the H stands for.

SX720

43

Re: HDR without clipping

« Reply #2 on: 19 / May / 2020, 17:41:54 »

This seems to suggest that the true limits are not query-able. I agree with the idea of allowing for a user defined limit but it will be important that the users of the limit understand that they are making a trade-off in doing so. By setting an upper bounds on the exposure they will have more noise in the shadows.

I made another version of this script where the user is expected to set the maximum exposure that they could tolerate waiting for using the CHDK override menu then the script only has the highlight loop. I found that the final images were noisy because some of the shadows were too dark.

Logged

reyalp

14079

Re: HDR without clipping

« Reply #3 on: 20 / May / 2020, 00:08:18 »

Quote from: SX720 on 19 / May / 2020, 17:41:54

This seems to suggest that the true limits are not query-able.

That's correct, the shutter limits are camera dependent, and often unknown or ill-defined.

Logged

Don't forget what the H stands for.

SX720

43

Re: HDR without clipping

« Reply #4 on: 23 / May / 2020, 18:07:41 »

I've updated the script using reyalp's suggestions plus some additional changes. His ideas improved the script a lot.

Here is what has changed from v1 to v2:

1. Optional upper and lower bounds on shutter speed. One thing to note: I decided to make entry simpler for a majority of use cases. This means that the lower and upper bound are set differently. For the lower bound you enter the reciprocal value of what shutter speed you want (the number after the 1/). For the upper bound enter how many seconds you would like the script to go up to. This way you won't need to bring a pocket calculator with your camera.

2. When setting an upper bound on shutter speed, the script may (if enabled) increase the ISO and continue to take brighter pictures that meet the upper bound requirement on exposure time. In this mode you are required to set a limit on the ISO so the script should still have a guaranteed end to it. It's based on what the script thought it set the ISO to, not what your camera actually did so if you set the max ISO to something higher than what's supported by your camera it won't be an issue in terms of the brighter loop ending at some point.

3. The script now first takes the darker images before taking the brighter images. The reason why is because I've found it worse to be missing the dark images than missing some or even all of the bright images. Usually the worse that can happen when missing the bright images is more noise in the shadows whereas if you miss the bright images it can be very bad depending on how much of the detail is missing in the clipped highlights. This way if the script were terminated while running you will more likely have something usable.

4. You may now disable the bright loop completely if desired.

5. I switched to the new parameter syntax.

I'd be interested in knowing why this script does not seem to perform as well on an SX150 IS vs other cameras.

« Last Edit: 23 / May / 2020, 18:12:23 by SX720 »

Logged

reyalp

14079

Re: HDR without clipping

« Reply #5 on: 23 / May / 2020, 20:27:50 »

Quote from: SX720 on 23 / May / 2020, 18:07:41

I'd be interested in knowing why this script does not seem to perform as well on an SX150 IS vs other cameras.

It would be easier to help if you can narrow it down to a simpler test case and indicate exactly what part fails.

Quote

150 IS If the exposure is too long it will not take the dark frame, instead it will time out on the raw hook.

What length exposure fails?
What does it time out in the dark frame, or in the preceding shot?

In my scripts, I usually log to a csv, one row per shot with timestamps, values used etc. You can find this code in rawoptint or fixedint between "-- log module" and "-- end log module"

Logged

Don't forget what the H stands for.

c_joerg

1248

Re: HDR without clipping

« Reply #6 on: 25 / May / 2020, 12:14:29 »

I once played with the script on the EOS M3.
I wonder how to display the long texts in the parameter list?

Quote from: SX720 on 18 / May / 2020, 20:29:39

Also it takes a dark frame after each picture which is very important for long exposures.

Is it really necessary for short exposure time? No parameter to switch off?

I‘m wondering why you use startRealISO=45. Dues your cam support that ISO value?
I can’t remember any more if I tried isoinc.lua on my M3. I’m wondering that I found ISO 50 in the EXIF data…

20200525_174857.jpg (899.95 kB, 3032x1978 - viewed 65 times.)

20200525_174805.jpg (541.58 kB, 3297x2272 - viewed 68 times.)

Logged

M100 100a, M3 121a, G9x II (1.00c), 2*G1x (101a,100e), S110 (103a), SX50 (100c), SX230 (101a), S45,
Flickr https://www.flickr.com/photos/136329431@N06/albums
YouTube https://www.youtube.com/channel/UCrTH0tHy9OYTVDzWIvXEMlw/videos?shelf_id=0&view=0&sort=dd

SX720

43

Re: HDR without clipping

« Reply #7 on: 06 / June / 2020, 21:22:16 »

reyalp, I've made a test script that is reproducible. It takes one regular image and one dark frame then fails when trying to take the next regular image. On my SX 150 IS it fails every-time I run it. You can see the hook timeout by using the display last console option. I ran the test in "P" (Program) mode with the ISO set to 80.

c_joerg, you are right. I modified the script to support taking dark frames only after meeting a minimum exposure time threshold. Please re-download the attachment in the first post to get this feature.

Also I discovered a bug in the script and I do not yet have a solution for this: When the ISO ramps up the shutter time ends up being one second for some reason. I do not understand the cause of this yet.

sx150_hook.lua (2.38 kB - downloaded 25 times.)

« Last Edit: 06 / June / 2020, 21:24:09 by SX720 »

Logged

SX720

43

Re: HDR without clipping

« Reply #8 on: 04 / September / 2020, 11:26:40 »

Very important update: I've fixed the bug where when the ISO starts ramping up the shutter speed gets forced to one second. The reason it's very important is because I was incorrectly passing very large values to tv96_to_usec where I should have been using usec_to_tv96. Please re-download the attachment in the first post.

Logged