Tips on improving shooting speeds for Long Exposures & HDRs

ShahJahan

30
King Of The World

« on: 13 / March / 2016, 01:08:29 »

Advertisements

This thread was inspired from this thread https://chdk.setepontos.com/index.php?topic=12786.0 but since my problems are different, perhaps I might need different solutions?!

I shoot HDR (I use 9 images shot in RAW with 1 EV spacing = +4 +3 +2 +1 0 -1 -2 -3 -4) then merge to HDR. Some of this shots go upto 2 mins or more in shooting, but I still need them as I often shoot in a varying dynamic range where even such long exposures are required. When they are NOT, then I shoot 7/5/3 shots, rather than nine.

My trouble is that when I shoot lets say a shot with 2 mins exposure, then after shooting that "wait" screen is ALSO of 2 mins! So that is 4 mins a shot! Can I get rid of it?!
I only need RAWs, but I was told I cannot get rid of JPGs, but can I?
Since I don't need the JPGs, I don't care about how the camera processes them, i.e. I want to save time on noise reduction, red eye removal etc. (from the thread mentioned above, I found that disabling those "intelligent features" will help save time. Also shooting on a lower ISO will help. So I shoot on 100 ISO (lowest on my Cam). What are all those intelligent features? Are they on CAM or on CHDK? Do I have to manually turn them on or are they on by default?
I keep the JPG to 0.3 MP with Fine JPG compression. (the lowest is 0.3MP, and other compression is SuperFine)
I manually lock focus and White Balance
I want to shoot in the series of -4 -3 -2 -1 0 +1 +2 +3 +4 (to decrease any motion between shots), but unfortunately my cam only shoots it like 0 +1 -1 +2 -2 +3 -3 +4 -4 (The Magic Lantern Hack for Canon DSLRs has this feature, and you can see that feature in this video at around 0:55 min mark https://www.youtube.com/watch?v=R_pMSbl_MLQ

The maximum grudge I have is with the "wait" screen which is always exactly the same as for the shot. That is, for 15 sec exposure, there is a 15 sec "wait". Now what else can I do to save time, kindly guide?
Thanks, I sincerely appreciate it!

Logged

The name ShahJahan comes from Persian, where Shah means King and Jahan means World,
so the name ShahJahan in its literal sense means King of the World.

reyalp

14117

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #1 on: 13 / March / 2016, 03:00:57 »

Quote from: ShahJahan on 13 / March / 2016, 01:08:29

My trouble is that when I shoot lets say a shot with 2 mins exposure, then after shooting that "wait" screen is ALSO of 2 mins! So that is 4 mins a shot! Can I get rid of it?!

This is "dark frame subtraction" done by the canon firmware. On most CHDK ports, you can disable it by setting "Dark Frame Subtraction" to off in the "Enhanced Photo Operations" menu, but quality may suffer unless you do dark frame subtraction on the raw later. Google "dark frame subtraction" to learn more about this process.

Quote

I only need RAWs, but I was told I cannot get rid of JPGs, but can I?

You cannot disable jpeg using CHDK. You can set them to small and lowest quality (as you described), but this will likely not improve shooting speed much. On some cameras, using lower resolution jpeg is actually slower than the full resolution L size.

Quote

but unfortunately my cam only shoots it like 0 +1 -1 +2 -2 +3 -3 +4 -4

This is how CHDK built in bracketing works currently. You could write a script to do it the other way.

Logged

Don't forget what the H stands for.

waterwingz

12140

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #2 on: 13 / March / 2016, 11:12:29 »

Quote from: reyalp on 13 / March / 2016, 03:00:57

This is "dark frame subtraction" done by the canon firmware. On most CHDK ports, you can disable it by setting "Dark Frame Subtraction" to off in the "Enhanced Photo Operations" menu, but quality may suffer unless you do dark frame subtraction on the raw later. Google "dark frame subtraction" to learn more about this process.

Quote

but unfortunately my cam only shoots it like 0 +1 -1 +2 -2 +3 -3 +4 -4

There is a script that manages dark frame subtraction - disabling it for each shot but occasionally collecting a reference dark frame for use in post processing. It would not be hard to modify it to shoot in your desired sequence with a reference dark frame taken at the beginning and end of the sequence.

link > Meteor Intervalometer with Dark Frame Management

Logged

Ported : A1200 SD940 G10 Powershot N G16

ShahJahan

30
King Of The World

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #3 on: 14 / March / 2016, 12:06:02 »

Quote from: reyalp on 13 / March / 2016, 03:00:57

This is "dark frame subtraction" done by the canon firmware. On most CHDK ports, you can disable it by setting "Dark Frame Subtraction" to off in the "Enhanced Photo Operations" menu, but quality may suffer unless you do dark frame subtraction on the raw later. Google "dark frame subtraction" to learn more about this process.

Does CHDK apply "Dark Frame Subtraction" in its RAW?! I can see a lot of coloured noise (red, green, blue pixels) in the RAW files, but the JPGs are clean. So until now I've been using Dark Frame Subtraction and wasn't even getting any benefit from it, since I only use RAW!

Quote from: reyalp on 13 / March / 2016, 03:00:57

You cannot disable jpeg using CHDK. You can set them to small and lowest quality (as you described), but this will likely not improve shooting speed much. On some cameras, using lower resolution jpeg is actually slower than the full resolution L size.

I'm using lowest quality and size. If on some cameras it can slow it even further, then I'll have to time it and see which option is better. Thanks for pointing it out.

Quote from: reyalp on 13 / March / 2016, 03:00:57

Quote
but unfortunately my cam only shoots it like 0 +1 -1 +2 -2 +3 -3 +4 -4
This is how CHDK built in bracketing works currently. You could write a script to do it the other way.

I'm NOT very good with scripts and hence I try to avoid it as much as I can. I only use a modified interval. lua script and will this script work with that HDR script?! And is there such a script available, since I don't know programming and I can't write any codes?!

Quote from: waterwingz on 13 / March / 2016, 11:12:29

Quote from: reyalp on 13 / March / 2016, 03:00:57
This is "dark frame subtraction" done by the canon firmware. On most CHDK ports, you can disable it by setting "Dark Frame Subtraction" to off in the "Enhanced Photo Operations" menu, but quality may suffer unless you do dark frame subtraction on the raw later. Google "dark frame subtraction" to learn more about this process.
Quote
but unfortunately my cam only shoots it like 0 +1 -1 +2 -2 +3 -3 +4 -4
There is a script that manages dark frame subtraction - disabling it for each shot but occasionally collecting a reference dark frame for use in post processing. It would not be hard to modify it to shoot in your desired sequence with a reference dark frame taken at the beginning and end of the sequence.
link > Meteor Intervalometer with Dark Frame Management

I've read that the Dark Frame has to be of the same settings (ISO, Ap, Tv) for ALL the frames?! Will shooting just one in the beginning and one in the end do?! Since I shoot SEVERAL pics in HDR, I was wondering if I can somehow subtract the dark frame noise using the series of all those pics (without an actual dark frame) and I found this article http://starcircleacademy.com/2012/10/darkframes/

Unfortunately StarStax DOESN'T support RAW and a curious thing happens when I load the RAW in Photoshop and save it as TIFF. For some reason, it removes all the noise and coloured noise (the one we need dark frame for) even when I do NOT process anything! Although it removes noise, but it kills the sharpness (since the image is NOW processed!)

Wondering that maybe loading the image in PhotoShop's Camera RAW, might be responsibe for this (like it might be loading some last used settings like some apps do) I decided to go into batch processing and then converting the RAWs to TIFFs from there, without actually opening PS's Camera RAW and the results were the same!

For some reason Camera RAW processes the RAW files and it kills the noise, but along with that softens and smudges the frame (like all processed images look). The reason to shoot RAW is to circumvent this "processing"

Also, apart from PhotoShop, all other RAW managers and RAW viewers show the noise and the coloured noise in the image, it is just PS that has this curious problem. The only way I circumvent it, is by processing the RAWs in Photomatix HDR software first (it can handle RAW) and then loading it to PS.

What should I use on a Windows Machine to convert the RAWs to TIFFs, where it DOESN'T process ANYTHING from the image, just convert it to TIFF, so that I can use that technique from that aforementioned article as it claims, it not only removed coloured noise but also,
"

The hot pixels were almost completely annihilated
The sky gradient caused by lights glowing in the distance was also almost eliminated.
The contrast in the sky and elsewhere was improved
The red bias on the railing was mostly removed."

Thanks for all your replies, I sincerely appreciate it.

Logged

The name ShahJahan comes from Persian, where Shah means King and Jahan means World,
so the name ShahJahan in its literal sense means King of the World.

reyalp

14117

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #4 on: 14 / March / 2016, 13:28:47 »

Quote from: ShahJahan on 14 / March / 2016, 12:06:02

Does CHDK apply "Dark Frame Subtraction" in its RAW?! I can see a lot of coloured noise (red, green, blue pixels) in the RAW files, but the JPGs are clean. So until now I've been using Dark Frame Subtraction and wasn't even getting any benefit from it, since I only use RAW!

Dark frame subtraction is done by the Canon firmware. It is done on the raw data, so it affects both CHDK raw and Canon jpeg.

Dark frame subtraction only deals with with things dark current, hot pixels, amp glow. The noise you see in the raw is from other sources like shot noise. The canon firmware does noise reduction when it creates the jpeg. You should be able to do similar in your raw software.

Quote

I've read that the Dark Frame has to be of the same settings (ISO, Ap, Tv) for ALL the frames?!

The shutter speed and ISO should be the same as the shot you are subtracting from. It's also best if the sensor is at the same temperature. Av shouldn't matter. With CHDK raw, you might need different dark frames for different zoom levels, because some cameras do vignette correction in a way that affects raw.

Some software (raw therapee, for example) lets you keep a collection of dark frames and picks the closest matching Tv/ISO.

Quote

For some reason Camera RAW processes the RAW files and it kills the noise, but along with that softens and smudges the frame (like all processed images look). The reason to shoot RAW is to circumvent this "processing"

Sounds like it is doing noise reduction for you. There is always a trade off between how much detail you keep and how much noise you see. Raw doesn't change that, it just lets you decide where to make the trade.

Logged

Don't forget what the H stands for.

ShahJahan

30
King Of The World

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #5 on: 15 / March / 2016, 02:21:24 »

Quote from: reyalp on 14 / March / 2016, 13:28:47

Dark frame subtraction is done by the Canon firmware. It is done on the raw data, so it affects both CHDK raw and Canon jpeg.

Dark frame subtraction only deals with with things dark current, hot pixels, amp glow. The noise you see in the raw is from other sources like shot noise. The canon firmware does noise reduction when it creates the jpeg. You should be able to do similar in your raw software.

I'm NOT sure if it is doing Dark Frame Subtraction (DFS) on my RAWs as my RAWs have a lot of coloured noise (not 'normal' noise, but the coloured pixels of dark current, hot pixels, amp glow etc)
I tested a shot of two 15 Sec exposures one with DFS on and one with Off, I didn't notice any difference in their RAW! Both of them were, IMO, not DFSed.

Quote from: reyalp on 14 / March / 2016, 13:28:47

The shutter speed and ISO should be the same as the shot you are subtracting from. It's also best if the sensor is at the same temperature. Av shouldn't matter. With CHDK raw, you might need different dark frames for different zoom levels, because some cameras do vignette correction in a way that affects raw.

Quote from: reyalp on 14 / March / 2016, 13:28:47

Some software (raw therapee, for example) lets you keep a collection of dark frames and picks the closest matching Tv/ISO.

This DFS is causing me a lot of problems! I'm now contemplating of getting rid of it completely and see if skipping it makes much of a difference. Since my shots already have a lot of coloured pixels, I think I'm getting them already!

Quote from: reyalp on 14 / March / 2016, 13:28:47

Sounds like it is doing noise reduction for you. There is always a trade off between how much detail you keep and how much noise you see. Raw doesn't change that, it just lets you decide where to make the trade.

Why is it doing that noise reduction, when I'm NOT telling it to?! I tried RAW Therapee and even it processes the images by default! I need something that DOESN'T process the image, just convert it from RAW to TIFF, without any loss in data.

Thanks again for your help. I appreciate it.

Logged

The name ShahJahan comes from Persian, where Shah means King and Jahan means World,
so the name ShahJahan in its literal sense means King of the World.

reyalp

14117

Re: Tips on improving shooting speeds for Long Exposures & HDRs

« Reply #6 on: 15 / March / 2016, 13:59:33 »

Quote from: ShahJahan on 15 / March / 2016, 02:21:24

I'm NOT sure if it is doing Dark Frame Subtraction (DFS) on my RAWs as my RAWs have a lot of coloured noise (not 'normal' noise, but the coloured pixels of dark current, hot pixels, amp glow etc)

Canon DFS is done on raw data. This is a fact, not a matter of opinion.

It is possible that a bug in the CHDK port could cause the raw to be saved before DFS takes place, but IIRC you have an sx160, and in my experience DFS works correctly on this camera.

On this camera (and most others of the same age or newer), the difference between images with and without DFS is not very significant for exposures shorter than a few tens of seconds.

Quote

This DFS is causing me a lot of problems! I'm now contemplating of getting rid of it completely and see if skipping it makes much of a difference.

As I mentioned above, if your exposures are less than several tens of seconds, just turning off DFS will probably not hurt quality much.

Quote

Why is it doing that noise reduction, when I'm NOT telling it to?!

First of all, I don't know what your software is doing, I am only guessing based on your description. When you say the image doesn't have noise but looks soft, that certainly sounds like noise reduction. If you upload examples somewhere, we would have a much better idea of what you are seeing.

Quote

I tried RAW Therapee and even it processes the images by default!

Again, it's not clear what you are referring to. You can turn noise reduction on or off. By default, raw therapee does auto levels (which often looks bad), but you can undo this by hitting the "neutral" button in the exposure tab.

Quote

I need something that DOESN'T process the image, just convert it from RAW to TIFF, without any loss in data.

There really isn't such a thing. Converting a raw 12 bit bayer filter image to an RGB image with appropriate colors requires processing.

If you really do want the raw data in tiff format, you can do that with programs like dcraw, but I suspect that is not actually what you want.

Logged

Don't forget what the H stands for.