Night sky shooting - too much noise... - Hello, I'm a NEWBIE - HELP!! (Newbies assistance, User Guides and thank you notes)

pianoman

4

Night sky shooting - too much noise...

« on: 09 / May / 2016, 09:36:39 »

Advertisements

Hello, my beloved photography-related forum!

Could you please take a look at my night sky attempts and say, what is the nature of the little, mostly greeny dots? I'm a newbie at nightsky shoting, but I was reading you guys a lot and manage to understand the concept of the dark frame and now I use this method to substract hot pixels

And it goes from this:

to this:

But there is still a little bit of noise

So I'm wondering, if this is a Badpixel.bin/.txt problem or a high 800 ISO of my camera(Canon SX230)?

I was desperately trying to understand the usage of Badpixel.bin/.txt, but it is unclear how Photoshop(Camera Raw 7.0) should interact with Badpixel.bin: should I put it somewhere in the Ps folder?..

These are links to the sample picture, if that would be of any help:

RAW photo: https://www.dropbox.com/s/9vxuk48hstbvhum/CRW_9830.DNG?dl=0
RAW dark frame: https://www.dropbox.com/s/tbdxgfduiac6064/CRW_9836.DNG?dl=0

JPEG version as shoot by Canon: https://www.dropbox.com/s/7iuy66auq9z2785/IMG_9830.JPG?dl=0
JPEG dark frame: https://www.dropbox.com/s/65uxr63i20pqmzr/IMG_9836.JPG?dl=0

Thank you for your attention and have a nice day!
-Miron

Logged

reyalp

14118

Re: Night sky shooting - too much noise...

« Reply #1 on: 09 / May / 2016, 20:08:59 »

Quote from: pianoman on 09 / May / 2016, 09:36:39

So I'm wondering, if this is a Badpixel.bin/.txt problem or a high 800 ISO of my camera(Canon SX230)?

I was desperately trying to understand the usage of Badpixel.bin/.txt, but it is unclear how Photoshop(Camera Raw 7.0) should interact with Badpixel.bin: should I put it somewhere in the Ps folder?..

badpixel.bin and badpixel/badpixel.txt are files use by CHDK. Some raw processing programs (like raw therapee) do allow you to use a list similar to badpixel.txt but their slight differences is the formats.

badpixel.bin is only used if you shoot DNG 1.1. You will need to create it on camera using the the "create badpixel.bin" before you switch to DNG 1.1.

If you use DNG 1.3 (the default) with programs that don't support some advanced DNG features (DNG opcodes) you might see some colored spots. However, I would expect Camera Raw to handle opcodes correctly, and the remaining spots in your image don't look like this problem because they are too bright.

Your image looks like the dark frame subtraction didn't subtract everything it should. Using a program with that directly supports dark frame subtraction in raw might help. Attached is a crop of your DNG, with the dark frame you provided selected in the raw therapee raw processing options. There is still lots of noise (as you should expect in this kind of camera) but almost all the bright green spots are gone.

This doesn't mean you necessarily need to switch to raw therapee, any program that lets you use a raw dark frame on the raw data should be fine.

« Last Edit: 09 / May / 2016, 20:16:21 by reyalp »

Logged

Don't forget what the H stands for.

pianoman

4

Re: Night sky shooting - too much noise...

« Reply #2 on: 13 / May / 2016, 16:43:02 »

Wow.

I have installed RawTherapee and it produces an amazing result with dark frame subtraction, simple and effective. But there are a couple of nuances:

I set 'processing profiles' as Neutral and the image is much lighter than Camera Raw Defaults, here's the comparison(all dark-framed):

RawTherapee left, Camera Raw right, here's the full-sized JPEG if needed: https://www.dropbox.com/s/tpevhhtqfkgjg44/RawTh-Left-CamRaw-right.JPG?dl=0

I would like to know how to set RawTherapee so my image would be as neutral as it was shot, because if you zoom it up to 100% you will see a lot of noise, apparently not bad pixel noise, but an ISO noise:

compared to Camera Raw:

I see in RawTherapee there is an exposure compensation scale, but you obviously have to minus it down, at least to -2.00, so I'm wondering what is the most neutral value.

There is also some kind of a border on all sides of the image, what could it be?

And one last thing, if you have some advice on what settings(crucial) I should pay attention(because there is a lot!) to get a good result, I would really love to hear it from you reyalp!

Thank you for you time, it was VERY helpful, I'm starting to love RawTherapee... amazing

« Last Edit: 13 / May / 2016, 16:48:34 by pianoman »

Logged

reyalp

14118

Re: Night sky shooting - too much noise...

« Reply #3 on: 14 / May / 2016, 00:25:03 »

Quote from: pianoman on 13 / May / 2016, 16:43:02

I set 'processing profiles' as Neutral and the image is much lighter than Camera Raw Defaults, here's the comparison(all dark-framed):
...

I would like to know how to set RawTherapee so my image would be as neutral as it was shot, because if you zoom it up to 100% you will see a lot of noise, apparently not bad pixel noise, but an ISO noise:

Neutral is a good start, RT auto levels often goes badly wrong if the original image exposure is not normal. However, I find RT neutral is very flat: low contrast, dull colors compared to camera jpeg, so you probably want to add some contrast and saturation.

I can't tell you specifically how to get the same levels you get from Camera RAW. There a LOT of knobs, and I don't fully understand how they all work and interact. So what follows is just my experience messing around with it.

On the exposure tab
The "black" control can help you bring down the stuff that is so dark it's all noise. I think just increasing this will give you something a lot closer to your Camera RAW example.
You may want to add some contrast in top area and/or L*a*b. The top one seems to affect color more.
I also like the L* "parametric" adjustment in the L*a*b section to adjust the contrast.

If sky brightness isn't even (brighter near the horizon from light pollution for example) the gradient filter can be helpful. The "black" option mentioned above works much better if the sky background is fairly even.

Detail tab
Sharpening can really make the noise stand out, you may find the results better with it turned off.
The median filter can also help with noise.

Crop tab
The vignette and lens distortion correction settings can be helpful

RAW tab
Different demosaicing algorithm may give less noise. I usually use lmmse
You can correct chromatic aberration (On the version of RT I'm using, there are CA controls on both the raw and crop tabs, I find the raw one works better, but sometimes doesn't have enough range)

Quote

There is also some kind of a border on all sides of the image, what could it be?

This is normal. The sensor includes some areas that are dark, but not fully black. How dark these borders are may depend on zoom, because some cameras do basic vignette correction in raw. By default, "active area" in CHDK DNGs includes this. If you set active area to jpeg in DNG settings, the active area will be set to the same dimensions as the jpeg, and not include this. But sometimes you can salvage a little more image by including the borders.

FWIW, you can see some images I've processed in raw therapee in the examples at http://chdk.wikia.com/wiki/Lua/Scripts:_Fixed_Exposure_Intervalometer

The "master dark" section might also be useful.

Logged

Don't forget what the H stands for.

pianoman

4

Re: Night sky shooting - too much noise...

« Reply #4 on: 15 / May / 2016, 18:36:01 »

Extraordinary post, I definitely owe you a beer: you come to Moscow - you will have it all, my friend:)

I also like that picture of Polaris you took, aren't you supposed to be at the North Pole for this to happen and were you? o.O

fixedint.lua is my official summer choice as well, instead of weird two-dark-frame-taking-wasting-my-time-meteor2.lua

Okay, I only have this one ultimate thing to sum it all up, do you think there will be some notable benefits worth bothering with badpixel.txt (in tandem with RawTherapee)? Because, you didn't say, but as I understand you RawTherapee is working perfectly with DNG 1.3(with bulit-in badpixel.bin), no pitfalls. Because I was working with showbad tool last summer and barely managed it to recognize my DNGs (or even didn't manage, can't remember for sure, but I still got some random-threshold badpixel.txt with coordinates, maybe another command-line program). Because the result, you taught me, is very good, but I don't know... these ~20k bad pixels, they are still somewhere, I don't even know if I see them or something else...

Logged

reyalp

14118

Re: Night sky shooting - too much noise...

« Reply #5 on: 15 / May / 2016, 19:30:04 »

Quote from: pianoman on 15 / May / 2016, 18:36:01

Extraordinary post, I definitely owe you a beer: you come to Moscow - you will have it all, my friend:)

I'll keep this in mind

Quote

I also like that picture of Polaris you took, aren't you supposed to be at the North Pole for this to happen and were you? o.O

It's always at the pole, how far north you are just changes the elevation. FWIW, I like stellarium http://www.stellarium.org/ to see where things are in the sky.

Quote

Because, you didn't say, but as I understand you RawTherapee is working perfectly with DNG 1.3(with bulit-in badpixel.bin), no pitfalls.

It's possible they've added it, but last time I checked, raw therapee didn't support DNG 1.3 badpixel opcodes. The "dead pixel" option on the raw tab should help a lot but might not fix them completely.

Personally, I use chdkptp to fix the badpixels after I download images, like

Code: [Select]

dngbatch some/directory {mod -patch; save -over -keepmtime}

Logged

Don't forget what the H stands for.