How to get Bayer raw using dcraw - RAW Shooting and Processing - CHDK Forum
« previous next »
Pages: [1] 2   Go Down

How to get Bayer raw using dcraw

  • 14 Replies
  • 26962 Views
*

Offline yardleydobon

  • *
  • 35
How to get Bayer raw using dcraw
« on: 24 / March / 2010, 19:42:17 »
Advertisements
By "Bayer raw" I mean a non-demosaiced, completely unprocessed grayscale image that is very close to the intensities seen at the different photosites (i.e. R,G;B,G, individually). This website gives a good explanation of Bayer raw. I haven't tried out their software though.

First I'm posting because I'd like to share this how to with those who don't know how; however, I'm certain this isn't new so I'd like feedback, from someone more experienced, to see if I'm doing it right.

From reading the documentation I believe this is the correct incantation:
dcraw -v -r 1 1 1 1 -o 0 -D -W -g 1 1 filename.crw

However, when I use the -W option the image is very dark with a max pixel intensity of 3 (even though picture was taken in normal lighting and the jpeg is not dark at all).
Without the -W option the image is scaled with the brightest pixel becoming 255.
dcraw -v -r 1 1 1 1 -o 0 -D -g 1 1 filename.crw

Is there a better way to do this?
Logged
*

Offline reyalp

  • ******
  • 14128
Re: How to get Bayer raw using dcraw
« Reply #1 on: 25 / March / 2010, 04:37:56 »
If you tell us what you want to do with the resulting files, we may be able to offer more useful advice.

In my experience, dcraw will not ever give you a completely raw conversion. That's part of why I wrote the rawconvert tool found in the tools directory of the CHDK source. Depending on what you are doing, you may find this useful. Described here http://chdk.wikia.com/wiki/CHDK_Tools and http://chdk.setepontos.com/index.php/topic,2361.msg42533.html#msg42533

If you want to use dcraw
dcraw -D -W -4
Will give you about as close as you can get to a completely raw conversion. Note that this outputs a 16 bpp file. If you output in an 8 bit format, some scaling/truncating must happen somehwere along the line, since the actual raw is 10 or 12 bpp!

edit:
If you want to access the pixel values from code (I'm not sure what else you would be doing given your requirements), you can just borrow the readpixel code from CHDK or rawconvert.
« Last Edit: 25 / March / 2010, 04:49:07 by reyalp »
Logged
Don't forget what the H stands for.
*

Offline yardleydobon

  • *
  • 35
Re: How to get Bayer raw using dcraw
« Reply #2 on: 25 / March / 2010, 13:57:12 »
I emailed David Coffin, maker of dcraw, and explained what I wanted to do. He said to use "dcraw -4 -D".

I want to use my camera as a spectrometer. I want a very simple, unprocessed grayscale image where an individual pixel represents an individual photosite from the color filter array. I don't want a group of photosites merged to form a pixel. I want the gray level of a pixel to represent the voltage the ADC measured at the photosite. In short, I want to convert the raw file CHDK gives me into a format I can use in MATLAB without changing the raw data. The PGM dcraw puts out seems to be exactly what I need, but "seems to be" isn't the same as "is" especially since you say "dcraw will not ever give you a completely raw conversion."

Will rawconvert give me what I need?
Can you give me some more information on how the ouput of rawconvert is formatted?
Logged
*

Offline yardleydobon

  • *
  • 35
Re: How to get Bayer raw using dcraw
« Reply #3 on: 25 / March / 2010, 18:54:06 »
I thought I had to setup the arm cross-compiler to compile rawconvert.c. However, that failed. After trying a few different things, I actually looked at the code, didn't see any weird .h includes, so I tried compiling with plain old gcc. It compiled, no problem. Sigh, I wish I had those three hours back.

To get a .crw raw from my A590 into Octave/MATLAB I did the following:

rawconvert -10to16 -w=3336 -h=2480 -noshift filename.crw filename.raw

_octave_
fid = fopen('filename.raw', 'r');
%a row of the source image becomes a column vector of matrix I
[I, count] = fread(fid , [3336, 2480], 'uint16'); %[nr, nc] = [width, height]
I = I.'; %transpose I so it has the same orientation as the source image.
imagesc(I)

Simple.
« Last Edit: 25 / March / 2010, 22:40:30 by yardleydobon »
Logged
*

Offline reyalp

  • ******
  • 14128
Re: How to get Bayer raw using dcraw
« Reply #4 on: 25 / March / 2010, 19:25:09 »
Sorry about that. If you try to use the CHDK build process, it will expect a working arm toolchain, but rawconvert itself is obviously a native tool. As you have found out, it's very plain vanilla C.

I don't know octave/matlab, but the output in 16 bit is just a sequence of 16 bit ints.
Quote
but "seems to be" isn't the same as "is" especially since you say "dcraw will not ever give you a completely raw conversion."
On some cameras at least, dcraw appears to always interpolate over zero value pixels. Otherwise, -D -4 should give you the raw pixel values in your PGM output.

You will probably want to do something with the < CAM_BLACK_LEVEL pixels anyway, although discarding them may be more to the point. The canon firmware appears to set pixels that are in it's bad pixel list to the value zero.

You may find it useful to look at the CHDK include/camera.h entries for your camera.
Logged
Don't forget what the H stands for.
*

Offline yardleydobon

  • *
  • 35
Re: How to get Bayer raw using dcraw
« Reply #5 on: 25 / March / 2010, 22:54:40 »
I will process both dcraw raws and rawconvert raws through my code to see if there is a difference. I doubt there will be since I am summing column-wise the effect of a zero or it's interpolated replacement is probably miniscule. However, I am very grateful to have your code. It puts my mind at ease to know I have the rawest raw I can get (and still use). Thank you.

I'm not familiar with CAM_BLACK_LEVEL. Is there a lowest level that is considered true black? Then what is less than CAM_BLACK_LEVEL? Noise?
Logged
*

Offline reyalp

  • ******
  • 14128
Re: How to get Bayer raw using dcraw
« Reply #6 on: 25 / March / 2010, 23:34:45 »
Quote from: yardleydobon on 25 / March / 2010, 22:54:40
I'm not familiar with CAM_BLACK_LEVEL. Is there a lowest level that is considered true black?
Yes. CHDK generally takes this as 31 for 10 bit cameras, and 127 or 128 on 12 bit cams.
Quote
Then what is less than CAM_BLACK_LEVEL? Noise?
The canon firmware sets known bad pixels (assumed to be set in ROM by the factory) to be zero. Different tables may be used based on temperature or other factors. It's not clear to me if there are other circumstances where a pixel would be between blacklevel and zero. Possibly the border pixels (which you may want to consider too). Each camera has an active area, which is less than the raw image size. This defined in camera.h, but not necessarily correct on every camera. In fact, what is "correct" isn't entirely clear, because pixels at the edge of the active area sometimes have data but don't respond quite the same as normal pixels.

Caveat for all of the above, this is based on observation and reverse engineering.
Logged
Don't forget what the H stands for.
*

Offline PS

  • ***
  • 157
  • A610 1.00f
Re: How to get Bayer raw using dcraw
« Reply #7 on: 31 / March / 2010, 15:45:52 »
Use -d.
-D -4 -T will create almost identical file to DNG (except header and cropped borders).
Logged
*

Offline yardleydobon

  • *
  • 35
Re: How to get Bayer raw using dcraw
« Reply #8 on: 04 / April / 2010, 21:49:06 »
dcraw -d scales the image to 8 bits. dcraw is a good tool but I don't believe it is the right tool for this job. rawconvert does exactly what I need.

@reyalp
Have you considered adding an option to add a simple header to the raw data? I know for most users this wouldn't be useful, so it would be not be a default option. However, for people like me who want to read the data into Matlab a header containing the width, height, and bpp is very useful. I've actually already modified your code to do this myself. I used a header similar to the netpbm format: BayerRaw\nwidth height\nmaxcolor\n. Do you intend for rawconvert to have any sort of license? Am I free to share un/modified versions of it?
Logged
*

Offline reyalp

  • ******
  • 14128
Re: How to get Bayer raw using dcraw
« Reply #9 on: 04 / April / 2010, 22:31:04 »
Quote from: yardleydobon on 04 / April / 2010, 21:49:06
@reyalp
Have you considered adding an option to add a simple header to the raw data?
I'd rather not create a new format (although there's a fairly good argument for creating a CHDK native raw format.) I'd suggest storing it in some existing format that supports 16 bit gray scale (or bayer) if you need that. For things that need the raw data in a well specified format, DNG is the obvious choice.

If you are just using it for your own work, use whatever format makes thing easiest for you!
Quote
I know for most users this wouldn't be useful, so it would be not be a default option. However, for people like me who want to read the data into Matlab a header containing the width, height, and bpp is very useful. I've actually already modified your code to do this myself. I used a header similar to the netpbm format: BayerRaw\nwidth height\nmaxcolor\n. Do you intend for rawconvert to have any sort of license? Am I free to share un/modified versions of it?
Of course, it should be GPL like the rest of CHDK. I guess we are supposed to have GPL notices in all the source files, but see COPYING in the top level directory.
Logged
Don't forget what the H stands for.
Pages: [1] 2   Go Up
« previous next »
 

Related Topics


SimplePortal © 2008-2014, SimplePortal