RANDOM ZERO-VALUED (BLACK) PIXELS ?

SticK

« on: 13 / August / 2012, 15:05:15 »

Advertisements

SX110IS. I'm new to CHDK. Everything controls perfectly, except something very strange in RAW (CRW and DNG formats): random zeroed pixels ... most noticeable if shot with flash of white paper 6" away, but there in all shots, even non-flash. The result is the same whether badpixels.bin is applied or not. I tried all the settings possible with no difference. The artifact is not visible in the Canon JPG, ie, parallel JPGs are clean white [Fig 1]. Only *one* of RGB channels gets zeroed. There appears to be some processing or interpolation going on because the visual result is what looks like 3x3 dot [Fig 2], with the dot color & morphology depending on the zeroed channel.

Fig 1: Canon JPG in Registax, 1:1 zoom
Fig 2: DNG file in Registax, 1:1 zoom same position in image

Eg white paper with flash. R G B
0 65535 65535 = cyan, round
65535 0 65535 = violet, cross-shaped
65535 65535 0 = yellow, round
In Fig 2, mouse arrow is pointing to the center of a violet dot artifact. See pixel values at bottom.

EVEN STRANGER: When moving the CHDK SD card to another SX110 (I have 2 cameras), the artifact pattern result is *exactly* the same. I made sure it was not related to file position on SD card (many sequential tests). To me, this behavior does not appear related to dead or hot pixels in the camera hardware (CCD etc).

It *feels* as though CHDK is introducing a fixed random pattern of zeroed pixel values, regardless of camera A or camera B.

Fig 1.PNG (73.63 kB, 1152x864 - viewed 134 times.)

Fig 2.PNG (78.09 kB, 1152x864 - viewed 117 times.)

Logged

reyalp

13840

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #1 on: 13 / August / 2012, 16:44:57 »

As far as we know, the canon firmware sets known bad pixels to zero, and interpolates over them in the jpeg process.

CHDK DNG interpolates over them if you use DNG version 1.1. This requires you to create badpixel.bin from from the menu first.

If you use DNG 1.3, then CHDK uses DNG opcodes instead. Unfortunately, most non-adobe programs do not understand DNG opcodes.

This doesn't explain how two different cameras would have the same set of bad pixel... that's interesting if true, but I would expect experimental error.

Quote

It *feels* as though CHDK is introducing a fixed random pattern of zeroed pixel values, regardless of camera A or camera B.

There is no mechanism for this that I can think of, what we do boils down to write(file,*framebuffer)

edit:
Also, the badpixel.bin creation process just reads the frame buffer looking for zero valued pixels, without modifying it. So that really rules out CHDK creating them.

edit:

Quote

Only *one* of RGB channels gets zeroed. There appears to be some processing or interpolation going on because the visual result is what looks like 3x3 dot [Fig 2], with the dot color & morphology depending on the zeroed channel.

CHDK raw (and DNG) is in the cameras bayer pattern. Bad pixels are individual R, G or B photosites on the sensor. The "dot color & morphology" you are seeing is a product of how your software is de-bayering the raw data.

edit:
I'd be interested to hear from anyone else who owns multiple identical cameras if the badpixels are identical. Comparing badpixel.bin should be an easy way to check.

« Last Edit: 13 / August / 2012, 17:14:43 by reyalp »

Logged

Don't forget what the H stands for.

SticK

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #2 on: 13 / August / 2012, 17:23:05 »

Thank you for taking the time and for your professional reply. This CHDK is the latest posting for the SX110 & uses DNG 1.3. The same happens with CRW files. That is, I did both "DNG format [ ]" generating a *.CRW file and "DNG format

" to generate a DNG file. I also tested DNG->TIFF (with dcraw) using another viewer other than Reigstax to mitigate experimental error, and no change in the pattern of color dots.

At first when I noticed this behavior with camera A, that was when I thought the same as you, something in the hardware I was not addressing by setting/using the appropriate bad pixel options in CHDK. I tried every reasonable combination and opticn CHDK offers in the RAW menu to no avail, really stuck. For the DNG format, I even tried "Manual bad pixel removal" with all three options, having generated badpixel.bin, to see if I could come up with a different pattern. No change.

So that is when it occurred to me to run CHDK on camera B where I did same maneuvers but in different sequences, and got the same results.

CHDK in Camera A reports 6100 bad pixels and camera B has 13000, so I thought as you, if the intrinsic bad pixels were being processed somehow, there should have been (and I was expecting) a difference in the density and distribution of the colored dots (zeroed pixels). Again I tried the whole slew of option combinations.

To my astonishment the two cameras produced exactly the same results, that is, the positional pattern and colors of these specific artifacts are static in the image space, from shot to shot and camera to camera. Very perplexing. So I ask a question only under dire circumstances.

Intensity of exposure doesn't matter; the pattern is there every time. It's just camouflaged (ET in the closet) as soon as the image is other than bright white.

Registax 5.1 reads CHDK CRW and DNG, by the way (DNG in my figures).

Logged

reyalp

13840

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #3 on: 13 / August / 2012, 17:51:32 »

Quote from: SticK on 13 / August / 2012, 17:23:05

Thank you for taking the time and for your professional reply. This CHDK is the latest posting for the SX110 & uses DNG 1.3.

You can select 1.1 or 1.3. If you use 1.1, then badpixels should be interpolated by the camera.

Quote

At first when I noticed this behavior with camera A, that was when I thought the same as you, something in the hardware I was not addressing by setting/using the appropriate bad pixel options in CHDK. I tried every reasonable combination and opticn CHDK offers in the RAW menu to no avail, really stuck.

CHDK has two different badpixel systems. One is badpixel.bin, which is only used for DNG 1.1. It works like this
1) When you create the file, it finds all zero valued pixels in the framebuffer, and records their locations.
2) When you take a DNG (version 1.1), the camera interpolates over those pixels

The second badpixel system is for manually marking pixels as bad (the "manual badpixel" option). It uses the badpixel file (no .bin extension) and has two different modes of operation. You must create this file yourself, with the locations of the pixels you want to mark as bad. It is not in the same format as badpixel.bin. In "Average" mode the pixel is interpolated over in the camera, and in "RawConv" it is just set to zero, on the assumption that the camera jpeg process and the raw workflow will understand it as bad.

The description on the wiki should be reasonably accurate: http://chdk.wikia.com/wiki/CHDK_User_Manual#Bad_pixel_removal

Quote

CHDK in Camera A reports 6100 bad pixels and camera B has 13000

If this is the case, then they can't have identical patterns of bad pixels

It's possible that A could have a subset of B I guess. If you can post a CHDK raw from each one, without any badpixel processing turned on, I'd be interested to see it.

Note that at least some cameras have multiple badpixel lists in the Canon firmware, and different ones may be used depending on temperature or other factors.

Logged

Don't forget what the H stands for.

SticK

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #4 on: 13 / August / 2012, 18:57:09 »

Nice. DNG Version 1.1 did it.

Here's what happened FYI .....

I made a mistake, apologies, -- finger-trouble -- with a Camera B file, so the flag I raised about two cameras doing the same thing has been rendered moot. I took your original suggestion and very carefully tracked the badpixel.bin file creation on B. And yes, it is twice the size on B. And yes I did get a different and denser pattern that I hoping to see this morning, and soon as I saw that, I became a bit more relaxed. So correct ... in part experimental error. Nonetheless, I feel that I probably would not have found the solution anyway, and would have had to ask.

Enabling DNG 1.1 did create a visually-perfect white image, so does it does work OK. But I can't recall seeing the relevant explanation in the CHDK docs. If it is not there, perhaps a developer or site admin could write it in.

I did certainly had read about badpixel and its differences, thank you suggesting it anyway. That is a very good mechanism because I do long exposures ... I understand it operates on soft-defective pixels (hot/warm) as opposed to dead (black) pixels.

Thus some related questions:

Q1. Does badpixel and badpixel.bin work together in DNG 1.1?
Q2. Is there a menu option that I have to set ?
Q3. What are the differences between 1.3 and 1.1? A link?
Q4. Is there a way I can read the DNG files directly from the SD card in the camera without having to remove it (and not use PTP, yet)? Using the Canon driver, File Explorer does not allow seeing anything other than JPG.

So yes, this all looks very good. Slick coding too on CHDK. Thanks for the immediate and excellent help. I am very impressed.

Logged

reyalp

13840

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #5 on: 13 / August / 2012, 22:59:54 »

Quote from: SticK on 13 / August / 2012, 18:57:09

Enabling DNG 1.1 did create a visually-perfect white image, so does it does work OK. But I can't recall seeing the relevant explanation in the CHDK docs. If it is not there, perhaps a developer or site admin could write it in.

DNG 1.3 support is new, so it isn't in the manual yet. That should be updated...

Quote

I did certainly had read about badpixel and its differences, thank you suggesting it anyway. That is a very good mechanism because I do long exposures ... I understand it operates on soft-defective pixels (hot/warm) as opposed to dead (black) pixels.

You can use it for any pixel you want. As far as we know, the zero value pixels are ones the Canon firmware knows are bad.

Quote

Q1. Does badpixel and badpixel.bin work together in DNG 1.1?

Yes, it should

Quote

Q2. Is there a menu option that I have to set ?

You must create the file, and set the manual badpixel removal option. If you want the pixels fixed in camera (like DNG 1.1 badpixel), use "average".

Quote

Q3. What are the differences between 1.3 and 1.1? A link?

From a CHDK perspective, the difference is you don't need badpixel.bin to create a valid DNG file. Other differences you can find in the spec: http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf

Quote

Q4. Is there a way I can read the DNG files directly from the SD card in the camera without having to remove it (and not use PTP, yet)?

Maybe. What you can see over USB depends on the prefix and extension used for the file, and possibly what software is used on your PC. If you pick the right combination, you should be able to see the files. "Right" depends on the camera, but I would suggest starting with CRW_*.CRW or CRW_*.CR2
See this post http://chdk.setepontos.com/index.php?topic=2045.msg18862#msg18862 for some information. Those notes were made with a much older camera, so will probably not be the same as yours. In particular, newer cameras recognize CR2 rather than (or in addition to) CRW.

A couple other things you should know:
- If you save your DNGs with an extension other than DNG, you may need to rename them on your PC to get software to recognize them correctly.
- The part of the Canon firmware that lists the files over USB only sees the files created by CHDK at startup. So to transfer DNGs, you'd need to restart the camera between taking the shot and plugging in the USB cable.

Logged

Don't forget what the H stands for.

SticK

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #6 on: 14 / August / 2012, 00:56:25 »

Thanks for answering my questions and giving extra detail. That really helped. My final objective is to get the camera to work with RemoteCapture like it does under Canon stock firmware, but with the features of CHDK, including RAW files, longer exposure times etc, which the SX110 does not support out-of-box. For example, turn on live viewfinder, set Tv, Av, and ISO etc, shoot from computer ... and have a form of "CHDK++" (perhaps PTP, scripts, etc) transfer RAW files directly to computer (cleanly without an intervening save to SD), like the S70 did, and like the SX110 does out-of-box but only with native JPGs.

The above works in part as-is. So far with your expert help, I have been able to do the following:
a) set up CHDK to shoot from RemoteCapture,
b) have it save DNG files to the SD card, and
c) cloak the DNG files using CR2 extension and copy them over by hand without removing the SD card.

The USB cable does not need to be unplugged, but camera has to be shut OFF and back ON again to have FileExplorer report the new CR2. Unfortunately, that still requires the need to physically attend to the camera, moving it and retracting the lens each time. By contrast, manipulating the camera using RemoteCapture under stock firmware is not needed, and the camera does not need to be switched OFF. Nonetheless, I can confirm this albeit limited remote fundamentally does work on the SX110 for folks wishing to do the same and retain the Canon device driver.

However, there may be a caveat in CHDK itself that a developer might consider looking at: DNG file numbering, DNG file numbers don't get incremented on their own. I suspect that because the parallel JPG file gets automatically transferred to the computer after processing the shot leaving no trace of the IMG_* file in the folder. Hence, perhaps for that reason, CHDK does not increment the file number. In fact, if I don't delete the last shot (leave an empty folder), then more than one DNG file with the same number gets saved.

At present, even though CHDK is running and saving DNGs by computer control, I don't have access to CHDK camera features via RemoteCapture. For example, the shutter speed drop-down does not have the extended Tv values.

Q. In the context above, what would I need to obtain full seamless computer-controlled remote operation? I think there may be no way out but to change the device driver, at least, and then add extensions to CHDK, a kind of CHDK++, scripts, etc ?

Logged

reyalp

13840

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #7 on: 14 / August / 2012, 01:33:12 »

Quote from: SticK on 14 / August / 2012, 00:56:25

However, there may be a caveat in CHDK itself that a developer might consider looking at: DNG file numbering, DNG file numbers don't get incremented on their own. I suspect that because the parallel JPG file gets automatically transferred to the computer after processing the shot leaving no trace of the IMG_* file in the folder. Hence, perhaps for that reason, CHDK does not increment the file number. In fact, if I don't delete the last shot (leave an empty folder), then more than one DNG file with the same number gets saved.

If I understand this correctly, you are saying that when you use Canons remote capture app, the DNG file number doesn't get incremented, but it does increment normally if you shoot manually ? CHDK just uses the camera file counter, so if Canon doesn't increment it then CHDK won't either.

Assuming this is a correct description, CHDK could probably be modified to work around this in some way, but I don't have any cameras that support Canons remote capture so I can't really test it.

Quote

At present, even though CHDK is running and saving DNGs by computer control, I don't have access to CHDK camera features via RemoteCapture. For example, the shutter speed drop-down does not have the extended Tv values.

Of course, the canon firmware and remote software doesn't know anything about CHDK

Quote

Q. In the context above, what would I need to obtain full seamless computer-controlled remote operation? I think there may be no way out but to change the device driver, at least, and then add extensions to CHDK, a kind of CHDK++, scripts, etc ?

The only way I could see doing this is using the CHDK PTP extension. This would also solve your downloading problem, since CHDK can download any file on the camera.

Logged

Don't forget what the H stands for.

SticK

Re: RANDOM ZERO-VALUED (BLACK) PIXELS ?

« Reply #8 on: 14 / August / 2012, 09:12:09 »

Your understanding of the file numbering caveat is correct. To further elaborate, the image file gets transferred to RemoteCaptureTask without being recorded on the SD card, which is the ideal condition and makes sense of course. When that happens, the camera does not name the image file. Instead, RemoteCaptureTask names it and increments according to the last file number in the computer folder it assigned. There are 2 versions of RemoteCapture. The version that works with S50 & S70 notifies ZoomBrowser that a new file has arrived, and ZoomBrowser automatically updates its contents so you can see the photo. That is, you're watching the live viewfinder on computer, you click on the shutter release, and the photo appears: it's all seamless, and after so many years I find that Canon feature amazing. The version that works with the SX110 does not notify ZB of a new file, so you have to press F5 for a manual refresh of the ZB filesystem to see the photo. Otherwise, both versions work the same. The default naming format by RemoteCapture is Capture_NNNNN.[JPG or RAW], which can be user-modified in preferences of RemoteCapture for name style and starting number.

So the fact that the camera does not name the "remoted" file, the camera file counter likely remains fixed. That seems to agree with your CHDK operation description. Hence a pileup of RAW files with the same number on the SD card. CHDK does not overwrite a existing file with the same name, instead it adds it to the SD card folder. The result is that filenames that have the same name on the SD card are actually different, and one can transfer the files one-by-one to the host with a rename each time.

One option a developer may consider is when CHDK detects RemoteCapture, it uses its own counter to increment local RAWs, based on the last existing RAW on the SD card. So if the SD folder contains no RAW files and RemoteCapture is detected, CHDK can assign #000 to the first file etc. Hence if I create an empty folder on my computer and point RemoteCapture to it, and create an empty folder on the camera, the JPG file number that arrives in my computer folder will be synchronized with RAW filenumber on the SD card, even though prefixes may be different. It will then be up to me to manage both folders. Eg, if I delete the last rejected JPG in ZB, I must delete the last RAW in the camera to maintain synchronicity.

That would work reasonably well, with the exception of course that I mentioned before of having to power OFF/ON the camera each I want FileExplorer to see the new RAW files.

Please be aware that the RemoteCapture feature Canon had for Powershots only works up to version ZB 6.1.1, and worse, is no longer supported by the current PowerShots. The last high-performance PowerShot to support RemoteCapture is the SX110 (also uses the amazing AD9920A correlated double sampler). SX120 for example, no longer has that capability.

Over the next several days I will try your suggestion of PTP on another computer and might have some questions later, for I which I'd start a new thread. You seem very knowledgeable ... I noticed a C code snippet in a reply ... are you one of the developers?

Logged