Line Scan Camera emulation mode

whitis

4

« on: 17 / August / 2009, 04:24:38 »

Advertisements

I have seen a number of applications for a line scan camera. However, since many off the shelf models are expensive and many don't offer resolution greater than the width of a regular digital camera, it would make sense for many applications to use a digital camera. A line scan camera has a resolution of, for example, 4096x1. Or 10200x3 (color). It is used for applications such as scanning an object of relatively infinite length.

Suppose the camera resolution is 4096x3072. Then we would program the window on the sensor for 4096x2 window (we need two lines to debayer the image - should have option to use just 1 line, or two to get the third color, and output without debayering). The camera should be able to read those in about 50 microseconds. Plus whatever integration time you use, which would normally be much faster than normal framerates. High light levels are a definite plus.

Some uses:
   - non-stitched Panorama mode using rotary table, pan tilt mount, etc.
   - High resolution scans of lumber, veneer, etc. to make public domain high resolution texture maps.
   - Many industrial uses, some of which require immediate access to data.
   - scanning movie film as a continuous strip with frame alignment being handled by image processing rather than mechanical means.
   - scanning objects as they go by on a conveyor, etc or in a continuous sheet.
   - barcode scans
   - election ballot scanning (continuous roll). Multiple cameras controlled by different parties can record the same ballot roll simultaneously as they are fed past. Why have sporadic recounts when you can have multiple automatic recounts for next to no cost?
   - digitizing strip chart recordings from seismographs, lab equipment, etc.
   - Astronomy: The sky moves past at sidereal rate.
   - cloud photography.
   - special effects
   - one dimensional vs time recording, such as position of a pendulum or seismograph beam. General position sensing.
   - high speed time measurements.
   For example, you mount over the finish line and you have high time resolution (not limited by normal frame rates)
   results. You also get a distorted image of objects crossing the finish line.
   - set the camera to free run (or use a wheel encoder) and point at the road.
   - Point it at an analog oscilloscope with the X sweep disabled for continuous recording (requires precise alignment).
   - various weird effects by setting to free run and waving the camera around.
   - continuous strip areal photography
   - spectroscopy - put a diffraction grating in front of the sensor.
   - applications which might not even have been thought of yet.
   - microscopy
   - chromatography
   - biology. For example, point it at one place and count the ants going by.
   - point it at the sky and catch the birds going by - or the planes.
   - point it down at a river from a bridge and catch everything floating by.
   - various QA/QC applications
   - scan and shred
   - digitizing labels on cans and bottles. You rotate the product and the camera unrolls the label.
   - prototyping and proof of concept for applications that will ultimately require a better line scan camera.
   - data recovery from magnetic tape using a ferofluid to reveal the flux.
   - scanning large blueprints, paintings, etc. without as many stitching artifacts.
   - fluid level monitoring.
   - data logging anything that can be converted to a linear motion
   - counting the number of people entering/exiting an area.
   - book scanning. In one mode, a line scan camera is used like a digital camera back. However, using full frame
   mode works better unless your line scan camera has higher resolution. However, see below for the tilt method.
   - rotate the camera about its optical axis and produce an image in polar rather than cartesian coordinates.
   - matte/still images for filmmaking (application of panarama mode).
   - take pictures that would just be too blurry with the longer integration times of full frame. Such as of, or from,
   an amusement park ride spinning or moving at high speed.
   - point sideways while driving to capture continuous images of buildings, scenery, etc.
   - 3D reconstruction (uses a different type of line scan, see below).
   - high spot maps while hand scraping a lathe bed, etc. (this is the art of making a metal surface flat to
   within one ten thousandth of an inch or better by comparing it to a reference surface by coating one surface
   with dye and pressing it against the other, the dye transfer spotting indicates which portions need to be scraped
   away, as well as providing certain surface properties).

(split into four parts due to message size limits)

« Last Edit: 17 / August / 2009, 04:42:19 by whitis »

Logged

whitis

4

Re: Line Scan Camera emulation mode - Part 2

« Reply #1 on: 17 / August / 2009, 04:25:50 »

Some of these can be accomplished by stitching, and actually may benefit from higher light levels. However, it isn't always possible to sync up the images or get the dimensional accuracy this way. One of the real advantages, that wouldn't apply here, is high resolution on both axes because some sensors are available with high res on one the one line
(a sensor found in 1200dpi scanner might be 10,000 pixels). In line scan mode, you can make accurate linear measurements without parallax/perspective distortion and other distortions on one axis. Full frame works better for some applications and line scan for others; having both capabilities in one device gives you more flexibility, better bang for the buck, and opens the world of line scan to people who otherwise might not be able to afford an additional device.

Subsequent lines would be joined together into a very large image and written to flash that way. They would not be buffered in memory other than a handfull of lines as part of the idea is to take images of any length.

Synchronization:
I am under the impression that you can sense the levels on D+ and D- lines on the USB port. Edge triggered is even better. Maybe even assert them independently.
- Mode 1: Acquire 1 line per pulse on D+ line.
- Mode 2: Quadrature encoder inputs, A+B on D+ and D-. Similar to Mode 1, but it decodes the quadrature signals so any inadvertant backward travel is ignored.
A B
1 0
1 1
0 1
0 0
repeat
Easy way to decode this is to use A, B, last A, last B to index into a 16 element table containing values of -1, 0, 1 to add to position. When position > last position, scan a new line.
- Mode 3: output quadrature on D+ and D- lines.
- mode 4: output step and direction on D+ and D- lines, where direction would be a fixed value in this application but
in variations you might use both (for example, in panorama mode you might rotate the rotary table back to initial position after taking a fixed number of steps.
- mode 5: like mode 1 except that a pulse on D+ causes the current file to be ended and a new one started.
You would wait for a configurable number of pulses or output a configurable number of pulses between scans, allowing for ratios which are not 1:1. When outputting pulses, you would do so at a programmable rate which might be slower than the frame rate.
- mode 6: constant time interval. No actual sync. like Mode3/mode4 but doesn't tie up usb port. Requires smooth constant rate motion.

Logged

whitis

4

Re: Line Scan Camera emulation mode - Part 3

« Reply #2 on: 17 / August / 2009, 04:30:52 »

(Part 3)
There is another mode which would be partway between linescan and regular mode. In this mode, you capture a programmable window and save it to a sequentially numbered file or an mjpeg frame. You do this each time you receive a preprogrammed number of pulses (or integrating the A/B lines increments by a fixed amount) or issue a preprogrammed number of pulses. These would be stitched together. Allows longer integration times while still minimizing distortion. The frames would overlap some.

There are numerous other uses if you have another way to get the pulses in while retaining use of the USB port. Data can then be streamed continuously over USB. Could use USB Video Class mode (which is something that is needed, anyway) and insert a dummy frame boundary after a specified number of scan lines or emulate a flatbed scanner with a very long page size (but flatbed scanners aren't standardized). The result in UVC video mode would look a lot like a very high resoultion webcam with a slow frame rate except that there would be exact alignment between the images. There also may be some ways to transfer long images in real time using picture tansfer protocol by simply reporting a very large image resolution and streaming the data and then abort the transfer but keep the data when you come to the end.

As far as the display is concerned, you would scroll the display up (or down, user selectable) one line at a time and insert the new data. Or display just the pixels being scanned. User selectable.

Raw uncompressed output should be an option. Fixed/Auto focus.

A degenerate form might be a single pixel mode. Some image sensors can be read at 80Mhz pixel rates. So, you could have modes that are limited only by the speed of writing to flash or the amount of memory for storage at full speed (and, of course, the integration time). Offers extremely high time resolution and the ability to record for a very long time. One application, point it at a rotating shaft with an index mark, stripes, or an optoencoder disk and measure speed vs. time. Another is to measure scintilation. You can point the camera at a remote control (the sensor is usually sensitive to the near infrared) and measure the pulse waveform generated when you press each key. Using a small window instead of a pixel, you can measure tracking accuracy (telescopes, etc) as well as get around integration time limits for rotary shaft applications.

Another variation is to alternately scan a horizontal line and a vertical line (or a narrow horizontal window and a narrow vertical one)- allows two dimensional position logging with higher time resolution than with normal full frame images, video, etc. If the image sensor allows you to control integration and readout separtely, you can readout both from a single integration.

Tilt mode: In this mode you scan a narrow window. For example, 4096x16. The camera is deliberately rotated very slightly. This effectively gives you much higher resolution, for some apps, since each scan line is shifted in position relative to the others. It also means that each portion of the object will travel past a red blue and green pixel. Unless the rotation is tightly controlled, resolution may be somewhat variable.

There is another type of camera also sometimes called line scan. In this type, a line from a laser is projected on the object being scanned and the object is imaged by a camera at another angle. This is used for 3D reconstruction. Some applications include inspection of solder paste and solder joints on printed circuit boards, as well as general 3D digitization. The camera needs to acquire more than a single raster line but, depending on the height variation, may need less than a full frame. Such applications can take advantage of the flexible window sizes and the stepper/opto-encoder features described here.

Note that some applications for line scan (as well as full frame scan) require removing the lens. It is desirable to have an option for disabling the lens self test so the lens doesn't have to be set asside and connected by wires. And so a camera with a broken lens (very common) can still be used for other applications.

Alternate I/O pin sources without tying up the USB port:
   - Indicator LEDs may be used for output by optical coupling or hardwiring.
   - If the lens is dismounted, you have two to four wires normally used for lens communication that can be used
   for GPIO, SPI, I2C/SMbus/TWI, etc.
   - If there is a shutter release switch input...
   - If the USB is needed, the SD card connector may not be needed. This provides a number of pins that may be
   used for GPIO or SPI.
   - Alternatively to freeing up the USB port (but requires an external circuit in most cases, limiting many users), image data can be blasted as a continuous stream over the SD card slot in SPI 1 bit mode, SD 4-bit mode, or (if camera supports SDHC) as 8 bits plus strobe. If the camera doesn't support USB high speed, this may provide faster output - possibly around 50megapixels per second (which would be faster than USB full speed) over very short wires. This mode has some
other possibilities as well, such as 1080p 25fps raw video.

Some applications require very accurate time reference. One GPIO pin can also be used for a PPS (one Pulse Per Second) input from a GPS receiver, rubidium clock, TCXO, OCXO, etc. The data from this pin can be recorded along with the image data (if you have no other place to put it, just add one pixel to the image in raw mode). In addition, the crystal normally built into the camera can be removed and replaced with a TCXO. Unused I/O pins may also be used to communicate with a GPS receiver providing position and time info.

Acquiring more than one scan line (i.e. a narrow window) can be used to calibrate out motion without an external motion sync or with an inadequate one.

Just on general principles, the hardware can do it the software should be able to, too. Why should users be limited by the lack of imagination of camera manufacturer?

Someone might even hack a real line sensor in place of the normal image sensor. Other applications might be enabled by removing the lens.

(continued in part 4)

Logged

whitis

4

Re: Line Scan Camera emulation mode - Part 4

« Reply #3 on: 17 / August / 2009, 04:31:46 »

Likely a separate project, Bar code decoding could be integrated with this mode and with normal full frame mode. In one mode the results would be reported over the USB port (USB HID keyboard emulation). In another, any decoded barcodes are stored in the image EXIF header such that you can search for images that match a barcode. In line scan mode, you simply run the barcode decoder software along the line. In full frame mode, you draw a pattern of lines at various angles and offsets. The camera can be programmed to beep or to store the image only when the barcode can be clearly decoded. In another mode, you only record the barcodes to a file. In another, you allow scanning barcodes and taking pictures. The barcode file may store the barcodes and filenames in the order they were acquired. So you can scan the barcode, then take a picture of the object. Good for taking inventory of your books and DVDs with cover images, or taking product photos for a web store. And again, why buy a digital camera, line scanner, and barcode scanner when one device can do all three?

Another variation is multiple window readout mode. This depends on the image sensor letting you control integration and readout seperately. In this mode, you readout multiple small windows from a single integration. This has uses such as calibrating a fabry-perot imaging photospectrometer (I have done this using a box in the center and one in each of 4 corners), star tracking, and tracking the movement of an object by just watching the corners. In star tracker (auto-guider, done this too), mode you calculate the centroid in one or more boxes. The result can be sent over the USB port (USB communications device class - i.e. USB to serial adapter mode), output over four lines as north, south, east, west pulses (pulse width varies in proportion to the centering error), or as a pair (two axis) of step/direction or A/B quadrature outputs, as selected by the user. You may also record the image of the boxes. If boxes are the same size, they can be concatenated vertically or horizontally to make a simple image for USB or SD card image/movie transfer.

I won't have a chance to work on this anytime soon, but maybe someone will take the idea and run with it. Good student project. Also, someone could publish a paper based just on the implementation and many more papers could possibly be written using the result as a tool (which might reference the original paper). Good for you publish or perish folks. For someone who is already familiar with using CHDK, this shouldn't be too hard. Most of the code already exists to do other things, you just need a small app to put the pieces together. It may expose some limits in the API, however, as there may not be entry points for certain functions such as incremental image compression (8 lines at a time). And almost everything needed here can also be used for other applications, as well. For example, the stepper motor control can be used to take regular stitched panoramas with a rotary table.

Logged

Microfunguy

4037

Re: Line Scan Camera emulation mode - Part 4

« Reply #4 on: 17 / August / 2009, 05:56:21 »

Quote from: whitis on 17 / August / 2009, 04:31:46

I won't have a chance to work on this anytime soon

Why do I hear that so often ?

« Last Edit: 17 / August / 2009, 06:51:56 by Microfunguy »

Logged

Anaglyphic

129
Anaglyphic lives!

Re: Line Scan Camera emulation mode

« Reply #5 on: 17 / August / 2009, 11:03:13 »

Hmm I wonder what normal user this is under an alias? No one pops up with a thread this well though out as a first-time poster jumping straight to feature requests?? :b

Quote from: whitis on 17 / August / 2009, 04:24:38

And again, why buy a digital camera, line scanner, and barcode scanner when one device can do all three?

While you might be able to hack in a barcode scanner as long as it didn't have too many calculations for angle and such, a line scanner is another story, and a high-resolution line scanner is... ambitious.

Quote from: whitis on 17 / August / 2009, 04:24:38

The camera should be able to read those in about 50 microseconds. Plus whatever integration time you use, which would normally be much faster than normal framerates.

Stop. Is that a hard limit? There is no feasible way to implement this in software via reverse engineering the camera firmware. I don't think anyone but Canon design engineers know what's possible, but from a CHDK standpoint, you'd have to fire the image capture process to get a high-resolution still which is of course slow, and even if the LCD live preview was enough resolution, it updates a few times a second, and is nowhere near 20,000 times a second.

Now if you butchered the camera for the CCD/aperture/optics/motors and wired up your own microcontrollers, you might have a plausible project - although harder than designing one from scratch, it's probably cheaper. (as long as you consider your time free.)

For the sake of conversation, let's assume the opposite is true - that everything you propose is possible and you could turn these cameras into "a very high resolution webcam" at their best continuous speed, and assuming they'd run more than a week (that's 604,800 seconds and probably WAY outside of their durability testing) - do you think Canon would tolerate it? It would happen once, ever. They would recall all current models, it would cost them millions, but they would have to get them back from the vendors and "fix" them, as giving away ~$4000 machine vision cameras for ~$200 is bad business. And once bitten, no design will be signed off on until this "flaw" is fixed. So the next consumer-level cameras released would be bloated with highly encrypted routines and intentionally crippled hardware, effectively ending this project. Meanwhile, "Canon announces it's new line of cost-effective machine vision cameras..."

Logged

Since we cannot know all that there is to be known about anything,
we ought to know a little about everything. -- Blaise Pascal

RaduP

926

Re: Line Scan Camera emulation mode

« Reply #6 on: 19 / August / 2009, 02:58:54 »

Quote from: Anaglyphic on 17 / August / 2009, 11:03:13

For the sake of conversation, let's assume the opposite is true - that everything you propose is possible and you could turn these cameras into "a very high resolution webcam" at their best continuous speed, and assuming they'd run more than a week (that's 604,800 seconds and probably WAY outside of their durability testing) - do you think Canon would tolerate it? It would happen once, ever. They would recall all current models, it would cost them millions, but they would have to get them back from the vendors and "fix" them, as giving away ~$4000 machine vision cameras for ~$200 is bad business. And once bitten, no design will be signed off on until this "flaw" is fixed. So the next consumer-level cameras released would be bloated with highly encrypted routines and intentionally crippled hardware, effectively ending this project. Meanwhile, "Canon announces it's new line of cost-effective machine vision cameras..."

That is not at all how business works. If Canon would do that, some other company (Panasonic, for example) would fill in the gaps and release a similar camera, but better, for the same price as the cheap canon cameras.

Logged