1. Introduction

Three-dimensional (3D) imaging technologies have begun to enter the commercial market. These systems employ many different techniques such as depth recovery from defocus [1, 2], stereoscopic camera systems [3], and structured light systems [4]. Each of these systems has strengths and weaknesses depending upon the application.

We present here a new 3D imaging technique that uses an aberrated projected pattern and a conventional camera to retrieve 3D images. The advantages of this technique are the ability to have both projector and camera aligned to the same axis, which could open up 3D imaging in space confined systems such as endoscopes, and the ability to maintain the full resolution one would expect from a camera.

By calculating a depth value for objects in the scene, we will enable a range of advanced post processing options. When implemented fully, this modification and its subsequent software package will allow the selection of planes of focus in an image, the blurring of objects outside or inside a specific depth, and produce outputs capable of being displayed with stereoscopic displays.

Section 2 outlines some of the current methods used to measure 3D information, as well as benefits and problems associated with these methods. Our solution to meet the required goals are presented in Section 3. The test results are presented in Section 4. Finally, a discussion of the results and future work is discussed in Section 5.

2. 3D depth sensing methods overview

3D photography includes all aspects of depth-related imaging, ranging from full-depth information (such as in stereoscopic 3D) to depth of focus control. Many different system architectures are available for 3D optical depth measuring. Among these systems are time of flight [5], laser scanning [6, 7], interferometry and laser speckle pattern systems [8, 9], stereoscopic imaging systems [10–12], depth from defocus [13, 14], plenoptic camera systems [15], and structured light [4].

Table 1 contains a summary of the benefits and disadvantages of some of these modalities. We have chosen these features of 3D imaging systems since most modalities commonly used in 3D imaging excel in some features and are weaker in others. For systems like time of flight measurements a laser source is generally used. This method delivers excellent 3D information over long distances, but can be expensive and require sensitive electronics. Stereoscopic systems are some of the most common 3D imaging devices. Stereoscopic imaging can use many different illumination sources and uses few complicated devices, driving the low cost of most of these systems, but require angle diversity between the two imaging devices to gather 3D information. This can become problematic in space confined 3D imaging situations. Plenoptic systems seem to solve many difficulties in 3D imaging, offering a wide range of illumination options, no additional electronic equipment asides from a camera, but these cameras generally fail to retain a full resolution 2D image of the scene. Structured light systems also solve many problems encountered in 3D imaging, such as the ability to use incoherent illumination and the inexpensive components of most systems, but this modality requires angular diversity between the projected pattern and the imaging device. We will discuss some of these modalities that use readily available optics and electronics in more detail to understand how they compare to our proposed method.

Table 1. Overview of the benefits and disadvantages of several optical 3D measuring techniques.

View Table

2.1. Stereoscopic imaging system

Stereoscopic systems utilize two independent imaging systems to gather parallax information of a scene. Commercial advances have recently been made with the introduction of stereoscopic camera systems for 3D home imaging. There are significant benefits to using a stereoscopic system and the underlying concept is well established [10–12]. However, there are some significant disadvantages to using stereoscopic systems to gather 3D information. The software analysis necessary to perform 3D imaging using stereoscopic systems requires the identification of similar features within a scene. This correspondence problem is an active and heavily discussed topic within the 3D stereoscopic imaging field [16–19]. A more basic issue with stereoscopic cameras is that a commercial stereoscopic system requires two distinct imaging systems. Using the same quality of lenses in a stereoscopic system results in a significant size and cost increase compared to a similar quality 2D camera.

2.2. Plenoptic camera

Plenoptic systems use an imaging system and a lenslet array to create multiple smaller images of the imaging pupil [15, 20, 21]. Plenoptic systems allow the user to capture both spatial and angle information, unlike standard cameras that capture only spatial information [22]. This technology uses post processing to achieve changes in depth of focus as well as perspective shifts. Attractive features of this method include the integration of already well designed imaging optics and the use of a relatively inexpensive lenslet array. Further advantages include the ability to capture the spatial and angle information in one snapshot, giving the user a wide variety of post processing options [23].

Disadvantages of the plenoptic camera architecture include loss of reconstructed image resolution due to the fundamental effects of capturing both spatial and angular information on a single sensor array. Both the optics and post-processing reconstruction algorithm can be optimized for maximum image resolution, but there will always be a fundamental reduction in image quality when compared to a camera with the same sized sensor.

2.3. Structured light systems

Structured light systems project a pattern onto a target and image the apparent change of this pattern on the target from the point of view of the camera. There are many ways to gain 3D information using structured light, either modifying the pattern spatially or temporally [24], color coding the pattern [25], and a variety of other methods [26–28]. The attractive features of a structured light system is the large signal gained by using an active system, as well as the ability to design a pattern that can be easily analyzed via software. There are many applications of structured light systems, such as facial recognition [29] or gaming applications, and these systems are occasionally utilized in conjunction with stereoscopic cameras to help solve correspondence problems [10, 25].

However, structured light systems rely on angular separation of the projector and imaging system. This can lead to a trade off between large angular separation of both the pattern projector and imaging optics, and large separation between the two systems. Many systems that employ structured light use reasonably large, quasi-permanent system setups that are acceptable for a machine vision inspection routine, one-time scientific imaging situation, or even home commercial entertainment systems, but are more difficult to utilize in small imaging systems due to the angular separation requirement.

2.4. Depth from defocus

Depth from defocus relies on prior knowledge of the imaging optics and uses single or multiple images to estimate target distances from the camera based on the magnitude of defocus [13,30]. It is a convenient modality for depth detection since it uses only computational methods to retrieve a basic depth map of the target and doesn’t require changes in the optical design of the imaging system. However, depth from defocus relies on contrast elements in the scene that can be clearly differentiated as the image becomes more focused [13]. Low contrast objects such as white backgrounds are difficult to determine defocus from. Later in the results presented in this work we show a series of scenarios that would likely be difficult for depth from defocus methods to accurately reconstruct a depth map.

3. Depth through controlled aberration

To meet the goals of this project we have developed a new method of depth detection called depth sensing through controlled aberrations. This method uses standard DSLR camera and a projector system with astigmatic optics. The concept for controlled aberration depth sensing is relatively simple. By purposely aberrating a projected pattern it is possible to retrieve depth information from the scene. In this paper this is done by using an extraction algorithm to isolate the projected pattern information from a background subtracted image. The projected pattern is chosen such that it is possible to extract via an algorithm such as the wavelet transform. We chose to use the wavelet transform instead of the more common Fourier transform because wavelet transforms preserve the location of the projected pattern features within the original image, while Fourier transforms only tell what frequencies are present in an image, not where in the image they are located. This means that more image processing must be performed when using Fourier transforms to isolate specific pattern features in the image. In this example we used a projected pattern that had information that could be extracted using the 5th level wavelet transform with the Haar wavelet, but there is no hard requirement to use either the 5th level wavelet or the Haar wavelet specifically. It is also possible to project a pattern that contains mixed sets of information and rely on different level wavelets to extract pertinent pattern information.

Our method relies on two concepts: projected patterns and differential focus. The differential region of focus is created by mounting a cylindrical lens onto the projector, which adds a known amount of astigmatism to the illumination system. The projected pattern has components with spatial frequencies in directions both perpendicular and parallel to the axis of the cylinder lens. Without the cylinder lens present both the parallel and perpendicular spatial frequencies would focus at the same location in Z-depth from the camera. However, the cylinder lens creates a differential focus region depending upon the orientation of the spatial features with respect to the cylinder lens axis. The cylinder lens effectively encodes depth into the pattern projected onto the target, with vertical lines having high contrast close to the camera and horizontal lines having high contrast far from the camera.

To gather the 3D information of a target the projector displays a pattern with known structure as in Fig. 1. The camera is set such that the entire scene is in focus, and takes an image of the object with the projected pattern. The camera then captures a second image, the background image, that does not have a pattern projected onto the target and used as a high resolution image for image processing. The software then subtracts both these images and performs a wavelet transform. The transform produces two images that contain the wavelet coefficients in the X direction as well as the wavelet coefficients in the Y direction. By taking the ratio of the wavelet coefficient in the X and Y direction we effectively create a ratio that captures X and Y line contrast values, which have become a function of depth due to the astigmatic focus of the projected pattern.

 

Fig. 1 The projected pattern used to attain the results displayed later in this paper. The reason for this pattern’s particular shape is discussed in the text.

Download Full Size | PDF

There are many benefits when using astigmatic depth sensing over the other common 3D sensing methods profiled earlier. Projected light techniques require large amounts of hardware, as well as a large projecting area over which a system of multiple cameras and projectors are typically fixed. The depth through controlled aberration method requires only a conventional camera and an illumination channel with added aberration. Plenoptic systems produce excellent 3D images, but suffer from a loss of resolution due to the capturing of both angular and spatial information on the sensor array. The resolution of the image is unaffected by the setup, and the longitudinal resolution is theoretically only limited by the sensitivity of the detector itself. Finally, the controlled aberration method does not require the design of an entirely new camera like that of a stereoscopic system, only a simple attachment to an already designed imaging system.

3.1. Uniform field calibration

An experiment was performed that measured the ability to quantitatively relate wavelet ratio coefficients to distance ranging from the camera. In this setup we use a 1 diopter cylinder lens oriented along the Y spatial frequency direction. This causes the spatial frequencies in the Y direction of our pattern to be at best focus approximately 30 inches from our camera, while the spatial frequencies in the X direction are at best focus approximately 60 inches from our camera.

Ideally the projector should be co-axial with the camera, but for this proof of concept the camera and projector were mounted as close as possible to one another to simulate a flash attachment. The position of the projector does not need to be known; however, it is necessary that the projector and camera system separation remains constant after calibration.

To calibrate the camera we imaged a flat test target across the full field of projected light at distances from 25 inches to 60 inches from the camera, in 5 inch step sizes.

Subtracting the images in Fig. 2(a) and 2(b) and thresholding this difference to reduce noise produces Fig. 2(c). This step is necessary for two reasons: reducing the effects of the background environment on the actual depth measurement, and preparing the image for image segmentation in the following steps.

 

Fig. 2 Figure 2(a) shows the astigmatic pattern projected on to a flat target. Figure 2(b) shows the background image of the flat target. These two images are captured so that they can be subtracted from each other and the projected pattern can be retrieved, as seen in Fig. 2(c).

Download Full Size | PDF

The 5th level wavelet transform using the Haar wavelet was performed on Fig. 2(c) in both the X and Y directions.

The ratio of these two images seen in Fig. 3 are then segmented using the Watershed image segmentation algorithm [31, 32]. This method allows for minimal user input, while forming the base of the depth map output by our algorithm. The algorithm then examines each of the individual crosses of the projected pattern and divides the average value of the vertical wavelet coefficient by the average value of the horizontal wavelet coefficient to create a metric that is directly related to the distance from the camera.

 

Fig. 3 Figure 3(a) shows the horizontal component of the wavelet transform performed on the subtracted pattern see in Fig. 2(c). Figure 3(b) shows the vertical component of the wavelet transform.

Download Full Size | PDF

Figure 4 is the experimental data used to test the initial uniform field calibration. The value of the wavelet coefficient ratio is applied as the value of the entire segmented region to produce a visual representation of depth. This flat target was moved over a range of 25 inches from the camera to 60 inches, with 5 inch increments.

 

Fig. 4 After segmentation ratio of X and Y wavelet coefficients at different depths from the camera. Red colormap values are closer to the camera, while blue colormap values are farther from the camera. Non-uniformities are discussed in the text.

Download Full Size | PDF

Note the non-uniform value across the test targets at a fixed depth. This is due to many different factors such as non-uniform projector intensity and non-equal optical paths causing intensity fall off. There is also a small residual pattern present in the image due to the wavelet transform itself. We analyzed the sensitivity of this method to determine the accuracy of a uniform field calibration. Plotting the depth versus average ratio of vertical to horizontal wavelet coefficient produces Fig. 5. The errors bars are defined by plus and minus the standard deviation of the average of the ratio of the wavelet coefficients across the full projected field.

 

Fig. 5 Calibration data set using the astigmatic depth sensing method. Error bars are plus and minus one standard deviation of the average ratio of the wavelet coefficients in the X and Y directions for an entire image at a single depth.

Download Full Size | PDF

After examining the calculated depth values of each point within a flat surface it becomes clear that significant residual structure exists even when only measuring a flat surface. This produces a large standard deviation of the calculated depth values for a flat surface and necessitates correction before the depth map can be used in this form for quantitative depth measurements. The lack of sensitivity seen in these results necessitated the development of a calibration procedure.

3.2. Field of view calibration

The field of view calibration algorithm was subsequently designed to correct for the nonuniform projected light from the illumination system. The following calibration procedure presented a series of five full field of view images of flat targets, starting at 33 inches from the camera and moving to 45 inches from the camera. The calibration procedure then re-centers each set of calculated depth values with respect to a user defined calibration surface. This is necessary due to the slight angular misalignment of the projector and camera systems. After data realignment the reference data set of depth values are then divided through the remaining data sets. This action is intended to both normalize the nominal reference distance, and to remove the underlying structure in the calculated depth value ratios due to the projector.

Significant improvement of the measured data can be seen after calibration is performed. Figure 6(a) shows the calculated depth values of every point measured projected onto the X–Z plane, where Z is the distance from the camera. It is evident that a residual structure exists within the calculated depth map, even though these surfaces are flat and perpendicular to the camera system. Figure 6(b) shows the mean calculated depth value for each distance from the camera, along with the standard deviation shown as the red error bars. Note the overlapping standard deviation of the separate data sets. Figure 6(c) shows the same plot as Fig. 6(a), but after calibration. The residual structure has been removed from the calculated depth points, and clear separation can be seen between three inch target separation. Figure 6(d) is the same plot as Fig. 6(b). Note that the standard deviation error bars have been reduced dramatically.

 

Fig. 6 Figure (a) shows the depth ratio for the each target projected onto the X–Z plane before calibration, where the X–Y plane is the pixel X and Y coordinates of a measured depth point, and the Z axis is the distance from the camera. Figure (b) shows the average depth ratio for all points, with the red error bars denoting plus and minus one standard deviation of the points at a particular depth. Figure (c) shows the post-calibration projection of the depth ratios for each flat target. Figure (d) shows the average depth ratio of these calibrated data points with the red error bars denote plus and minus one standard deviation of all the depth points at a particular depth. Note the significant decrease of the standard deviation of the depth values calculated for a flat surface, from approximately 0.13 before calibration to an average standard deviation of 0.027 after calibration, as well as the separability of the 3 inch shifts after calibration. This post-calibration standard deviation corresponds to 1.06 inch of depth.

Download Full Size | PDF

4. Results

To demonstrate the potential of the depth sensing through controlled aberration method we present three example cases to review: imaging a tilted target, a scene with a collection of objects, and simulated stereoscopic images. No modifications to the process outlined in this text was used in the processing of these images. No user input was required other than the name of the images being analyzed.

4.1. Tilted surface

A surface tilted approximately 27 degrees with respect to the camera system was placed approximately 30 inches from the camera. The depth sensing procedure as outlined previously in this document was performed. Examining Fig. 7 from left, the closest side of the target, to right, the farthest side, shows a uniform increase in depth value across each row of the image.

 

Fig. 7 The reconstructed depth map using the methods outlined in this paper.

Download Full Size | PDF

The residual structure seen in the depth map values of Fig. 7 arises due to a slight mismatch between the wavelet size and the projected pattern, resulting in aliasing of the processed data. Altering the wavelet size such that it more closely matches that projected pattern would be a method to remove these effects. We have left further matching of the projected pattern and wavelet size to future advancements of this work.

4.2. Collection of objects

A scene with multiple objects at different depths from the camera system is examined next. In designing this scene we found objects with different tilts, curvatures, sizes, and objects that had some variation of reflectivity. Also to be noted is the presence of text on some of the objects such as a book in the foreground. Figure 8 shows both the original 2D image, the reconstructed depth map, as well as the contrast change between close horizontal lines and far vertical lines. The tilted box in Fig. 8(a) appears in Fig. 8(b) to have the correct tilt being captured. Similarly, the background flat target is tilted and this tilt is clearly captured in the visual depth map. Cylindrical objects such as the coffee cup are captured and easily identified in Fig. 8(b).

 

Fig. 8 (a) Shows the original 2D image captured by our camera. (b) Shows the measured depth mask. Darker blue colors represent distances close to the camera while red colors represent distances farther from the camera. Note that cylindrical objects are captured correctly with this technique, as well as the more specular cup being imaged correctly. (c) Is a zoomed view of the projected pattern incident upon nearer targets, while (d) is a zoomed view of the pattern incident upon farther targets. Note that the vertical and horizontal line contrast changes as distance from the camera increases.

Download Full Size | PDF

We have stated earlier that a key benefit to this technology is the ability to have both projected pattern axis and optical axis concurrent with each other. To achieve this coaxial alignment the camera system would require some modification to a commercial camera, so we have instead chosen to demonstrate this technology with the camera and projector axis close to one another, as in a flash attachment. This is the source of the shadows seen on the objects in Fig. 8.

4.3. Simulated stereoscopic images

Almost all 3D displays use stereoscopic image pairs to give the appearance of a 3D image to the user. Since the astigmatic depth sensing method outputs a depth map, a method was developed that could translate this depth map into a stereoscopic image. Once the depth map of a scene is captured, it is possible to use this information, as well as the original 2D image, to create simulated left and right stereoscopic pairs of images. Each individual region of the measured depth map is shifted based on the magnitude of the depth map value at that location. The mathematics of stereoscopic imaging is worked backwards, with the user suppling the simulated camera pair separation.

Figure 9 shows a simple red-cyan anaglyph is reconstructed after applying the simulated stereoscopic algorithm to Fig. 8(a). Though displayed as an anaglyph, it is equally feasible to display such simulated images on more modern polarized 3D displays. Finally, a unique benefit derived from the algorithm is the total user control on the magnitude of the 3D effect, allowing individuals to tailor the effect to their personal comfort.

 

Fig. 9 The reconstructed stereo image, presented as an anaglyph. A 3D image can be seen using a red-cyan pair of glasses.

Download Full Size | PDF

5. Discussion of results and future work

We have shown that quantitative depth measurements can be achieved using a single camera and aberrated projected pattern. Through a more complex calibration equation that takes into account projector non-uniformities we are able to achieve resolution of 1.06 inch. To increase the distance over which we can determine depth a lower powered astigmatic plate could be used to increase the region of differential focus.

Future work will include optimizing the setup and image processing algorithms for use with video capturing. Post processing would also allow for changes of focal planes after the video is recorded. A unique wavelet could also be developed that is developed specifically for the projected pattern. Interpolation of the depth map is possible, removing the blank spaces between individual watershed regions, as well as permitting a more detailed pixel by pixel simulated stereoscopic image simulation algorithm.

Other more advanced aberrations could also be used in conjunction with different sets of wavelets to simultaneously gather depth data beyond a single pair of wavelet ratios outlined in this text. It is likely that a series of aberrated patterns and matching wavelets could be developed that could produce either redundant information sets or extend the depth imaging range of such a system.

A final system would likely use a modified version of the projected light system described here. We presented taking alternate picture of the scene with the projected pattern and the background image. The system could be streamlined by projecting the pattern in the near infrared (NIR). Using a NIR beam splitter in the optics, the pattern information could be split into a side channel that is imaged onto a lower resolution silicon detector.

6. Conclusion

We have outlined a new technique for 3D imaging we call depth measurements through controlled aberrations of projected patterns. Through the use of an astigmatic projected pattern we capture two images of a target: one of a projected pattern and one of the background. By subtracting these two images we are able to examine only the aberrated projected pattern incident upon a target. We then take the wavelet transform of this subtracted image in both the X and Y directions, and ratio the results. The output is a unique ratio that corresponds to a target distance from the camera.

In this example we were able to achieve approximately 1 inch depth resolution. A key advantage of this technology is that the illumination and imaging paths can be coincident.