This paper presents a new high-resolution computational integral imaging system employing a pickup with the axial movement of a lenslet array and a computation reconstruction algorithm with pixel-to-pixel mapping. In the proposed method, a lenslet array and its image sensor are moved together along the z-axis direction (or axial direction) and a series of elemental image arrays are obtained while moving. The elemental image arrays are then applied to pixel-to-pixel mapping without interpolation for the reconstruction of 3D slice images. Also, an analysis of the proposed reconstruction method is provided. To show the usefulness of the proposed method, experiments are conducted. The results indicate that the proposed method is superior to the existing method such as MALT in terms of image quality.
© Optical Society of America
The integral imaging technique, which was proposed by G. Lippmann for the first time in 1908 , has been actively studied since it provides important features such as 3D display without eye-glasses, full parallax, and continuous viewing points [2–8]. Thus, it is considered as one of next-generation three-dimensional techniques. Normally, an integral imaging system has two stages: pickup and display/reconstruction. Pickup devices such as a camera attached with a lenslet array capture perspectives of 3D objects into an image array, which is called as the elemental image array. Display or reconstruction devices show us the 3D objects by simply placing the same lenslet array in front of the 2D screen displaying the elemental image array. Besides 3D display, a computational reconstruction method that simulates the geometrical optics using the pinhole model produces a series of slice images representing a volume of 3D space. This reconstruction, called as the computational integral imaging reconstruction (CIIR), is considered as a substitute for optical reconstruction .
Despite advantages over optical reconstruction, the image quality of the slice images from CIIR still depends on the resolution of each elemental image; CIIR suffers from low image quality due to the limitation of capture devices. To remedy the problem, some studies have been discussed [9–16]. One approach is to increase the sampling rate 3D scene by moving a lenslet array in two-dimensional directions, which is called as the moving array-lenslet technique (MALT) . In the technique, shifting a lenslet array and multiple snapshots provides a set of elemental image arrays. The technique however requires a very accurate xy-moving stage. Also, a double snapshots-based technique was recently addressed . Another approach is to increase the resolution of elemental images by moving a camera [13–15], which is called as synthetic aperture integral imaging (SAII). Every snapshot from a camera moving is considered as an elemental image, where the resolution of each elemental image is much higher than that of the lenslet array-based method. However, the number of elemental images in SAII is much smaller than that of the lenslet array-based method. A much lower number of elemental images may restrict the usefulness of the SAII technique.
This paper focuses on longitudinal or axial movement of a lenslet array, which is compared to the lateral movement in MALT. Also, this paper discusses a method to reconstruct high-resolution 3D images from multiple elemental image arrays obtained by moving a lenslet array along the z-direction or axial-direction. In the literature, axially moving a lenslet array was introduced for 3D display , not for computational integral imaging. Recently, axial movement of a camera was applied to the SAII technique [13–15], but not for a lenslet array. Consequently, applying axial movement of a lenslet array to computational integral imaging requires another analysis and its reconstruction methods, compared with previous methods.
It can be seen that the axial movement, one of one-dimensional movements, has advantages over the two-dimensional movement that is engaged in MALT; the former has much simpler design than the latter in terms of physical designs. In addition, the digitization error and the limitation of optical pickup devices destroy a subtle movement of a lenslet array in MALT (for example, a typical moving step is about a few hundred micrometers); thus, two elemental image arrays can be almost the same in spite of being obtained from different snapshots. Therefore, the axial movement of a lenslet array and its corresponding reconstruction deserve to be addressed; thus, this paper describes analyses on the axial movement of a lenslet array and proposes a high-resolution computational reconstruction method. This work also reveals that a pixel-to-pixel mapping without interpolation is suitable for the axial movement.
2. Overview of computational integral imaging
The CII system, as shown in Fig. 1, is composed of two processes: a pickup process of 3D objects and a computational reconstruction process from elemental images [2,4,6,10]. In the pickup process of CII, elemental images are recorded by use of a lenslet array and an image sensor. On the other hand, in the computational reconstruction process 3D images are reconstructed from the elemental images by use of a digital computer; in the computer domain, 3D images are easily reconstructed by simulating ray-optics without optical devices. As shown in Fig. 1(b), the elemental images are back-projected on an image plane that is located away from the virtual pinhole array. Here, the elemental images are magnified by a factor of z/g, where z is the distance between the reconstruction plane and the virtual pinhole array and g is the distance between the elemental images and the virtual pinhole array. Then, the magnified elemental images are superimposed on the reconstruction plane. After normalization to compensate intensity irregularity, a slice image for the 3D image is finally produced with respect to the reconstruction plane at a distance z. To completely generate the volume data for 3D space, this process is repeatedly conducted adjusting the distance z.
3. Ray analysis of axial movement of a lenslet array
Figure 2 illustrates the operational principle of the proposed method. The method consists of two stages; optical pickup and computational reconstruction. In the optical pickup stage, as shown in Fig. 2(a), a lenslet array and an image sensor array are moved together along the longitudinal direction (the z-direction) with a moving step.
While moving, multiple snapshots produce a number of elemental image arrays, where each array is captured at a different distance. Thus, each elemental image array has different perspectives of 3D objects. This can increase the sampling rate of rays for 3D objects and gather lots of 3D information.
In the computational reconstruction stage, as shown in Fig. 2(b), a set of elemental image arrays from the different locations are used to computationally reconstruct 3D slice images. The conventional CIIR method produces a slice image at a certain reconstruction plane by overlapping all magnified elemental images in an elemental image array. Here, many slice images are given at the same reconstruction plane because multiple elemental image arrays are prepared. In this case, MALT provides a resulting slice image at a certain reconstruction plane by averaging the slice images. This multiple slice images however can be utilized in a different way.
In the proposed method, a pixel-to-pixel mapping algorithm is applied to our computational reconstruction method. We tested several variant CIIR algorithms including the conventional method [2,4,6,7]. Among them, it turned out that the pixel-to-pixel mapping without interpolation matches the multiple elemental image arrays. Basically, the pixel-to-pixel mapping method , one of our previous works, has a problem of so called empty pixels or dot-pattern. The problem arises from the lack of the number of pixels in an elemental image array. To overcome the problem, an interpolation method for the empty pixels was proposed. However, interpolation causes other problems such as the interpolation error and computational load. Here, the proposed method that uses multiple elemental image arrays can remedy the empty pixel problem without interpolation. Thus, our approach can improve a CIIR method in terms of image quality and reconstruction speed.
To understand the proposed method using multiple elemental image arrays and pixel-to-pixel mapping, let the p-th elemental image of the k-th elemental image array be Epk(x) and an overall slice image on the reconstruction plane at a distance z be Rz(x), as shown in Fig. 3. For simplicity, one-dimensional notation is here considered, but two-dimensional analysis is straightforward. Let dk be the distance between the lenslet array and 3D objects for the k-th pickup position; this is the same as the object location z in Fig. 2(a). Let g be the gap between the image sensor and the lenslet array. Now, let us consider a scaling parameter Sk to compensate the object size in each elemental image array. The scaling parameter Sk is easily determined by dk/g. Then, an overall slice image from all elemental image arrays is written as7]. Our previous study proposed an interpolation method to eliminate the empty pixels since the problem is unavoidable for a single array of elemental images. However, the situation is different if multiple elemental image arrays are given.
4. Sufficient interpolation-free conditions
As shown in Fig. 3, let us consider a longitudinal moving step ∆, and two parallel rays coming from two adjacent arrays of elemental images, which are projected onto the same reconstruction plane. Also, let a value δ be the distance between two projected pixels in the reconstruction plane. If the value δ is the pixel size, the reconstruction plane is completely covered by projected pixels. Then the empty pixels no longer exist. Thus, the maximum moving step ∆ avoiding empty pixels is given by
Another sufficient condition to eliminate the empty pixels is possibly obtained by putting the moving step to be the focal length of the lenslet. This is basically from the experimental results in . The results indicated that at least one slice image among the slice images located at multiples of the focal length does not have empty pixels at all. Conversely, this fact is the same that all pixels in a slice image located at a certain distance can be one-to-one mapped from at least one elemental image array among multiple elemental image arrays which are located at multiples of the focal length. In the proposed method, the multiple elemental image arrays are obtained by moving a lenslet array. This yields a flexible and larger moving step, compared with the moving step in (2).
5. Experiments and discussions
To show the usefulness of the proposed method, a preliminary experiment with a partially occluded 3D object is carried out. Figure 4(a) shows an experimental setup for the optical pickup using a lenslet array. A target object, a toy car, is occluded by an occluding object, a tree. The tree and car are longitudinally located at z = 3 mm and z = 21 mm, respectively. The lenslet array with 30 × 30 lenslets is initially located at z1 = 0 mm. Each tiny lens has a size of 1.08 mm, a pixel array of 30 × 30, and a focal length of 3 mm. Then, each elemental image array has a size of 900 × 900 pixels. The lenslet array and a camera or a CCD array are moved together backward with a step of 3 mm, which is the same as the focal length, and multiple snapshots produces a series of twenty one elemental image arrays. For examples, the first and twenty-first elemental image arrays are shown in Figs. 4(b) and 4(c), respectively. It is seen that the recorded elemental images shows the car in different scales.
Note that the lenslet array and CCD array are moved together; thus, while moving, tiny lenses in the lenslet array are observed with a constant size. Therefore, the size of all elemental images captured is the same regardless of the pickup locations. However, different elemental images from different locations are shows the object in different scales, as depicted in Figs. 4(b) and 4(c).
To evaluate the proposed method, the conventional CIIR with an elemental image array and the MALT with the twenty one are compared with the proposed method. Figure 5 shows three slice images reconstructed at z = 21 mm, where the object was located originally. The conventional CIIR with an elemental image array provided a low image quality, as shown in Fig. 5(a). The MALT with the 21 elemental image arrays improved the quality of the reconstructed slice image, as depicted in Fig. 5(b). However, the slice image from MALT still suffers from blurring since MALT still has interpolation errors by magnification and the effect of the occluding object. On the contrary, the proposed method with the 21 arrays enhanced the image quality because pixel-to-pixel mapping without interpolation can reduce those errors, as indicated in Fig. 5(c). Especially, the zoomed area, showing the word ‘car’ in the object, indicated that the proposed method substantially improved the image quality of the slice image. Therefore, it is the much more useful in object recognition than the existing technique such as MALT.
A novel high-resolution reconstruction method in computational integral imaging has been proposed, employing a pickup process with longitudinal or axial movement and a pixel-to-pixel mapping algorithm without interpolation. Also, sufficient conditions to eliminate the empty pixels have been introduced. An experiment indicated that the proposed method based on the pixel-to-pixel mapping without interpolation with a set of elemental image arrays improves the image quality of the slice images. In addition, the pixel-to-pixel mapping without interpolation is very fast naturally; thus, it is expected that the proposed method can be applied to various applications using integral imaging, such as 3D object recognition.
References and links
1. G. Lippmann, “La photographic integrale,” C.R. Acad. Sci. 146, 446–451 (1908).
3. A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006). [CrossRef]
7. D.-H. Shin and H. Yoo, “Computational integral imaging reconstruction method of 3D images using pixel-to-pixel mapping and image interpolation,” Opt. Commun. 282(14), 2760–2767 (2009). [CrossRef]
8. B. Lee, S.-Y. Jung, S.-W. Min, and J.-H. Park, “Three-dimensional display by use of integral photography with dynamically variable image planes,” Opt. Lett. 26(19), 1481–1482 (2001). [CrossRef] [PubMed]
12. H. Navarro, J. C. Barreiro, G. Saavedra, M. Martínez-Corral, and B. Javidi, “High-resolution far-field integral-imaging camera by double snapshot,” Opt. Express 20(2), 890–895 (2012). [CrossRef] [PubMed]
15. D.-H. Shin and B. Javidi, “Three-dimensional imaging and visualization of partially occluded objects using axially distributed stereo image sensing,” Opt. Lett. 37(9), 1394–1396 (2012). [CrossRef] [PubMed]
16. A. Stern, B. Javidi, A. Stern, and B. Javidi, “Three-dimensional image sensing and reconstruction with time-division multiplexed computational integral imaging,” Appl. Opt. 42(35), 7036–7042 (2003). [CrossRef] [PubMed]