1. Introduction

Integral imaging has attracted a great deal of attention in recent years [1]–[14]. Full color, full parallax moving three-dimensional (3D) images of integral imaging distinguish the technology from conventional stereoscopic displays that are based on binocular parallax. Compact system configuration is another advantage of integral imaging which makes it a practical alternative to other volumetric or holographic 3D displays.

One of the crucial issues for the practical implementation of an integral imaging system is the mismatch of the system parameters, especially the lens arrays. An integral imaging system consists of pickup and display parts as shown in Fig. 1(a). In the pickup part, an object is imaged through a lens array. Each elemental lens forms a corresponding image of the object, called an elemental image, and these elemental images are captured by a charge-coupled device (CCD) camera. In the display part, the captured elemental images are displayed on the display panel and integrated by the lens array into a 3D image at the same depth as the original object was located. In this configuration, the specifications of lens arrays used in the pickup and display parts should be same, if elemental images are to be integrated correctly. If we use a different lens array in the display part, as shown in Fig. 1(b), the elemental images cannot be integrated properly and images that is displayed is not meaningful. This restriction of the lens array severely limits the utility of a 3D display based on integral imaging. Since we cannot expect all display systems to have lens arrays of the same specifications, it becomes necessary to prepare various sets of the elemental images for different lens arrays by repeating pickup processes numerous times with various lens arrays, which is obviously impractical. This is especially important in 3D television broadcasting systems where a television set at the receiver end would likely have a different panel size and, hence, have a lens array consisting of elemental lenses with a different number or different pitches. Moreover the desired specifications of the lens array are different for the pickup and display. In the pickup process, the lens array that has a small overall size, implying a small elemental lens pitch and small number of elemental lenses, would be preferred to make the overall pickup system compact comparable to common two-dimensional (2D) camera systems. On the other hand, in the display process, a lens array with a large overall size is preferred for the display of large 3D images. This conflicting need for the lens arrays between the pickup and display processes leads to severe restrictions regarding lens array.

 

Fig. 1. Concept of integral imaging and lens array mismatch (a) when the same lens arrays are used in the pickup and display (b) when different lens arrays are used in the pickup and display

Download Full Size | PDF

In order to alleviate the lens array mismatch problem, the use of elemental image shifting [15] or ratio-conserving scaling [16] have been proposed. Elemental image shifting avoids the lens array mismatch problem by shifting each elemental image with respect to its center. In elemental image shifting, however, the integrated image is compressed or expanded in the depth direction. Consequently the integrated image is not the same as the original object. In the case of the ratio-conserving scaling algorithm, each elemental image is magnified or reduced by a factor determined by the system parameters of the pickup and display systems. With magnified or reduced elemental images, a 3D image with the same longitudinal-transverse ratio as the original object can be integrated. The ratio-conserving scaling algorithm, however, generally locates the integrated 3D image out of the range where it can be integrated to achieve the best resolution by the display system. Moreover the ratio-conserving scaling algorithm cannot cope with a change in the number of elemental lenses.

In this paper, we propose a novel scheme incorporating the extraction of 3D information between the pickup and display processes. Figure 2 shows the concept of the proposed method.

In the proposed method, 3D information of the object is extracted from elemental images obtained in the pickup process and the extracted 3D information of the object is then transmitted to each display system. At each display system, the 3D profile of the object such as thickness, depth, transverse size can be modified for the specific needs of that display system. The elemental images are finally generated for modified 3D information and given the system parameters of the display system. Therefore the proposed method completely eliminates the restriction from the mismatch of the system parameters. Since the full 3D information of the object can be modified by digital processing, excellent flexibility on the 3D profile of the displayed 3D image can be achieved for given set of the elemental images. The pseudoscopic image problem of integral imaging is also solved naturally by inverting the depth of the object. Though some parts of the proposed method - depth extraction and the generation of elemental images - have been reported [4], [11], [17]–[19] separately, this is, to the authors’ knowledge, the first report on a framework comprising them as well as the digital manipulation of the extracted 3D information and an experimental verification of the resultant alleviation of the lens array mismatch problem along with excellent flexibility of the 3D profile of the displayed 3D image.

 

Fig. 2. Concept of the proposed method

Download Full Size | PDF

In the following, the extraction of the 3D information and conversion to the elemental images with the 3D profile modified appropriately is discussed. We demonstrate the extraction and conversion to elemental images and integrate them as a 3D image experimentally.

2. Description of the proposed method

2.1 Extraction of 3D information from elemental images

In the integral imaging pickup process, elemental images are formed by the elemental lenses in the lens array and captured by the CCD. Each elemental image is actually a perspective of the 3D object at the specific viewing direction determined by the relative position of the corresponding elemental lens to the object. Therefore the 3D information on the object is stored in the form of a set of the elemental images in the pickup process. By analyzing the elemental images, 3D information on the object can be extracted.

Figure 3 shows the geometry of the pickup part of the integral imaging system. The object point (y,z) is imaged by q-th elemental lens at y _i,q in the elemental image plane and captured at y _c,q in the CCD plane. The position y _c,q of the q-th elemental image of the object point (y,z) is given by

where f_a is focal length of the elemental lens, f_c the focal length of the camera lens, φ the elemental lens pitch, and l the distance between the focal plane of the lens array and the camera lens. The difference in position, or disparity, between the q ₁-th and q ₂-th elemental images (q ₂>q ₁) becomes

where d denotes the fundamental disparity as shown in Fig. 4. Since f_a , f_c , l and φ are known parameters of the pickup system, the depth information z of the object can be extracted by detecting the disparity d. Once the depth z is known, its transverse information y can also be extracted using Eq. (1) and thus full 3D information on the object is obtained.

 

Fig. 3. Geometry of the integral imaging pickup part

Download Full Size | PDF

Note that, in the simple configuration shown in Fig. 3, the pickup directions of the elemental images are not parallel so that the integration of those elemental images is accompanied by distortion in the display process [20]. To make the pickup directions parallel in a conventional integral imaging system, it is necessary to use an additional convex lens that is sufficiently large to cover the lens array. In the proposed method, however, the simplified integral imaging pickup system that is shown in Fig. 3 can be easily used, since the elemental images obtained in the pickup system are not directly used for the display but analyzed for the extraction of the 3D information. No variation in the pickup configuration shown in Fig. 3 or supplementary devices for optical pseudoscopic-orthoscopic conversion is needed in the proposed system, as explained in the following section.

In the extraction of the 3D information, depth is the first information extracted and is found by detecting the disparity given by Eq. (2). The detection of the disparity between two or more perspectives was developed in the field of machine vision or computer vision decades ago. Such techniques are typically called stereo matching algorithms [21]–[23]. These stereo matching algorithms can be directly applied to the elemental images obtained in the pickup process because the elemental images are set of different perspectives of the object [17]. The main difference between the usual stereo matching configuration and the integral imaging pickup system is that stereo matching utilizes two or more cameras to capture the perspective of the object, while an integral imaging pickup system incorporates a lens array and a single camera. Therefore the integral imaging pickup system provides many more perspectives (as many as the number of the elemental lenses) than the usual stereo matching system. Since all perspectives are captured by one camera, however, the resolution of each perspective is typically low. Generally, the accuracy of disparity detection depends on the number of the input images and the resolution of each. Hence, two counteracting characteristics of the integral imaging pickup system - large number of perspectives, low resolution of each perspective - determine the reliability of disparity detection and the extracted 3D information on the object.

 

Fig. 4. Disparity of the elemental images

Download Full Size | PDF

In addition to the low resolution of each elemental image, the narrow field of view of the elemental lens is another restricting factor in the acquisition of the 3D information of an object. The field of view of each elemental lens is given by

When the lateral dimension of the object exceeds the field of view of the elemental lens, 3D information on the object can be fully found by collecting the partial 3D information from each reference elemental image and combining them. In this case, however, a small error in depth extraction from each reference elemental image can result in large abrupt discontinuities and distortion in the reconstructed image. Therefore an object with a small lateral dimension is desirable and for a larger one, a more sophisticated technique is required.

Some techniques have been proposed to fully utilize the large number of perspectives obtained in the integral imaging pickup process and overcome its low resolution and narrow field of view. One involves the generation of sub-images from the elemental images obtained in the pickup process [18]. The sub-image is an image consisting of pixels at the same position in every elemental image. Disparity detection is performed on the sub-images instead of the elemental images. Using this technique, the low resolution and narrow field of view of elemental images can be overcome to some extent. Since the resolution of the generated sub-image is determined by the number of elemental images, this technique is suitable for an integral imaging system in which the lens array has a very small elemental lens pitch and a large number of elemental lenses. In another technique, sub-images are formed only along one direction [19]. Hence the generated images have different disparity characteristics along the two directions. This different disparity characteristic increases the detectable range of the depth information of the object and eliminates ambiguities in disparity detection.

2.2 Modification of the extracted 3D information

The acquired 3D information of the object is stored and transmitted to each display system. In the display system, given the 3D information, the 3D profile of the object such as thickness, depth, transverse size, and so on can be modified so as to be suitable for the parameters of the specific display system and the display conditions. In the integral imaging display system, the 3D profile of the integrated image that can be displayed with tolerable resolution depends on the system parameters. For example, in a display system with a lens array with a large elemental lens pitch, the resolution of the integrated image is good but a thick image cannot be displayed due to the small depth of focus. The distance, or depth, of the integrated image from the lens array also should be selected based on the focal length and the elemental lens pitch of the lens array. If a 3D image is integrated too far from the lens array compared to the focal length, the resolution of the integrated image becomes severely degraded due to the large magnification of the pixels of the display panel through the lens array. Therefore each display system has its own appropriate depth range and image size for integrating a 3D image with the best quality and it is important to maintain the 3D profile of the object in that condition. Since, in the proposed method, the 3D profile of the object can be modified arbitrarily, these specific needs for each display system can be fully satisfied. Consequently the proposed method provides integral imaging systems with excellent flexibility with respect to system parameters.

 

Fig. 5. Concept of the pseudoscopic image problem

Download Full Size | PDF

One thing that should be noted is that the proposed method can solve the pseudoscopic image problem of the integrated image in natural way. In the pseudoscopic image problem, the depth of the integrated image is inverted due to the opposite observation directions in pickup and display processes, as shown in Fig. 5. Conventional integral imaging systems avoid the pseudoscopic image by rotating each elemental image with respect to the center of each elemental image region by means of a gradient index lens array [24] or by digital image processing [20]. The integrated image using rotated elemental images is located behind the lens array with an inverted depth, thus solving the pseudoscopic image problem. However this conventional method is accompanied by the distortion of the integrated image in the depth direction. Let us assume that the pickup system consists of a lens array and a film for simplicity, as shown in Fig. 6(a).

The position of the elemental image y _f,p of the object point at (y,z _c +b) is given by

where z_c is the central depth plane of the object and g_p is the gap between the lens array and the film. After the rotation of each elemental image, the elemental image position $y_{f,q}^{\prime }$ becomes

In the display system, the rotated elemental image is imaged by the corresponding elemental lens at (y _i,q, $z_c^{\prime }$ +b ^′) behind the lens array as shown in Fig. 6(b) and y _i,q is given by

where g_d is the gap between the lens array and the display panel. In order for the elemental images to be integrated, the first parenthesis should be zero. Assuming that the gaps in the pickup system g_p and in the display system g_d are adjusted so that the central depth planes of the object and integrated image are in focus, g_p =z_cf_a /(z_c -f_a ), g_d =$z_c^{\prime }$f_a /($z_c^{\prime }$ +f_a ),

should be satisfied. If we set b ^′=0 when b=0, we obtain

and by substituting Eqs. (8) and (9) to Eq. (6) we obtain

Equations (9) and (10) indicate that the transverse magnification is 1 while the longitudinal magnification is (z_c -2f_a )/z_c . Hence, though pseudoscopic image problem is solved, the integrated image is reduced along the depth direction compared with the original object. Therefore in the conventional integral imaging system, the integrated 3D image should be located behind the lens array and it is impossible to integrate orthoscopic 3D image without depth distortion. In the proposed method, however, the pseudoscopic image problem is easily solved without any distortion. Since 3D information of each voxel of the object is extracted, the orthoscopic image is easily generated by inverting the depth of the voxels with respect to the central depth plane of the object. There is no distortion in any direction, since the depth information of voxels is accessed directly.

 

Fig. 6. Conventional method for overcoming the pseudoscopic image problem (a) simplified pickup system (b) display as a virtual image

Download Full Size | PDF

2.3 Generation of the elemental images

The generation of elemental images with 3D information of the object is performed based on the principle of computer-generated integral imaging (CGII) [25]. The elemental image position of each voxel of the object with modified 3D information is calculated by simple geometric optics, on the basis of the system parameters of the integral imaging display system. The calculated elemental image points are accumulated and stored in depth order.

2.4 Quality of the generated 3D image by the proposed method

One drawback of the proposed method is that the quality of the generated 3D image can be degraded. In conventional integral imaging, the object is sampled to produce an elemental image set and this set of elemental images is directly integrated into a 3D image. Hence the sampling process occurs one time. In the proposed method, however, 3D information is extracted from the elemental image set which is already sampled from the object. As a result sampling is performed a second time to produce new set of elemental images from the 3D information of the object after modification. Thus sampling occurs two times in the proposed method, and this additional sampling process may result in a degradation of the quality of the generated 3D image.

In order to alleviate the degradation of image quality, the resolution of the extracted 3D information should be enhanced. Besides interpolation, the super-resolution techniques [26], [27] developed in the field of image processing represent a possible alternative for this purpose. Since super-resolution techniques produce high resolution image from a set of low resolution images, they can be applied to elemental images obtained by integral imaging [28], [29] to enhance the resolution. By extracting 3D information from high resolution elemental images obtained by the application of super-resolution techniques, the image degradation problem of the proposed method can be relaxed.

3. Experimental results

We demonstrated the proposed method experimentally. Two longitudinally separated dice were picked up, their 3D information extracted and modified variously. The elemental images for various lens arrays were then generated and among them, two elemental images for different lens arrays were integrated using corresponding lens arrays to verify the validity of the proposed method. We used the pickup system shown in Fig. 3. Two lens arrays are prepared in the experiment. We picked up the object using lens array 1 and displayed the integrated image using lens array 1 and lens array 2. The specifications of the experimental setup are listed in Table 1.

 

Fig. 7. The portion of the elemental images captured by CCD

Download Full Size | PDF

Figure 7 shows a part of the elemental images obtained by the pickup. The elemental image for a die with 5 spots on its face is larger than that for a die with 6 spots because it is closer to the lens array in the pickup. The elemental images were then digitally processed for disparity detection. In the experiment, we applied a multiple-baseline stereo algorithm, a type of intensity-based block matching technique [17], [23]. In the multiple-baseline stereo algorithm, the disparity map on the reference elemental image was obtained by using multiple elemental images collectively. If more elemental images are used in the disparity detection, the accuracy is enhanced but more computation time is required. Although the accuracy of disparity detection relies on the number of elemental images used, it tends to be saturated at a certain level. As a compromise of accuracy and computation time, we used elemental images located on the cross that is centered at the reference elemental image in the experiment. Our experimental result shows little enhancement in the accuracy of depth detection when all elemental images were used.

Table 1. Specifications of the experimental setup

View Table

Figures 8 and 9 show the detected disparity map and the calculated 3D information for the object. In Figs. 8 and 9, x_r and y_r are the coordinates with respect to the left-upper corner of the reference elemental image. Figure 8(a) shows the detected disparity map for the reference elemental image. We can see that the detected disparity of the die with 6 facing spots is smaller than that of the die with 5 spots. This is a direct result of the fact that the depth of the die with 6 facing spots is larger than that of the die with 5 facing spots in our experiment, since the disparity is inversely proportional to the depth, as given by Eq. (2). We can also see that the detected disparity map contains some irregular noise. This noise in the disparity map can induce much larger errors and discontinuities in the x, y, and z information on the object as Eqs. (1) and (2). In order to reduce the irregular fluctuation in the disparity map, the disparity map was regulated by a disparity gradient constraint that limits the disparity gradient to under a certain value [30]. The regulated disparity map is shown in Fig. 8(b). We can see that the fluctuation is reduced considerably in the regulated version. With the detected disparity map, the x, y, and z information of each voxel of the object is calculated using Eqs. (1) and (2). Figure 9 shows the 3D information obtained. Note that Figs. 9(a)–(c) are depicted with respect to the reference elemental image so that they show x, y, and z information for the object voxel corresponding to each pixel in the reference elemental image, respectively. Considering the real depth of two dice, 100 mm for the die with 5 facing spots and 130 mm for the die with 6 facing spots, we can see that Fig. 9(c) correctly reveals the depth of two dice.

 

Fig. 8. Detected disparity map (a) initial disparity map (b) regulated disparity map

Download Full Size | PDF

 

Fig. 9. 3D information obtained (a) x position (b) y position (c) z position

Download Full Size | PDF

 

Fig. 10. Examples of the generated elemental images

Download Full Size | PDF

We modified the 3D information on the object and generated elemental images with the modified 3D information and the given specifications of the lens array in the display system. The transverse size, thickness, and depth were modified in various manners and the corresponding elemental images were generated. In the generation of the elemental images, the extracted 3D information was interpolated to prevent under-sampling of the elemental images.

Figure 10 shows some examples of the elemental images that were generated. The origin of some of the noise in the generated elemental images is errors in disparity detection. Note that the conventional integral imaging system cannot cope with any change in the specifications of the lens array and 3D information on the object should be fixed. The results shown in Fig. 10 convince us of the excellent flexibility of the proposed method on system parameters and display conditions.

 

Fig. 11. Integrated image with depth inversion (a) integrated image (b) diffused image by a diffuser at 9.5 cm (c) diffused image by a diffuser at 13 cm

Download Full Size | PDF

 

Fig. 12. Integrated image with transverse magnification (a) diffused image by a diffuser at 9.5 cm (b) diffused image by a diffuser at 13 cm

Download Full Size | PDF

Using the generated elemental images, we integrated the 3D images and some examples are shown Figs. 11–13. Figure 11(a) shows the integrated image using lens array 2 and Fig. 11(b) and (c) show its diffused images obtained by locating the diffuser at corresponding positions. The diffuser was used to precisely determine the depth of the integrated image. We moved the diffuser longitudinally to determine the location where the diffused integrated image has the maximum resolution. In the generation of the elemental images used in Fig. 11(a), we inverted the depth of the object with respect to the central depth plane, which is calculated by averaging the detected depth of the two dice. Therefore we can see that the die with 5 facing spots is integrated farther than the die with 6 facing spots although the die with 5 spots was originally located nearer to the lens array in the pickup. Figure 12 shows the diffused image of the 3D image integrated by lens array 1 with transverse magnification by a factor of one and a half. In this case, the size of the two dice images is enlarged compared to the original size shown in Fig. 11, while the depths of the two integrated images are not inverted. Figure 13 shows the integrated image of Fig. 12 observed at different viewing angles without a diffuser. Since the integrated image of a die with 6 facing spots (left die) is located farther from the lens array (closer to the observer) than the die with 5 spots (right die), their relative positions are changed according to the viewing direction, clearly demonstrating the 3D nature of the integrated image.

 

Fig. 13. Integrated image observed from different directions

Download Full Size | PDF

4. Conclusion

In this paper, an integral imaging system is proposed involving the extraction of 3D information. The proposed method extracts 3D information on the object from elemental images that are picked up by an integral imaging pickup system, modifies it variously, and generates elemental images for an integral imaging display system with different lens array specifications. Since the extracted 3D information of the object, rather than the original elemental image, is transmitted and processed for a specific display system, the proposed method provides much flexibility with respect to system parameters in the integral imaging pickup and display systems and controllability of the 3D profile of the object. The experimental results confirm the feasibility of the proposed method.