1. Introduction

Integral imaging (InIm) is one of the most feasible technologies for displaying a three-dimensional (3D) object since full parallax field is reconstructed without viewing aid such as glasses [1–3]. InIm display uses a lens array to define the viewing directions of correspondent elemental images. Each elemental image is a projection view of 3D object and projection view volumes are positioned periodically according to the centers of lenses in lens array. The field of view in InIm display is able to display both virtual and real objects when the gap between the lens array and elemental images is equal to the focal length of lens. Therefore, the pickup method of both real and virtual objects is necessary. Various computational generations of elemental images for InIm display have been proposed [4–6]. But it is a challengeable issue to embody the actual optical pickup system with the view volumes equivalent to those of InIm display.

In order to pickup actual objects, many optical systems with lens array have been proposed. Generally, the pseudoscopic problem occurs in the pickup with a single lens array since the view volume of an individual lens in the pickup system has mirror symmetry to the view volume in the display [7, 8]. To solve the pseudoscopic problem, each elemental image is flipped and the pseudoscopic real image is changed into an orthoscopic virtual image. Therefore, in these techniques, only virtual objects could be captured orthoscopically. To improve the quality of pickup images, the numerical compensation is necessary [9, 10]. Some pickup systems with the optically correct view volumes have been proposed [11–13]. However, these optical solutions also capture only virtual objects even though they have the advantages of optically corrected pickup without unnecessary data processing.

For capturing the real objects, the depth conversion optics has been studied intensively. The depth conversion optics makes it possible to capture real objects for InIm. Typically, the depth conversion is executed by two sequential steps. In the first step, real objects are captured pseudoscopically and in the second step, the pseudoscopic real images are converted to orthoscopic virtual images [14–16]. And in the same sense, the micro-convex-mirror arrays can be applied as depth conversion optics [17]. These processes could be simply understood as converting the real field by mirror symmetry twice. Therefore, by adjusting the distance between two lens arrays in the second step, the depth of objects is controllable [18]. However, the only real objects can be captured with these techniques.

The pickup techniques for both virtual and real objects have been proposed by some research groups. The main idea of these techniques is the application of a large converging lens before the lens array [19, 20]. In these optical systems, all view volumes defined by the lens array converge with the help of the large convex lens. Here, the plane where each view volume is focused is a critical plane to separate virtual and real fields during pickup [21]. However, the converging lens distorts view volumes and the compensation is necessary for corrected pixel matching. Thus, as a numerical compensation method, the undistorted pickup system with the view volumes equivalent to those of InIm display is proposed by Martinez-Corral et al [22].

This paper is organized as follows. In Sec. 2, the distortion problem is described and the imbricate view volumes of InIm display are expressed analytically. In Sec. 3, the undistorted pickup with imbricate view volumes is proposed. In Sec. 4, experimental results are presented and discussed. In Sec. 5, conclusion and perspective are given.

2. Distortion problem between pickup and integral imaging

The conventional pickup of both virtual and real objects with a large convex lens results in a distortion as shown in Fig. 1. In this conventional pickup, there exists a critical plane which defines virtual and real fields separately. Here, the pickup directions (red arrows in Fig. 1(a)) of elemental images are not parallel while the display directions (green arrows in Fig. 1(b)) of elemental images are parallel to one another in a general InIm display. Therefore, the reconstructed objects in InIm display from the conventional pickup are distorted as shown in Fig. 1(b). That is, the virtual object shrinks in comparison with the real object. These phenomena result from the convergence of pickup directions of elemental images. And thus virtual objects are magnified relatively. To capture elemental images correctly, the pickup method with the view volumes equivalent to those of InIm display is necessary.

 

Fig. 1. Distortion in conventional pickup of both virtual and real objects with a large convex lens: view volumes of (a) pickup and (b) InIm display.

Download Full Size | PDF

The view volumes of InIm display are the set of sub-view volumes by every lens in the lens array. Figure 2 shows a sub-view volume of an individual lens which is represented as the viewing angle and display direction of elemental image. Each lens has an aperture stop defined by the interval between lenses. And the width of this aperture stop is usually the same as the width of an elemental image. Therefore, the viewing angle, θ_display is given by

Here, w and f mean the width of an elemental image and the focal length of each lens respectively. For the ith lens, the corresponding sub-view volume is given by

Here, (x _ci,z _ci) is the center position of aperture stop for the ith lens. That is, the individual sub-view volume generated by a single lens is shaped as two equilateral triangles with the vertexes in contact. And the direction of elemental image is parallel to the optical axis.

 

Fig. 2. Sub-view volume of an individual lens in InIm display.

Download Full Size | PDF

Every sub-view volume defined by lens array forms an imbricate view volume as shown in Fig. 3. One sub-view volume is overlapped with others. And the display directions of elemental image are parallel to one another and positioned apart periodically.

 

Fig. 3. Imbricate view volumes of InIm display.

Download Full Size | PDF

3. Pickup method with imbricate view volumes

The pickup method with the equivalent view volumes is realized by 4f optics and a telecentric lens. In this proposed system, the imbricate view volumes are transformed by 4f optics and the directions of view are defined by the telecentric lens normal to the lens array plane. Therefore, pickup directions of elemental image are parallel to one another. Figure 4 shows a schematic of the proposed pickup system with imbricate view volumes. The individual sub-view volume is shaped as two equilateral triangles with the vertexes in contact and each sub-volume is overlapped with others. These view volumes are equivalent to those of InIm display. In these imbricate view volumes, the virtual and real fields are divided by the critical plane and this critical plane is located at the focal plane of the front lens in 4f optics.

 

Fig. 4. Schematic of pickup system with imbricate view volumes.

Download Full Size | PDF

The imbricate view volumes are separated spatially after transformed by 4f optics and the lens array is aligned at the position where the every view gets close and contacts with adjacent others. The distance between the back lens in 4f optics and the lens array is the sum of the focal lengths of two serial lenses. In Fig. 4, f ₁ and f ₂ represent the focal lengths of front and back lenses in 4f optics respectively. And f ₃ is the focal length of lens array. In this proposed pickup, the viewing angle is defined by

Here, w ₃ means the width of aperture stop of an individual lens in the lens array. Therefore, in order to keep the proper relation between pickup and InIm display, two viewing angles, θ_display and θ_pickup, should agree with each other.

The CCD is relatively small in comparison with objects to be captured. Hence, the magnification optics is necessary. In this proposed pickup system, a telecentric lens is applied for magnification and it filters out any rays with undesirable directions. Therefore, the only rays with normal direction to the lens array plane are able to be captured by CCD. This filtering by the telecentric lens is reasonable since only directions of view normal to the lens array plane are captured.

In this proposed system, a point to be considered is that every captured elemental image is flipped in comparison with desirable elemental images of InIm display. Figure 5 shows the relationship between demanded elemental images and a pickup image. Since the pickup system and InIm display have the same view volumes, the demanded elemental image for InIm display is located in front of the critical plane. After 4f optics, the rays passing through demanded elemental images are disordered when they arrive at the pickup plane. Therefore, a rotation process is necessary to organize the set of elemental images for InIm display. By the rotation process each elemental image is rotated 180 degrees around its center. This process simply changes the coordinates of an individual elemental image in reverse order and requires negligible computational costs.

 

Fig. 5. Relationship between demanded elemental images and pickup image.

Download Full Size | PDF

Figure 6 shows the photograph of embodied pickup system with imbricate view volumes. The 4f optics are composed of two Fresnel lenses with focal length of 152.4mm, and the lens array has 1.0mm periods and 3.3mm focal lengths. The telecentric lens with 0.09×magnification is applied, which was manufactured by Edmund optics. The CCD is Sony XCD-SX90CR with 3.75 µ m pixel size. Therefore, the resolutions of each elemental image are 24×24 pixels.

 

Fig. 6. Photograph of embodied pickup system with imbricate view volumes.

Download Full Size | PDF

4. Experimental results

The proposed pickup method provides elemental images of virtual and real fields for InIm without distortion. In this section, we present two experimental results, where one shows the simultaneous pickup of both virtual and real objects and the other shows the comparison of the proposed method with a conventional pickup.

Figure 7 shows the elemental images captured by the proposed system. In Fig. 7(a), the raw image on pickup plane is shown. The set of elemental images after rotation process is shown in Fig. 7(b). Here, a red smile face is a real object and a yellow smile face is a virtual object. Therefore, the normal outer directions in boundary of real object concentrate inside but those of virtual object spread outside as expected. Figure 8 shows the perspective views of reconstructed image.

 

Fig. 7. (a) Raw pickup image and (b) set of flipped elemental images.

Download Full Size | PDF

 

Fig. 8. Reconstructed image: (a) the perspective views and (b) (Media 1).

Download Full Size | PDF

In order to compare the proposed method with a conventional pickup, we configure the objects where three of them have fixed positions and one is movable as shown in Fig. 9. Here, the virtual and real objects are marked with letters ‘V’ and ‘R’ respectively. The object on the critical plane, z=0mm is marked with a letter ‘C’. And the object with a letter ‘M’ is able to move from the virtual object position, z=-30mm to the real object position, z=30mm.

In the conventional method, the pickup system has the same schematic as that shown in Fig. 1(a). In this system, the specifications of optical components are the same as the proposed system except that 4f optics is replaced with a single Fresnel lens with focal length of 152.4mm. This Fresnel lens is the large convex lens controlling the position of critical plane. Figure 10 shows pickup images with the conventional method. In Figs. 10(a)–(c), the object marked with ‘M’ is positioned at z=-30mm, z=0mm, and z=30mm respectively. Figs. 10(d) and (e) show the elemental images of the virtual and real objects respectively. The elemental images of the movable object are shown in (f) and (g) where they are the parts of (a) and (c) respectively. In this conventional method, the letters on virtual and real objects are hard to be recognized. These phenomena result from the perspective error of a large convex lens.

 

Fig. 9. Configuration of objects with different positions.

Download Full Size | PDF

 

Fig. 10. Conventional pickup images with the movable object positioned at (a) z=-30mm, (b) z=0mm, and (c) z=30mm respectively. The elemental images of virtual and real objects are shown in (d) and (e) respectively. And the elemental images of the movable object are shown in (f) and (g) when it is positioned at z=-30mm and z=30mm respectively.

Download Full Size | PDF

 

Fig. 11. Proposed pickup images with the movable object positioned at (a) z=-30mm, (b) z=0mm, and (c) z=30mm respectively. The elemental images of virtual and real objects are shown in (d) and (e) respectively. And the elemental images of the movable object are shown in (f) and (g) when it is positioned at z=-30mm and z=30mm respectively.

Download Full Size | PDF

Figure 11 shows the pickup images with the proposed method. Figures 11(a)–(c) show the pickup images when the movable object is located at the same positions as Fig. 10(a)–(c). And Figs. 11(d)–(g) show the elemental images of virtual, real, and movable objects respectively. The transform of virtual and real fields with 4f system shows much smaller perspective errors in elemental images than the conventional method.

As previously discussed, the distortion appears in the conventional pickup. Figure 12 shows the movie of reconstructed images by the conventional and proposed methods. In the conventional method, the virtual object appears smaller than the real object. And the movable object grows as its position changes from z=-30mm to z=30mm. On the other hand, in the proposed method, the reconstructed images show virtual and real objects with the same size. And the size of movable object is also unaffected by its position. Therefore it is proved that the distortion which occurs in the conventional pickup method cannot be found in the proposed method.

 

Fig. 12. (Media 2) Pickup and reconstructed images with the conventional pickup (left) and the proposed pickup (right) according to the position of the movable object.

Download Full Size | PDF

Figure 13 shows the perspective views of reconstructed images with representative positions of the movable object. When the letter ‘M’ is positioned at z=-30mm, it is swinging with the constant gap between itself and the letter ‘V’. When it is positioned at z=30mm, it is swinging with the constant gap between itself and the letter ‘R’. As expected, the proposed pickup method provides correct perspective views according to the change of the position of the letter ‘M’.

 

Fig. 13. (Media 3) Perspective views of reconstructed images at z=-30mm, z=0mm, and z=30mm respectively.

Download Full Size | PDF

5. Conclusion

Undistorted pickup method of both virtual and real objects for InIm is proposed. The proposed pickup system is realized by 4f optics and telecentric lens and it has the same imbricate view volumes as those of InIm display. In this system, there is no distortion resulting from the discord in directions of elemental image between pickup and display. Hence additional numerical compensations are unnecessary. Even though the rotation process of each elemental image is required, this simple calculation needs negligible computational cost. Therefore, this proposed method makes real-time pickup feasible without the apparent distortion. It is expected that the proposed technique can be one of the most promising real-time pickup techniques for InIm.