Open Access
Add to BibSonomyAbstract
Integral three-dimensional (3D) imaging provides full-motion parallax, unlike other conventional stereoscopy-based techniques. To maximize this advantage, a 3D system with a wide view along all directions is required. We propose and demonstrate a new integral imaging (InIm) method to enhance the viewing angle along both horizontal and vertical directions. Elemental lens switching is performed by a combination of spatial and time multiplexing by use of double display devices and orthogonal polarizations. Experimental results show that the viewing angle of the system is enhanced along all directions without any mechanical movement or any cross talk between afterimages. We believe that the proposed method has the potential to facilitate practical use of the wide-viewing InIm system.
©2003 Optical Society of America
1. Introduction
Integral imaging (InIm), which is also called integral photography (IP), has attracted much attention recently as an attractive three-dimensional (3D) display technology because of its advantages over other methods. For example, InIm does not require any special viewing aids for seeing the 3D images, and it has continuous viewpoints within a specific viewing angle. In addition, whereas conventional stereoscopy-based methods give only horizontal parallax, InIm provides full parallax along both horizontal and vertical directions. Back when this technique was first proposed, the recording and display media were film or photographic plates [1, 2]. But recent studies have demonstrated its usefulness with a high-resolution charge-coupled device (CCD) or liquid-crystal display (LCD) as a pickup or display medium [3]. Therefore the term “integral imaging” is more widely used than “integral photography” because of the possibility for implementing real-time imaging. Continuous efforts to enhance viewing resolution or reality of depth in InIm have been widely studied [4–8].
However, one of the main bottlenecks in InIm is the limitation of viewing angle. In general, the viewing angle of a 3D display system using InIm has not been as wide as that of a conventional two-dimensional (2D) display. According to previous studies, the viewing angle of the InIm system in experiments was ~20 deg along horizontal or vertical directions [9]. The narrow viewing angle of the InIm system results because the maximum viewing angle is limited by the physical properties of the lens array. Although some schemes to enhance the viewing angle of InIm were proposed recently, enhancement of the viewing angle could be obtained along only one direction (horizontal or vertical direction). Therefore it is desirable that we improve the viewing angle along all directions in order to take full advantage of InIm. Full parallax is one of the important features that distinguish the InIm from other techniques. In this paper, we propose and demonstrate what to our knowledge is a new scheme for enhancing the viewing angle of InIm along all directions. Another benefit of our scheme is that we do not need any mechanical movement to implement the proposed system. We believe that the proposed method will facilitate the practical use of the wide-viewing system and extend the applicability of optical systems for integral 3D imaging displays.
2. Viewing-angle-enhanced integral 3D imaging
2.1 Limitation of viewing angle in integral imaging system
The basic concept first proposed by Lippmann in 1908 is quite simple. The rays containing the information of a 3D object are sampled as the form of a 2D image array through a lens array or a pinhole array. Then the 2D image array, called an elemental image array, is recorded and displayed on a proper medium. During the display process, the rays from the 2D image array retrace the original routes after transmission through a lens array. As a result, an observer can feel as if a 3D object were present at the same place where it was. But the major problem of InIm in the early days was the limitation of implementing dynamic 3D images. The use of passive devices as recording and display media prevented integrating 3D images in real time. With the recent rapid development of active devices, CCD and LCD have replaced film. Consequently, InIm becomes applicable to a 3D TV system or a 3D animation display. To enrich displayed images, the elemental image array can also be generated by computer graphics (rather than by pickup process), which we call the computer-generated (CG) InIm.
The viewing angle is a very significant factor in the display system because observers want to see the image without restriction on viewing positions, and the maximum number of observers is reduced if the viewing angle is narrow. InIm is advantageous in that the viewpoint is not fixed, and this fact gives an observer freedom of movement. However, the viewing angle of the InIm is basically restricted because the range of presentable elemental images is confined. Moreover, the image overlapping or image duplication disturbs the original images. Figure 1 illustrates the factor that limits the viewing angle. Generally, the viewing angle in InIm depends on the characteristics of the elemental lens in an array. That is, it is desirable to adopt a lens array whose elemental lens has a small f-number. This fact can be figured out easily from Fig. 1.
The limitation first arises because the observer views the fringe of each elemental image at the fringe of the viewing angle. Each elemental lens in an array has its own area on the display panel, and in the conventional InIm each elemental image cannot be displayed out of that area, as in the figure. This limitation on the size of an elemental lens gives a viewing angle as shown in the following equation:
where w is the width of an elemental lens and g is the gap between the lens array and the elemental images. The gap is determined by the relation between the object distance and the focal length of a lens array. Assuming that g is approximated by the focal length of a lens array f, we can find that a lens array with small f-number (f/w) is advantageous for increasing the viewing angle. However, as f-number decreases, aberration due to short focal length also increases [10]. This trade-off between viewing angle and aberration makes it important to choose a lens array with optimum f-number. Another factor that limits the viewing angle of InIm is interference through neighboring elemental lenses. During the pickup process, the elemental images captured through the corresponding elemental lens and neighboring lenses may overlap. This is because the light rays from different points may converge at the same point. Hence they cannot be separated successfully during reconstruction because the elemental images originate from different object points. Also, in the CG InIm, a similar effect can cause image overlap during the reconstruction, and the maximum viewing angle is reduced.
2.2 Viewing-angle enhancement by use of elemental lens switching
In an effort to increase the viewing angle in InIm, several methods have been proposed. For example, a scheme using a phase-conjugate readout beam was reported recently [11]. However, this method is not practical because it is based on the volume holographic scheme, which uses a laser as a light source. To solve the problem of the interference through neighboring elemental lenses, the use of a gradient-index lens array that functions like an optical barrier during the pickup and reconstruction process has also been suggested [12]. However, a gradient-index lens array is not easy to obtain in the commercial market. Some other ideas that we have proposed for wide-viewing InIm adopt the concept of elemental lens switching by mechanical or nonmechanical methods [13, 14, 15]. Recently, the viewing-angle-enhanced scheme by use of moving lenslet arrays with low fill factor has also been proposed [16]. In that scheme, the viewing angle is enhanced along all directions by motion of tilted lens arrays; however, this involves mechanical movement.
The basic idea of our previous research is quite simple. If we block some parts of neighboring elemental lenses with a proper method, then overlapping or interference does not occur. The lost information of the blocked area is compensated for if we multiplex opened area time sequentially or spatially. Figure 2 shows why we can observe the integrated image around a wider viewing region if we use this scheme. In the elemental lens-switching scheme, the system operates at two modes as the figure shows. At mode I, the elemental image point P can be seen by the observer A through lens 1, while it is blocked from observer B. And the reverse is true at mode II; the elemental image point Q can be seen by observer B through lens 2, while it is blocked from observer A. The image points P and Q are parts of different elemental images. In both modes, the elemental image array for the corresponding mode is displayed in synchronization with the change of the switching state. If two modes change alternately so fast that an afterimage effect is produced, the image integration at one mode can be performed without interrupting another mode. As a result, the viewing angle becomes wider because an observer can see undistorted 3D image at the viewpoint A.
In fact, in the conventional InIm, each elemental image for each elemental lens should exist on only the area that covers the region of the corresponding elemental lens. If the observer moves away from the proper viewing region, the elemental image for a certain elemental lens is seen through the neighboring elemental lens. This causes integrated image overlapping or duplication, which limits the viewing angle of the system. But in the viewing-angle-enhanced scheme, the elemental image arrays can exist on the area that covers not only the total region of the opened mask but also the half-region of the neighboring closed mask. Hence the area is twice as wide as that of the conventional InIm. Figure 3 shows an example of the procedure to generate elemental image arrays for the two modes. As the figure shows, the elemental images are displayed on the area that expands to the half-region of neighboring elemental lenses. The switching mask plays the role of a switching device that switches the elemental lenses on and off, and this is realized by several methods.
The elemental lens switching can be performed by mechanical or electrical methods. First, we can switch elemental lenses by moving an on-off patterned mask that moves alternately along one axis [13]. In this scheme, the moving interval is exactly the same as the width of an elemental lens. The mask has alternative on-off patterns, and it is in the form of a vertical aperture array. If the mask is shifted to the left or the right by the aperture interval, the vertical array of lenses is opened or blocked in turn. However, this method requires fast mechanical movement to obtain sufficient speed for the afterimage effect. Ideally an electrically controllable liquid-crystal (LC) shutter array instead of a moving mask is most appropriate for the nonmechanical scheme. But its implementation is difficult because that kind of a shutter array is not commercially available. Electrical switching by use of orthogonal polarization has also been proposed [15]. The system consists of an orthogonal polarization sheet attached to a lens array and orthogonally polarized elemental image arrays with a polarization shutter screen. The polarization shutter screen that is widely used in the field of stereoscopy can modulate two orthogonal polarization states electrically. The dynamic speed is fast enough to exceed the eye’s response time if the screen operates with a high refresh rate. Hence we can say that this scheme is more practical than the mechanical switching method. Another possible method for elemental lens switching is the spatial multiplexing of two modes for elemental image arrays [14]. In this approach, the total system consists of two sub-InIm systems with double display devices and on-off patterned masks. The corresponding elemental image arrays for each state are displayed in subsystems, and the integrated images from each subsystem are combined spatially. Hence this scheme does not depend on the afterimage effect or mechanical movement. However, all the above schemes are for viewing-angle enhancement along only one direction (for example, along only the horizontal direction). To see the full parallax 3D images more freely, it is obvious that the viewing angle of the system should be enhanced along all directions (both horizontal and vertical directions). This necessity of viewing-angle enhancement along all directions is the motivation of our research.
3. System configuration of the proposed method
In this paper, we propose and implement an integral 3D imaging system that has an enhanced viewing angle along both horizontal and vertical directions with no mechanical moving parts. The basic idea to increase the viewing angle is similar to the concept of elemental lens switching that we suggested previously. But the viewing-angle enhancement along all directions by use of the combination of spatial and time-multiplexed methods is the major and important difference between the proposed scheme and our previous research in Refs. [13]–[15]. To obtain viewing-angle enhancement, we have to relax the rule that the elemental images exist on the area of corresponding elemental lenses. Therefore, to enhance the viewing angle along all directions, the existing region of elemental images should be enlarged along both horizontal and vertical directions. And the temporal-spatial multiplexing of elemental images by elemental lens switching makes the incomplete integrated image combine successfully. The schematic diagram of the proposed method is shown in Fig. 4(a). The system consists of two subsystems combined by a beam splitter. Each subsystem has a display device, a variable polarizer, and a lens array attached with polarizing mask cells and completely off-patterned mask cells. Here off-patterned mask cells are attached in the form of alternating (every-other-column) vertical arrays and have the opposite patterns for the two subsystems. If we look at the overall system in front of it, all the elemental lenses seem to be opened because they are combined by a beam splitter. The rest of the regions (the regions that are not blocked by the off-patterned mask cells) are filled up by the polarizing mask cells that have two orthogonal polarization states alternately along the vertical direction as shown in Fig. 4(a). The system operates at two modes according to the polarization states that can be controlled by a variable polarizer. The variable polarizer modulates two orthogonal polarization states, which are the same as those of the polarizing mask cells. As a result, the status of elemental lenses in each system becomes like that of Fig. 4(b). The elemental lenses are opened or blocked by the polarization modulation and have four overall statuses as in the figure. The white regions denote opened lenses. Four types of elemental image arrays are displayed according to each status. The possible regions in which the elemental images are displayed are denoted with yellow square boxes in Fig. 4(b). That is, each elemental image can exist in the area including corresponding elemental lens and some regions of neighboring lenses along both horizontal and vertical directions. Another noticeable fact is that the proposed system has no mechanical moving part because it utilizes polarization switching and spatial multiplexing. This nonmechanical structure makes the system more advantageous for adoption in real applications.
The process that generates the elemental image array in our proposal is similar to that of viewing-angle-enhanced InIm along one direction except that the current case expands into the 2D area. The same rule for displaying elemental images on the area including neighboring elemental lens regions is applicable. If we assume that the object point is P(x, y, z) and the center of the (n,m)th elemental lens (m, column; n, row) is L(Lm, Ln, g), the corresponding elemental image should be displayed on the point E(Ex, Ey, 0). Here, the plane of the display panel is located at z=0 and the lens array is separated from the display panel by g. The rays passing through the (n,m)th elemental lens satisfy the following equation by simple geometric calculation:
where g is the gap between THE lens array and THE display panel. In conventional CG InIm, the possible position of the elemental image point E is restricted by following conditions:
where Wh is the horizontal width and Wv is the vertical width of an elemental lens. These restraints on the location of elemental images are relaxed in our proposed method. The elemental image point E can exist if it satisfies the following conditions:
Therefore, the area where the elemental images are displayed is four times wider than that of the conventional method.
The important feature of the proposed method is that there is no cross talk between afterimages. Because the elemental lens switching by use of orthogonal polarization depends on the afterimage effect, the unwanted image from the surrounding elemental lenses may appear. This cross talk is prevented from appearing in our proposed scheme by proper arrangement of opened elemental lenses. The cross talk occurs when both integrated images from one status and another status seem to be seen simultaneously. This kind of cross talk is another problem that should be distinguished from the interference between neighboring elemental images. The interference between adjacent elemental images is prevented effectively by switching or blocking of elemental lenses. However, if the f-number of the elemental lens or the size of elemental image is not so small, the interference between elemental images for the opened neighboring lenses (across the blocked lenses) can still take place. In addition to the interference between elemental images, the cross talk between the afterimages can also occur if the opened region of the elemental lens is arranged inadequately. For example, we can assume different types of masks in two subsystems as in Fig. 5(b). In fact, this scheme is not so practical to implement because we need an electrically controllable switching device such as a LC shutter array, which is not commercially available. In this scheme, mode I is for enhancing viewing angle along the horizontal direction and mode II is for enhancing viewing angle along the vertical direction. The elemental image arrays are generated in the same way as in Fig. 3; that is, they are produced to enhance the viewing angle along one direction. The possible area for displaying elemental images is within the yellow square boxes of Fig. 5(b). If the two modes are time multiplexed with a fast speed that exceeds the eye’s response time, the observer experiences the afterimage effect. There is no cross talk between systems 1 and 2 in the same mode. One may wonder whether the elemental images for the (3,3)th elemental lens of mode II-system 1 and the (2,3)th lens of mode II-system 2 have cross talk resulting from the overlapping area in Fig. 5(b); however, the observer does not experience any cross talk, for the reason illustrated in Fig. 2.
Let us consider the (3,3)th elemental lens in mode I-system 1 and mode II-system 1 in Fig. 5(b). From the figure we can find that the elemental images for both horizontal mode (mode I-system 1) and vertical mode (mode II-system 1) may be seen through the same (3,3)th elemental lens, which is opened in this status. Although time multiplexed, the integrated image from these two statuses seems to be seen simultaneously, owing to the afterimage effect. This overlapping of the opened elemental lens between different statuses is the main cause of the cross talk. Similarly, as Fig. 5(b) shows, the images integrated from the (2,3)th elemental lens of mode II-system 2 may be also seen when we are to see the images integrated from the (2,3)th elemental lens of mode I-system 1. The black circle in the Fig. 5(b) indicates these situations. In contrast, in Fig. 5(a), any opened elemental lenses are not overlapped or omitted through total status in our proposed scheme. This suitable arrangement of opened regions prevents the cross talk between afterimages that appear in the time-multiplexing-based scheme.
The elemental image arrays for enhancing viewing angle along all directions can be produced not only in the CG InIm scheme but also in the pickup/display InIm scheme. We have recently proposed a viewing-angle-enhanced (along single direction) pickup/display InIm that uses elemental lens resizing and elemental lens switching [17]. The basic idea is that the elemental image array is taken at a longer object distance and image postprocessing is performed by resizing and cropping. A similar approach can be applied to our proposed scheme. Figure 6 shows the process of generating modified elemental image arrays from the pickup images. If we are to reproduce the original object at the distance of z, we pick up the object at the distance of z’ that is longer than z. Here we assume z’=2z as a matter of convenience. To obtain four sets of elemental image arrays according to the polarization and subsystem as in Fig. 4, the original elemental image arrays are magnified by z’/z along both horizontal and vertical directions. Then the elemental images that correspond to opened elemental lenses in Fig. 4 are collected and rearranged by the four types of situation. In Fig. 6 the pickup images distinguishable by different colors are magnified by a factor of 2 and are reassembled. In the previous case [17], a vertical crop was needed to fit the image size because the elemental images were for enhancing viewing angle along only one direction. But in our proposed scheme, the image cropping is not required. In this way, the viewing-angle enhancement along all directions is possible not only in the CG InIm but also in the pickup/display InIm.
Fig. 6. Image processing of elemental images obtained by pickup. (a) Classification, (b) magnification and reassembling.
4. Experimental results
For demonstrating the feasibility of the scheme in Fig. 4, we performed a basic experiment to investigate the viewing-angle enhancement along all directions. We used a square-type lens array that has 13×13 elemental lenses, and each elemental lens has a focal length of 22 mm and a width of 10 mm. If we consider Eq. (1), the maximum viewing angle for the conventional method is 10.05 deg along both horizontal and vertical directions. However, actual viewing angle is more reduced in the conventional method, owing to the integrated image overlapping by neighboring interference. Figure 7 shows elemental image arrays generated by computer graphics following the procedure of Eqs. (2) and (3) and Eqs. (6) and (7). Four types of elemental image array sets exist according to the polarization status and subsystem. The possible area for displaying each elemental image is enlarged by a factor of four (2×2).
Ideally the orthogonal polarization states should be modulated by an active device such as a polarization shutter screen that works with high speed. But, in these preliminary experiments, we proved our idea by changing the polarization of the input field with in sufficient speed for the afterimage effect. Figure 8 shows the integrated images observed from different viewing directions in the conventional and the proposed InIm. From the figure, we can determine that the shape of the integrated image in the conventional scheme is distorted at the observation angle of left 9.6 deg and up 13.4 deg, which exceeds the maximum value of viewing angle of the system. In contrast, in the proposed scheme, we can see that the image distortion does not appear in both directions from the same viewing positions. The contrast of the image is partly different (the image integrated from subsystem 2 seems to be darker) because the reflectance of the beam splitter is lower than the transmittance and the contrast of the display panels is not the same. This problem can be solved if we use a more efficient beam splitter and adjust the contrast of the display panels in advance. Thus we can confirm that our proposed method increases the viewing angle along all directions without any mechanical movement.
However our system also has some drawbacks, unfortunately, in spite of its advantages. First of all, the system complexity increases because a pair of subsystems are necessary. The reduction of light efficiency caused by the use of beam splitter and polarizer is another problem. In addition, the signal bandwidth, which is important in real-time broadcasting such as a 3D TV, is reduced to half if we assume a conventional frame rate. This is because the system depends on the time-multiplexing scheme that divides the timeslot into two. Therefore, the polarizing shutter with high transmission efficiency and high refresh rate is recommended for applying our system to practical applications. Further study on reducing system complexity is also needed.
5. Conclusions
We have proposed and demonstrated a new integral imaging (InIm) method to enhance the viewing angle along both horizontal and vertical directions. Elemental lens switching is performed by a combination of spatial and time multiplexing with double display devices and orthogonal polarizations. Experimental results show that the viewing angle of the system is enhanced along all directions without any mechanical movement or any cross talk between afterimages. Furthermore, this method can also be applied in the direct pickup/display InIm scheme if some image postprocessing is adopted. We believe that the proposed method has the potential to facilitate practical use of the wide-viewing InIm system.
Acknowledgments
This research was supported by the Ministry of Science and Technology of Korea through the National Research Laboratory Program.
References and links
1. M. Lippmann, “Epreuves reversible donnant la sensation du relief,” J. Phys. 7, 821–825 (1908).
2. H. E. Ives, “Optical properties of a Lippmann lenticular sheet,” J. Opt. Soc. Am. 21, 171–176 (1931). [CrossRef]
3. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36, 1598–1603 (1997). [CrossRef] [PubMed]
4. B. Lee, S. Jung, S.-W. Min, and J.-H. Park, “Three-dimensional display using integral photography with dynamically variable image planes,” Opt. Lett. 26, 1481–1482 (2001). [CrossRef]
5. L. Erdmann and K. J. Gabriel, “High-resolution digital integral photography by use of a scanning microlens array,” Appl. Opt. 40, 5592–5599 (2001). [CrossRef]
6. J.-S. Jang and B. Javidi, “Improved viewing resolution of three-dimensional integral imaging by use of nonstationary micro-optics,” Opt. Lett. 27, 324–326 (2002). [CrossRef]
7. S. Manolache, A. Aggoun, M. McCormick, N. Davies, and S. Y. Kung, “Analytical model of a three-dimensional integral image recording system that uses circular and hexagonal-based spherical surface microlenses,” J. Opt. Soc. Am. A. 18, 1814–1821 (2001). [CrossRef]
8. T. Naemura, T. Yoshida, and H. Harashima, “3-D computer graphics based on integral photography,” Opt. Express 8, 255–262 (2001), http://www.opticsexpress.org/ abstract.cfm?URI=OPEX-8-4-255 [CrossRef] [PubMed]
9. F. Okano, J. Arai, H. Hoshino, and I. Yuyama, “Three-dimensional video system based on integral photography,” Opt. Eng. 38, 1072–1077 (1999). [CrossRef]
10. S.-W. Min, S. Jung, J.-H. Park, and B. Lee, “Study for wide-viewing integral photography using an aspheric Fresnel-lens array,” Opt. Eng. 41, 2572–2576 (2002). [CrossRef]
11. S.-H. Shin and B. Javidi, “Viewing-angle enhancement of speckle-reduced volume holographic three-dimensional display by use of integral imaging,” Appl. Opt. 40, 5562–5567 (2001).
12. J. Arai, F. Okano, H. Hoshino, and I. Yuyama, “Gradient-index lens-array method based on real-time integral photography for three-dimensional images,” Appl. Opt. 37, 2034–2045 (1998). [CrossRef]
13. B. Lee, S. Jung, and J.-H. Park, “Viewing-angle-enhanced integral imaging using lens switching,” Opt. Lett. 27, 818–820 (2002). [CrossRef]
14. S. Jung, J.-H. Park, B. Lee, and B. Javidi, “Viewing-angle-enhanced integral 3D imaging using double display devices with masks,” Opt. Eng. 41, 2389–2390 (2002). [CrossRef]
15. S. Jung, J.-H. Park, H. Choi, and B. Lee, “Wide-viewing integral three-dimensional imaging by use of orthogonal polarization switching,” Appl. Opt. 42, 2513–2520 (2003). [CrossRef] [PubMed]
16. J.-S. Jang and B. Javidi, “Improvement of viewing angle in integral imaging by use of moving lenslet arrays with low fill factor,” Appl. Opt. 42, 1996–2002 (2003). [CrossRef] [PubMed]
17. J.-H. Park, S. Jung, H. Choi, and B. Lee, “Viewing-angle-enhanced integral imaging by elemental image resizing and elemental lens switching,” Appl. Opt. 41, 6875–6883 (2002). [CrossRef] [PubMed]
