Open Access
Add to BibSonomyAbstract
We present a method to display an integral three-dimensional (3D) image without gaps between multiple display active areas by using multiple liquid crystal display (LCD) panels and multi-image combining optical systems (MICOS). We designed a MICOS to improve the resolution characteristics and decrease the luminance unevenness corresponding to the viewpoint. Furthermore, we developed a method for correcting the distortion of the integral 3D image by using image processing. We prototyped an integral 3D display using four 8K dual-green (8KDG) LCD panels and the improved MICOSs. The prototype display achieved to magnify the display area about 5.66 times more than when a single LCD panel was used.
© 2017 Optical Society of America
1. Introduction
The original integral photography (IP) technique for capturing and displaying three-dimensional (3D) photographs was developed by Lippmann [1] in 1908. An IP image enables 3D images to be seen without special glasses and full-parallax images to be presented in accordance with viewpoints. Various studies have reported IP-based systems for capturing and displaying 3D images [2–6]. We have been advancing the research and development of an integral 3D television based on the IP principle that can enable viewers to see 3D images naturally and easily [7,8].
The basic principle of IP system to capture and display 3D images is as follows. First, an object is captured through a lens array aligned by many small lenses horizontally and vertically as shown in Fig. 1(a). Light rays can be captured from various directions at once by using the lens array. Next, as shown in Fig. 1(b), the captured image (elemental image) is displayed on the display, and a lens array is placed in front of the display. Therefore, the light rays emitted from the object are reproduced, and the 3D image is reconstructed spatially. However, as shown in Fig. 1, since the capturing direction and viewing direction are reversed, a problem arises that the depth of the 3D image is reversed. This problem can be solved by rotating each elemental image 180 degrees with respect to its center [4]. A liquid crystal display (LCD) panel, a projector, and so on are used to display the elemental images.
The integral 3D television reconstructs the light rays from various directions that are emitted from the objects. Thus, to improve the quality of the 3D images, elemental images that have many pixels are required [9]. We have reconstructed an integral 3D image that has about 100,000 elemental lenses by using a projector that made the resolution equivalent to 16K image (15,360 × 8,640 pixels) by wobbling technology using display elements of 8K Super Hi-Vision (SHV) [10,11]. However, since it is currently difficult to display more than 8K on a single flat display, improving the quality of an integral 3D image by a single display without using time division technology is considered to be difficult. Therefore, we have attempted to increase the pixels by combining multiple displays and studying techniques to improve the quality of the integral 3D image.
Researchers have attempted to display 3D images by combining multiple projectors [12–15]. When multiple projectors are used, the screen size and number of pixels of a 3D image can be flexibly changed. However, since this is done by using the projectors, the projection length becomes long and the equipment becomes thick. We assume that 3D televisions will be watched in general households in the future. Thus, the equipment needs to be made thin.
Therefore, we developed a method for connecting images of multiple LCD panels without gaps between multiple display active areas by using a multi-image combining optical system (MICOS) and increasing the number of elemental images for reconstructing an integral 3D image. The equipment can be made thin because thin LCD panels and MICOS are used.
First, the entire proposed constitution is described in section 2. Next, the elemental technologies to connect multiple images and construct a high quality integral 3D image are explained in sections 3 and 4. Next, the prototype integral 3D display using four LCD panels and the reconstructed integral 3D image is detailed in section 5. Finally, the conclusion and the future work are given in section 6.
2. Integral 3D display using multiple LCD panels
An integral 3D image was reconstructed by placing a lens array in front of a single LCD panel as shown in Fig. 2(a). In this study, multiple LCD panels are arranged side by side to increase the number of elemental images as shown in Fig. 2(b). However, since the LCD panel has a bezel at which the image is not displayed, if LCD panels are simply arranged side by side, the reconstructed image will contain gaps.
Fig. 2 Integral 3D display using LCD panel. (a) Display using single LCD panel (conventional method). (b) Display using multiple LCD panels (proposed method).
Therefore, a connected 2D image that has the increased pixels is produced by placing the MICOS in front of each LCD panel, magnifying the image of the LCD panel, and connecting the magnified images without the gaps. The diffuser plate is placed at the imaging position of the magnified image. Then, the integral 3D image is reconstructed by placing a lens array in front of the diffuser plate.
Applying this method enables the total number of pixels of the 3D image to be increased by increasing the number of LCD panels and using a MICOS corresponding to the number. In addition, thinning the optical system makes the whole device thinner than the device using the projector, so it will be easy to apply to future home televisions.
3. Multi-image combining optical system (MICOS)
3.1 Constitution of MICOS
A MICOS is composed of an erect unmagnified optical system and a concave lens. As shown in Fig. 3(a), the erect unmagnified optical system is composed of two sets of biconvex lens arrays. Then, as shown in Fig. 3(b), combining the optical system and a concave Fresnel lens makes it possible to expand the light ray and create a magnified image. As shown in Fig. 3(c), the MICOS is placed in front of each display, and connecting each magnified display image without a gap makes it possible to generate an image that has more pixels than a single display. Since the light ray through between each LCD panel becomes stray light and reduces the quality of the image, the unnecessary light is prevented from entering the optical system by placing a shielding mask between each LCD panel.
Fig. 3 Constitution of MICOS. (a) Erect unmagnified optical system. (b) MICOS. (c) Connecting multiple display images by using MICOS.
Moreover, since the MICOS is composed of multiple lens arrays, when the diffused light ray enters it, part of the light ray enters not the corresponding pair lens but the neighboring lens. The unnecessary light ray becomes stray light and produces multiple images at the diffuser plate. Thus, to prevent this, a parallel light is used as the backlight of the LCD panel. In the prototype, we produced the parallel light by placing a convex lens at the position of the focal length away from the light source.
The same focal length and lens pitch are used for all lenses of the lens arrays used for the MICOS. If the length between the display plane and the first lens array plane is a in the erect unmagnified optical system as shown in Fig. 3(a), the length between the last lens array plane and the erect unmagnified image similarly becomes a. The same thickness b is used for two lens arrays. Since the light emitted from the LCD panels is condensed in the middle of the two lens arrays once, the lens formula is applied as
where f+, n and R are the focal length, the index of refraction and the radius of curvature of the single lens array, respectively. In addition, if a concave lens with a focal length f- is placed so as to be in close contact with the rear of the lens array, the composite focal length f' of the single lens array and the concave lens is expressed by the following equation,where d is the distance between the lens array and the concave lens (d ≈0). The distance c between the magnified image and the concave lens is expressed as the following lens formula,The magnification ratio m is expressed as3.2 Improvement of resolution characteristics
In the prototype integral 3D display, to suppress degradation of the resolution, the display was arranged as closely as possible to reduce the magnification ratio. In the MICOS using the lens array of minute lens pitch and short focal length, the distance from the display surface to the magnified image can be made smaller than that in the MICOS not using the lens array. Therefore, overall thickness can be reduced, making it suitable for future home 3D televisions. Additionally, a discontinuity is likely to be perceived around the image connecting portion due to luminance unevenness and a minute deviation of correction accuracy. If a lens array that has a delta arrangement is used for reconstructing a 3D image, the lens crosses the connecting portion and the discontinuity is emphasized more. Therefore, a lens array that has a square arrangement is used for reconstructing a 3D image, and the boundary between the lens and the image are made to coincide to make the discontinuity of the connection portion inconspicuous.
First, to investigate this principle, we previously prototyped an integral 3D display that connects the images of four HD LCD panels using the MICOS (first prototype MICOS) [16]. However, the resolution characteristics were deteriorated by the MICOS used in the equipment because it was designed only in consideration of the magnification ratio.
Therefore, we have developed a lens array of the MICOS that improves the resolution characteristics (improved prototype MICOS). Table 1 lists the specifications of the first and improved prototype MICOSs. The erect unmagnified optical system constituting MICOS was designed by optical simulation software to have a modulation transfer function (MTF) value of 10% or more at the same spatial frequency as the pixel pitch of the LCD panel used. In particular, increasing the curvature radius of the lens reduced the effect of the spherical aberration. Since the focal length of the lens was increased to reduce the spherical aberration, the thickness of the optical system has increased. It is possible to design a thin MICOS with high resolution characteristics by using an aspheric lens.
Table 1. Specifications of MICOS of first and improved prototype MICOSs
Figure 4 shows the MTF of the erect unmagnified optical system of the first and improved prototype MICOSs, which is calculated using the optical design simulation software. The simulation was performed assuming light passing through a pair of lenses by setting the diameter of the entrance pupil to the pitch of the lens array. The MTF values are about 8.8% and 13.7% in the first and improved prototype MICOSs for a pixel pitch of 55.5 μm (spatial frequency conversion: 9.01 cycles/mm) for the LCD panel used for the prototype integral 3D display. In fact, we investigated the resolution characteristics at the imaging position by using the resolution chart and the MICOS. Figure 5 compares the resolution characteristics the first and improved prototype MICOSs. In the Fig. 5, a value of 1000 corresponds to a width of 153 μm per line. We evaluated whether we can discriminate on the resolution chart or not and found the resolution characteristics were improved approximately 1.75 times both horizontally and vertically.
Fig. 4 Simulations of MTF value of erect unmagnified optical system. (a) First prototype MICOS. (b) Improved prototype MICOS.
Fig. 5 Comparison of resolution characteristics between first and improved prototype MICOSs. Value of 1000 corresponds to width of 153 μm per line. (a) Horizontal direction. (b) Vertical direction.
3.3 Uniformity of luminance
In the first prototype MICOS, luminance unevenness corresponding to the viewpoint is caused by magnifying the image. When the length between the MICOS and the diffuser plate (working distance) is short, the difference in the angle of the light rays that enter the diffuser plate between the different LCD panels is increased. Therefore, luminance unevenness appears large depending on the viewing position. Figure 6 shows the state of the luminance unevenness when imaging on the diffuser plate using the white parallel light and the first prototype optical system and viewing from an oblique direction. Thus, we changed the MICOS and the diffuser plate to the optimum specifications and improved the quality of the integral 3D image. In the improved prototype MICOS, to reduce luminance unevenness, we made the working distance longer and used a diffuser plate that has a wide diffusion angle (FWHM angle: 100 degrees). Extending the working distance makes it possible to suppress the spread angle of the magnified image. Furthermore, increasing the diffusion angle of the diffuser plate can suppress the luminance unevenness. Figure 7 shows the connected image of the two magnified images (white light) on the diffuser plate using the first and improved prototype optical systems when viewed from the center and the right. Moreover, the luminance graph on the red line in the Fig. 7 is shown below each image. The horizontal axis and the vertical axis show the position of the horizontal direction and the luminance value (256 gradations), respectively. The luminance unevenness index value is defined as
where L is a luminance unevenness index value and Lmax and Lmin are the maximum and minimum luminance values on the display. The luminance unevenness index value is calculated for each graph. The values for the first and improved prototype MICOSs are 0.81 and 0.46 for views from the center and 0.94 and 0.44 for views from the right. This reveals that the luminance unevenness is greatly improved for views from any direction.Fig. 7 State when viewing the connected image of two magnified images (white light) on diffuser plate using first and improved prototype MICOSs. Luminance graph on red line is shown below each image (horizontal axis: position of horizontal position, vertical axis: luminance value). (a) View from center. (b) View from the right.
4. Distortion correction
The distortion is generated in the image magnified by the MICOS because of the lens aberration and erroneous placement of the optical system. If the distorted elemental images are placed in front of a lens array, an exact integral 3D image is not reconstructed because the mutual relationship between the elemental lens and the elemental image breaks down [17]. Thus, we numerically correct this distortion by computer processing to combine two image transformations: a projective transformation and an affine transformation.
First, the diffuser plate is placed at the imaging place of the magnified image by the MICOS. The triangle mesh image is displayed on the LCD panel and imaged at the diffuser plate through the MICOS. Then, the triangle mesh sheet (reference sheet) of the correct shape is placed at the diffuser plate, and the distortion is corrected so that the magnified image and the reference sheet match. The correction work is carried out while visually checking to see whether the magnified image is changed dynamically by using the interactive program and the magnified image and the reference sheet match.
Here, a projective transformation and an affine transformation are described in detail. First, the four corners of the magnified image are corrected by a projective transformation using the four corners as the control points. Thus, the whole image size is corrected roughly. The equation expressing this projective transformation is given as
where x and y are the coordinates before the correction and x’ and y’ are the coordinates after the correction. A projective transformation matrix, H, is calculated by using the coordinates of the four corners.Next, the image is corrected by an affine transformation using the vertex of each triangle as the control points. The transformation is performed individually for all triangles; thus, the image is corrected more precisely than is possible with a projective transformation. In the case of the prototype display, the distortion was corrected by dividing the whole image into 20 × 20 sections. The equation expressing an affine transformation is given as
Affine transformation matrix A is calculated by using the coordinates of the vertex of each triangle. The equation for combining the two transformations is given asExamples of the correction using both transformations are shown in Figs. 8(a) and 8(b).Fig. 8 Example of correction using projective and affine transformations and applying them to elemental images. Red solid lines, black dotted lines, and green points indicate magnified image, reference sheet, and control points, respectively. (a) Projective transformation. (b) Affine transformation. (c) Application to elemental images.
Finally, the elemental images are transformed by using transformation matrixes as shown in Fig. 8(c). The integral 3D image is reconstructed by using the corrected elemental images. By using this method, all element image positions can be corrected with an accuracy of about 0.1 pixels or less.
5. Prototype integral 3D display and reconstruction of integral 3D image
We built the first prototype integral 3D display using four HD LCD panels of 4.8 inches diagonally and the first prototype MICOS and reconstructed an integral 3D image that has about 25,000 elemental images [16]. The distortion correction method described in section 4 was able to reconstruct 3D images without distortion. The changes in the appearance of the image from different viewpoints can be seen in Visualization 1 in Fig. 9. Because the connecting parts of multiple images overlap slightly, bright lines are seen. It is assumed that it can be made inconspicuous by making the luminance value uniform by pixel using image processing.
Fig. 9 Changes in the appearance of image from different viewpoints (see Visualization 1). (a) View from upper position. (b) View from left position. (c) View from right position. (d) View from lower position.
To further increase the number of pixels, we built the improved prototype integral 3D display using four 8K dual-green (8KDG) LCD panels of 9.6 inches diagonally. Figure 10 shows the appearance of the improved prototype. Furthermore, the improved prototype MICOS is used in which the resolution characteristics are improved and the luminance unevenness corresponding to the viewing position is suppressed as mentioned in sections 3.2 and 3.3. Table 2 lists the specifications of the improved prototype integral 3D display using four 8KDG LCD panels. The 8KDG panel has 7,680 × 4,320 pixels. The pixels are in a Bayer arrangement. The 2 × 2 pixels are composed as one set, and the green filter is arranged on one pair of diagonal pixels, and the red and blue filters on are arranged the remaining pairs of diagonal pixels. Regarding the reconstruction of the 3D image, the numbers of elemental lenses were 420 horizontally and 236 vertically, the number of the elemental images was 99,120, and the viewing angle was 28°. The reconstructed image was displayed at 60 frames per second.
Table 2. Specifications prototype integral 3D displays.
Figure 11 compares integral 3D images reconstructed by using the first and improved prototype integral 3D displays. The elemental images in Fig. 11 captured by using a camera that made the resolution equivalent to 16K image by using four 8K imaging elements [11] were converted to match the specifications of the integral 3D displays. Figures 11(a) and 11(c) show the images reconstructed by using the first and improved prototype integral 3D displays, respectively, and Figs. 11(b) and 11(d) show the enlarged images of the face portions of the respective images. Only the dotted parts of Figs. 11(a) and 11(c) could be viewed when a single LCD panel was used. Using four LCD panels made it possible to enlarge the displayable image area about 5.66 times since the magnification ratio of the MICOS is 1.19. As described in Sections 3.2, since the quality of the 3D image was improved by using the improved MICOS, even fine detail could be expressed. Furthermore, as described in Sections 3.3, by using the improved MICOS that has a diffuser plate with a large diffusion angle, the luminance unevenness was suppressed within the range of the viewing angle, and the luminance unevenness of the image connecting portion became less conspicuous. Figure 11 shows a comparison result when using panels of different resolutions. Even if the same panel is used, it is considered that the quality of reconstructed integral 3D image is improved since the resolution characteristics of the MICOS is improved as shown in Section 3.2.
Fig. 11 Comparison of reconstructed integral 3D image by using first and improved prototype integral 3D displays. Dotted part shows displayable area when using single LCD panel. (a) Image using first prototype. (b) Magnified image of Fig. 11(a). (c) Image using improved prototype. (d) Magnified image of Fig. 11(c).
To upgrade the first prototype MICOS to the improved prototype MICOS, its lens array was made thick. Therefore, a stray light was made by entering the neighboring lens, and multiple images were more likely to occur. To prevent the generation of multiple images, it is necessary to use a light source with high parallelism or use a gradient index (GRIN) lens or a shielding mask to prevent light from entering the neighboring lens. Furthermore, the contrast of the reconstructed 3D image was lowered by using a diffusers plate with a large diffusion angle to suppress luminance unevenness. In this regard, it is thought that by placing a convex lens instead of the diffuser plate and collimating the divergent light, luminance can be made uniform in accordance with the direction and scattering of light by the diffuser plate can be eliminated.
6. Conclusion
In this study, a method was presented for increasing the number of pixels of an integral 3D image by using multiple liquid crystal display (LCD) panels and a multi-image combining optical system (MICOS). We designed and prototyped an improved MICOS to improve the resolution characteristics and to suppress the luminance unevenness corresponding to the viewing position and verified that it actually improved on the first prototype MICOS. We built an improved prototype integral 3D display using the improved MICOS and four 8KDG LCD panels and found that it could magnify the area in which the integral 3D image is displayed by approximately 5.66 times more than when a single LCD panel was used. It is possible to display a higher quality integral 3D image in accordance with a set number by increasing the number of a set of an LCD panel and using the improved MICOS. Furthermore, the constitution of an LCD panel and a thin optical system enables the device to be made thin, so it can be expected to be applied to future home televisions.
Since the proposed constitution of MICOS uses a diffuser plate, the light is scattered on the diffuser plate and the contrast of the reconstructed 3D image is lowered. Therefore, we will investigate an optical system that combines multiple images without using a diffuser plate. Furthermore, in the proposed constitution of MICOS, parallel light had to be used as the backlight of the LCD panels. Therefore, we will consider an optical system that can use diffused light in order to further thin the whole display.
Acknowledgments
The authors acknowledge the support of Ortus Technology Co., Ltd for providing the 8KDG LCD panels. In addition, the authors would like to thank K. Hisatomi, K. Ikeya and H. Hiura of NHK (Japan Broadcasting Corporation), Tokyo, Japan, for their technical support in creating the elemental images.
References and links
1. M. G. Lippmann, “Épreuves, réversibles donnant la sensation du relief,” J. Phys. 7(1), 821–825 (1908).
2. Y. Igarashi, H. Murata, and M. Ueda, “3-D display system using a computer generated integral photograph,” Jpn. J. Appl. Phys. 17(9), 1683–1684 (1978). [CrossRef]
3. N. Davies, M. McCormick, and L. Yang, “Three-dimensional imaging systems: a new development,” Appl. Opt. 27(21), 4520–4528 (1988). [CrossRef] [PubMed]
4. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997). [CrossRef] [PubMed]
5. H. Arimoto and B. Javidi, “Integral three-dimensional imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001). [CrossRef] [PubMed]
6. B. Javidi, I. Moon, and S. Yeom, “Three-dimensional identification of biological microorganism using integral imaging,” Opt. Express 14(25), 12096–12108 (2006). [CrossRef] [PubMed]
7. J. Arai, M. Okui, T. Yamashita, and F. Okano, “Integral three-dimensional television using a 2000-scanning-line video system,” Appl. Opt. 45(8), 1704–1712 (2006). [CrossRef] [PubMed]
8. K. Suehiro, M. Yoshimura, Y. Haino, M. Sato, J. Arai, M. Kawakita, and F. Okano, “Integral 3D TV using ultrahigh-definition D-ILA device,” Proc. SPIE 6803, 680318 (2008). [CrossRef]
9. H. Hoshino, F. Okano, and I. Yuyama, “Analysis of resolution limitation of integral photography,” J. Opt. Soc. Am. A 15(8), 2059–2065 (1998). [CrossRef]
10. T. Yamashita, M. Kanazawa, K. Oyamada, K. Hamasaki, Y. Shishikui, K. Shogen, K. Arai, M. Sugawara, and K. Mitani, “Progress report on the development of Super-Hi Vision,” SMPTE Motion Imaging J. 119(6), 77–84 (2010). [CrossRef]
11. J. Arai, M. Kawakita, T. Yamashita, H. Sasaki, M. Miura, H. Hiura, M. Okui, and F. Okano, “Integral three-dimensional television with video system using pixel-offset method,” Opt. Express 21(3), 3474–3485 (2013). [CrossRef] [PubMed]
12. H. Liao, M. Iwahara, N. Hata, and T. Dohi, “High-quality integral videography using a multiprojector,” Opt. Express 12(6), 1067–1076 (2004). [CrossRef] [PubMed]
13. Y. Takaki and N. Nago, “Multi-projection of lenticular displays to construct a 256-view super multi-view display,” Opt. Express 18(9), 8824–8835 (2010). [CrossRef] [PubMed]
14. M. Kawakita, S. Iwasawa, M. Sakai, Y. Haino, M. Sato, and N. Inoue, “3D image quality of 200-inch glasses-free 3D display system,” Proc. SPIE 8288, 82880B (2012). [CrossRef]
15. J.-Y. Jang, D. Shin, B.-G. Lee, and E.-S. Kim, “Multi-projection integral imaging by use of a convex mirror array,” Opt. Lett. 39(10), 2853–2856 (2014). [CrossRef] [PubMed]
16. N. Okaichi, M. Miura, J. Arai, and T. Mishina, “Integral 3D display using multiple LCDs,” Proc. SPIE 9391, 939114 (2015). [CrossRef]
17. M. Kawakita, H. Sasaki, J. Arai, M. Okui, F. Okano, Y. Haino, M. Yoshimura, and M. Sato, “Projection-type integral 3-D display with distortion compensation,” J. Soc. Inf. Disp. 18(9), 668–677 (2010). [CrossRef]
