Open Access
Add to BibSonomy
Abstract
In this paper, we suggest the improved integration of a holographic display and a Maxwellian-view display using time-division multiplexing and describe an image rendering process for the proposed system. In general, the holographic displays have a resolution limit when used to represent a virtual 3D scene. In the proposed system, the holographic display processed relatively few layers of the virtual 3D scene, while the remaining objects were processed with a Maxwellian-view display to which was applied a Gaussian smoothing filter. Hence, we obtained the retaining holographic image quality, expanding the field of view, and reducing the computation time of the proposed system. The holographic display of the proposed system had an image size of 28 mm × 28 mm with a field of view of 1.02° and a 10.8 mm eye box. The Maxwellian-view display had an image size of 230 mm × 230 mm with a field of view of 22.6 ° and a 0.9 mm eye box diameter. Each display was integrated in time-division multiplexing of 40 Hz, and the proposed system was successfully verified.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Currently, augmented reality and virtual reality are highly impressive technologies for next generation displays. Augmented reality allows interaction with real-life items through a screen such as that on a smart-phone or computer. Virtual realty provides a perfect 3D experience, but it is isolated from real life. Thus, recently, a mixed reality display that combines augmented reality and virtual reality has been studied. The mixed reality display facilitates interaction with virtual 3D objects in real life. Although there are numerous related products, such as Magicleap (Google), Hololens (Microsoft), and Smart Glass (Sony) capable of can interacting with virtual 3D objects in real life, most of them do not overcome the vergence-accommodation conflict (VAC) issue. Decoupling of accommodation and vergence corresponding to monocular vision and binocular vision, respectively, which is called vergence-accommodation conflict induces a user uncomfortable physiological functions such as fatigue, double vision, and eyestrain. Thus, they are difficult to use for a long time because VAC causes problems [1, 2]. Researchers have suggested ideas to solve the VAC issue, such as the use of multi-plane display [2], a light field display [3], a focal surface display [4], a holographic display [5], and a Maxwellian-view display [6]. Although there are pros and cons with regard to each technology, we focused on the holographic display and Maxwellian-view display types in this paper.
The holographic display creates a cloud of voxels in the air by controlling the amplitude and phase of the light. Theoretically, holographic displays can create perfect virtual 3D objects. However, practical holographic displays cannot create perfect 3D objects owing to the finite spatial bandwidth of the spatial light modulator (SLM) used withe these displays [7, 8]. In addition, most hologram calculation methods are based on iterative algorithms, which require significant computing power [9,10].
The Maxwellian-view display is also a well-known technology capable of solving the VAC issue [6, 11]. When we put a pinhole in front of a pupil, we can see a relatively clear image, as the numerical aperture of the crystalline lens becomes larger. For the same reason, when a plane wave is focused on the pupil, each ray of the plane wave has an extremely large numerical aperture and the depth of focus (DOF) is extended [6,12]. Consequently, the observer sees clear images regardless of the eye focus.
When we see an object in real life, we observe sharp images of the object while the area around the center is blurry as shown in Fig. 1. The acuity of a normal person is highest in the fovea region(±5°) and it drops rapidly as the eccentricity angle increases. Thus, content containing relatively more information is concentrated in the fovea region [13–15]. This forms the main motivation behind the proposed system. In this paper, a sharp image which corresponds to the center of the fovea region is rendered in the form of a hologram, and a blurry image which corresponds to the vicinity around the fovea region is rendered in a Maxwellian-view. Thus, by processing only a small region of a virtual scene, holographic objects retain their resolutions and this method reduces the computation load.
In fact, two cases should be considered, where the object is located inside of the hyperfocal distance and outside of the hyperfocal distance. If the object is inside the hyperfocal distance, the depth of field of the eye and foveated image processing should be reflected in a point spread function. On the other hand, if the object is outside the hyperfocal distance, only foveated image processing is considered as the point spread function. In this paper, although the object is located within the hyperfocal distance, we only consider the depth of focus of the eye as the size of the represented image is 50 mm × 50 mm and is inside the fovea region with the proposed system and considering the fovea region in the paper.
We believe that the technology proposed here can provide a key with which to overcome obstacles related to current forms of holographic near-eye display technology. In a previous study, we verified two versions of optical systems using focus-tunable lens to integrate the holographic display and Maxwellian-view displays into a single system [12, 16]. In this paper, present in more detail an improved version of the holographic and Maxwellian-view display and the image rendering process in the proposed system.
2. Structure and theoretical background of the proposed system
2.1. Structure and specifications of the proposed system
The proposed display system is illustrated in Fig. 2. This system has three parts.
First, a light-emitting diode (LED, 645nm) with a 100 μm pinhole was used as a light source. Lasers have high temporal and spatial coherence and are therefore widely used in holographic displays. However, speckle, safety issues, and the bulky sizes of lasers are stumbling blocks hindering their use with holographic near-eye displays. Thus, a LED with a pinhole was used as a light source for the holographic display instead of a laser. LEDs have very poor spatial and temporal coherence, but the spatial coherence can be increased by using a pinhole [17]. We captured the holographic image by varying the pinhole size. Although spatial coherency was not measured in this paper, we confirmed that a 100 μ pinhole produced a sharper image than other cases as shown in Fig. 3.
Fig. 3 Captured results of a holographic image (a),(d) without a pinhole, (b),(e) with a 100 μ pinhole, and (c),(f) with a 200 μ pinhole
Second, a transmissive liquid crystal panel (Holoeye, LC-2012) with a pixel pitch of 36 μm and a number of pixels of 1024×768 was used as the SLM. Linear polarizers were also used to set the phase and amplitude mode of the SLM, and a liquid crystal half-wave retarder which has a rise/falling time of 10.2 ms / 310 us and voltage/frequency of ±25 Vac / 2 kHz was placed between the linear polarizers. The SLM is consist of an LC panel between two polarizers in Fig. 2 which are the polarizer and the analyzer. A rotated angle of polarizer and analyzer determines the amplitude mostly mode and the phase mostly mode. In this paper, the Maxwellian-view display is operated by the amplitude-mostly mode which can modulate transmission, whereas the holographic display is operated by the phase-mostly mode which can modulate the phase. when the angles of the polarizer and analyzer are 90 and 180 degree, the LC panel is operated in the amplitude mostly mode. Thus, if the input gray level is changed from 0 to 255, the transmission is modulated from 0 to 1 and a small amount of phase is shifted. On the other hand, when polarizer and analyzer are 18 and 66 degree, the LC panel is operated in the phase mostly mode. If the input gray level is changed from 0 to 255, the phase is shifted from 0 to 2π and a small amount of transmission is varied. Thus, a half-wave retarder switched the phase and amplitude mode of the SLM which corresponds to holographic display and Maxwellian-view display as shown in Fig. 4.
Third, a polymer membrane lens (Optotune, EL-10-30-TC) was used as focus-tunable lens. Although the polymer membrane lens undergoes deformation due to gravity, it is the largest and the fastest focus-tunable lens we have found thus far. It has an operating frequency of 1kHz, a diameter of 10 mm, and a diopter range of 8.3 D to 20 D. In order to adjust the diopter tunable range from 0 D to 20 D, a concave lens of −5 D was attached to the focus-tunable lens.
The proposed system was implemented in the optical cage system shown in Fig. 2(b), and all parts were controlled simultaneously by computer.
2.2. Holographic image resolution
Generally, holographic image reconstruction can be described in terms of Fresnel propagation [18,19],
where λ and d denote the wavelength and distance from the object plane to hologram plane, respectively. h(x,y) is the complex amplitude of the virtual 3D object and (Δu,Δv) is the pixel pitch on the object plane.Nx, Ny and Δx represent number of pixels of axis of x and y on the hologram plane and the pixel pitch on the hologram plane. According to Eq. (2), the pixel pitch on the object plane Δu and Δv depends on the reconstruction distance d. In addition, Δu and Δv are the half diameter of the airy disk or speckle diameter on the object plane, as the hologram is identical to the optical system of the aperture NxΔx × NyΔy. Thus, the number of pixels of the holographic image depend on the SLM in discrete Fresnel diffraction. However, unfortunately, a theoretical analysis is difficult when used in a direct comparison with a practical result, as, Eq. (1) assumed that the reference wave and the optical system were perfect. Hence, the modulation transfer function was measured as shown in Fig. 4. Vertical periodic line images were prepared as a test pattern, and holographic images of the test patterns were formed at 0.6028 m which is the critical sampled distance to avoid the natural scaling problem. Unfortunately, in this paper only 300×300 pixels of the SLM were allocated to the hologram owing to the focus-tunable lens. The reduced number of effective pixels leads to a reduction of the resolution of the holographic image. This corresponds to the increase in the speckle size due to the reduced aperture during the optical hologram reconstruction process. In addition, holographic image resolution is described with the optical capacity [20],
where Q is the number of quantization levels, Nx and Ny are correspondingly the numbers of pixels in the hologram and the object plane, and S/N is the signal-to-noise ratio. The pixel pitch was not directly reflected in Eq. (3). However, by decreasing the pixel pitch with a certain number of pixels leads reducing numerical aperture of the SLM. As a result, image resolution will be degraded by increasing airy disk size which indirectly corresponds to S/N value. In other words, the resolution of the holographic image always has a limitation. Therefore, when representing a highly complex image with a hologram, degradation of the holographic image resolution is inevitable Fig. 5.2.3. Holographic image rendering process
There are typically four methods which can be uesd to calculate the complex amplitude of the holographic image. These are a point cloud, a depth layer, ray reconstruction and a triangular patch. Although each method has advantages and disadvantages, the depth layer method is easier to implement and is faster than the other methods. Therefore, it was used here. The calculation of the complex amplitude of the depth-layer-based hologram was done using the equation below by [12],
Here, is the transfer function, k and z are repectively the wave number and propagation length and exp(iϕgs(u,v)) is the output phase after the Gerchberg-Saxton iteration algorithm (GS algorithm). In general, the Fourier spectrum in the image contains numerous low-frequency components, hence high-frequency component should be added to the Fourier spectrum to reconstruct the holographic image properly by adding a random phase or passing through an iterative phase retrieval algorithm [21]. In this paper, an iterative phase retrieval algorithm was used because it creates a more accurate phase map compared to that when adding a random phase [21,22].
Each layer of the virtual 3D object was passed through the GS algorithm until the peak signal-to-noise ratio (PSNR) reached the point of saturation. The saturated PSNR and the number of iterations depend on the image complexity [23]. As described in chapter 2.1, the saturated PSNR is proportional to the number of pixels in the SLM. Although the rendered holographic images appear to be relatively clear in Figs. 6(a) and (b), the PSNRs were approximately 15 dB and 18 dB, respectively, as shown in Fig. 6(c). Because only 300 × 300 pixels were used in this paper, a high PSNR cannot be obtained. It took about 31 seconds to calculate one layer. Although our proposed method reduces the calculation time by processing a small area of the image, it is not enough for representing the holographic image with real-time, hence it needs another idea to reducing calculation time. The specifications of the computer used in the calculation are as follows: an Intel i5-4670 processor, 16 GB of ram, and a GTX 560 2 GB graphics card.
2.4. Maxwellian-view image rendering process
As noted in the introduction, when the eye focuses on a foreground object, background objects become blurry as shown in Fig. 1. The optical system in this case is illustrated in Fig. 7.
The blur effect is proportional to the blur circle diameter (w) which can be determined with simple optical theory as follows [14],
where l and f are the aperture and the focal length of the crystalline lens, respectively. m is the distance between the focal plane the crystalline lens and z is the distance between the defocused object and the crystalline lens. Thus, if the distance from the defocused obejcts to the crystalline lens is longer, the blur effect becomes more serious. In terms of image processing, the blur effect is the mean square of the point spread function. Thus, the defocused object O′(x,y) is the convolution between the original sharp image O(x,y) and the point spread function h(x,y).The point spread function can be modeled as a 2D Gaussian function.
Here, σ depends on blur circle diameter w, σ=kw, where the k is a constant value of the eye which is define by user. Thus, we can adjust blur effect of background object which corresponds to the vicinity of the eye’s focus as adjusted by changing w. In order to determine k, the optical system identical to that in Fig. 7 was modeled by a computer graphics tool (Blender). The alphabets were then placed at 1.5 m, 1 m, and 0.5 m, after which the camera was focused at 0.5 m. When k was optimized to 12870 in our camera condition (Canon EOD 60D, f/5.6 with 44mm focal length), the result from the customized Gaussian smoothing and computer graphics tool was nearly identical to that shown in Figs. 8(a) and (b), the peak signal-to-noise ratio was 32 dB.
Fig. 8 (a) Image modeled by computer graphics image, (b) image with the customized Gaussian smoothing filter applied, and (c) result of the image in (b) applied to a Maxwellian-view display.
3. Time-division multiplexing results from the proposed system
For time-division multiplexing, all devices (liquid lens, light source, half-wave retarder, SLM) are connected to a computer and they are operated sequentially with an in-program delay time under 0.00003s. Therefore, although the synchronization accuracy was not measured, the in-program delay is already small enough that it is negligible as shown in Visualization 1. The focus-tunable lens requires a stabilization time to saturate lens surface movements. With the polymer membrane lens (Optotunes, EL-10-30-TC) used here, the stabilization time is approximately 25 ms (= 40Hz) according to device manual. Although the liquid lens can operate with 60 Hz, we followed the manual to deplete aberration of the liquid lens as possible in this study. Therefore, we allocated 40 Hz to the holographic display and the Maxwellian-view display for frame by frame. Although 40 Hz is not high enough to see clear video without flickering, the feasibility of the proposed idea was successfully verified, as shown in Fig. 13 and Visualization 1.
A box image and an alphabet image were prepared as shown correspondigly in Figs. 10 (a) and (d). The box ’B’ and alphabet ’B’ was calculated according to Eq. (4). Then, the Gaussian smoothing filter applied to the remaining objects corresponding to each distance, as described in chapter 2.4. As shown in Fig. 9, each image was well represented. However, the higher order diffraction of the holographic image was not removed. In general, 4F optics is required to eliminate higher order diffraction, but it is not suitable for near-eye displays because it makes the system larger. Thus, spatial filters were not used, and we expect that higher order diffraction will naturally deviate from the viewing window of the pupil when the pixel pitch of the SLM becomes smaller [24].
Fig. 10 (a), (c) Rendered Maxwellian-view image and (b), (c) holographic image applied to proposed system.
The maximum image size of the Maxwellian-view image was 230 mm × 230 mm, as shown in Fig. 10 (a). The size of the holographic image depends on the distance according to Eq. (2) and it was 28 mm × 28 mm at the critical sampled distance as shown in Fig. 11 (b).
The field of view (θh) and the eye box’s diameter (wh) of the holographic display can be described as follows [12],
where dx and λ are correspondingly the pixel pitch and wavelength, and Nx is the number of pixels which are allocated. The field of view of the holographic display, which is equivalent to the diffraction angle, was 1.02°, and the eye box width was 10.8 mm, sufficient to cover the eye pupil. Therefore, considering the diameter of the pupil (approximately 5 mm), the eye box’s diameter should not be less than 4mm. In addition, the holographic image reconstructed at 0.5m, 1.5m, and 5.5m as shown in Fig. 12. As a result, the images represented well without noticeable quality degradation. Thus, the holographic display can reconstruct the image for all range depth of human-recognizable.Fig. 13 (a),(b) Intended scene image, (b),(e) result of hologram only rendering, and (c),(f) result of the proposed system when digitally merged using an image processing tool (see Visualization 1).
The field of view (θm) and the eye box’s diameter (wm) of the Maxwellian-view display can be described as follows [12,25,26],
where NA and f denote the numerical aperture and focal length of the focus-tunable lens. Therefore, the field of view of the Maxwellian-view display is 22.6°. The eye box’s diameter of the Maxwellian-view display is proportional to the focal length and inverse proportional to pixel pitch and it was 0.9 mm in our case which is too small to cover the eye pupil. According to a relation between the eye box’s diameter and the depth of field, the accommodation effect is eliminated from 30 cm to infinity at the eye box’s diameter of 0.9 mm.The entire scene in Figs. 13(a) and (d) were also rendered using a hologram. As shown in Figs. 13(b) and (e), the entire scenes were not only too small, but the background and objects as well were not properly represented, whereas the proposed system represented our intended scene well, even if it was a digitally overlaid result using an image processing tool. Images represent frame by frame in time-division multiplexing method. Hence, we attached the Visualization 1 in the manuscript, because the camera cannot capture both images in a time as shown in Fig(c) and (f).
4. Conclusion
The holographic displays are a promising type of technology for next-generation 3D displays and the Maxwellian-view display is also receiving attention as a near-eye display. However, As discussed earlier, each display has problems. Thus, for the first time, a holographic display and a Maxwellian-view display were integrated using time-division multiplexing. Consequently, expanded field of view and reasonable resolution of a virtual 3D image was achieved without the vergence-accommodation conflict. We revealed thoretically and with the transfer modulation function of the proposed system that the holographic image resolution is limited. The rendering process of each display was also discussed. Finally, the results showed a Maxwellian-view image 230 mm × 230 mm in size and a holographic image 28 mm × 28 mm in size. Each display was time-division multiplexed at a frequency of 40 Hz, and the proposed system clearly showed better image quality than the image rendered with only a hologram. Although the operating frequency of the proposed system and holographic image quality is still insufficient to cover the eye box and the field of view. Expanding the field of view(up to 4.85°), improving the holographic image quality, and depleting of the higher order diffraction are possible by using the SLM of 3.74 um pixel pitch with 4K resolution (Holoeye, GAEA-2LCOS SLM) [24,27]. In addition, even though several issues such as the eye box, form factor, and computation time, etc still remain to be solved for the perfect holographic near-eye display, we believe that the proposed system can be the key to the realization of an advanced holographic near-eye display.
Funding
Institute for Information & Communications Technology Promotion (IITP) funded by the Korean Government (MSIT) (2017001803, Development of Fundamental Technology of Core Components for Augmented and Virtual Reality Devices) and KAIST (Venture Research Program for Master’s and PhD Students in the College of Engineering).
References
1. B. Zou, Y. Liu, M. Guo, and Y. Wang, “EEG-Based Assessment of Stereoscopic 3D Visual Fatigue Caused by Vergence-Accommodation Conflict,” IEEE/OSA J. Disp. Technol. 11, 1076–1083 (2015). [CrossRef]
2. S. Ravikumar, K. Akeley, and M. S. Banks, “Creating effective focus cues in multi-plane 3D displays,” Opt. Express 19, 20940 (2011). [CrossRef] [PubMed]
3. D. Lanman and D. Luebke, “Near-eye light field displays,” ACM Transactions on Graph. 32, 1–10 (2013). [CrossRef]
4. N. Matsuda, A. Fix, and D. Lanman, “Focal surface displays,” ACM Transactions on Graph. 36, 1–14 (2017). [CrossRef]
5. S. Tay, P. A. Blanche, R. Voorakaranam, A. V. Tunç, W. Lin, S. Rokutanda, T. Gu, D. Flores, P. Wang, G. Li, P. St Hilaire, J. Thomas, R. A. Norwood, M. Yamamoto, and N. Peyghambarian, “An updatable holographic three-dimensional display,” Nature 451, 694–698 (2008). [CrossRef] [PubMed]
6. R. Konrad, N. Padmanaban, K. Molner, E. A. Cooper, and G. Wetzstein, “Accommodation-invariant computational near-eye displays,” ACM Transactions on Graph. 36, 1–12 (2017).
7. T. Kozacki, “On resolution and viewing of holographic image generated by 3D holographic display,” Opt. express 18, 27118–29 (2010). [CrossRef]
8. W. Zaperty and T. Kozacki, “Numerical model of diffraction effects of pixelated phase-only spatial light modulators,” Speckle 2018: VII Int. Conf. on Speckle Metrol. p. 106 (2018).
9. B. R. W. Gerchberg and W. O. Saxton, “A Practical Algorithm for the Determination of Phase from Image and Diffraction Plane Pictures,” Optik 35, 237–246 (1972).
10. M. Makowski, M. Sypek, I. Ducin, and A. Fajst, “Experimental evaluation of a full-color compact lensless holographic display,” Opt. express 17, 270–275 (2009). [CrossRef]
11. G. Westheimer, “Retinal light distribution for circular apertures in Maxwellian view,” J Opt Soc Am 49, 41–44 (1959). [CrossRef] [PubMed]
12. J. S. Lee, Y. K. Kim, and Y. H. Won, “See-through display combined with holographic display and Maxwellian display using switchable holographic optical element based on liquid lens,” Opt. Express 26, 19341–19355 (2018). [CrossRef] [PubMed]
13. G. Tan, Y.-H. Lee, T. Zhan, J. Yang, S. Liu, D. Zhao, and S.-T. Wu, “Foveated imaging for near-eye displays,” Opt. Express 26, 25076 (2018). [CrossRef] [PubMed]
14. H. Sun, Z. Zhao, X. Jin, L. Niu, and L. Zhang, “Depth from defocus and blur for single image,” in IEEE VCIP 2013 – 2013 IEEE International Conference on Visual Communications and Image Processing, (2013).
15. F. W. Campbell, “The Depth of Field of the Human Eye,” Opt. Acta: Int. J. Opt. 4, 157–164 (1957). [CrossRef]
16. J. S. Lee, Y. K. Kim, and Y. H. Won, “Time multiplexing technique of holographic view and Maxwellian view using a liquid lens in the optical see-through head mounted display,” Opt. Express 26, 2149 (2018). [CrossRef] [PubMed]
17. Y. Deng and D. Chu, “Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays,” Sci. Reports 7, 1–12 (2017).
18. C. J. Bouwkamp, “Diffraction theory,” Reports on Prog. Phys. 17, 35–100 (1954). [CrossRef]
19. C. J. R. Sheppard and M. Hrynevych, “Diffraction by a circular aperture: a generalization of Fresenel diffraction theory,” J Opt Soc Am 9, 274–281 (1992). [CrossRef]
20. C. Slinger, C. Cameron, and M. Stanley, “Computer-generated holography as a generic display technology,” Computer 38, 46–53 (2005). [CrossRef]
21. T. Shimobaba and T. Ito, “Random phase-free computer-generated hologram,” Opt. Express 23, 9549–9554 (2015). [CrossRef] [PubMed]
22. C.-Y. Chen, W.-C. Li, H.-T. Chang, C.-H. Chuang, and Chang Tsung-Jan, “3-D modified Gerchberg-Saxton algorithm developed for panoramic computer-generated phase-only holographic display,” J. Opt. Soc. Am. B 34, B42–B48 (2017). [CrossRef]
23. P. Sun, S. Chang, S. Liu, X. Tao, C. Wang, and Z. Zheng, “Holographic near-eye display system based on double-convergence light Gerchberg-Saxton algorithm,” Opt. Express 26, 30368–30378 (2018).
24. M. Makowski, I. Ducin, K. Kakarenko, J. Suszek, and A. Kowalczyk, “Performance of the 4k phase-only spatial light modulator in image projection by computer-generated holography,” Photonics Lett. Pol. 8, 26–28 (2016).
25. H. Takahashi and S. Hirooka, “Stereoscopic see-through retinal projection head-mounted display,” (2008), p. 68031N.
26. Y. Takaki and N. Fujimoto, “Flexible retinal image formation by holographic Maxwellian-view display,” Opt. Express 26, 22985 (2018). [CrossRef] [PubMed]
27. S. Jolly, N. Savidis, B. Datta, V. M. Bove, and D. Smalley, “Progress in off-plane computer-generated waveguide holography for near-to-eye 3D display,” in Proc. SPIE 9771, Practical Holography XXX: Materials and Applications, (2016), March 2016, p. 97710L.
