Open Access
Add to BibSonomy
Abstract
The optical design process of conventional stereoscope-type head mounted displays for virtual and augmented reality applications typically neglects the inherent aberrations of the eye optics or refractive errors of a viewer, which misses the opportunity of producing personal devices for optimal visual experiences. Although a few research efforts have been made to simulate the retinal image formation process for some of the emerging 3D display systems such as light field displays that require modeling the eye optics to complete the image formation process, the existing works generally are specific for one type of display methods, unable to provide a generalized framework for different display methods for the benefit of comparison, and often require the use of at least two different software platforms for implementation which is challenging in handling massive data and implementing compensation of wavefront aberrations induced by display engine or eye refractive errors. To overcome those limits, we present a generalized analytical model for accurately simulating the visual responses such as retinal PSF, MTF, and image formation of different types of 2D and 3D display systems. This analytical model can accurately simulate the retinal responses when viewing a given display system, accounting for the residual eye aberrations of schematic eye models that match with the statistical clinical measurements, eye accommodative change as required, the effects of different eye refractive errors specific to viewers, and the effects of various wavefront aberrations inherited from a display engine. We further describe the numerical implementation of this analytical model for simulating the perceived retinal image with different types of HMD systems in a single computational platform. Finally, with a test setup, we numerically demonstrated the application of this analytical model in the simulation of the perceived retinal image, accommodative response and in the investigation of the eye refractive error impacts on the perceived retinal image based on the multifocal plane display, integral imaging based light field display, computational multilayer light field display, as well as the stereoscope and natural viewing for comparison.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
The optical design process of conventional head mounted displays (HMDs) typically concentrates on the optimization of the optical elements inserted between a display source and the eye to deliver a high-quality two-dimensional (2D) virtual image for a standard observer with 20/20 normal vision. The optimized optics does not attempt to compensate for either the inherent aberrations of the eye optics or refractive errors of a viewer, neglecting the fact that more than half of the adults all over the world have eye refractive errors such as myopia, hyperopia or astigmatism [1]. With the ever-growing interests of making HMDs as a personal device, it becomes necessary to properly account for the impacts of the residual aberrations and refractive errors of the eye optics on the display performance during the optical design process and to implement further optimization to the display hardware and rendering algorithms for better display quality and user experiences. Moreover, several of the emerging 3D display approaches [2–8], such as multifocal plane (MFP) displays and light field displays, go beyond simply rendering 2D images per eye and require accurate modeling of the visual optics to complete the image formation process in which multiple samples of a 3D scene either at different depths or in different perspective directions enter the eye pupil and are integrally summed up to render retinal image effects. In such displays, the perceived retinal image quality is no longer directly determined by the individual pixels on the display panel and the visual optics itself becomes an integral part of the overall optical system optimization as it plays critical roles to the perceived quality of the display. Therefore, the conventional optical design process without integrating the visual optics becomes inadequate for the need of optimizing these innovative 3D display systems, and integrating an eye model with such display systems becomes a necessity in order to simulate the perceived retinal image and the accommodative response for the purpose of evaluating and optimizing their performance.
Several previous studies have investigated the perceived retinal image formation of several emerging 3D display systems [2,9–12] or attempted to simulate the perceived retinal image of a 3D scene through schematic human eye optics models [13]. However, most of these studies required the use of a commercial optical design software, such as CODEV or Zemax [11,12], to model the optical properties of a display system and eye optics to simulate a set of perceived retinal point spread functions (PSF) corresponding to different views and ocular parameters. These simulated PSFs for different views are then imported into another software tool, such as MATLAB, to compute the accumulated PSFs and modulation transfer functions (MTF) and produce simulated retinal images [2,9,11,12]. For instance, Huang and Hua [9] used CODEV to simulate the elemental view PSFs for an integral imaging based light field display and then used MATLAB to integrate the elemental views for the accumulated PSF as well as the perceived retinal image. Qin and Huang implemented the fundamental Rayleigh Sommerfeld diffraction integral in Zemax for simulating elemental view PSFs and then use MATLAB for image simulation [11,12]. Although modeling a schematic eye model along with the HMD optics in optical design software is effective and straightforward for some of the 3D display methods, the requirement for cross-platform computation is not only cumbersome and inflexible, but also can be prohibitively challenging and inadequate for some of the computationally intensive 3D display methods. For instance, in a computational multilayer light field (MLLF) display, the elemental views are selected by the pixels on the modulation layers for different directions, thus there are thousands of elemental views enter the human pupil simultaneously to drive the eye to accommodate at a rendered depth, which makes massive cross-platform computation challenging. Moreover, modeling the visual optics through optical design software lacks the flexibility of integrating wavefront aberrations inherited from a display engine or eye refractive errors from a viewer. In this sense, it is highly desirable and essential to develop an analytical model as a generalized framework for the computational convenience of numerically simulating the perceived retinal image of a wide range of 3D display methods based on a single computational platform such as MATLAB for better scalability and flexibility.
In this paper, we present a generalized analytical model that integrates a schematic eye model with the imaging properties of a display system to analytically simulate the visual responses such as retinal PSF, MTF, and image formation of 3D display systems. More specifically, for a range of 3D display systems, this analytical model can accurately simulate their retinal responses accounting for the residual eye aberrations of schematic eye models that match with the statistical clinical measurements, eye accommodative change as required, the effects of different eye refractive errors specific to viewers, and the effects of various wavefront aberrations inherited from a display engine. We further describe the numerical implementation of this analytical model for simulating the perceived retinal image with different types of HMD systems. The remaining paper is structured as follows. Section 2 is to describe the mathematical expression and validity of the analytical model, where we introduce the analytical expression for the monochromatic retinal PSF in responding to a point source and compare the simulated results of the analytical model with those simulated with Zemax. Section 3 describes the implementation of the analytical model to simulate the perceived retinal responses to different types of 3D display methods, while Section 4 demonstrates several application examples of the analytical model where we use a MFP display, InI-based LF display, computational MLLF display, and stereoscope as display examples and natural 3D viewing configuration as reference to characterize the accommodative responses of the these displays and compare the perceived retinal image appearance for normal vision viewer and eyes with refractive errors.
2. Analytical model for retinal image simulation of display systems
To build an analytical model as a generalized framework for simulating the perceived retinal image formation of various 2D and 3D display systems, four major requirements should be satisfied. First of all, this analytical model should employ a robust schematic eye model which can predict both the on-axis and off-axis eye aberrations matching with the statistical, clinical measurements. Secondly, the eye model shall allow convenient adjustment of its accommodation level as needed in order to investigate various visual responses to different types of display technologies. Thirdly, the analytical model shall have the ability to incorporate different types of eye refractive errors specific to users in order to investigate the visual responses under different eye conditions. Lastly, the analytical model should support the modeling of residual aberrations from a display engine to account for the impacts of the display to the perceived retinal image and should be flexible enough to be adapted to different display methods. Due to these requirements, choosing a robust and flexible schematic eye model is essential for establishing an analytical model for accurately simulating the perceived retinal image of any display systems. Among the current available schematic eye models as listed in the Table 1 in Ref. [14], only the Arizona eye models [15,16] and Navarro eye models [17,18] can adjust accommodation as required and the on-axis and off-axis aberrations as well as the chromatic aberration of these models match with the population study results. Therefore, these schematic eye models are eligible to be used to in the analytical model and they produce very comparable results.
Table 1. Display system parameters for test setup.
Figure 1 shows the perceived retinal image formation for an on-axis point, B, and an off-axis point C through an eye model, where the points can be part of a physical object or rendered by a display engine. The figure is not drawn in scale as the object distance is substantially larger than the dimensions of the eye model. The perceived retinal image formation process takes both the display and ocular factors into consideration, including the display mechanism, parameters, and residual aberrations of the display engine as well as the choice of a schematic eye model, accommodation state, eye pupil size, refractive errors. For convenience, we define a reference coordinate system OXYZ for the visual space and O'X'Y'Z’ for the retina space, with O and O’s being the origins of the two references, respectively. As shown in Fig. 1(a), the OXY plane is collocated with the entrance pupil plane of the schematic eye model, the O'X'Y’ plane is parallel to the OXY plane and centered at the fovea center, and z’ is the distance from the exit pupil of the eye to retina planes. The object distances, denoted as zB and zC, respectively, are measured from the eye entrance pupil plane to the object locations. As the separation between the entrance and exit pupils of the eye is much smaller than the typical object distance to a viewer. Figure 1(a) neglected the separation and labelled both planes as eye pupil plane for simplicity. The characteristics of the object wavefront incident upon the eye pupil plane, which are affected by both the wavefront property of the object points such as the residual aberrations of a display engine and by the refractive errors of a viewer’s eye, can be modeled by a wavefront map on the eye pupil plane as shown in Fig. 1(b). The retinal image of a point of interest can be formed by propagating the input wavefront map through the schematic eye model under a given eye accommodation state. In the illustration, the eye model accommodation distance, zA, was set to coincide with the object distance, zB. Correspondingly, the object point B is in focus and its retinal image B’ is sharp and object point C is out of focus and its retinal image C’ is blurred as illustrated by Fig. 1(c) and also shown by their retinal PSF simulation in Fig. 1(d).
Fig. 1. (a) is the schematic illustration of a generalized configuration for simulating retinal image formation process for on-axis and off-axis object points. (b) is the wavefront map to model the characteristics of the wavefront incident upon the eye pupil plane when the eye model accommodates at the distance of the object point B. (c) is an illustration of the perceived retinal images for object points B and C. (d) shows the normalized PSF for object points B and C when the eye model accommodates at the depth of point B.
Without loss of generality, the monochromatic retinal PSF for a single object point of interest may be calculated by applying the Fourier transform of the corresponding wavefront map on the eye pupil plane, and can be analytically expressed as [19]
For incoherent display systems, the normalized polychromatic PSF for a single object point can be calculated by summing up the weighted monochromatic PSF incoherently as shown in
For the display systems, such as multi-focal plane displays or LF displays, in which the light rays from multiple source points either located at different depths or different angular directions are integrally summed up to form the perception of a single retinal image, the retinal response can be modeled by a normalized accumulated retinal PSF. Assuming incoherent condition among the elemental views, the normalized accumulated retinal PSF can be calculated by the weighted sum of the retinal PSF of each elemental view and expressed as [19]
To validate the accuracy of the analytical model described above for simulating the perceived visual responses, we set up a simple configuration for both the eye model and display source and compared the simulation results obtained from using the analytical model and the Zemax. In the system configuration, we chose the Arizona eye model as the schematic eye model [15]. The pupil function P(x,y) was set as being a 3mm circular shape centered on the origin O and the wavelength of the source as 549nm. Three on-axis object points located at 0, 1, and 2 diopters from the eye pupil were chosen as the independent image sources. In this example, when configuring the wavefront error map for the analytical model, no wavefront error from the object sources or eye refractive errors were included, except the residual wavefront aberrations of the schematic eye model when it is accommodated at a given depth. Both the analytical model and the Zemax optics model were configured accordingly. Figure 2 shows the comparison of the monochromatic PSF results based on the analytical model and Zemax ray tracing. Figure 2(a) plots the in-focus PSFs of the three object points where the eye accommodation distance, zA, was set to coincide with the depths of the object points. Figure 2(b) plots the out-of-focus PSFs for the objects points located at 0 and 2 diopters where the eye accommodation distance is fixed at 1 diopter. In both the in-focus and out-of-focus conditions, the monochromatic PSF results based on the analytical model match accurately with the Zemax results.
Fig. 2. Comparison of monochromatic PSF results from the analytical model and from Zemax ray tracing for the (a) in-focus and (b) out-of-focus conditions.
To numerically demonstrate the application of the analytical model, Figs. 3,4 and 5 demonstrate examples of applying the analytical model for investigating the effects of eye accommodation, eye refractive error, and aberrations of display sources on the retinal responses, respectively. These examples also help to demonstrate how these ocular or display factors can be modeled through the analytical model. In all these examples, the parametric configurations of the analytical model were the same as those used for the example shown in Fig. 2 except the wavefront map function, Weye. In all three examples, besides a single object point fixed at the depth of 1 diopter away from the eye pupil for the purpose of simulating the retinal PSFs based on Eq. (1) for a single-view source, we also place a target object at the same depth. The target object consists of three pairs of Snellen letters of different sizes, corresponding to the spatial frequencies of 3,5,8 cycles/degree, respectively, at 1 diopter of viewing distance.
Fig. 3. Effects of eye accommodation change on the retinal responses simulated via the analytical model: Monochromatic retinal PSF results (Top row) and perceived retinal images (Bottom row) for a 2D resolution target located at 1 diopter while the eye accommodation distance was varied from 0.1, 0.5, 1, to 2 diopters, respectively, under the normal vision condition.
To simulate the effects of eye accommodation change through the analytical model in Eq. (1), we need to adjust the eye model parameters including the radius, refractive index and thickness of the optical elements according to the Arizona eye model for a given eye accommodative state, perform real ray tracing from the object point through the accommodated schematic eye model to determine the aberrations of the eye optics on the eye pupil plane, and encode the aberration into the wavefront map Weye. It is worth noting that the wavefront map includes both the residual aberration of eye as well as the defocus term introduced by the mismatch between the object distance and eye accommodation distance. Figure 3 demonstrates the effects on the retinal responses with the eye accommodation depth set at 0.1, 0.5, 1 and 2 diopters, respectively. The top and bottom rows of Fig. 3 plot the monochromatic normalized PSFs and the simulated retinal images corresponding to the different accommodation distances of 0.1, 0.5, 1 and 2 diopters, respectively. It can be observed that when the eye accommodation distance matches with the object distance of 1 diopter, the retinal PSF has the narrowest distribution and the corresponding retinal image has the highest contrast. As the eye accommodation distance shifts further or closer than the object depth, the retinal PSF is broadened and the perceived retinal images show increasing levels of blur due to defocus.
To demonstrate the ability to simulate the effects of eye refractive errors of a viewer on the retinal responses, we take myopia as an example in this simulation, while other types of refractive errors such as hyperopia or presbyopia can be implemented in a similar fashion. The degree of myopia in diopters is the reciprocal of the far point or focal point of the eye measured in meters. To simulate myopia in the schematic eye model, we can always introduce the refractive error by changing the thickness of the vitreous humor, denoted as tvit. Take Arizona eye model as an example, the vitreous thickness for the desired level of refractive error ${\Phi _{ref}}$ can be expressed as,
For instance, for an eye with 1D myopia, the corresponding vitreous thickness changes to 17.0819 mm. To numerically demonstrate the impact of myopia to the visual response, we simulate the retinal PSFs and the corresponding perceived retinal images for the same target plane of 1 diopter and eye accommodation distances as shown in Fig. 3 but add 1 diopter myopia into the schematic eye model. To implement the simulation of myopia impact on visual response based on analytical model in Eq. (1), we adjust the eye model parameters based on the desired eye accommodation state first and then adjust the thickness of vitreous humour in Eq. (4) for a given degree of myopia. Perform the exact ray tracing from object point through the adjusted schematic eye model towards the retina to covert the eye aberrations on pupil plane into the corresponding wavefront map Weye. The eye aberrations compose the residual aberration from eye optics, defocus term from the discrepancy between eye accommodation distance and object distance and the aberrations introduced by eye refractive errors. Figure 4 demonstrates the effects of myopia on the retinal responses. The top and bottom rows of Fig. 4 plot the monochromatic normalized PSFs and the simulated retinal images corresponding to the different accommodation distances of 0.1, 0.5, 1 and 2 diopters, respectively, while the eye model was added with 1 diopter of myopia using Eq. (4). We expect that the added myopia shifts the in-focus distance by 1 diopter toward the eye. It can be observed from Fig. 4 that the PSF for accommodation distance of 0.1 diopters has the narrowest distribution and the corresponding perceived retinal image has the highest image contrast, while the target plane was located at 1 diopter from the eye. As expected, when the accommodation distance shifts from 0.5 to 2 diopters, the PSFs are significantly broadened due to myopia and defocus, leading to the increasing blurriness of the corresponding perceived retinal images.Fig. 4. Effects of 1-diopter myopia on the retinal responses simulated via the analytical model: Monochromatic retinal PSF results (Top row) and perceived retinal images (Bottom row) for a 2D resolution target located at 1 diopter while the eye accommodation distance was varied from 0.1, 0.5, 1, to 2 diopters, respectively, with 1 diopter myopia for the eye model.
To demonstrate the ability to integrate the residual optical aberrations of a display engine into the analytical model and investigate their impacts on the visual response, we can simply characterize the residual aberrations of the display optics in terms of wavefront errors on the defined pupil plane and then encode the residual aberrations into the wavefront map Weye of Eq. (1). We use a simple monocular display as an example. The monocular display consists of a microdisplay and a magnifying eyepiece, through which the virtual display plane is located at 1 diopter from the eye pupil. The target object rendered through the display is composed of three groups of Snellen letters with spatial frequencies 6,10,16 cycles/degree, respectively. The sampled wavelengths are 464 nm, 549 nm, 611 nm and the wavelength weightings are determined by the eye photopic response. Based on the analytical model in Eq. (2), we simulated the retinal PSFs and perceived retinal images for two different eyepiece configurations while the Arizona eye model was configured to accommodate at the depth of 1 diopter with normal vision. In one configuration, the eyepiece is modeled as an ideal lens with a focal length of 10 mm without any residual aberrations, and in the second configuration the eyepiece is a bi-convex singlet of the same focal length with residual aberrations. Figure 5 shows a comparison of the two eyepiece configurations to demonstrate the impacts of the display optics aberration on visual responses. In this example, the Arizona eye model is implemented with its accommodation distance as 1 diopter. Figures 5(a) and 5(c) show the polychromatic PSF and the perceived retinal image of the target for the configuration with an ideal lens as the eyepiece and the configuration using a bi-convex singlet as the eyepiece, respectively. It can be observed that for the bi-convex eyepiece the corresponding perceived retinal image becomes blurrier and the polychromatic PSF is broadened mainly due to the spherical aberration and chromatic aberration introduced by the singlet eyepiece. Figure 5(b) plots the measured wavefront aberrations of the bi-convex eyepiece for the three sampled wavelengths.
Fig. 5. Effects of the residual aberrations from the display optics on the retinal responses simulated via the analytical model: (a) is the PSF and perceived retinal image for the ideal lens as eyepiece (b) is the wavefront aberration for sampled wavelengths introduced by the singlet eyepiece (c) is the PSF and image for the singlet eyepiece condition
3. Implementation of analytical model in 3D display systems
To apply the generalized analytical model to different displays under various visual conditions, in this section, we explain how the different display setups such as different display mechanisms, display parameters, display aberrations as well as the ocular setups including the eye aberrations, eye model accommodation, eye refractive errors are implemented in the analytical model for the purpose of simulating visual responses. Though we implemented the simulation in MATLAB, other programming platform can be used as well.
Figure 6 illustrates the implementation flow chart of the analytical model described in Section 2 to form perceived retinal image. The first step of the implementation is to initialize the system, including the ocular setup, the display setups, as well as the target 3D scene. The initialization of the ocular factors includes the selection of eye model, eye accommodation status, the refractive errors of a viewer, the adjustment of the optical elements of the eye model according to the accommodation state and eye refractive error, the entrance pupil size, the sampled wavelengths and the corresponding weighting factors. The display setup initialization varies widely with the different display mechanisms. For instance, for a simple stereoscope-type display that projects a 2D image of a scene through an eyepiece for each eye, the display initialization only requires specifying the distance of the 2D virtual image plane from the eye as well as the field of view (FOV) and angular resolution of the virtual image. For a MFP display, the initialization step needs to specify the number and dioptric placements of the sampled focal depths and the corresponding depth fusing weighting functions, as well as the FOV and angular resolution of each focal planes. For an InI-based LF display, the initialization step needs to specify the central depth plane location in the visual space, the number of elemental views encircled by an eye pupil, the geometric arrangement of the views, the footprint size, shape and location of each elemental views, as well as the FOV and angular resolution of each elemental view. The initialization of a 3D target scene includes the specifications of both the geometric and radiometric properties of the 3D scene to be rendered through the system.
Fig. 6. Flow chart for implementing the analytical model and rendering perceived retinal image for different display systems
Following the initialization step is the image rendering that produces the appropriate image or images to be viewed through a display system. The complexity of the image rendering largely depends on the display setups. For instance, the image rendering in a simple stereoscope-type display is performed by simply projecting the 3D scene onto the 2D virtual image plane using the well-established computer graphics rendering pipeline. For a MFP display, the rendering requires steps for establishing a depth map of the 3D scene, segmenting the 3D scene into discrete zones according to the depth map, projecting the segmented 3D scenes onto a stack of discrete virtual image planes of different depths, and applying depth-fusing algorithms to the rendered image stacks [2,21]. For an InI-based LF display, the rendering step requires rendering an array of elemental images of the 3D scene from each of the positions corresponding to the footprints of the views on the eye pupil plane [9,19]. For a computational multi-layer LF display, the rendering step requires the factorization of the LF scene to generate the attenuation masks for the attenuation layers [6,7,10].
Based on the display and ocular configurations and the target object, the following step is to perform the exact ray tracing from the object point through the display optics and the optics of adjusted schematic eye based on accommodation state and refractive errors to calculate the wavefront map Weye. Then the monochromatic retinal PSF is calculated based on Eq. (1) for each object point rendered by a given display or for each elemental view. This step needs to be repeated for each of the sampled wavelengths. If the display setup requires the integration of multiple elemental views, which can be either rendered at different depths or different viewing directions, to reconstruct a 3D point in space, the step needs to be repeated for each of the views. In the calculation, ($\Delta {x_0},\Delta {y_0}$) are determined by the off-axis object point location or the elemental view directions. The sampled wavelengths are determined by the display system such as the primary wavelength in sRGB color space which is commonly used in the customer electronics display devices and the wavelength weighting is applied to the polychromatic PSF calculation. The aperture function P(x,y), is determined by the projected footprint of the rays on the eye pupil plane from a given object point. The footprint can either be the whole eye pupil or a sub-region of the eye pupil. For displays rendering a single view of the 3D scene, Eq. (2) is further applied to compute the polychromatic PSF, while for displays rendering multiple views or multiple depth planes, Eq. (3) is further applied to compute the accumulated PSF which apply the Stiles-Crawford effect, the wavelength weighting as well as the weighting factor based on the display. Specifically, in the case of MFP displays, the display weighting factors are the depth fusion functions while for LF displays, the display weighting factors are the elemental view luminance values.
For displays rendering a single view of the scene, the perceived retinal image can be directly calculated by convolving the rendered 2D image with the retinal PSF calculated with Eq. (2). For displays rendering multiple views of a 3D scene, the calculation of the perceived retinal image may be classified into two categories based on the scene characteristics and display mechanisms. For 3D scenes composed of objects that suggest strong angular dependency on luminance such as specular objects and for LF displays in which each elemental view may render a distinct angular sample of the scene from an observation angle, the perceived retinal image, following the green path in Fig. 6, is more accurately calculated by the integration of the elemental views in all directions weighted by the corresponding luminance values. For 3D scenes with low viewing- angle dependency or for displays in which the elemental views renders the scene of different depths, the accumulated PSFs is a fairly accurate modeling of the retinal response and the perceived retinal image can be calculated through convolution between the rendered images and the accumulated PSF, following the red path in Fig. 6.
To better demonstrate the implementation of the analytical model, we use several 3D display methods as examples. Among the emerging 3D display approaches which can render nearly correct focus cue to overcome the vergence-accommodation cue conflict (VAC) problem, a depth-fused multifocal plane display (MFP) [2] can sample the projection of a 3D scene at different depths, while an InI based LF display [22] and a computational multilayer light field display [7,8] can reconstruct the angular sampling of a 3D scene in the directions of the light rays emitted by this 3D scene and viewed from different positions [23]. For a depth-fused MFP display, it samples a 3D scene at different depths and these depth-sampled projections may be considered as the elemental views of the display. These elemental views, however, usually are rendered from a single viewpoint and are additive along the depth direction, and the ray footprint of each view covers the entire eye pupil. The perceived retinal image of such displays can therefore be computed by the convolution of the rendered images on the retina with the depth-weighted accumulated PSF of each object point [2]. The aperture function P(x,y) is the whole eye aperture. Varying with the object distance, a residual defocus term, which is calculated based on exact ray tracing, the aberrations of the display optics, as well as the refractive errors of a viewer are all programed in the wavefront map Weye. To model an InI-based LF display [9] and computational multilayer light field display [7,8], the perceived retinal image is the integration through all the modulated elemental views. For each elemental view, the shape, size and location of the sub-aperture function P(x,y) are determined by the footprint of each elemental view on the eye pupil, as described in Huang and Hua [9]. The wavefront aberration Weye(x,y) includes the display aberration and eye aberrations, which is implemented by the exact ray tracing from the display plane through all the optical elements including the lenslet and eyepiece and the accommodated schematic eye model to the retina. It has a residual defocus term that is due to the axial location shift between the CDP and the reconstruction plane. Meanwhile, for the multilayer LF display, as described in Xu and Hua [10], the sub-apertures are determined by the pixel pitch of the attenuation layer and the wavefront aberration contains a residual defocus between the accommodated depth and optical infinity due to the assumption that each elemental view is a narrow parallel beam. In both of those two display systems, the residual defocus PSF degrades the perceived retinal image quality and hence leads to the accommodation cue errors.
When the eye model changes its accommodation distance, zA, the elemental view PSF changes with the wavefront aberration Weye. In an MFP display, besides the PSF change, the corresponding depth fusing functions change as well, leading to the change of perceived depth. In an InI-based LF display and multilayer light field display, the eye accommodation changes the chief ray location of each elemental view. Thus, the displacement between each elemental view changes. Both elemental view PSF and displacement lead to a change to the perceived retinal image. Such dependency on eye accommodation state thus leads to repeating computation of the perceived retinal image whenever the eye accommodation state changes.
4. Applications of the analytical model to emerging 3D display systems
To numerically demonstrate the application of the analytical model in different display systems or under different visual conditions to evaluate the perceived retinal image and to predict the accommodative response, in this section, we introduce a test setup including five different configurations of display systems, visual parameters, and a target 3D scene. For comparison purpose, the five different display configurations include a simulated 3D display that preserves all natural-viewing 3D cues, a stereoscope that renders the 3D scene as a binocular pair of 2D images, an MFP display that renders the 3D scene as a discrete stack of depth-fused projections at different depths, an InI-based LF display that renders the scene as a dense array of elemental views sampled from different viewing angles, and an MLLF display that renders the scene decomposed as two-layer factorized attenuation maps for reconstructing the LF of the scene. Based on these display configurations, both the normal vision and the eye condition with refractive errors are compared and the effects of eye accommodative response are examined.
4.1 Test setup
For the purpose of visually examining the retinal responses to contents of different spatial resolutions and depths, we artificially construct a target 3D scene consisted of three 2D resolution target planes, as shown in Fig. 7. All three resolution target planes have the same angular size but are located at different depths. The object distances denoted as Zobj1, Zobj2, and Zobj3, are 1.25 diopter, 1 diopter, and 0.75 diopters from the eye model entrance pupil, respectively. The targets are also laterally displaced from either other so that, as shown in Fig. 7(b), each target plane occupies 1/3 of the horizontal field of view when they are projected on the retina, arranged from left to right as the depths of the target planes move further from 1.25 to 0.75 diopters. The retinal image is geometrically flipped for convenience. Each plane is a Lambertian surface covered by a resolution target that is composed of 5 groups of tri-bars with spatial frequencies of 5, 10, 15, 20, 30 cycles/degree arranged from the top to bottom of the plane, respectively.
Fig. 7. Illustration of an artificially constructed target 3D scene: (a) the scene is composed of three target planes placed at three different depths from the eye model and each target plane covers 1/3 of the horizontal FOV; (b) the projection of the 3D scene on the retina based on geometric optics.
Table 1 summarizes the main parameters for the display configurations except the natural viewing condition which is essentially the same as shown in Fig. 7. In the stereoscope, MFP, and InI-based LF display configurations, we neglect the pixel sampling effects by assuming the display source to have infinitely high resolution. Detail discussions about the effects of pixel sampling for InI-based and multi-layer-based LF-displays can be found in our prior works [10,19]. We further assume the eyepiece and other optical elements required for these display configurations are aberration-free for simplicity. For the stereoscope, the screen or the virtual image plane is placed at 1 diopter away from the eye pupil. For the depth-fused MFP display configuration, a dual-focal plane setup is constructed with the front focal plane at 1.3 diopters and the back focal plane at 0.7 diopters, respectively, which provides a dioptric spacing of 0.6 diopters for adjacent focal planes as suggested in [2,4]. The corresponding weighting factors, denoted as w1 and w2, for the 3D target scene described above are based on the non-linear fusing functions suggested in [2], where w1 are [0.846,0.5,0.154], and w2 are [0.154,0.5,0.846], respectively. For the InI-based LF-3D display configuration, the central depth plane is set at the 1 diopter away from the eye entrance pupil, and the elemental view number, denoted as Nv, is 2 by 2, which means 4 views enter the eye pupil simultaneously. Therefore, the elemental view footprint size is $aD/\sqrt {{N_v}} $, where D is the entrance pupil diameter of the eye model, a is a scalar factor between 0 and 1 defining the fill factor of each elemental view as introduced in [9] and a is set as 1 here so the sub-aperture diameter of the elemental views on eye pupil is D/2. For the multilayer light field display, we use a dual-layer configuration with the front and back layers located at 1.15 and 0.85 diopters, respectively. The pixel pitch of both layers is 0.2327 mm, which corresponds to an angular resolution of 1arc min per pixel for the test plane at 1.25 diopter. The number of elemental views encircled by the eye pupil is determined by pixel pitch as well as the display layers’ locations, which could result in thousands of elemental views. For the purpose of factorization [6,10], the eyebox size is set as 3mm, and there are 3 by 1 reference views in horizontal and vertical directions respectively, of the target light field are assumed to calculate the transmittance of the display layers.
For the ocular parameters, as shown in Table 2, we choose the Arizona eye model for simulation with the entrance pupil diameter as 3mm, which is the typical pupil size when viewing displays of luminance around 200 cd/m2. Regarding the Stiles-Crawford effect, a gaussian apodization filter is applied on the eye pupil with an amplitude transmittance coefficient as β = -0.116mm-2. The sampled wavelengths are 464nm,549nm,611nm, and the wavelength weightings are based on the photopic response [15] of the human eye for those chosen wavelengths. Both a standard observer with 20/20 normal vision and an observer with 1 diopter myopia condition will be simulated in the paper.
Table 2. Ocular parameters.
4.2 Characterization of perceived retinal images
Following the analytical model in Section 2 and the implementation steps in Section 3, we simulated the perceived retinal images of the target 3D scene described in the test setup for the five different configurations of displays. Figure 8 shows the perceived retinal images for three of the configurations: natural viewing, stereoscope, multi-focal-plane display based on a standard observer of normal vision, while the eye accommodation depths are set at 1.25, 1, and 0.75 diopters corresponding to the left, middle and right columns of the figure, respectively.
Fig. 8. Perceived retinal images for three different display configurations, including natural viewing, stereoscope and multifocal display, while the eye model is set at three different accommodation levels, 1.25, 1, and 0.75 diopters, corresponding to left, middle, and right columns of the figure, respectively.
As shown in the top row of Fig. 8, for natural viewing where light rays emit from the points of interest at different depths, when the eye accommodates at the depth of a resolution target plane, the corresponding perceived retinal image appears sharp while the object planes at other depths appear blurry on the retina due to defocus. This observation is true for all three accommodation depths. The perceived retinal images for natural viewing serve as the references to evaluate the depth rendering ability of other display systems.
In the case of the stereoscope display which renders a virtual image at a fixed distance of 1 diopter in the test setup, as shown in the second row of Fig. 8, when the eye accommodates at the depth of 1 diopter, all of the three resolution targets in the scene appear sharply focused on retina, while when the eye accommodates at the depths of 1.25 or 0.75 diopters, all the target planes become blurry due to defocus and they all have the same amount of blurriness. As expected, there is no distinctive blurriness among targets of different depths to drive the eye to accommodate at the depths of targets but at the screen or the virtual image plane.
The perceived retinal images of the MFP configuration are shown in the bottom row of Fig. 8. When the eye accommodates at 1.25 and 0.75 diopters, the image blurriness of all the target planes is comparable with that observed in the natural viewing condition. When the eye accommodates at 1 diopter, which is in the dioptric middle point of the two display focal planes, the perceived retinal image for the target plane of 1 diopter does not appear as sharp as the corresponding image in natural viewing condition, and the perceived retinal images for the object planes of 1.25 and 0.75 diopters show very little difference of blurriness from the target of 1 diopter. This observation is consistent with the prediction in the work [2,21] because the focus cue accuracy rendered by an MFP degrades as the distance between the rendered depth and the physical focal planes increases and the midpoint depth between the focal planes tends to be least accurate and depends on the dioptric separation between the focal planes.
4.3 Investigation of eye accommodative response
To evaluate the performance of a 3D display, besides the perceived retinal image quality, another critical factor is the depth rendering accuracy. When viewing a 3D display, the accommodation cue from the display system is considered to be accurate if the display is able to render focus cues driving a user’s eye to accommodate at a rendered depth. Otherwise, the difference between the rendered depth and the actual eye accommodation distance is considered as accommodative error. The image blur on the retina, which is characterized by a progressive loss of contrast, is the primary stimulus to the eye accommodation [24]. Therefore, the contrast amplitude and gradient change of the retinal image are typically considered as the primary visual information to predict the changes responding to eye accommodative response. In this subsection, we apply the analytical model to investigate the eye accommodative responses to two of the display configurations, the InI-based LF display and stereoscope, in comparison to that of the natural viewing configuration.
Figures 9(a) though 9(e) illustrate the contrast change of the perceived retinal images for the InI based LF-3D display, natural viewing. More specifically, Figs. 9(a)–9(c) are the contrast change plots of reconstructed on-axis 3D points rendered at the depths of 0.75, 1, and 2 diopters, respectively, through the InI-based LF-3D display, Fig. 9(d) is for the contrast change for an on-axis point at the depth of 1 diopter in the natural viewing configuration, and Fig. 9(e) is for the contrast change of an on-axis point rendered at the depth of 0.75 diopters in the stereoscope configuration. These contrast plots were generated by applying Fourier transforms to the retinal PSFs calculated with the analytical model while the eye accommodation depth is scanned by ±0.3 diopters from the rendered depths at an increment of 0.05 diopters. The contrast changes responding to the eye accommodation scanning for the spatial frequencies of 5,10,15, and 20 cycles per degree are plotted in these figures. For each frequency, the location corresponding to the maximum image contrast during an eye accommodation scan is marked by a black arrow. In the InI LF display configuration, as shown in Fig. 9(b), when rendering the target at 1 diopter, which is the same depth as the central depth plane (CDP), the peaks of the contrast profiles for all the frequencies coincide with the rendered depth, which is consistent with the plot shown in Fig. 9(d) for natural viewing configuration. In contrast, when the rendering depths are shifted away from the CDP, such as the ones shown in Figs. 9(a) and 9(c), the peaks of the contrast profile deviates from the rendered depths and the magnitude of deviation increases with the decrease of spatial frequencies, which indicates a non-zero accommodation shift for the accommodative response. In the stereoscope configuration, as shown in Fig. 9(e), the peaks of the contrast profile are all shifted toward and coincide with the depth of the virtual image plane for all the spatial frequencies.
Fig. 9. (a)-(c) Perceived retinal image contrast for different accommodation shifts in InI based LF-3D display when the rendered targets are reconstructed at the depths of 0.75, 1, and 2 diopters, respectively; (d) Retinal image contrast for different accommodation shifts when viewing an object located at 1 diopter for natural viewing; (e) Retinal image contrast for different accommodation shifts in stereoscope when a target is rendered at the depth of 0.75 diopters. (f) Accommodation errors for different rendering depths shift for InI based LF-3D display, natural viewing, and stereoscope.
The deviation between a rendered depth with zero accommodation shift and the accommodation depth corresponding to the maximum image contrast is considered as the accommodation error, all of which is further summarized in Fig. 9(f) for the three configurations. In Fig. 9(f), the red circle line illustrates the natural viewing condition which is the reference producing no accommodation error. The stereoscope configuration is plotted in blue line with triangle markers. The accommodation error increases linearly as the rendering depth shifts from the focal plane. The accommodation errors for the InI-based LF-3D display are in-between the natural viewing and stereoscope, and vary according to the spatial frequencies of the rendered targets. High-frequency targets tend to yield less accommodation error but also yield lower image contrast as suggested in Figs. 9(a)–9(c) and 9(f).
4.4 Effects of eye refractive errors on retinal image quality
In this last example, we demonstrate the application of the analytical model to investigate the impacts of eye refractive errors to the perceived retinal image of a 3D display. More specifically, we demonstrate the perceived retinal image simulation for natural viewing, stereoscope and computational multilayer light field display. In this simulation, the schematic eye model is set with 1 diopter myopia while its accommodation depth is set at 1.25, 1 and 0.75 diopters, corresponding to the depths of three 2D target planes in the test scene, respectively.
The first row in Fig. 10 shows the perceived retinal image for the natural viewing while the observer has 1 diopter myopia. Compared with the first row in Fig. 8, which is the perceived retinal image for natural viewing under normal vision condition, when users have 1 diopter myopia, the in-focus object distance all shifts 1 diopter towards the eye. For example, when the eye accommodates at 0.75 diopters, with 1 diopter myopia, the object at 1.25 diopters appears sharper on retina than the target objects at 1 and 0.75 diopters.
Fig. 10. The perceived retinal images for natural viewing, stereoscope and computational multilayer display systems for eye model with 1 diopter myopia and accommodate at 1.25, 1 and 0.75 diopter, respectively.
As shown in the second row of Fig. 10, the retinal images of the stereoscope with 1 diopter myopia shifts the in-focus distance towards the accommodation depth of 0.75 diopters. At each of the accommodation states, there is still no variation of blurriness or focus cues among the three targets of different depths.
The perceived retinal images of the multilayer LF display for an observer with 1 diopter myopia are shown in the third row of Fig. 10 to demonstrate the impacts of eye refractive errors on a 3D display which can render nearly correct focus cues. The overall images suffer from very low contrast due to both the inherent limits of the multi-layer LF rendering mechanism [10] and the defocus blur introduced by myopia. However, the corresponding perceived retinal images could still offer some focus cues for the low-frequency contents since the trends of the retinal image blurriness are similar to that of the natural viewing. For a viewer with 1 diopter myopia, when the accommodation distance is set at 0.75 diopters, the perceived retinal images of all three target objects located at the distances of 1.25, 1, and 0.75 diopters, respectively, appear sharper than those corresponding to the accommodation depths of 1.25 or 1 diopters. At each accommodation distance, the perceived retinal image becomes sharper as the object moves closer to the eye pupil. For instance, when eye accommodate at 0.75 diopters, with 1 diopter myopia, the in-focus distance is 1.75 diopters. Therefore, the perceived retinal image of the object target located at 1.25 diopters appears sharper the object plane at 0.75 diopters.
5. Conclusion
We presented a generalized analytical model for simulating the visual responses such as retinal PSF, MTF, and image formation of different types of display systems, from natural viewing, stereoscopes, to emerging 3D light field displays. This analytical model can accurately simulate the retinal responses when viewing a wide range of 2D and 3D display systems, accounting for the residual eye aberrations of schematic eye models that match with the statistical clinical measurements, eye accommodative change as required, the effects of different eye refractive errors specific to viewers, and the effects of various wavefront aberrations inherited from a display engine. We further describe the numerical implementation of this analytical model for simulating the perceived retinal image with different types of HMD systems in a single computational platform. Finally, with a test setup, we numerically demonstrated the application of this analytical model in the simulation of the perceived retinal image, accommodative response and in the investigation of the eye refractive error impacts on the perceived retinal image based on the multifocal plane display, integral imaging based light field display, computational multilayer light field display, as well as the stereoscope and natural viewing for comparison.
Disclaimer
Dr. Hong Hua has a disclosed financial interest in Magic Leap Inc. The terms of this arrangement have been properly disclosed to The University of Arizona and reviewed by the Institutional Review Committee in accordance with its conflict of interest policies.
Disclosures
Dr. Hong Hua has a disclosed financial interest in Magic Leap Inc. The terms of this arrangement have been properly disclosed to The University of Arizona and reviewed by the Institutional Review Committee in accordance with its conflict of interest policies.
References
1. H. Hashemi, A. Fotouhi, A. Yekta, R. Pakzad, H. Ostadimoghaddam, and M. Khabazkhoob, “Global and regional estimates of prevalence of refractive errors: systematic review and meta-analysis,” J. Curr. Ophthalmol. 30(1), 3–22 (2018). [CrossRef]
2. S. Liu and H. Hua, “A systematic method for designing depth-fused multi-focal plane three-dimensional displays,” Opt. Express 18(11), 11562 (2010). [CrossRef]
3. S. Liu, H. Hua, and D. Cheng, “A novel prototype for an optical see-through head-mounted display with addressable focus cues,” IEEE Trans. Visual. Comput. Graphics 16(3), 381–393 (2010). [CrossRef]
4. X. Hu and H. Hua, “High-resolution optical see-through multi-focal-plane head-mounted display using freeform optics,” Opt. Express 22(11), 13896–13903 (2014). [CrossRef]
5. H. Huang and H. Hua, “High-performance integral-imaging-based light field augmented reality display using freeform optics,” Opt. Express 26(13), 17578 (2018). [CrossRef]
6. G. Wetzstein, D. Lanman, W. Heidrich, and R. Raskar, “Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays,” ACM Trans. Graph. 30(4), 1–12 (2011). [CrossRef]
7. G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar, “Tensor Displays: Compressive Light Field Synthesis Using Multilayer Displays with Directional Backlighting,” ACM Trans. Graph. 31(4), 1–11 (2012). [CrossRef]
8. F. Huang, K. Chen, and G. Wetzstein, “The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-eye Light Field Displays with Focus Cues,” ACM Trans. Graph. 34(4), 60 (2015). [CrossRef]
9. H. Huang and H. Hua, “Systematic characterization and optimization of 3D light field displays,” Opt. Express 25(16), 18508–18525 (2017). [CrossRef]
10. M. Xu and H. Hua, “Systematic method for modeling and characterizing multilayer light field displays,” Opt. Express 28(2), 1014–1036 (2020). [CrossRef]
11. Z. Qin, P. Chou, J. Wu, Y. Chen, C. Huang, N. Balram, and Y. Huang, “Image formation modeling and analysis of near-eye light field displays,” J. Soc. Inf. Display 27(4), 238–250 (2019). [CrossRef]
12. Z. Qin, P. Chou, J. Wu, C. Huang, and Y. Huang, “Resolution-enhanced light field displays by recombining subpixels across elemental images,” Opt. Lett. 44(10), 2438–2441 (2019). [CrossRef]
13. T. Lian, K. J. MacKenzie, D. H. Brainard, N. P. Cottaris, and B. A. Wandell, “Ray tracing 3D spectral scenes through human optics models,” J. Vis. 19(12), 23 (2019). [CrossRef]
14. M. Xu and H. Hua, “Method for evaluating 3D display systems based on perceived retinal image,” Proc. SPIE 11310, 113100A (2020). [CrossRef]
15. J. Schwiegerling, Field guide to visual and ophthalmic optics (SPIE, 2004).
16. J. Schwiegerling, “The Human Eye and Its Aberrations,” in Advanced Optical Instruments and Techniques Vol. 2, D. Malacara -Hernández and B.J. Thompson, eds. (Taylor & Francis, 2018).
17. R. Navarro, J. Santamaría, and J. Bescós, “Accommodation-dependent model of the human eye with aspherics,” J. Opt. Soc. Am. A 2(8), 1273–1280 (1985). [CrossRef]
18. I. Escudero-Sanz and R. Navarro, “Off-axis aberrations of a wide-angle schematic eye model,” J. Opt. Soc. Am. A 16(8), 1881–1891 (1999). [CrossRef]
19. H. Huang and H. Hua, “Effects of ray position sampling on the visual responses of 3D light field displays,” Opt. Express 27(7), 9343–9360 (2019). [CrossRef]
20. M. Xu and H. Hua, “Methods of optimizing and evaluating geometrical lightguides with microstructure mirrors for augmented reality displays,” Opt. Express 27(4), 5523–5543 (2019). [CrossRef]
21. X. Hu and H. Hua, “Design and Assessment of a Depth-Fused Multi-Focal-Plane Display Prototype,” J. Display Technol. 10(4), 308–316 (2014). [CrossRef]
22. H. Hua and B. Javidi, “A 3D integral imaging optical see-through head-mounted display,” Opt. Express 22(11), 13484 (2014). [CrossRef]
23. H. Hua, “Enabling Focus Cues in Head-Mounted Displays,” Proc. IEEE 105(5), 805–824 (2017). [CrossRef]
24. K. F. Ciuffreda, “Accommodation and Its Anomalies,” in Vision and Visual Dysfunction: Visual Optics and Instrumentation, W. N. Charman, ed. (JR Macmillan, 1991).
