1. Introduction

In recent years, see-through head mounted displays (HMD) that overlay computer output over the user’s real view have gained increasing importance. They have been used in a wide range of application ideas, such as maintenance, medicine and mobile networking, as they offer two principal advantages: First, they allow the user to access computer information while (at least partially) paying attention to his environment. Second, they allow computer generated information to be attached to real life objects creating a mixture between the real and virtual world referred to as augmented reality [1]. However, to reduce technical complexity today’s available see-through displays produce a virtual image at a fixed optical distance. This leads to the result that the overlaid computer input gets out of focus, as soon as the eye accommodates to a real object at a different distance. Investigations by Edgar et al. demonstrated such accommodation problems for test people [2]. Thus, to overcome this drawback a see-through HMD should ideally be accommodation-free or at least have a very large depth of focus. This should be achieved while preserving high resolution, luminance and contrast.

1.1. Related work

Today’s most common HMDs consist of a microdisplay unit illuminated by incoherent backlight. An optical transmission system generates a virtual image of the display unit at a fixed focal distance: typically one to two meters from the user’s eye [3, 4]. According to Rayleigh’s quarter-wavelength criterion for an average 4 mm diameter eye pupil the depth of focus in this setup is limited to about 0.14 diopters (D) [5]. By integrating an artificial aperture stop in the display’s optical setup the DOF can be extended, resulting however in a severe reduction of display luminance and light efficiency. This is not a feasible way especially when considering that today’s HMD systems suffer anyway from luminance and resolution limitations. Recently, we proposed an LCD-based coherent wearable projection display where the LCD is illuminated by partially coherent light rather than by the common used incoherent backlight and where the LCD-image is projected directly onto the retina [6, 7]. In this setup the display luminance as well as the DOF can be extended significantly due to the direct retinal projection and due to the use of partially coherent light. Another possible technology is the retinal scanning technique where the image is not projected, but scanned onto the retina [8, 9]. For such a system, de Wit et al. discussed the effects of a small exit pupil when the eye is well accommodated [10].

1.2. Paper contribution

In this paper we explore the potential of the scanning technology for providing a see-through display with high depth of focus. The defocusing properties of a retinal scanning display are evaluated by using an accommodation-dependent schematic eye model. For characterizing the image quality the contrast function in terms of eye accommodation, spatial frequency and eye rotation is calculated and discussed. Based on that design trade-offs are derived.

The first part of the paper gives an overview over the retinal scanning technology as well as the used schematic eye model. In the second part the simulation background is discussed and the mathematical relations for the specific case of retinal scanning are derived. The third part focuses on the evaluation of the display’s defocusing properties for central image points as well as for peripheral points. The paper closes with a discussion of the final design trade-off between in-focus resolution, depth of focus and maximum image field.

2. Simulation background

2.1. Virtual retinal display

The virtual retinal display (VRD) technology was proposed by the Human Interface Technology Lab, Washington [8, 9]. It is based on a light beam which is scanned directly onto the retina in a raster pattern while being intensity modulated to form an image. The principle is depicted schematically in Fig. 1. As modulated light source a collimated, low-power laser diode is normally used. The tiny laser beam is subsequently deflected in x- and y-direction by two uniaxial scanners. The horizontal scanner operates at several kHz whereas the vertical one operates at the image refresh rate (at least 50Hz). Finally, a viewing optics projects the laser beam through the center of the eye’s iris onto the retina.

 

Fig. 1. Schematical illustration of the VRD principle (after [11]). The left zoom shows the scanning mirror setup (with S_H and S_V indicating the horizontal and vertical mirror, respectively). The right zoom focuses on the region, where the laser beam enters the eye. σ_cor signifies the radius of the beam at the eye’s cornea. This parameter will play an important role in the subsequent simulations.

Download Full Size | PDF

In the VRD system the pixels are projected serially in time on the retina. As the image refresh rate lies above the temporal resolution limit of the human eye, the user does not perceive any flickering effects. In optical terms, however, the temporally sequential pixel projection has the important effect that the pixels are mutually incoherent despite the use of coherent laser light. This has significant influence on the analysis.

2.2. Accommodation-dependent schematic eye model

Several different schematic eye models have been proposed in the literature¹ - from the simple and famous Gullstrand eye [13] up to a sophisticated one which models the eye’s anatomy very accurately [14]. For our purpose, ideally, the eye model should reproduce both anatomy and optical properties (such as first order aberrations) for on- and off-axis beams with a minimum of fitting parameters. Moreover, the model should incorporate the increment of refractive power of the eye during accommodation. Therefore, the simulations presented in this paper are based on the accommodation-dependent wide-angle schematic eye model as proposed by Navarro et al. and Escudero et al. [15, 16].

 

Fig. 2. The accommodation-dependent schematic eye model as used for all simulations. The numbered labels indicate the surfaces as parameterized in Tab. 1. CR indicates the center of rotation as explained in the text.

Download Full Size | PDF

This model - depicted in Fig. 2 - consists of three conical optical surfaces plus two spherical surfaces whose second one represents the retina. Table 1 shows the geometrical parameters and the used refractive indices in the unaccommodated state for a reference wavelength λ=632.8nm. Note in particular, that in this model all parameters defining the eye’s geometry in the unaccommodated state are based on anatomical data so that there is no need to fit original values to match experimental results.

Table 1. Geometry of the schematic wide-angle eye model in the unaccommodated state

View Table | View all tables in this article

The image surface is intersected by the optical axis at the paraxial focus for the reference wavelength λ=632.8nm. The focal length f of this schematic eye is 22.18mm in the image space; the refractive power in the unaccommodated state is 60.2 diopters (D).

When the natural eye accommodates², the increment in refractive power is mainly caused by two contributions: first, a geometrical variation of the eye lens shape and secondly, a change of the graded-index structure within the eye lens. In this eye model, the geometrical shape variations are incorporated accurately as proposed by [15]. They are listed in Tab. 2. The second contribution, the variation in the graded-index structure, however, is replaced by an effective, accommodation-dependent refractive index n ₃(ΔD), for simplification.

To get the dependence of the effective index n ₃(ΔD), a cubic adjustment has been made to fit the refractive powers for 2D, 4D and 6D of accommodation at the reference wavelength λ=632.8nm, similar to the procedure in Ref. [15] (cf. Tab. 2). Consequently, it is the only parameter which is not based on anatomical data. For details of the modeling process as well as comparisons of simulated aberration results with experimental data see Ref. [15, 16].

Table 2. Dependence of the lens parameters on changes of the eye accommodation ΔD (in diopters)

View Table | View all tables in this article

The natural eye rotates in its socket under the action of six muscles. Because of the way these muscles are positioned and operate, there is no unique center of rotation (CR). However, a mean position for this point can be assumed to lie 8.5mm behind the iris pupil and along the optical axis (as indicated in Fig. 2) [17].

2.3. Retinal properties

The resolution of the human retina differs from point to point [18, 19]. The highest retinal resolution is found in the retinal center (known as the fovea) where the receptors are most densely concentrated. The fovea’s diameter is approximately 5deg of visual angle. Just a few degrees from the fovea the resolution falls by an order of magnitude. Thus, when looking at a picture, the eye moves quickly from the current point of gaze to a new location to bring selected image parts to the fovea (referred as saccades).

 

Fig. 3. Due to the high foveal resolution the eye moves to bring selected image parts to the fovea. The cross in the grey rectangle indicates the corresponding image part which the eye is currently gazing at.

Download Full Size | PDF

When looking at a scanned image from a VRD system these eye rotations are also needed to gaze at all parts of the scanned image. In other words: to gaze at peripheral image points with a field angle α the eye has to rotate by α to make sure that these peripheral beams hit the retina in the fovea (illustrated in Fig. 3). Note, however, that these fast saccades do not interfere with the scanning process as the image refresh time (≈ 20ms) is about an order of magnitude faster than the characteristic saccadic latency (≈200ms) [5].

3. Contrast function

To characterize the quality of the scanned image and especially the defocusing properties of the retinal scanning display, a MTF-similar contrast function has been calculated in terms of accommodation ΔD, spatial frequency f and corneal beam radius σ_cor . This contrast function (CF) is defined as the ratio of the contrast of the retinal scanned image (c′) and the original image (c):

where c and c′ stand for the respective Michelson contrast:

Here I _max and I _min indicate the maximum and minimum of the intensity level in the original image. I′_max(f,ΔD,σ_cor ) and I′_min(f,ΔD,σ_cor ) indicate the respective levels in the scanned retinal image.

The calculation of the CF is processed in two steps: In a first step for each image pixel the accurate beam shape and position on the retina is evaluated. These beam properties determine the local resolution and contrast of the scanned image. In a second step the CF is calculated based on these retinal beam data. The viewing optics (as illustrated in Fig. 1) is not considered in the simulations since it turned out that possible aberrations do not affect the display’s defocusing properties significantly due to the tiny laser beam.

3.1. Beam raytracing

The goal of the first step is to calculate the shape and position of the scanning beam on the retina. Thus, the incoming scanned beam is raytraced from the cornea to the retina in the eye model. To perform this process the ray-tracing software package OSLO [20] is used. Note that due to the scanning mode these raytracing results correspond to the local Point Spread Function PSF(x_i ,y_i ) at a certain retinal pixel point (x_i ,y_i ). For all simulations it is assumed that the incoming scanned laser beam is Gaussian beam shaped with a variable beam radius σ_cor at the corneal plane (see Fig. 1). This ray-tracing calculation is processed for all retinal pixel positions of the scanned image. Further, the refractive power of the modeled eye as well as the eye orientation in terms of rotational movements are varied.

3.2. Contrast calculation

In a second step the CF has been calculated based on these local PSF-values. Considering that the human eye has a time-integrating behavior while scanning the beam onto the retina, the averaged retinal intensity I(x_i ,y_i ) can be written as an integral over the time of one frame T [21]:

with g(t) being the temporal intensity function of the scanning beam and C a factor of proportionality. Every time a x-line is finished, a new line is started at x=0 but with a slight shift δy in y-direction. Therefore, the Eq. (3) can be rewritten as

where n signifies the total number of lines in one frame and v(x,y) the horizontal scanning velocity at a retinal point (x,y)³.

CF in x-direction

For the CF-calculation in x-direction we assume that vertically oriented, dark and bright bars of contrast 1 with a certain spatial frequency f are scanned onto the retina. The corresponding intensity function g(x,y) is defined as follows:

Note that the spatial frequency of this bar pattern is 1/(4Δx). According to Eq. (1) the contrast can now be received by calculating the intensity levels I(x_i ,y_i ) of the dark and bright bars in the scanned retinal image. Symmetry considerations show that the maximum level is reached at the middle of the bright bar (x_i =-Δx) while the minimum intensity value lies at the middle of the dark bar (x_i =Δx), respectively.

Based on Eq. (4) and Eq. (5) and leaving out constant factors the maximum intensity at x_i =-Δx can be expressed as:

Due to the discrete scanning step δy in y-direction the intensity distribution in y-direction varies slightly within the range of δy and, thus, it is not totally homogeneous. Consequently, the intensity level $I_{\mathit{\text{max}}}^x$ (y_i ) is averaged in y-direction over a range of δy. Further, it is assumed that PSF(x,y)=PSF_x (x)·PSF_y (y) which is true for every Gaussian-beam shaped PSF-functions.

After inserting g(x,y) as defined in Eq. (5) and after some substitutions <$I_{\mathit{\text{max}}}^x$ > is given as follows:

In analogous manner <$I_{\mathit{\text{min}}}^x$ > turns out to be:

so that the contrast can be calculated by using Eqs. (1) and (2). Thereby, the integrals over y_i in Eqs. (8) and (9) are cancelled.

CF in y-direction

In similar manner the CF in y-direction can be calculated by defining an intensity function g(x,y) with bright and dark horizontally oriented bars. However, due to the the discrete scanning step δy in y-direction, the thickness of the bars must be an integer multiple of δy. Thus, g(x,y) can be set as follows and extended periodically:

with q∈ℕ₊. This pattern corresponds to a spatial frequency of 1/(2qδy).

Based on Eq. (4) the maximum and minimum intensity I^y (x_i , y_i ) at $y_i=-\frac{q+1}{2}\delta y$ and $y_i=\frac{q-1}{2}\delta y$ can now be figured out with similar considerations as above. It follows:

4. Contrast analysis

4.1. General considerations

The goal of the contrast analysis is to explore the potential of the retinal scanning technology to provide a wearable display with high depth of focus. The exploration space is defined by the following four parameters: spatial frequency f, corneal beam radius σ_cor , change in the eye accommodation ΔD and eye rotation angle α=(α_x ,α_y ) to gaze at peripheral points. By varying these four parameters the potential, the limits and the design trade-offs will be investigated. Sometimes the spatial frequency will be kept constant. In these cases the contrast values are evaluated and shown for the spatial frequency f=6.2cyc/deg referred as “standard frequency”⁴.

To quantify the depth of focus (DOF) several different criteria are used in the literature. Frequently, the DOF is defined as the dioptric range for which the contrast value does not fall below a certain amount (normally between 0.5 and 0.8) relative to its in-focus optimal value [12, 22]. In our case this definition is not very helpful as the in-focus contrast value depends substantially on the used corneal beam radius. Thus, the definition adopted in this paper is slightly different: the DOF here is defined as the dioptric range for which the absolute contrast value does not fall below 0.7 [23]. The limit of 0.7 seems to be rather arbitrary, but a different limit has an effect on the quantitative results only, but not on any qualitative ones.

Note further that the display is assumed to provide best focusing for an eye accommodated to infinity. Consequently, the DOF in this paper is measured as the dioptric power difference from optical focusing to the maximum defocusing, only. By adjusting the best focusing distance to closer values than infinity the depth of focus could be increased since in this case depth of focus is defined as the dioptric power difference between the maximum defocusing values on both sides around optimal focus.

For the first CF-simulation it is assumed that the modeled eye is unrotated and perfectly aligned to the scanning optics (i.e. α=0). That means that the exit pupil of the scanning optics is concentrical to the entrance pupil of the modeled eye and the eye is gazing at a central image point. The eye’s iris radius is set to 1.5mm. Note that this radius has no effect on the contrast results as long as the eye is aligned and the corneal beam radius σ_cor is much smaller than the iris radius. Variable parameters are, thus, the corneal beam radius σ_cor , the change in the eye’s refractive power ΔD due to accommodation and the spatial frequency f.ΔD=0 is set to the unaccommodated, emmetropic eye with D=60.2D, where the eye is focused to infinity.

4.2. Resolution analysis

As the CF in x- and y-direction are not identical (see Eqs. (7–11)), both directions have to be calculated. For the analysis the respective lower value is to be considered. Figure 4(a) shows the corresponding through-focus CF in x- and y-direction in terms of changes in eye accommodation ΔD for three different spatial frequencies f at a fixed corneal beam radius σ_cor =350µm.

 

Fig. 4. (a) shows the through-focus CF in x and y-direction for three different spatial frequencies. The bright circles (∘) indicate the x-direction whereas the filled ones (∙) signify the y-direction. Here the scanning step δy was 15µm. (b) shows the respective CF in x-direction in terms of eye accommodation and spatial frequency. In both cases the corneal beam radius σ_cor has been set to 350µm.

Download Full Size | PDF

As it can be seen from Fig. 4(a) the contrast-values in x-direction are lower than the ones in y-direction for any values of accommodation and spatial frequency. It turns out that this relation is valid for any values of the corneal beam radius and for any scanning steps δy>0, as long as the eye is perfectly aligned to the optics. This asymmetry between x- and y-direction is caused by the fact that the scan pattern in y-direction is discrete while in x-direction the image is scanned continuously. Thus, the subsequent simulation results for central image points deal with the CF in x-direction, only. For illustration, Fig. 4(b) shows the CF in x-direction in terms of eye accommodation and spatial frequency.

Note that the resolution in y-direction is limited due to the discrete scanning step δy. The cut-off frequency $f_{\mathit{\text{cut}}}^y$ in y-direction is $f_{\mathit{\text{cut}}}^y$ =11.9cyc/deg for δy=15µm. Certainly, the cut-off frequency can be extended when reducing the scanning step δy. However, this results in either a decrease of the vertical image field or in a higher x-scanning frequency. A reduction of the image refresh rate - as the third possibility - is not feasible as this leads to a flickering image.

4.3. Corneal beam variation

It is well known that the depth of focus of a scanning system can be increased when decreasing the corneal beam radius. However, this comes along with a decrease of the in-focus image contrast as the retinal radius of a well focused beam increases due to diffraction effects.

Figure 5 illustrates this effect for the standard frequency f=6.2cyc/deg: The lower the corneal beam radius is chosen the higher is the depth of focus which results in a lower decrease of the contrast values when the eye changes its refractive power. But similarly, the contrast values for ΔD≲1.5D decrease.

To evaluate this trade-off, Fig. 6(a) shows the isolines for different contrast values as a function of ΔD and σ_cor . The bold line in Fig. 6(a) symbolizes the depth of focus (DOF) at CF=0.7, as defined above. As expected the DOF increases when reducing σ_cor as long as σ_cor ≥250µm. Reducing the corneal beam radius further, however, does not result in a further increase in DOF, but in a abrupt decrease as diffraction effects become more and more important. Thus, for each spatial frequency an optimum beam radius σ_cor exists where the DOF is maximized. As presented exemplarily in Fig. 6(a) the optimum for a resolution target of f=6.2cyc/deg lies at σ_cor ≈230µm where a maximum DOF of about 4D is achieved.

 

Fig. 5. The through-focus CF for different values of the corneal beam radius σ_cor . The spatial frequency f was set to the standard frequency f=6.2cyc/deg.

Download Full Size | PDF

 

Fig. 6. a) CF-isolines for f=6.2cyc/deg as a function of corneal beam radius and changes in eye accommodation. The CF-level values are indicated at the right edge. The bold line signifies the DOF at CF=0.7. b) The circles (∙) show the optimum corneal beam radius in terms of spatial frequency as explained in the text. The triangles (▾) indicate the corresponding maximum depth of focus. The dashed lines symbolize the discussed example at f=6.2cyc/deg.

Download Full Size | PDF

This analysis can be repeated for any spatial frequencies. The results are depicted in Fig. 6(b): The circles (lower plot, right axis) show the corneal beam radius at which the system is optimized for the corresponding spatial frequency. The triangles (upper plot, left axis) indicate the resulting DOF at that resolution and beam radius. It can clearly be seen that if the system should be optimized to higher spatial frequencies the corneal beam must be widened to reduce diffraction effects. This, however, results in a lower depth of focus. Consequently, Fig. 6(b) shows the fundamental limits of a retinal scanning technology with regard to the trade-off between large depth of focus and high resolution and thus, gives first important criteria for the display design.

5. Image field analysis

So far, only the contrast for the unrotated eye - and consequently for the central part of the scanned image - has been analyzed. Thus, the following section will focus on effects arising when gazing at peripheral points. Of special importance will be the maximum angle α which the eye can be rotated horizontally and vertically without a significant loss in image quality. These angles define approximately the maximum image field within which the eye can still completely recognize the image without any pupil tracking systems.

The simulation procedure is identical to the section before. However, in contrast to the previous simulations the beams of interest are no longer the beams from the middle of the image, but from peripheral points where the eye has to be rotated in order to gaze at. Furthermore, the eye pupil radius now affects the image quality as it might block the scanned beam. Consequently, the smaller the pupil, the more it affects the image quality when the eye rotates. Therefore, for any subsequent simulations the iris’ radius was set to 1.5mm which corresponds to the minimum value occurring in the human eye for normal ambient light conditions [5].

5.1. In-focus analysis

First, the effect of eye rotations on the in-focus CF is analyzed. When the eye is rotating the entering laser beams diffracts more and more at the iris edge causing a widening of the beam at the retina. For horizontal eye rotations the retinal beam is widened in x-direction, while for vertical rotations the same happens in y-direction, correspondingly.

 

Fig. 7. Isolines of the CF for ΔD=0 and with σ_cor =350µm as a function of spatial frequency and eye rotation. The CF level values are indicated in the plot. The iris’ radius was set to 1.5mm.

Download Full Size | PDF

Figure 7 shows the CF-isolines in terms of spatial frequency f and corresponding eye rotation α. The corneal beam radius was set to 350µm and the eye accommodation to 0D. Qualitatively similar patterns can be seen for any other values of beam radius. For low spatial frequencies (f≲6cyc/mm) the eye rotating has no significant effect on the contrast up to the limit of α_max ≈12deg where the laser beam begins to be blocked by the iris pupil. This results in an abrupt fall of the CF to zero. For higher spatial frequencies this abrupt decrease of the CF is less pronounced. In return, however, the decrease in the contrast starts already at smaller rotating angles. Analyzing this for all corneal beam radii leads to Fig. 8 where the vertical and horizontal maximum image field is plotted as a function of σ_cor for the standard frequency f=6.2cyc/deg. Note that the maximum image field is defined as twice the rotating angle α_max where the contrast value has decreased to 0.7.

 

Fig. 8. The maximum image field as a function of corneal beam radius for a retinal scanning system without pupil tracking. The data are evaluated for f=6.2cyc/deg with a iris’ radius of 1.5mm. The maximum image field is defined as twice the rotating angle where the CF has decreased to 0.7.

Download Full Size | PDF

5.2. Defocusing analysis

In a second step, the defocusing properties for peripheral points should be investigated. Thus, Fig. 9 shows the contrast values in x-direction for f=6.2cyc/deg in terms of eye accommodation and horizontal eye rotation for two selected corneal beam radii. For σ_cor =200µm the CF is rather constant for any eye rotations up to the maximum rotating angle α_max ≈9deg. As known from the in-focus analysis, at α_max the beams are blocked by the iris pupil resulting in a abrupt fall of the contrast and the loss of the scanned image. Note especially that the maximum rotating angle (and consequently the image field) are nearly independent of the eye’s refractive power. For σ_cor =350µm the CF looks differently: For small eye rotations the CF is also rather constant. However, for higher eye accommodation and for eye rotations close to the limit α_max the contrast values increase before falling abruptly. This comes from the fact that first only a part of the laser beam is blocked resulting effectively in a smaller beam radius and thus a higher depth of focus. For higher corneal beam radii this effect is even more pronounced. Again, the maximum rotation angle is not dependent on the eye accommodation.

 

Fig. 9. CF in horizontal direction in terms of eye rotation α and changes in eye accommodation ΔD for the standard frequency f=6.2cyc/deg.

Download Full Size | PDF

6. Final trade-off between resolution, DOF and image field

The results on the depth of focus from the previous section can finally be combined with the latest image field results. Figure 10 shows this fundamental trade-off between resolution, depth of focus and maximum image field for different spatial frequencies. For instance, for f=6.2cyc/deg the maximum DOF in x-direction is ≈4D and is reached - as seen in the Fig. 6(b)- for σ_cor ≈230µm, but with a limited image field of ≈18deg (see Fig. 8). This image field can be extended to e.g. ≈23deg at σ_cor ≈450µm, but at the expense of the depth of focus which decreases to 2.8D. Similar results have been evaluated for other frequencies.

 

Fig. 10. Trade-off between depth of focus (DOF) and maximum image field for three different spatial frequencies. The lines connect points of the same frequency, but with different σ_cor -values. In contrast the symbols indicate points with the same σ_cor -value at different frequencies. Unfilled symbols indicate data in x-direction, whereas filled symbols signify the y-direction. The selected frequencies correspond to the capital letter “E” of font sizes 10/14/18 pt when being viewed from 50cm.

Download Full Size | PDF

Further, it can be seen, that the DOF in y-direction is normally larger by approximately 1D due to the discrete scanning pattern in y-direction. This has beneficial effects on the the design process: e.g. a display optimized for f=6.2cyc/mm can be designed with an elliptical corneal beam shape (σ_x ≈230µm,σ_y ≈310µm) rather than with a circular one. Such a display provides a DOF of about 4D with an image field of 18.4×20.7deg. For higher spatial frequencies (e.g. f=8.7cyc/mm) the use of elliptical beam shapes has even more effect.

7. Conclusion and outlook

In conclusion, the following results of the analysis can be highlighted:

When the human eye ages, accommodation decreases gradually, so that corrective spectacles are needed to focus on closer objects. Further simulations have shown that such corrective spectacles do not affect the defocusing properties of the retinal scanning display significantly.

Although the analysis is based on an accurate and sophisticated eye model it is not able to replace real measurement with a corresponding display implementation and with the real human eye. In particular, some human eye factors such as the appearance of eye floaters - as reported in [10] for small exit pupils - are not considered in this analysis. Thus, to take all these additional factors into account, extended tests with a bench model and a statistical set of test people will be needed. Note further that by integrating a sophisticated pupil tracking system which adjusts the display’s exit pupil to the current eye’s position in real-time the image field constraint can be extended. However, this causes additional technical expenses and solves only the image field constraint, but not the trade-off between depth of focus and in-focus resolution.