Open Access
Add to BibSonomyAbstract
A head-mounted display system with fully-integrated eyetracking capability offers multi-fold benefits, not only to fundamental scientific research but also to emerging applications of such technology. A key limitation of the state-of-the-art eyetracked head-mounted display (ET-HMD) technology is the lack of compactness and portability. In this paper, we present an innovative design of a high resolution optical see-through ET-HMD system based on freeform optical technology. A prototype system is demonstrated, which offers a goggle-like compact form factor, non-obstructive see-through field of view and true high-definition image resolution for the virtual display. The see-through view, via the combination of a freeform prism and corrector, achieved better than 0.5 arc minute of angular resolution for the central region of approximately 40-degrees to ensure minimal impacts on the see-through vision of an HMD user.
©2013 Optical Society of America
1. Introduction
Optical see-through head-mounted display (OST-HMD) is an enabling technology to augmented reality (AR), allowing superposition of digital information onto the real-world view, while optically maintaining see-through view. Along with the increased bandwidth of wireless networks and the prevailing cloud computing, a lightweight and compact OST-HMD is considered as a transformative technology which may penetrate through many fields of applications, ranging from military training, medicine practice, to education and entertainment. In recent years a few promising commercial OST-HMDs, such as Google Glass (http://www.google.com/glass/start/), demonstrated very compact and lightweight form factors and the high potential of widespread public use. These emerging commercial OST-HMDs, however, are limited by their fairly narrow display field of view (FOV) and low image resolution. For instance, the current version of Google Glass has approximately a 15° FOV with an image resolution of 640x360 pixels. A compact OST-HMD with a much wider FOV and higher resolution is required to fully exploit the full range of benefits afforded by AR technologies. Moreover, none of these existing displays provides a user interface that enables hands-free interaction with displayed information, which can be essential for many wearable applications.
A lightweight, high-resolution OST-HMD system with fully integrated eyetracking capability, referred to as an eyetracked head-mounted display (ET-HMD), has long been desired for many fields of applications. It is able to display virtual images as a classical HMD does, while additionally tracking the gaze direction of the user. An ET-HMD system offers not only a means of hand-free interaction with the display, but also multi-fold benefits to a broad range of other applications. For instance, eyetracking capability may help to create HMD solutions addressing challenges such as the FOV-resolution tradeoff through fovea-contingent display schemes [1] and the accommodation-convergence conflict in stereoscopic HMDs by using vari-focal plane display methodology [2]. Moreover, an ET-HMD offers unique opportunities for novel interactive interfaces for people with proprioceptive disabilities where eye gaze instead of hands or feet can be used as a method of interaction and communication. Finally, an ET-HMD system adds a very valuable tool for scientists to quantitatively assess the effectiveness of various 3D visualization technologies for various specific tasks including training, education, and augmented cognition tasks.
The development of ET-HMD technology has been previously explored in various levels. For instance, Duchowski integrated an eyetracker by ISCAN Corporation with a V8 HMD by Virtual Research Corporation to study software-based fovea-contingent display scheme [3]; Vaissie and Rolland [4] and Hua [5] explored the benefits of a fully integrated design approach with low-level optimization and developed robust eyetracking methods and algorithms [6,7]; and Waternberg [8] presented an ET-HMD prototype based on a bi-directional microdisplay which embeds photodiodes within the pixel array of an organic light emitting display (OLED) panel to enable the functionality of display and eye image capture via the same device. Despite the significant advancements in stand-alone HMD and eyetracking technologies over the past decades, integrating these two stand-alone technologies remains challenging in terms of creating a compact, portable, and robust system. Few of the existing approaches offer a truly portable and lightweight system that conforms to the form factor of an eyeglass-style display.
In this paper, we present a novel design and implementation of a lightweight, high-resolution, optical see-through ET-HMD based on freeform optical technology. Our approach allows elegantly combining the display and eye imaging paths through a monolithic freeform wedge-shaped prism without adding significant weight or volume to the HMD system.
2. Optical design
The optical system layout of our ET-HMD system is shown in Fig. 1. It consists of a freeform wedge-shaped prism, a freeform corrector, and a singlet imaging lens. The freeform wedge-shaped prism is the key element consisted of three freeform optical surfaces, S1-1', S2, and S3. Though this type of freeform prism is well-known for its use as an eyepiece for HMD systems [9,10], to our best knowledge, for the first time the prism was adopted to serve as a core element shared by four unique optical paths: eye illumination, eye imaging, virtual display, and real-world see-through paths [11]. First, the prism serves as an illumination optics that collimates the light from one or multiple near infrared (NIR) light emitting diodes (LED). The NIR LEDs uniformly and non-invasively illuminate the eye area and form critical features (e.g. glints and darkened pupil) that are to be imaged for eyetracking. Secondly, the same freeform element serves as the core element in the eye imaging subsystem that captures NIR-illuminated eye images of a user and tracks eye movements using the captured eye images. Unlike a conventional imaging system, which typically employs rotationally symmetrical optical surfaces in the lens construction and typically requires the imaging lenses remain collinear with the detector and the objects to be captured, the freeform prism folds the light path within a single element so that the image detector is placed on the side of the element. This innovation enables an accurate and robust eyetracking capability. Thirdly, the prism serves as a single-element HMD eyepiece for viewing images on a microdisplay. Finally, the prism, when cemented with a freeform corrective lens to correct the viewing axis deviation and undesirable aberrations introduced by the prism, enables see-through capability of the system which offers low peripheral obscurations and negligible distortions to the real-world view. Overall, our unique optical scheme enables us to combine the optical paths for eye illumination, eye imaging, and HMD eyepiece through the same freeform prism and achieve the capabilities of eyetracking and display without additional hardware cost.
Fig. 1 Optical system layout of ET-HMD: (a) virtual display and optical see-through paths; (b) eye illumination and imaging paths
Based on this approach, a high-resolution optical see-through ET-HMD system was recently designed and implemented. The virtual display system (rays traced in solid lines in Fig. 1a) adopts a high-definition OLED microdisplay with 1920 by 1200 color pixels and a pixel size of 9.6um, which affords a spatial resolution up to 52lps/mm in the microdisplay space. The field of view (FOV) of the virtual display path is 45° diagonally, or 40° by 22° in the horizontal and vertical directions, respectively, which offers a virtual display of approximately 65 inches at a 6 feet distance, three times of the current Google Glass. In the visual space, the virtual display equivalently affords an angular resolution of 1.25 arc minutes per pixel. The effective focal length of the prism is 25.5mm. The eye clearance of the prism is 18.9mm while the eye relief is 20mm along the visual axis. The exit pupil diameter (EPD) of the virtual display is 10mm without vignetting and 13mm with 80% vignetting for the diagonal corner fields, which supports an eyebox as large as 14mm by 14mm. Such a large eyebox ensures good tolerance to eye movements and to the difference of interpupillary distance (IPD) among different users. The F-number of the system is approximately 1.9.
In the see-through path (rays traced in dashed lines in Fig. 1a), a freeform corrector was designed to achieve an elliptical see-through FOV of 80° horizontally by 50° vertically. The see-through FOV is much wider than that of the virtual display to minimize visual obstruction to the real-world view. The combination of the freeform prism and corrector was optimized to yield better than 0.5 arc minute of angular resolution for the central region of approximately 40-degrees to ensure minimal impacts on the see-through vision of an HMD user. It was further optimized for an 18mm non-vignetted EPD to ensure comfortable viewing of the real world. Low distortion is ensured for the central region of 50 degrees.
In terms of eye illumination and imaging, multiple low-power NIR LEDs of 850nm wavelength were selected to illuminate the eye through the prism (rays traced in dotted lines in Fig. 1b). Approximately 20° of light rays emitted by each of the NIR LEDs were collected and collimated by the prism to create a uniformly illuminated eye area of 30mm by 20mm. In the eye imaging path (Rays traced in dot-dash lines in Fig. 1b), the same illuminated eye area was then captured through the prism and a singlet imaging lens by a 1/2.5” NIR sensor (XIMEA MU9PM_MBRD subminiature camera) which has a maximum frame rate of 60 fps at 720p resolution. The F-number of the eye imaging path is approximately 3.5 which offers an overall high throughput in the NIR band.
The freeform prism was strategically optimized to achieve high optical performance in a broad band from 400nm to 900nm. The first reflection of light from microdisplay by surface 1 of the prism satisfies total internal reflection (TIR) in both visible and NIR spectrum, while a beamsplitting coating is required for the second reflection by the surface 2 of the prism.
The optical design of the ET-HMD system was very challenging due to the requirements for a large FOV and high resolution in both the virtual display and see-through paths, low F-number in the virtual display path, and the unique requirements for a broad spectrum and the integration of four optical paths through a single prism. Particularly, the internal reflection via the surface 1' of the prism needs to satisfy TIR conditions for both the visible and NIR spectrum.
Figure 2(a) shows the field plots of the polychromatic modulation transfer function (MTF) of the virtual display path with a 3-mm centered pupil for the spatial frequency of 25 lps/mm and 40 lps/mm, denoted with red and green circles, respectively. Three representative wavelengths, 625nm, 550nm, and 460nm, were utilized with weights of 2, 3 and 2, respectively, for calculating the MTF values. 52lps/mm is the cut-off frequency of the OLED microdisplay utilized in the design. The sizes of the circles are proportional to the average of the MTF values in tangential and sagittal directions. Across the entire FOV, the average MTF value is 0.41 at 25 lps/mm and 0.32 at 40lps/mm, respectively. Additionally, we characterized the MTF performance of the virtual display with a 3-mm pupil offset from the center of the exit pupil by 1mm and 2mm along +/−y and +/−x directions, respectively. The average and standard deviation of the MTF values were summarized in Table 1.
Table 1. MTF Performance (Average/Standard Deviation) at Different Pupil Positions
Figure 2(b) shows the polychromatic MTF field plot of the see-through path with a 3-mm pupil for the spatial frequency of 0.5 cycles/arcminutes and 1.0 cycles/arcminutes, which corresponds to an angular resolution of 1 arcminute and 0.5 arcminute in the visual space, respectively. Same wavelengths were used for computing the see-through MTF values as those for the virtual display path. At normal visual acuity of 1 arcminute, the average MTF value of the see-through path is larger than 0.45 for the central 40-degrees of FOV.
3. System prototype
Figure 3(a) shows the prototype of the optics alone and Fig. 3(b) shows the integrated system prototype. Each of prism-compensator assembly measures approximately 55mm in width, 45mm in height, and 19mm in depth. Figure 3(c) shows an image of the virtual display captured with a wide-angle camera placed at the exit pupil position. It demonstrated superb optical quality. The noticeable distortion in the captured image demonstrated residual distortion of the system design as an expense to trade for higher image resolution. In Fig. 3c we applied neither electronic distortion correction nor lateral color correction to demonstrate the native image quality. However, practically both residual distortion and lateral color aberration can be digitally corrected using a pre-warping process.
Fig. 3 (a) freeform wedge prism prototype, (b) ET-HMD prototype (c) Display image captured with a camera placed at eye position.
Figure 4(a) shows an image of the see-through view captured with a camera for a resolution target placed at 2.5 feet away covering the FOV of approximately 40 degrees diagonally. Figure 4(b) shows an image of the see-through view for viewing a standard Snellen chart at 20 feet. The camera was configured to have an angular resolution of 0.23 arcminutes per pixel. Letters from Group 8 (20/20 vision) to Group 11 (20/10 vision) are clearly readable. These results demonstrated the see-through optics caused negligible degradation and distortion on the real-world view across a wide angle.
Fig. 4 (a) Captured see-through view of 40-degree resolution target, (b) Standard eye chart at 20ft seeing through the freeform prisms. (c) Captured eye image under IR illumination for eye-gaze tracking.
Figure 4(c) shows an image of the eye under NIR illumination captured by the integrated IR camera. The dark pupil and the glint reflections of the IR sources are the main features to be utilized for computing eye gaze direction.
4. Conclusion
In conclusion, we developed a high-resolution compact optical-through eyetracked HMD system. The system utilizes a wedge-shaped prism to combine four unique optical paths, virtual display, see-through, eye illumination, and eye imaging, which resulted in a very compact form factor for the entire system. With true HD-resolution microdisplays, the virtual display of the prototype system yields an angular resolution of 1.25 arc minutes per pixel, along with a 10mm nonvignetted EPD or 14mm EPD with vignetting. The see-through optics offered a FOV as wide as 80 degrees by 50 degrees and a spatial resolution better than 0.5 arc minutes, ensuring minimal obstruction to the real-world view. Compared to the prior art, the integrated eyetracking capability adds minimal weight and volume to the entire system. We are currently adapting this technology as an augmentative assistive communication device for patients with severe proprioceptive disability, such as ALS conditions.
For many demanding applications, lightweight and compactness are critical. For instance, in one of our applications to support patient communication with proprioceptive disabilities, the integrated system has to be lightweight so that the patients are able to bear the weight with their significantly weakened muscles and very limited mobility.
Acknowledgments
This work is funded by National Science Foundation Grant Awards 1115489 and 0915035. The authors would like to acknowledge Dr. Yuxiang Lin for his contribution in designing the mechanical housing and frames for our prototype system.
References and links
1. J. P. Rolland, A. Yoshida, L. D. Davis, and J. H. Reif, “High-resolution inset head-mounted display,” Appl. Opt. 37(19), 4183–4193 (1998). [CrossRef] [PubMed]
2. S. L. S. Liu, H. H. H. Hua, and D. C. D. Cheng, “A novel prototype for an optical see-through head-mounted display with addressable focus cues,” IEEE Trans. Vis. Comput. Graph. 16(3), 381–393 (2010). [CrossRef] [PubMed]
3. A. T. Duchowski and A. Çöltekin, “Foveated gaze-contingent displays for peripheral LOD management, 3D visualization, and stereo imaging,” ACM Trans. Multimed. Comput. Commun. Appl. 3(4), 1–18 (2007). [CrossRef]
4. L. Vaissie and J. Rolland, “Head mounted display with eyetracking capability,” U.S. patent US 6433760 B1 (2002).
5. H. Hua, “Integration of eye tracking capability into optical see-through head-mounted displays,” Proc. SPIE 4297, 496–503 (2001). [CrossRef]
6. H. Hua, P. Krishnaswamy, and J. P. Rolland, “Video-based eyetracking methods and algorithms in head-mounted displays,” Opt. Express 14(10), 4328–4350 (2006). [CrossRef] [PubMed]
7. H. Hua, C. W. Pansing, and J. P. Rolland, “Modeling of an eye-imaging system for optimizing illumination schemes in an eye-tracked head-mounted display,” Appl. Opt. 46(31), 7757–7770 (2007). [CrossRef] [PubMed]
8. J. Baumgarten, T. Schuchert, S. Voth, P. Wartenberg, B. Richter, and U. Vogel, “Aspects of a head-mounted eye-tracker based on a bidirectional OLED microdisplay,” J. Inf. Disp. 13(2), 1–5 (2011).
9. S. Yamazaki, K. Inoguchi, Y. Saito, H. Morishima, and N. Taniguchi, “Thin wide-field-of-view HMD with free-form-surface prism and applications,” Proc. SPIE 3639, 453–462 (1999). [CrossRef]
10. D. Cheng, Y. Wang, H. Hua, and M. M. Talha, “Design of an optical see-through head-mounted display with a low f-number and large field of view using a freeform prism,” Appl. Opt. 48(14), 2655–2668 (2009). [CrossRef] [PubMed]
11. H. Hua and C. Gao, “A compact eyetracked optical see-through head-mounted display,” in Proc. SPIE 8288, Stereoscopic Displays and Applications XXIII, A. J. Woods, N. S. Holliman, and G. E. Favalora, eds. (2012), Vol. 8288, p. 82881F–82881F–9.
