1. Introduction

Virtual reality (VR) display blocks the normal sight to the real world and provides a virtual realistic scenario rendered by the computer to viewers. It has promising applications in commercial, constructional, medical and educational domains, such as video game, architecture visualization, and model visualization.

Since the first head-mounted-display (HMD) device had been developed in 1968, numerous VR schemes have been developed. These display technologies can be roughly classified into two types which are based on binocular parallax and light field. For binocular parallax based display technology, the representative devices are Google Glass and Oculus Rift, which provide a three dimensional scene by displaying different views to the left and right eyes. Nevertheless, since only two views are presented, the reconstructed virtual scene has vergence-accommodation conflict [1]. In recent years, the near eye light field display technologies have been developed to improve the amount of presented information to eliminate the vergence-accommodation conflict and enhance the immersion. Nvidia has introduced a multi-view based near eye light field display precept by employing a high-resolution OLED layer and a microlens array to reconstruct a light field [2]. Although it achieves an approximate monocular focusing effect, a tradeoff between the spatial resolution and the angular resolution is imposed owing to the limited light field information provided by a single layer of OLED, which leads to a low resolution performance. Based on fiber scanning, Schowengerdt. et al. has proposed a near-eye light field display design by using a fast scanning fiber to display multi-view scene [3]. The amount of information is limited by the characteristics of the fiber, which results in a tradeoff between refresh rate and resolution. In addition, a near eye light field display strategy utilizing multi-layer LCDs as light modulations to reconstruct the light field has been exploited [4]. This strategy factorizes the original light field data into some patterns displayed by these LCDs, which makes it possible for providing a large amount of information by a simple device. It has a very high information utilization and presents a high resolution light field supporting accommodation. However, the light field factorization of this method is computational complexity and time-consuming. Moreover, a significant latency causes degraded performance and adverse users’ sense of immersion [5, 6].

In order to realize near-eye 3D display with a good sense of immersion, it is necessary to quickly reconstruct a 3D scene with a large amount of information and accommodation effect by a simple hardware device. Inspired by the multi-layer light field display technology, we propose a near eye light field display design with super-fast synthesis speed based on human visual features.

Human visual acuity over visual field is inconsistent. The highest acuity is confined to the fovea which is responsible for sharp vision, and the large peripheral area delivers low resolution information of the surroundings [7]. In this paper, we propose to reconstruct a multi-resolution light field whose resolution distribution is consistent with human visual acuity distribution. The proposed system provides an immersive high resolution light field supporting retinal blur in high reconstruction speed. A human vision based algorithm is used to synthesize the light field. Experiment results show that the acceleration of the proposed scheme is evident and escalates when the spatial resolution increases. These key aspects of our design are verified by simulation and a prototype device.

2. The display principle

In this section, the principle of the proposed near eye light field display is discussed. In the first part, the original light field rendering method is introduced. In the second part, we explains the human vision based reconstruction algorithm, which shows the light field data can be compressed by setting its resolution distribution identical to human visual acuity. The reconstruction speed can be accelerated by synthesizing a multi-resolution light field.

2.1The light field rendering

The general steps of multi-layer display technology are rendering or capturing the original light field data, then factorizing the light field into 2D patterns, and lastly displaying these patterns by multi-layer LCDs display equipment.

The Fig. 1 explains the arrangement of viewpoints. The origin of coordinate system is located at the center of the eyeball. The scene is located in front of the observer along Z axis. A reference plane is arranged for original light field rendering. The position of light ray is described by its intersection on the reference plane and the viewpoint. As Takaki et al. mentioned [8], there should be at least two light rays falling into the eye to achieve accommodation. On account of that, several viewpoints over the eye are arranged as shown in Fig. 1. For the purpose of simulating the distribution of the viewpoints in different viewing directions, the viewpoint is approximated as the point distributed on the eyeball, as shown in Fig. 1(b). The viewpoints are distributed at equal angle gap between each other in X and Y direction, respectively. The position of each viewpoint ($x_e,y_e,z_e$) is described by the Eqs. (1)-(3).

 

Fig. 1 The arrangement of viewpoints for original data rendering. (a) Illustration of the original data rendering in z-x plane. (b) Three-dimensional schematic of the arrangement of viewpoints, the eyeball is sketched as a sphere, and the schematic viewpoint located on the circular section of the eyeball.

Download Full Size | PDF

2.2 The factorization of light field

In this paper, the display system consists of a backlight, dual-layer LCDs and a lens, wherein the lens images the LCD and backlight as a far-enlarged virtual LCD and backlight, respectively (Fig. 2). The imaging relationship can be described as

 

Fig. 2 Illustration of the proposed display system

Download Full Size | PDF

The components of a conventional dual-layer display are dual-layer LCDs and backlight [9]. The front LCD and the rear LCD modulate these light rays emitted by the backlight to create a discrete light field $L[k,l,i,j]$, where ($k,l$) and ($i,j$) correspond to the pixel indices of the front LCD and the rear LCD, respectively. The matrixes $P_f[k,l]$ and $P_r[i,j]$ correspond to the display patterns of front LCD and rear LCD, respectively. The light field $L[k,l,i,j]$ can be expressed as the outer product of the matrixes $P_f[k,l]$ and $P_r[i,j]$, as described by Eq. (5).

The light field $L[k,l,i,j]$ is imaged by the lens to form a magnified light field $L_m[k,l,i,j]$, which can be written as

The update rule of factorization is given by [10]

 

Fig. 3 Geometric relationship of our proposed system. (a)Illustration of the geometric relationship in z-x plane (b) Magnification of inset

Download Full Size | PDF

2.3 Human visual features

It is a universal experience that we can obtain the sharp detail in the viewing direction, but the surrounding is less distinct. In fact, human gets part of information of the whole scene at one moment and changes viewing direction to obtain the whole information. Visual acuity represents the ability of visual resolution, which declines with the increasing visual eccentricity [11]. Visual acuity can be expressed as the resolution of different sizes of letters at different visual eccentricities. By measuring reading speed about a sequence of letters with distinct sizes at central vision and other visual field at different eccentricity, Chung et al. indicates that reading speeds should remain invariant with visual eccentricity, as long as the print is appropriately scaled in size [12]. That paper summarized that the angular resolution at the maximum reading speeds formulated as a function of visual eccentricity

 

Fig. 4 Relationships between the critical point size and the eccentricity. The red line shows the linear relationship between critical print size (minute) corresponding to maximum reading speed and the eccentricity. The green polyline is the discrete function applied in this paper.

Download Full Size | PDF

2.4 The reconstruction algorithm based on human visual features

It is different from the light field with inherent uniform resolution that the number of pixels per row in each view is not the same in multi-resolution light field. Therefore, the original light field should be sampled owning to the multi-resolution, and the sampling information of each view is collected to form a matrix.

The Fig. 5 shows the sampling process of each view. As shown in Fig. 5(a), for every view, firstly, the entire visual field is divided into N sub-regions according to discrete $R_a$ and the corresponding visual eccentricity $E_{cc}$. Secondly, for each sub-region, the sampling unit size is equal to the spatial resolution derived by the corresponding $R_a$. The sampling information in each sub-region is recorded in a column-wise manner. Lastly, the sub-region’s information is arrayed from outside to inside to form a new matrix $S(p,q)$(the compressed image), where the $(p,q)$ is the pixel index of matrix S. The number of the row and column of S is equal, and the extra pixels value of S are set as zero. For every view, $S(p,q)$ is the matrix that records the value of sampling points. All the $S(p,q)$ of each view are gathered as the compressed light field ${\tilde{L}}^{\text{'}}(p,q,n_x,n_y)$. The process of converting $\tilde{L}(m_r,m_c,n_x,n_y)$ to ${\tilde{L}}^{\text{'}}(p,q,n_x,n_y)$ is defined as the light field compression.

 

Fig. 5 The compression process of $\tilde{\text{L}}(m_r,m_c,n_x,n_y)$. (a) The illustration of sampling perceived image of $\tilde{\text{L}}(m_r,m_c,n_x,n_y)$. (b) The illustration of $S(p,q)$.

Download Full Size | PDF

The whole compressed light field is ${\tilde{L}}^{\text{'}}(p,q,n_x,n_y)$ which can be factorized as the compressed front LCD $P_{mf}^{\text{'}}\left(p_f,q_f\right)$ and rear LCD $P_{mr}^{\text{'}}(p_r,q_r)$, where the $\left(p_f,q_f\right)$ and $(p_r,q_r)$ correspond to the index of pixel of $P_{mf}^{\text{'}}\left(p_f,q_f\right)$ and $P_{mr}^{\text{'}}(p_r,q_r)$.

We construct a map image $S1(p,q)$ of the reference plane to record the original index ($m_r,m_c$) of the sampling point, which allows us to leverage GPU for fast sampling. The construction process of the $S1(p,q)$ is the same as $(p,q)$ . By contrast, the information to be recorded in $S1(p,q)$ is the original index ($m_r,m_c$) of the sampling point instead of the value of sampling point. The map image $S1(p,q)$ of $P_{mf}(k,l)$ and $P_{mr}(i,j)$ is formed in the same way as the generation process of $S1(p,q)$of the reference plane. Furthermore, we construct a map image $S2(k,l)$ of $P_{mf}(k,l)$ to record the index $(p_f,q_f)$ of sampling point on $P_{mf}^{\text{'}}(p_f,q_f)$. The map image $S2(i,j)$ of $P_{mr}(i,j)$ is formed in the same way as $S2(k,l)$. The map images $S1(p,q)$ and $S2(p,q)$ are utilized in the reconstruction process.

As mentioned in section 2.2, in order to factorize $\widetilde{L^{\text{'}}}$ by the Eqs. (7)-(9), the indexes of the sampling light rays’ corresponding pixels on the compressed front LCD $P_{mf}^{\text{'}}\left(p_f,q_f\right)$ and rear LCD $P_{mr}^{\text{'}}(p_r,q_r)$ are necessary. However, this correspondence cannot be directly derived by Eqs. (10) and (11). Figure 6 explains the process of obtaining the correspondence.

 

Fig. 6 The process of searching the corresponding pixels of $\widetilde{L^{\text{'}}}(p,q,n_x,n_y)$ on $P_{mf}^{\text{'}}(p_f,q_f)$ and $P_{mr}^{\text{'}}(p_r,q_r)$. (a) The original index ($m_0,n_0$) of sampling points B on reference plane is acquired by the $S1(p,q)$ of reference plane, all of the original index of sampling points can be obtained in this way. (b) Since light ray is positioned by the index of sampling point B and the viewpoint, the location of intersections on multi-layer LCD is gained by geometric mapping. The intersection on front layer is illustrated as A($i_1,j_1$). (c) The index of A($i_2,j_2$) on $P_{mf}^{\text{'}}(p_f,q_f)$ is recorded by the pixel indexed as ($i_2,j_2$) on $S2(k,l)$ . The index of A on $P_{mf}^{\text{'}}(p_f,q_f)$ is illustrated as ($m_2,n_2$).Index of C($i_2,j_2$) on $P_{mf}^{\text{'}}(p_f,q_f)$ can be obtained by the same way. (d) The corresponding pixel A on $P_{mf}^{\text{'}}(p_f,q_f)$ of light ray B is obtained by the index ($m_2,n_2$), the pixel C can be acquired by the same way.

Download Full Size | PDF

With regard to each view, firstly, the index ($m_0,n_0$) of sampling point B in original light field is obtained by the map $S1(p,q)$ of the reference plane (Fig. 6(a)), and the light ray is positioned by B and the viewpoint. Secondly, the intersections of light ray B on $P_{mf}(k,l)$ and $P_{mr}(i,j)$ corresponding to $\text{A}(i_1,j_1)$ and $\text{C}(i_2,j_2)$ can be calculated by Eqs. (10) and (11)(Fig. 6(b)). Thirdly, the index ($m_1,n_1$) of A on $P_{mr}^{\text{'}}(p_r,q_r)$ can be obtained by map $S2(k,l)$ (Fig. 6(c)). The index ($m_2,n_2$) of C on $P_{mf}^{\text{'}}(p_f,q_f)$ can be obtained in the same way by utilizing the map $S2(i,j)$ of $P_{mr}(i,j)$.In this way, the $\widetilde{L^{\text{'}}}(p,q,n_x,n_y)$s corresponding sampling points on $P_{mf}^{\text{'}}(p_f,q_f)$ and $P_{mr}^{\text{'}}(p_r,q_r)$ can be ascertained. By combining this ascertaining method and the Eqs. (7)-(9), the $\widetilde{L^{\text{'}}}(p,q,n_x,n_y)$ can be quickly factorized as $P_{mf}^{\text{'}}(p_f,q_f)$ and $P_{mr}^{\text{'}}(p_f,q_f)$.

The factorization patterns can be displayed until the $P_{mf}^{\text{'}}(p_f,q_f)$ and $P_{mr}^{\text{'}}(p_r,q_r)$ are converted to the multi-resolution form $P_{mf}(k,l)$ and $P_{mr}(i,j)$, respectively. The conversion process is defined as decompression which is shown in Fig. 7. It is fast to get the pixel value of $P_{mf}(k,l)$ and $P_{mr}(i,j)$ stored in $P_{mf}^{\text{'}}(p_f,q_f)$ and $P_{mf}^{\text{'}}(p_r,q_r)$ by the map image $S2(k,l)$ and $S2(i,j)$ respectively. In summary, the whole algorithm flow of factorizing the $\tilde{\text{L}}(m_r,m_c,n_x,n_y)$ into multi-resolution patterns $P_{mf}(k,l)$ and $P_{mr}(i,j)$ is depicted in Fig. 8.

 

Fig. 7 The decompression process of the $P_{mf}^{\text{'}}(p_f,q_f)$ or $P_{mr}^{\text{'}}(p_r,q_r)$. (a) The $S2(k,l)$ of $P_{mf}(k,l)$, or $S2(i,j)$ of $P_{mr}(i,j)$. (b) $P_{mf}^{\text{'}}(p_f,q_f)$ or $P_{mr}^{\text{'}}(p_r,q_r)$. (c) $P_{mf}(k,l)$ or $P_{mr}(i,j)$

Download Full Size | PDF

 

Fig. 8 The proposed reconstruction algorithm flow

Download Full Size | PDF

As discussed above, the $S1(p,q)$ and $S2(k,l)$ are used to record the indices of pixels on original light field or the LCD and the compressed light field, respectively. The information is recorded as image format to take full advantage of efficient GPU implementations. However, since the recorded index may be larger than 255, the format of the map image is set as portable network graphic format (.png) to insure the veracity. The relationship between the RGBA of the map images and recorded indices of the pixels is described by Eqs. (13)-(16).

3. Experiment

In this section, the details of constructing the near-eye light field display based on visual features are explored. Furthermore, we implement a comparison on acceleration between the conventional dual-layer LCD light field display and our method.

3.1 Hardware and software

We built a prototype (Fig. 9) to validate the proposed approach. The prototype includes a dual-LCDs with pixel size of 51um, a uniform backlight and a lens. The focal length of the lens is 54.25mm, and its diameter is 40mm. A light field with a resolution of 608*760 and 5*5 views is set as a target. The reconstructed light field is observed by a CCD with an FOV of ${54}^{\circ }$.

 

Fig. 9 Different perspective views of the experimental prototype.

Download Full Size | PDF

According to the optical imaging formula, the two physical LCDs are located at distances of 48.051mm and 53.355mm behind the lens, and the virtual LCDs are imaged at 370.00mm and 2000.00mm, respectively. The lens is placed at 21.8mm away from the CCD. The original light field data rendering, light field factorization, and display are wrote by GLSL and C + + . The algorithms are running on an Intel Core i3 PC with Intel HD Graphics 4400 GPU. A model composed of 4 cubes marked as a, b, c and d is exploited in our system to demonstrate the accommodation and multi-resolution. Their relative position is shown in Fig. 10. The distances from cube a, b, c and d to the observer are equal to 370mm, 770mm, 1185mm, and 2000mm, respectively.

 

Fig. 10 The geometric positions of the model for display. (a) The illustration of geometric positions in X-Y plane. (b) The illustration of geometric positions in Z-X plane.

Download Full Size | PDF

The light field is synthesized by the proposed reconstruction algorithm. We took 4 sets of images of the synthesized light field when observer gazed at 4 cubes and kept the corresponding cube within depth of field, respectively.

3.2 Optical distortion correction

Generally speaking, there is distortion degrading the display fidelity in optical system. The general form of the distortion is expressed as a combination of basis radial functions and tangential components [13], which can be comprehended as the form of nonlinear polynomial. Pre-distortion pattern generated by the computational distortion correction can compensate the distortion. A geometric mapping polynomial relationship between the optical object, ideal image and distorted image is established in this paper. The input images displayed on dual-LCDs need to be observed as the ideal images when applied the pre-distortion.

The distortion correction process is conducted as follow: firstly, a calibration image composed of regularly arranged grid points is created. Then this calibration image is displayed the on front LCD, and the rear LCD is set as white to obtain a distortion image. Another distortion image is also required by exchanging the display image of the front and real LCDs. The mapping relationship between optical object and distorted image is set up as

The ($x_i$, $y_i$) and ($u_i$, $v_i$) are the coordinates of the grid point and the corresponding distorted point, respectively, m and n are the powers of the polynomials, $k_{xmn}$ and $k_{ymn}$ are the coefficients of corresponding polynomials, which can be calculated using least squares method.

Since the distortion near the optical axis is minimal, the central region of the distortion image can be taken as the linear magnified image of the same region of the calibration image. The ideal image of the object is a linear magnified image. The magnification is calculated by the position of the central grid points of calibration image and the distortion image. An ideal magnified input image is acquired by applying the magnification to the input image. Afterwards we applied Eqs. (17) and (18) to the ideal input image to get the pre-distortion image, as described by Eqs. (19) and (20).

Here, ($x_i^{\text{'}}$, $y_i^{\text{'}}$) and ($u_i^{\text{'}}$, $v_i^{\text{'}}$) are the coordinates of the pre-distortion and the ideal input image, respectively. The distortion of dual-layer LCDs system is compensated by applying the approach to both of the LCD layers.

4. Results and analysis

4. 1 Accommodation

Depth discrimination is a fundamental issue for 3D display technology. Accommodation elicited by retinal blur can improve depth discrimination performance. The experimental results, shown in Fig. 11, demonstrate the ability of the proposed approach to achieve retinal blur. In Fig. 11(a), a set of imageries were taken by altering CCD’s focal depth from 370mm to 2000mm. The result shows that each object is in reasonable focus when the CCD is focused at the intended focal depth, and out of focus when the CCD is focused elsewhere.

 

Fig. 11 The sharpness of the perceived image of reconstructed light field varies with the focus depths and viewing directions. (a) The left column illustrates the discussed cubes. The perceived images in the right columns are the magnified insets in left boxes, which are taken when CCD focuses at different depths. (b) These experiment results are the magnified insets as illustrated in Fig. 11(a)’s left column. Perceived images of cube a and d recorded within depth of field but with different viewing directions. The images in the left and right columns are captured within and out of fovea, respectively. E.g. the top-right image was captured when CCD viewed toward cube c and kept the cube a within focus. So the cube is out of fovea and within focus.

Download Full Size | PDF

4.1 The Perceived images

The perceived sharpness of the real world is varied with the observer’s focal depth and viewing direction. Figure 11(b) are perceived images of cube a and d recorded within depth of field but with different viewing directions. Figure 11(b) demonstrates the sharpness contrast result of the images captured with varying view direction when the cube is within depth of field. It indicates that although the object is within depth of field, the sharpness of perceived scene is different when the viewing direction is varied. As a consequence, the sharp images are perceived unless the objects are located in reasonable focus and within fovea. This property is coincident with the human visual features.

The Fig. 12 shows the perceived results when camera is focus at the gazed cube. The result shows that the object within fovea is perceived sharply, by contrast, the perceived images in the peripheral part is low resolution and distinct. The Fig. 13 is the maps when the viewer gazed at the cube a. The Fig. 14 is the example multi-resolution layer patterns which is reconstructed as acceleration 3.47 times by proposed method compared to the conventional algorithm. Combined with Fig. 12 (a) and the division of light field which reflected by map S2 in Fig. 13, it is clear that the sharpness of the reconstructed light field declines with the increasing visual eccentricity. By comparing Fig. 12(a) with Fig. 12(d), it suggests that the image sharpness of the gazed cube d is lower than cube a. It is because the image quality of multi-layer LCDs is influenced by diffraction effect [4, 14, 15].

 

Fig. 12 Perceived images when corresponding cube within focus and within fovea.

Download Full Size | PDF

 

Fig. 13 Maps of each plane when gazing at cube a. (a) Maps of LCD1. (b) Maps of LCD2. (c) Maps of reference plane. The top and the bottom row correspond to S1 and S2, respectively.

Download Full Size | PDF

 

Fig. 14 Factorized patterns of multi-valued light field when the viewer concentrates at cube a. (a) The Factorized pattern for front LCD (b) The Factorized pattern for rear LCD

Download Full Size | PDF

4.3 Acceleration

Time consumption is an important issue for VR. In order to estimate the computational acceleration of this method, two sets of comparison experiments were conducted. To analyze the acceleration with different spatial resolution, a set of experiments were conducted by altering the spatial resolution with constant viewpoints. Ten light fields with 2x2 viewpoints and different spatial resolution were reconstructed by the proposed algorithm and conventional dual-layer reconstruction algorithm. These spatial resolutions were designed as different integer multiples of 276*276. The runtime was measured for 5 iterations, and the recorded time was the runtime for one iteration.

These results are plotted by Figs. 15 and 16. Figure 15 plots the runtime of two algorithms with different spatial resolution. Figure 16 plots the runtime’s ratio of conventional algorithm to proposed method with different spatial resolution, which reflects the acceleration directly. To summarize, with the increasing spatial resolution, the time consumption of conventional algorithm compared with the proposed algorithm increases rapidly, and a higher resolution results a faster acceleration.

 

Fig. 15 The time consumption of two algorithm with different spatial resolution and 2*2 viewpoints.

Download Full Size | PDF

 

Fig. 16 The time consumption ratio of conventional algorithm to the proposed algorithm with different spatial resolution and 2*2 viewpoints

Download Full Size | PDF

To discuss the influence of the angular resolution and spatial resolution on acceleration, a set of experiments were manipulated by altering angular resolution and spatial resolution with the amount of light field information as a constant. Table 1 shows the runtime taken by these two methods for one iteration of each light field reconstruction. It shows that the proposed method supports an obvious acceleration. In addition, a more prominent acceleration is gained with higher spatial resolution. In conclusion, the acceleration of the proposed scheme is evident and increases rapidly when the spatial resolution increases.

Table 1. The runtime of two methods for four light fields.

View Table

As discussed previously, the proposed display system supports sharp image in fovea and gradually blurred scene at the peripheral area. Human changes the viewing direction to bring the attention part into human’s fovea vision when viewer watches a scene. This watching behavior is comprised of fixation and saccades [16]. Saccades is fast eye movement. Saccades is not continuous, and Fixation is the interval between saccades. The general duration of fixation is 200~300ms [17]. In this paper, the reduced time consumption is much less than that time. It indicates that we can achieve a high resolution near eye light field display without latency by taking full use of that time to update the entire scene when the viewing direction is changed.

5. Conclusion

VR display as the next-generation display technology is gaining prevalence. For the near-eye light field display, updating the displayed images in accordance with the rotation of the eye or head is necessary, because it is hardly to keep the head or eye motionless. What’s more, high resolution without latency or low-latency is a crucial issue for immersion, which still has not been adequately addressed yet. In this paper, we introduced a new near eye light field display proposal that reconstructing a multi-resolution light field whose resolution distribution is consistent with human visual features for corresponding viewing direction in a super-fast synthesis speed. A human vision based reconstruction algorithm is introduced to reconstruct a high resolution light field with support accommodation. The prototype and experimental results indicates the reconstruction speed of the proposed system is accelerated obviously comparing to the conventional reconstruction system. By this method, it is possible to realize real time multi-layer light field near eye display with higher image quality with an eye tracking device.