Abstract
Compressive light field display with multilayer and multiframe decompositions is able to provide three-dimensional (3D) scenes with high spatial-angular resolution and without periodically repeating view-zones. However, there are still some limitations on the display performance, such as poor image quality and limited field of view (FOV). Compressive light field display with the viewing-position-dependent weight distribution is presented. When relevant views are given high weights in the optimization, the displaying performance at the viewing-position can be noticeably improved. Simulation and experimental results demonstrate the effectiveness of the proposed method. Peak signal-noise-ration (PSNR) is improved by 7dB for the compressive light field display with narrow FOV. The angle for wide FOV can be expended to 70° × 60°, and multi-viewers are supported.
© 2016 Optical Society of America
1. Introduction
Currently, three-dimensional (3D) display has made a great progress. Different kinds of glasses-free 3D displays are demonstrated, such as autostereoscopic display with lenticular sheet [1] and integral imaging [2]. Super multi-views, smoother parallax and high resolution are important for the high-performance 3D display [3–8]. However, the intrinsic spatial-angular resolution tradeoff and narrow field of view hinder further applications of the available 3D display [9].
In recent years, compressive light field display has been put forward [10]. It is a distinct kind of computational display which overcomes limits of purely optical designs by computation [9]. Enabled by multilayer and time-multiplexed multiframe decompositions, it can provide three-dimensional scenes with features of high spatial-angular resolution and without periodically repeating view-zones. There are several multilayer implementations, including Content-Adaptive Parallax Barriers [11], Layered 3D Displays [12], Polarization Fields Displays [13], Tensor Displays [14] and near-eye display [15]. Though these prototypes can present 3D contents efficiently, there are still some limitations on the displaying performance, such as poor image quality and limited field of view (FOV). To address the limitation of FOV, eyes tracking was introduced and the light field was optimized to create two narrow view cones directed to viewer’s eyes [16]. However, two problems occur for this method, including view switching jumps and vulnerable eye tracking precision.
Here, compressive light field display with the viewing-position-dependent weight distribution is presented. When the viewer’s position is located, a viewing-position-dependent weight distribution function can be established. High weights are given to the relevant views directed to the viewing-position. Such views would be emphasized in the layers’ optimization, and 3D displaying performance can be noticeably improved. Simulations and experimental results demonstrate the effectiveness of the proposed method, in situations of narrow FOV, wide FOV and multi-viewers. Peak signal-noise-ration (PSNR) is increased by for the compressive light field display with narrow FOV. The angle for wide FOV is expended to and four viewers are supported.
Several points for the proposed method are different from the previous basic method. A weighted objective function is used to replace the basic objective function, and related views can be assigned to high weights which enhance displaying performance. The image quality can be flexibly adjusted by modifying function parameters, even when the covered views are fixed. A smooth switching between adjacent views can be provided, even when the number of covered views is very small. The viewing-position-dependent weight distribution can be established in the dark surrounding, which is little influenced by the ambient light.
2. The weighted compressive light field
2.1 Theory
Generally, light rays can be modulated on different directions when passing through stacked LCD layers. In the basic compressive light field display, as shown in Fig. 1(a), multi-layers are optimized so that the emitted view can approach the target view. Multi-frames are used to improve precision. The basic objective function of tensor display is given by the following expression,
where is the target light filed, is the brightness, is the emitted light, and is also the average of vector product of N LCD layers. I belongs to views, and these views are evenly involved in optimization.However, in general, the viewer cannot watch all views at the same time. Here, the method of weighted compressive light field display is used, based on the viewing-position-dependent weight distribution, as shown in Fig. 1(b). Weighted compressive light field means that the weight of each viewpoint is decimal fraction instead of binary in the objective function. The weighted objective function can be written as the following expression,
where is the weight of emitted view and target view . This expression can be solved by applying weighted non-negative tensor factorization (WNTF), which was used to address overlapped subpixels in [17]. The update rules of are given as following expression,in which, the expressions of and are given as following,In Eq. (1), the basic compressive display evenly involves all views into computation, and the residue error of each view is even. Whereas in Eq. (2), views of the weighted compressive display are unevenly optimized. If a view’s weight is higher, its residue errors will be less and the image quality will be improved. Therefore, performance improvement of compressive light field display can be achieved.
2.2 Viewing-position-dependent weight distribution function
In the basic tracking method, Kinect was used to track eyes [16]. However, non-uniform illumination may reduce the precision of eye tracking. Here, Kinect is used to locate the viewer's head based on the depth camera for the viewing position, of which the precision is not easily influenced by the ambient light.
The position of viewer’s head is assumed as the viewing position, based on which a viewing-position-dependent weight distribution function can be established. After non-negative, symmetry and high central energy are taken into consideration, the function expression is written as the Gaussian weight function. The function’s center lies at the viewing position. It should be noted that, since the head tracking is used, the weight function should cover enough views and the size should be set as large as viewer’s face.
Typically, as shown in Fig. 2, there are three expressions of weight distribution. Figure 2(a) is the Even weight function for the basic method. All weights are set to 1, so that views are evenly involved in Eq. (1). Figure 2(b) is the Binary weight function for the basic tracking method. Weights are set to 1 or 0. All views can be selected to be evenly involved or not. Figure 2(c) is the Gaussian weight function for the weighted method. Weights are assigned to views based on the viewing-position-dependent weight function. The central view at the viewing position is assigned a higher weight and it is emphasized in the optimization.
The Gaussian weight function can be decomposed in two directions,
where is the center of viewer’s head. is the reference coordinate located on the viewing plane. The intensity of the distribution function is 1. c is a constant value. and are Gaussian functions in the x and y directions, respectively. and v are related to the function’s size, and the displaying quality is affected.3. Performance analysis of the weighted compressive display
3.1 Performance of the tracking, viewing and optimizing
The capacity of tracking system is enough for our method, since Kinect v2.0 supports tracking FOV and depth range. Head tracking is used to avoid high precision of eye tracking, which is not affected by the illumination. In addition, a large size weight function is constructed to cover the whole face, which is more tolerant for tracking errors.
The time cost of optimization is very short, since the iteration rate and the convergence rate are very fast. Therefore, real-time displaying is possible. The iterating rate of is about . The convergence rate of optimization is shown in Fig. 3. For the weighted compressive display, PSNR could achieve 30dB after 20 times, 35dB after 25 times, and 38dB after 50 times. In practice, after a viewer stands firmly, a relative good result could be shown in 1s, and an improved performance can be obtained. In addition, within the same iteration times, the result of the weighted method is better than that of the basic method. This is because the proposed method can optimize related views rapidly and a desirable performance can be displayed.
Since all views are separated on different positions, the 3D perception is not affected by crosstalk. In the light field computation, all target views are designed to be focused on the viewing plane, as shown in Fig. 4(a). When the watching distance z is located, for some target view , view position is . and weight can be assigned in the light of the viewing-position-dependent weight distribution function, . For displaying, the emitted view is reproduced on the same position , as shown in Fig. 4(b). So, when a viewer stands at, he would watch the view in the viewing area, and other views cannot be watched.
3.2 Performance of the weight distribution function
For the performance of the weight distribution function used in compressive light field display, the Gaussian weight function exhibits two advantages in quality adjustment and switching smoothness.
When the covered views are fixed, for the basic compressive display with the Even function, the displaying quality would be constant. However, for the weighted method with the Gaussian weight function, an improved displaying quality is provided and the quality can be flexibly adjusted by modifying function parameters and. As demonstrated in Fig. 5, PSNR of the Gaussian function is overall higher than 30dB. With the value of and increasing, the obtained quality declines. The reason is that, when the function size is enlarged, more weights are assigned to irrelevant viewpoints and central views gets harder to be emphasized. Differences between viewpoint weights become smaller, and eventually the weighted method becomes the basic method. In Fig. 5, has a greater influence than, because the parallax in vertical is larger than the horizontal.
When the number of covered views is very small, the basic tracking method with the Binary weight function would cause obvious vision jumps when switching between adjacent views. For the Gaussian weight function, a smooth switching can be achieved. For example, there are five viewpoints covered, such that , , ,, and . The distribution of Binary function is shown in Fig. 6(a). The proposed method with Gaussian weight function is shown in Fig. 6(b). When moving in X direction, compressive light field would synthesize corresponding views, and EPI result is generated. It can be seen from Fig. 6 that, EPI of the Gaussian function is much smoother on the boundary, but the EPI of the Binary function is serrated. The reason is that, in the process of the Binary function, views are locally synthesized by three target view groups of , and , which causes three jumps and three serrations on EPI. For the proposed Gaussian function, each view is synthesized by the target views around with distance Gaussian weights. None of them would appear or disappear instantly. When the weighted compressive displays the 3D image with the Gaussian function, switching between adjacent views is very smooth, and there are no any obvious jumps.
4. Implementation
4.1 Hardware
Our experimental setup consists of three-layer stacked LCDs and a tracking device, as shown in Fig. 7. Three 24-inch AOC G2460PG monitors are used as the LCD panels, which can operate at 144Hz with the resolution of . Each panel is stacked at a space of 3cm. Since the tracking area of Kinect v2.0 only supports horizontal and vertical field, the real-time display maximum FOV is also designed as .
The program of WNTF of Eq. (3) runs on an Intel Core i7 workstation with a NVIDIA GTX1080 card. In the processing, layers are initialized from the random value and optimized iteratively, which is programmed using CUDA and rendered on D3D. In displaying, layers are presented after each iteration. With the time increasing, the displaying results are refined constantly. After hundreds of iterations, optimization stops and LCD remains fresh at 144Hz . This operation may cost 1s. When the tracking device detects the viewer moving to a new position, a new weight distribution can be established and optimization starts over immediately.
4.2 Configuration
The weighted compressive light field display is demonstrated in three situations, including 1) narrow FOV, 2) wide FOV and 3) multi-viewers display. For a certain angle, the more viewpoints involved, the more accurate calculation [9]. The light field “monkey” used here, contains views, as shown in Fig. 8. It can be displayed on FOV which corresponds to the maximum tracking FOV. The watching distance in experiment is set to 150cm. Thus, the angle between each viewpoint is , and the interval between each one is 1cm. In order to exclude the influence of rank, all experiments are set 3 frames which means rank-3 in WNTF. Detailed configurations of weight functions are shown in the Table 1. PSNR of simulated light field results are calculated for estimation. It should be noted that, in practice, the watching distance is not limited to constant value. The viewer can move arbitrarily within the tracking area, because our method has the capability to update contains and display improved views in real-time.
5. Experiment and assessment
5.1 The weighted compressive display with narrow FOV
The result of compressive light field display with narrow FOV is shown in Fig. 9. is set as 1. PSNR is employed to qualify simulated views under basic and weighted situations. Figure 9(a) gives the scene of “car” with views and FOV. From the simulation, we can see that after the weights are applied, the image quality is improved from 31.42dB to 38.59dB. The blurred headlight of “car” becomes clear. Figure 9(b) is the scene of “monkey” with views and FOV, extracted from the wide scene. It can be seen that, PSNR is improved from 25.61dB to 30.40dB. The inner ear contour of “monkey” arises, while the inner ear of the basic result is all white. The reason for the improvement is that, after a high weight is given, the target view would have a higher proportion in the light filed synthesis and a better result can be achieved. When the viewing position moves, the peak of weight distribution would follow the movement and an improved performance can be always received.
5.2 The weighted compressive display with wide FOV
In the situation of narrow FOV, all views are involved in the multilayer decomposition. However, when the FOV and the number of views are very large, the computational time would be tremendous. For the weighted compressive light field display, in order to satisfy the demand of tracking viewers, a reduction of the view number should be applied. We settle a view range directed to the view’s head, as the following expression,
where and g is the weight of each view, is the position of each view. Therefore, only appreciable views are preserved and others are removed.The wide scene of “monkey” contains views and FOV. The view range here is set as . Experiment results of wide FOV are shown in Fig. 10. In wide FOV, the quality of the basic compressive light field with the Even function is only 21.73dB and most details are blurred. The reason is that multi-layers are hard to carry too much dissimilarity for large FOV, and it results in severe image quality degradation. In the second row of Fig. 10, the quality of the Binary function is improved to 28.90dB. That is because none irrelevant views are involved in computation, and dissimilarity becomes less. However, the contour of the ear is still a little blurred. As demonstrated in the third row, the quality of the weighted display with the Gaussian function is improved greatly. PSNR is promoted to 30.32dB. After weights are assigned based on viewing position, only relevant views inside the viewing range are concerned in the multilayer decomposition, and the central views are emphasized. Therefore, the weighted compressive display can provide a good result for the wide FOV. Figure 11 is about experiment results captured at different viewing angles. We can see that within the Kinect tracking area, the displaying quality is acceptable.
5.3 The weighted compressive display for multi-viewers
In the proposed system, multi-viewers are supported. Different viewers are tracked by a Kinect and their viewing positions can be converted into an unified coordinate system in real-time. Different viewing positions are to , and weight function can be written as the following expression,
In our experiment of the weighted compressive light field display system for multi-viewers, the viewer number is 4. The angle for the displaying system with wide FOV for four-viewers is set as and views. The brightness is set as 0.8 for the basic and weighted optimization. For four viewers, the weight distribution function is established as Eq. (8) and view ranges are set as Eq. (7). Weight distributions and PSNR distributions are shown in Fig. 12. From the second row, we can see that four viewing-positions correspond to four weight peaks and four PSNR peaks, which are higher than the basic method result. Figure 13 shows the relation between the viewer number and the reconstruction quality. It can be seen that with the viewer number increasing, the displaying quality of viewer #1 becomes degraded. The reason is that, for more viewers, more parallax views are introduced into the light field, which causes the optimization more difficult. When all the viewpoints are evenly involved, the displaying quality will be reduced to the same as the result of the basic method. For four viewers, an improved result can be achieved.
The experimental result indicates that PSNRs of perceived 3D images are noticeably improved, as shown in Fig. 14. Details of results obtained by the weighted method are mostly preserved. Despite some artifacts appear on the object border, the performance of the weighted method is much more acceptable, compared to the blurred results of the basic compressive light field display. The reason is that the weight peaks can effectively emphasize relevant views and make them better. Four viewers can perceive good 3D images.
6. Conclusion
In summary, compressive light field display with the viewing-position-dependent weight distribution is demonstrated. With a tracking device, the viewing position is captured and a weight distribution function can be established. The weight peak is able to effectively emphasize relevant views in the optimization and 3D image quality is greatly improved. Experimental results for situations of narrow FOV, wide FOV and multi-viewers are presented. Simulations and experimental results show that the weighted light field displaying system is suitable for 3D display, and an improved performance can be achieved.
Funding
National Natural Science Foundation of China (NSFC) (61575025); The fund of the State Key Laboratory of Information Photonics and Optical Communications.
References and links
1. F. E. Ives, “Parallax stereogram and process of making same,” U. S. patent 725,567 (1903).
2. G. Lippmann, “Epreuves reversibles donnant la Sensation du Relief,” J. Phys. 7(1), 821–825 (1908).
3. J. H. Park, K. Hong, and B. Lee, “Recent progress in three-dimensional information processing based on integral imaging,” Appl. Opt. 48(34), H77–H94 (2009). [CrossRef] [PubMed]
4. X. Yu, X. Sang, S. Xing, T. Zhao, D. Chen, Y. Cai, B. Yan, K. Wang, J. Yuan, C. Yu, and W. Dou, “Natural three-dimensional display with smooth motion parallax using active partially pixelated masks,” Opt. Commun. 313(4), 146–151 (2014). [CrossRef]
5. X. Yu, X. Sang, X. Gao, Z. Chen, D. Chen, W. Duan, B. Yan, C. Yu, and D. Xu, “Large viewing angle three-dimensional display with smooth motion parallax and accurate depth cues,” Opt. Express 23(20), 25950–25958 (2015). [CrossRef] [PubMed]
6. X. Gao, X. Sang, X. Yu, P. Wang, X. Cao, L. Sun, B. Yan, J. Yuan, K. Wang, C. Yu, and W. Dou, “Aberration analyses for improving the frontal projection three-dimensional display,” Opt. Express 22(19), 23496–23511 (2014). [CrossRef] [PubMed]
7. P. Wang, S. Xie, X. Sang, D. Chen, C. Li, X. Gao, X. Yu, C. Yu, B. Yan, W. Dou, and L. Xiao, “A large depth of field frontal multi-projection three-dimensional display with uniform light field distribution,” Opt. Commun. 354, 321–329 (2015). [CrossRef]
8. S. Xie, P. Wang, X. Sang, and C. Li, “Augmented reality three-dimensional display with light field fusion,” Opt. Express 24(11), 11483–11494 (2016). [CrossRef] [PubMed]
9. F. Heide, G. Wetzstein, R. Raskar, and W. Heidrich, “Adaptive image synthesis for compressive displays,” ACM Trans. Graph. 32(4), 876–884 (2013). [CrossRef]
10. G. Wetzstein, D. Lanman, M. Hirsch, W. Heidrich, and R. Raskar, “Compressive light field displays,” IEEE Comput. Graph. Appl. 32(5), 6–11 (2012). [CrossRef] [PubMed]
11. D. Lanman, M. Hirsch, Y. Kim, and R. Raskar, “Content-adaptive parallax barriers: optimizing dual-layer 3D displays using low-rank light field factorization,” ACM Trans. Graph. 29(6), 81–95 (2010). [CrossRef]
12. G. Wetzstein, D. Lanman, W. Heidrich, and R. Raskar, “Layered 3D: tomographic image synthesis for attenuation-based light field and high dynamic range displays,” ACM Trans. Graph. 30(4), 76–79 (2011). [CrossRef]
13. D. Lanman, G. Wetzstein, M. Hirsch, W. Heidrich, and R. Raskar, “Polarization fields: dynamic light field display using multi-layer LCDs,” ACM Trans. Graphic 30(6), 61–64 (2011). [CrossRef]
14. G. Wetzstein, D. Lanman, M. Hirsch, and R. Raskar, “Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting,” ACM Trans. Graph. 31(4), 13–15 (2012). [CrossRef]
15. F. Huang, D. Luebke, and G. Wetzstein, “The light field stereoscope - Immersive Computer Graphics via Factored Near-Eye Light Field Displays with Focus Cues,” ACM Trans. Graphics 34(4), 1 (2015). [CrossRef]
16. A. Maimone, R. Chen, H. Fuchs, R. Raskar, and G. Wetzstein, “Wide field of view compressive light field display using a multilayer architecture and tracked viewers,” J. Soc. Inf. Disp. 45(1), 525–534 (2014).
17. F. Heide, D. Lanman, D. Reddy, J. Kautz, K. Pulli, and D. Luebke, “Cascaded displays: spatiotemporal superresolution using offset pixel layers,” ACM Trans. Graph. 30(4), 70–79 (2014).