Open Access
Add to BibSonomyAbstract
A video see-through augmented reality three-dimensional display method is presented. The system that is used for dense viewpoint augmented reality presentation fuses the light fields of the real scene and the virtual model naturally. Inherently benefiting from the rich information of the light field, depth sense and occlusion can be handled under no priori depth information of the real scene. A series of processes are proposed to optimize the augmented reality performance. Experimental results show that the reconstructed fused 3D light field on the autostereoscopic display is well presented. The virtual model is naturally integrated into the real scene with a consistence between binocular parallax and monocular depth cues.
© 2016 Optical Society of America
1. Introduction
Augmented reality display allows the viewer to see the real scene meanwhile perceiving the virtual information which overlays on the real scene [1-3]. Augmented reality technology has been wildly spread in various fields, such as education [4], medical science [5], gaming [6] and teleconference [7]. There are varieties of display forms for the augmented reality display. The head-mounted video or optical see-through displays are the most common means, which can provide a 3D depth cue of the augmented virtual scene with stereo vision. Projection based displays are also used for augmented reality display, and the augmented reality scene can be viewed by several viewers without special eye-wearing equipment. Besides the two forms, the autostereoscopic 3D display is also an optional form for augmented reality display, with the wide field of view, glasses-free depth sense and free device size. The autostereoscopic 3D display is more convenient and comfortable to present augmented reality effect in a more natural way [8–11].
However, the large amounts of information in multi-view contents hinder the development of multi-view augmented reality. Traditional 2D or stereoscopic augmented reality display technologies are not suitable for the dense viewpoint case. It is impossible to process the content once a perspective while maintains a consistent augmented reality fusion among all the perspectives. Multi-view augmented reality 3D display technologies have draw attentions. Olwal et al. proposed an autostereoscopic projection system for optical see-though augmented reality which projects multi-view images to a transparent holographic optical element [12]. With increasing viewpoints, the system becomes significantly complex, and only two views stereoscopic display is implemented in the experiment. In [13] Yoshida et al. presented a full-parallax 3D projection display with high-density projector array and retro-reflector, and in [14] Yasuhiro et al. used windshield as a half mirror and composed a multi-view optical see-through system for car drivers. These multi-view augmented reality systems can present a correct depth relationship between real and virtual scenes, but a natural fusion effect is hardly produced because they are fused with a transflective optical manner, ignoring the occlusion between the real and the virtual scenes. Moreover, several augmented reality systems deal the occlusion problem with depth sensors, like KINECT [15,16]. In fact, it is inconvenient to obtain the accurate depth of the real scene, because jitter exists at the edge of the real scenes, which is an inherent problem of the structured light depth detection.
Here, an autostereoscopic video see-through augmented reality 3D display with light field fusion is presented, which correctly handles occlusion, and the depth information is perceived via the multi-perspectives themselves without the assistance of additional scene depth detection equipment. Starting with the basic work for the light field by Criminisi et al [17], a new light field processing pipeline for augmented reality method is proposed. The statistical depth estimation method and the edge truing method make the augmented reality performance effective and natural. The rest of this paper is organized as follows. Section 2 introduces the 3D light field and basic method for light field fusion. Section 3 and section 4 describe our augment reality framework in detail. Section 5 gives augment reality results and the auxiliary tool used in the experiment.
2. 3D Light Field
The light field is proposed by Gortler et al and Adelson et al, which can be expressed with a four-dimensional (4D) function called the Lumigraph or plenoptic function [18,19]. The content that generated for stereoscopic display should satisfy the stereo constraint: horizontal parallax only without the vertical displacement. The 3D light field can be used in stereo image processing without loss of the generality. The concept of the 3D light field is presented in Fig. 1, the 3D light field shown in Fig. 1(b) is assembled from a dense set of multi-perspective images. Different 2D sections of the 3D light field results in different 2D images that reflect various features of the scene. An x-y cut at particular parameter S extracts the original multi-perspective image, and an x-S cut leads to the epipolar plane image (EPI) which is composed of linear structures. The slope of linear structure is proportional to the depth of the corresponding scene point. The basic concept of real and virtual scenes fusion in the augment reality framework of this work is based on the linear structure, which is shown in Fig. 2. In Fig. 2(a), the scene captured by a pin-hole camera array with parallel arrangement, which leads to a linear offset when objects project to each image plane, and it further leads to a linear structure in the corresponding 2D light field or EPI as shown in Fig. 2(b). The linear structure is called the EPI-strip. As the slope of each EPI-strip is proportional to the depth of the corresponding object, the virtual object can be added to the light field with the corrected depth, and occlusion also can be handled easily. For example, the gray EPI-strip is added to the light field in Fig. 2(b) with corrected occlusion, because its slope is between the red one and the blue one after light field combination. The virtual object is located at the position of gray line in real scene in Fig. 2(a). The light field fusion process is implemented in each EPI of the light field with corrected depth and occlusion taking into account.
3. Light Field Understanding
The proposed framework of the augmented reality autostereoscopic display is illustrated in Fig. 3. After the real scene is acquired, its depth information is computed with light field understanding and the camera array position is calibrated. With the calibration information the 3D registration is done, thus virtual camera is set and the virtual model light field can be captured. Then according to the depth information in the virtual light field, the real scene in front of the virtual model is extracted out. Finally the light fields are fused and displayed on the autostereoscopic display. In section 3, The light field depth estimation method is analyzed in detail, and in section 4 the front scene extraction method and the light field fusion processes are analyzed in detail.
3.1. Local depth computation
The EPI of the light field is shown in Fig. 4(a). The orientation of the linear pattern texture represents the depth information of the scene and the same linear pattern has consistent intensity. As shown in Fig. 4(c), given as a pixel with the coordinates . The line across with an angle θ with x-axis is defined as
and the set of pixels Ф sampled by this line are defined aswhere n corresponds to the number of the viewpoint in the light field. To estimate the most reliable orientation of the linear pattern belongs to the pixel, an orientation score that represents the intensity consistence between the given pixel and the set of pixels in the line is defined as where the function calculates the intensity distance between the pixel and the pixel in normalized RGB space, and is a tent function and the bandwidth parameter t is set to 0.02. The orientation score of every admissible line with different θ is computed and an orientation consistence distribution can be obtained as shown in Fig. 4(d). The angle θ with the maximum score is the most responsible depth estimation of the pixel .
Fig. 4 Local depth computation (a) EPI, (b) region with significant texture in EPI, (c) directional statistics, (d) histogram of directional score, (e) local depth map
Only those regions with rich texture contain the effective depth information, and it is unreasonable to estimate the depth in texture-less regions. As shown in Fig. 4(d), the orientation score curve has a significant peak which indicants a reliable depth estimation at the region with rich texture, but a smooth distribution at texture-less region leads to ambiguous depth estimation. Fortunately, the edge of the front scene and the background always bring in significant texture, which is helpful for depth estimation and foreground extraction. For reliable depth computing, only those regions with significant texture are computed. The region with significant textures is shown in Fig. 4(b), which is decided by the significant degree of the horizontal gradient in the EPI. All the pixels in Fig. 4(b) are computed with the same method, and a local depth map is obtained as shown in Fig. 4(e). The color from dark green to bright green shows the scene is far away from the camera array.
3.2. Global depth statistics
Local depth pixels are isolated with each other in Fig. 4(e) and hardly reflect the global depth information among perspectives. Taking both the location and the orientation of the depth pixel into consideration, isolated pixels are contacted. Next, the responsible linear pattern that globally reflects the scene depth among each perspective is statistical linear regressed in EPI. As θ is proportional to the depth, it is convenient to define a depth pixel in depth map as a pixel with coordinates and with an orientation angle θ.
Each depth pixel is possible to expand to a global depth line that reflects the global depth in EPI,
for a reliable global depth line, it should be a linear regression with depth pixels around it, as shown in Fig. 5(a). The reliability of the linear regression is defined as a residual standard deviation σW(q) is a select window which select the depth pixels with orientation angle close to and locate near the global depth line (4),the bandwidth of the window is . Moreover, the set of depth pixels q in the selected window are the set of sample pixels for linear regression and for a reliable linear regression, samples should be sufficient for statistics. The density of samples is defined assmax and smin are the maximum and the minimum s in the set of sample pixels, respectively. Only the linear regression with sufficient sample pixels ρ and small linear regress residual standard deviation σ is considered as a reliable linear regression. The depth line is considered as a reliable global depth line.Fig. 5 Global depth statistics (a) the principle of global depth statistics, (b) depth line (c) global depth line after complement
For all the depth pixels, above global depth statistics are repeated, once a reliability depth line is determined, all its sample pixels are removed from the local depth map and the loop continues at another depth pixel in the local depth map. The final global depth statistical result is shown in Fig. 5(b), and a completed result by extending each depth line to every viewpoint is shown in Fig. 5(c). The depth line with larger θ cannot pass through the one with smaller θ according to the occlusion principle. Thus the edges of the EPI-strip in EPI are correctly detected, and their angle to x-axis θ represent the depth of the corresponding EPI-strip.
4. Front Scene Extraction and Light Field Fusion
As shown in Fig. 6, the light field is fused in EPI and the final fused EPI contains the corrected occlusion and depth. The principle of light field fusion is illustrated in section 2, with the depth estimation of real scene and the acquisition of virtual light field. The two light fields are fused in their EPI layers. Here, the process of light field fusion is discussed.
Fig. 6 Light field fusion in EPI layer, (a) EPI of real light field, (b) EPI of virtual light field, (c) fused EPI of the fused light field.
4.1. Front EPI-strip extraction and consistency refinement
EPI-strips of foreground in real light field are extracted according to the angle θ of the virtual EPI-strip. As shown in Fig. 7(d), the mask of front EPI-strips are decided by signing the pixels between depth lines with the smaller θ than the virtual EPI-strip. However, error occurs when front EPI-strips are extracted in a texture non-significant background, as shown in Fig. 7(d), because the texture non-significant background cannot provide effective depth information. In the 564th EPI layer, the EPI-strips of the two bottles as well as the background between them are fused together. This problem can be partly solved by adjusting parameters of local depth computation and global depth statistics, to loosen the significant texture decision and free the reliability of linear regression, but more noise is induced.
Fig. 7 (a) Middle perspective of the scene, (b) corresponding vertical gradient, (c) EPIs, (d) front EPI-strip extraction.
To eliminate this error, the consistency among adjacent EPIs is taken into consideration, and extracted front EPI-strips can be refined. Basic ideas of EPI consistency refinement are: 1. EPI is similar with its adjacent EPIs in the light field as shown in Fig. 7(c); 2. The structure of front EPI-strip significantly changes at the edge of the foreground. According to the basic ideas, the EPI layers consistency and the vertical gradient of the light field are taken into consideration.
As shown in Fig. 7(d), the extracted foreground EPI-strips changing from the 555th to the 561th EPI layers is reasonable since there are significant vertical gradients in corresponding regions of the multi-view image (the regions with true in Fig. 7(b)). The significant structure change between the 561th and the 564th EPI layers is an error as there is no vertical gradient in the corresponding region (the region with false in Fig. 7(b)). That is to say, the EPI-strips change only at the place with significant vertical gradients. So the region at the middle of the two front EPI-strips should be signed as background in the 564th EPI layer, and the two EPI-strips should be separated. The final consistency refining results are presented in Fig. 8. In Figs. 8(a) and 8(b), the fusion results without consistency refinement are in different perspective, and the earth is cut into slices. Each cut corresponds to a wrong front EPI-strip extraction in the corresponding EPI as the same circumstance in the 563th EPI and the 564th EPI. The results after consistency refinement are shown in Figs. 8(e) and 8(d), and it can be found that the fusion performances are fairly good. The Fig. 8(e) shows the detailed refined results in EPIs, where the consistency of adjacent EPIs is well kept.
4.2. Edge burrs elimination
According to section 3, the edge of EPI-strip is a statistical linear regression result, which is not the accurate edge of the foreground since random errors inevitably exist during recording the real scene. After consistency refinement, the performance should be further improved at the edge of the fused light field. Details of the EPI-strip are shown in Fig. 9(a), the red points represent the actual edge of foreground and the edge of EPI-strip is a statistical regression line according to the actual edges and it does not fit so well with the actual edge. The edge mismatch causes burrs at the edge of light field fusion if the linear regression edge is used directly as actual edge, which reduces the naturalness of light field fusion, as shown in Fig. 9(b).
Fig. 9 Edge burrs elimination, (a) details of EPI-strip edge, red points represent the actual edge of each multi-perspective image and the statistical regression result is the depth line, (b) fusion effect without edge burrs elimination, (c) fusion effect with edge burrs elimination
As the mismatch between the actual edge and the regression line is usually small, a watershed algorithm with the regression line as reference is used to find the actual edge, thus edge burrs can be eliminated. As shown in Fig. 9(a), an M*1 local window centered at the regression line (M = 5 in the figure) is defined as a region, where the horizontal gradient of EPI is computed. Then the position with the maximum horizontal gradient is considered as the actual edge, the red point in Fig. 9(a). Finally, regions are redistributed with treating the red point as the watershed. The red arrows and the black arrows in Fig. 9(a) represent the redistributive direction of the front EPI-strip region and the background region, respectively. After edge burrs elimination, edges of the EPI-strip fit well with the actual object edges, as shown in Fig. 9(c).
5. Experiment of the Augment Reality autostereoscopic 3D display and Discussions
5.1 3D Registration and Virtual Model Generation
The posture of virtual object in the real scene is important for a natural augment reality. The “ARToolkit” software is convenient, and the sample for augment reality is used for the posture estimation. In order to put the virtual model on the ground, the fundamental matrix between the ground and the camera array is obtained with a calibration plate on the ground, as shown in Figs. 10(a) and 10(b). To compute the fundamental matrixes of all the cameras more accurately, the linear arrangement of camera array is used. In our registration process, the fundamental matrixes of camera array are computed through statistics. The parameter of the horizontal shift is linearly regressed and other parameters perform the average. After 3D registration, the fundamental matrixes are sent to OpenSceneGrapth (OSG) to build a virtual scene, where the virtual light field can be recorded with the virtual camera array. Procedures are shown in Fig. 9, and the 3D registration obtains the camera array posture relative to the ground. Then the virtual scene can be set up with this relationship, and the virtual model in the virtual scene could be captured with the posture, that is consistent with the real scene.
Fig. 10 3D registration and virtual model Generation, (a) and (b) the rightmost and leftmost real scene perspectives, respectively, (c) the virtual scene build in OSG, (d) and (e) the rightmost and leftmost virtual scene perspectives, respectively.
5.2 Augment reality exhibition
The proposed augment reality performances show a natural augment reality with not only the actual depth scene but also correcting the occlusion relationship. The binocular parallax together with motion depth cues, such as posture of virtual object and occlusion, enhances the reality of the virtual scene in our autostereoscopic augment reality 3D display. Two kinds of simple applications are given with our augment reality display: light field presentation and model random walking.
Moreover, the proposed augment reality algorithm is treated as an off-line post-process algorithm for multi-view media. The local depth computation occupies the most computational complexity. To speed up, each pixel depth is parallel computed with GPU naturally, and only the pixels in the significant texture region as shown in Fig. 4(b) is taken into account. The main bottleneck of the speed is to put a mass of real light field data into the RAM. In the experiment, as much as 237 perspectives of the light field are used, and each perspective has a resolution of 1280*720. Given the light field data in the RAM in advance, the algorithm can be finished within several seconds according to the scene complexity.
5.2.1 Light field presentation
A light field with hundreds or thousands of perspectives is hardly completely presented to viewers. However, with the aid of head tracking, it is convenient to present the panorama of the light field, as shown in Fig. 11. The display is a 5 viewpoints autostereoscopic display with a resolution of 1920*1080. The composited images are changed according to the position of viewers’ head, which is obtained with a KINECT. All the 237 perspectives of the light field can be viewed by changing the viewing position, which enhanced the augment reality by introducing the viewer position interaction. With the viewer changing the viewing position, the perspective of light field changes and the earth rotates as shown in Fig. 11(b). When the autosterescopic display is observed, the image of every viewpoint presents correct perspective relation of scene and occlusion. When the observer walks around in front of the display to view the successive viewpoints of the display, the motion parallax can be obtained. It is without debate that the depth sense of the earth is natural behind the bottles, since both the monocular depth cues and the binocular depth cue are correct and consistent.
Fig. 11 Autostereoscopic augment reality display, (a) perspectives of light field from left to right, (b) zoom in each perspective.
5.2.2 Model random walking
Random walking, another application model is shown in Fig. 12, which allows a virtual model put onto the real scene freely. Since the position relationship between the ground and the camera array is registered, the light field of the virtual model at any positions on the ground is available with the aid of OSG, as shown in Figs. 12(a) and 12(b). A 50-inch autostereoscopic 3D display with 32 viewpoints based on a 3840 × 2160 LCD panel is used. Multi-perspective images are shown in detail in Fig. 12(c), and the cube is naturally fused into the real scene with a consistent augment reality effect among all the perspectives.
6. Conclusion
In summary, an augmented reality three-dimensional display method based on the light field fusing is presented. Different from previous videos based augment reality, the multi-perspective information of scene is organized as a 3D light field, which inherently contains the depth information of the scene. The scene fusion is carried out in the light field, and a consistent augment reality effect is maintained among all the perspectives. Experimental results illustrate the efficiency of the light field fusion framework and the proposed light field fusion method meets the demand of autostereoscopic augment reality 3D display.
7. Acknowledgment
This work is partly supported by the “863” Program (2015AA015902), the National Science Foundation of China (61575025), the fund of the State Key Laboratory of Information Photonics and Optical Communications, and the Program of Beijing Science and Technology Plan (D121100004812001).
References and links
1. D. W. F. van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” Int. J. Virtual Reality 9(2), 1-20 (2010).
2. R. T. Azuma, “A survey of augmented reality,” Presence (Camb. Mass.) 6(4), 355–385 (1997). [CrossRef]
3. F. Zhou, H. B. L. Duh, and M. Billinghurst, “Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR,” in Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality (IEEE, 2008), pp. 193–202.
4. H. K. Wu, S. W. Y. Lee, H. Y. Chang, and J. C. Liang, “Current status, opportunities and challenges of augmented reality in education,” Comput. Educ. 62, 41–49 (2013). [CrossRef]
5. V. Ferrari, G. Megali, E. Troia, A. Pietrabissa, and F. Mosca, “A 3-D mixed-reality system for stereoscopic visualization of medical dataset,” IEEE Trans. Biomed. Eng. 56(11), 2627–2633 (2009). [CrossRef] [PubMed]
6. B. H. Thomas, “A survey of visual, mixed, and augmented reality gaming,” Comput. Entertainment 9(3), 3 (2012).
7. D. Bingbin and X. Yang, “A low-latency 3D teleconferencing system with image based approach,” in Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry (ACM, 2013), pp. 243–248.
8. N. S. Holliman, N. A. Dodgson, G. E. Favalora, and L. Pockett, “Three-dimensional displays: A review and applications analysis,” IEEE Trans. Broadcast 57(2), 362–371 (2011). [CrossRef]
9. X. Sang, F. C. Fan, C. C. Jiang, S. Choi, W. Dou, C. Yu, and D. Xu, “Demonstration of a large-size real-time full-color three-dimensional display,” Opt. Lett. 34(24), 3803–3805 (2009). [CrossRef] [PubMed]
10. X. Gao, X. Sang, X. Yu, P. Wang, X. Cao, L. Sun, B. Yan, J. Yuan, K. Wang, C. Yu, and W. Dou, “Aberration analyses for improving the frontal projection three-dimensional display,” Opt. Express 22(19), 23496–23511 (2014). [CrossRef] [PubMed]
11. X. Yu, X. Sang, S. Xing, T. Zhao, D. Chen, Y. Cai, B. Yan, K. Wang, J. Yuan, C. Yu, and W. Dou, “Natural three-dimensional display with smooth motion parallax using active partially pixelated masks,” Opt. Commun. 313, 146–151 (2014). [CrossRef]
12. A. Olwal, C. Lindfors, J. Gustafsson, K. Jellberg, and L. Mattsson, “ASTOR: An autostereoscopic optical see-through augmented reality system,” in Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (2005), pp. 24–27. [CrossRef]
13. T. Yoshida, K. Shimizu, S. Kamuro, K. Minamizawa, H. Nii, and S. Tachi, “RePro3D: full-parallax 3D display with haptic feedback using retro-reflective projection technology,” in 2011 IEEE International Symposium on VR Innovation (ISVRI) (IEEE, 2011), pp. 49–54. [CrossRef]
14. Y. Takaki, Y. Urano, S. Kashiwada, H. Ando, and K. Nakamura, “Super multi-view windshield display for long-distance image information presentation,” Opt. Express 19(2), 704–716 (2011). [CrossRef] [PubMed]
15. O. Hilliges and D. Kim, S. lzadi, M. Weiss and A. Wilson, “HoloDesk: direct 3d interactions with a situated see-through display,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM, 2012), pp. 2421–2430.
16. F. de Sorbier and Y. Uematsu, Y. uematsu, I. Daribo and H. Saito, “Augmented reality for 3d tv using depth camera input,” in 16th International Conference on Virtual Systems and Multimedia (VSMM) (IEEE, 2010), pp. 117–123.
17. A. Criminisi, S. B. Kang, R. Swaminathan, R. Szeliski, and P. Anandan, “Extracting layers and analyzing their specular properties using epipolar-plane-image analysis,” Comput. Vis. Image Underst. 97(1), 51–85 (2005). [CrossRef]
18. S. J. Gortler, R, Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” in Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp 43–54.
19. E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” in Computational Models of Visual Processing (1991), pp. 1–18.
