1. Introduction

Integral imaging (InIm) was firstly proposed by Lippmann in 1908 [1]. It is widely considered as a promising commercial display technique of three-dimensionally visualizing medical and navigational volume data without wearing additional devices like eyeglasses and helmet [2,3]. InIm is based on the reconstruction of multiple light-ray samples for the propagating light rays with specific intensities and directions from each point on objects. Therefore, InIm is one kind of the light field display techniques [4]. The reconstructed light rays in InIm approximate the original light field of displayed objects with natural depth cues involving binocular disparity, motion parallax, color hint and correct geometric occlusion. However, the low sampling rate for the original light field leads to low viewing resolution of the reconstructed 3D image in the early stage of InIm, which is one of the main inherent defects of InIm [5,6]. One of other primary inherent drawbacks of InIm is its narrow viewing angle, due to the limited area where each elemental image can be displayed by its corresponding periodic optics unit [7,8]. These two defects greatly limit commercial application of InIm.

In recent years, numbers of researches have been devoted to overcoming the limitation of viewing angle for InIm [9–11]. Sang et al. recently proposed an interactive InIm based light field display with a large pitch compound lens-array and an optimally designed holographic functional screen (HFS), which can realize a large viewing angle of 45° owing to suppression of aberrations and wavefront recomposing of light distribution [9]. However, its viewing resolution deteriorates obviously because of the large pitch of the compound lens-array. Lee et al. developed a viewing angle enhanced InIm display with a large viewing angle of more than 39° by lens switching, but mechanical movement of the used lightproof mask is difficult and complex in practical implementation [10]. Baasantseren et al. proposed an improved InIm display with the enhancement of the viewing angle using two elemental image masks [11]. The viewing angle of this InIm display is 2 times larger than that of the conventional InIm display with the same lens array. However, the display luminance is reduced to a large degree due to the use of the two elemental image masks.

To overcome the viewing resolution limitation for 3D image presented with InIm, there have been also various related effects made in recent years [12–18]. Non-stationary micro-optics was introduced in InIm to overcome the Nyquist upper limit for the resolutions [12,13], but the practical setup with accurate mechanical movement is difficult to be realized like literature [10]. Spatiotemporally multiplexed projector with a large number of pixels, instead of 2D display panels, was employed in InIm to splice the entire set of high resolution elemental images, which can increase the viewing resolution of the 3D image [14]. However, the operation of projector needs a large space and its adjustment is complex. Among these resolution-improved InIm displays, the PLSA-based InIm display is an effective approach to enhancing the viewing resolution [15–17]. The viewing resolution of the PLSA-based InIm display is determined by the number of the PLSs in the array, which can be significantly increased with the optically designed backlight system. In addition, the PLSA-based InIm can be also used to enlarge the viewing angle. Kim et al. realized the improvement of both the viewing angle and the viewing resolution with a PLSA-based InIm display using electrically movable pinhole array [18]. However, the optical efficiency of the PLSA-based InIm display is low and additional specific brightness-enhanced backlight is required.

Here, a 15.6-inch InIm display prototype based on time-multiplexed lens stitching is demonstrated with effectively enhancing the viewing angle as well as the viewing resolution. The displayed floating full-parallax 3D image can be perceived with the correct geometric occlusion and smooth motion parallax in the viewing angle of 50°, where 7056 viewpoints are presented. Meanwhile, the viewing resolution of the 3D image is also enhanced up to 4 times that of the conventional InIm. It should be pointed out that, image flipping, a common inherent issue for the conventional InIm display implemented using the lens array, can be overcome in our proposed InIm display, because each area on the LCD panel is limited by the PLSA with only one corresponding illumination area.

2. Principle

2.1 Experimental configuration

Similar to the traditional PLSA-based InIm display, our proposed InIm display constructs the PLSA behind the LCD panel, and the light rays emitting from the PLSA are modulated by the LCD panel with the spatial light-field information. As a result, the 3D image is integrated and reconstructed in the space. Different from the traditional PLSA-based InIm display, our display configuration utilizes the DTS-BL to realize time-multiplexed lens stitching for the compound lens-array, and a dense PLSA with a large emitting-ray divergence angle is generated in the time-multiplexed way. In consequence, both the viewing angle and the resolution of the reconstructed 3D image are effectively enhanced.

Principal components of the demonstrated InIm display are a DTS-BL, a compound lens-array, a LCD panel, a HFS and a personal computer (PC) as shown in Fig. 1. In the demonstrated display prototype, the DTS-BL with a 4 × 4 light-emitting diode (LED) array is optically designed for producing 16 parallel light beams with different propagating directions in the time-multiplexed way. The 16 parallel light beams are produced alternately in the chronological order, and specific parallel light beams among them, as marked with yellow color or blue color in the Fig. 1, are generated simultaneously as a group. The parallel light beams from the DTS-BL illuminate the compound lens-array, and then converge and form PLSs with a designed emitting-ray divergence angle at specific locations in the front of the compound lens-array. Through periodically emitting parallel light beams with 16 different propagating directions, a dense PLSA is generated with a large emitting-ray divergence angle. Like the conventional InIm display, our proposed display configuration uses the LCD panel to load elemental image array (EIA). While in our display, synchronization of scanning different EIAs on the LCD panel at a high frame rate synchronizes with generating specific parallel light beams from the DTS-BL is realized, which guarantees that the reconstructed 3D image with enhanced viewing resolution and correct geometric occlusion are integrated using the corresponding viewpoint perspectives. The time synchronization signal, created by the dynamic time-synchronized controller (DTSC) operated in a PC (CPU: Intel Core i7-7700K, GPU: NVIDIA Geforce GTX 970), synchronizes scanning EIAs and generating corresponding parallel light beams. The compound optical components, instead of the standard optical components, are optically optimized and utilized to suppress the aberrations. To eliminate viewing blind areas of the displayed 3D image caused by the PLSA, the HFS is used to recompose the wavefronts of the light rays from the PLSA.

 

Fig. 1 The experimental configuration of the proposed InIm display.

Download Full Size | PDF

It should be noted that the refresh rate of the proposed display system is determined by the used LCD panel. Theoretically, the higher the refresh rate of the proposed display system is, the larger the enhancement of the viewing resolution would become. Unfortunately, current refresh rate of the LCD panel is limited due to the limited development of the display bandwidth for the LC display technology. Moreover, the display system with high refresh rate needs a LCD panel with sufficiently short response time to eliminate the crosstalk between subframes when alternately loading corresponding EIAs in the chronological order. However, the limited response time is a current primary bottleneck question for the LC display technology development. In addition, high refresh rate and short response time require high accurate time-synchronized controller and more complex drive circuitry supporting high frame rate. These requirements will lead to an obvious increase in the cost of the display system. Therefore, the trade-offs mentioned above should be carefully considered for the implementation of time-multiplexed lens stitching.

2.2 Time-multiplexed lens stitching

The viewing angle of the conventional InIm is restricted by the lens-pitch of the lens-array, which can be expressed as

In the demonstrated display prototype, principal components of the DTS-BL are a 4 × 4 LED array and a compound CFL (Circular Fresnel Lens) with two pieces of CFLs, which are used to generate 16 parallel beams with different propagating directions in the time-multiplexed way. It should be pointed out that the use of the compound CFL, instead of the standard CFL, is because the optically optimized compound CFL can efficiently suppress the aberrations. Additionally, in order to effectively alleviate the stray lights emitting from the DTS-BL, the draft facet of each saw-tooth of the compound CFL is covered with a black paint coating in the fabricating process of the compound CFL. Every specific 4 LEDs as a unit in the 4 × 4 LED array are synchronously turned on, noted as the LED-unit k (k from 1 to 4). To simplify description of the time-multiplexed lens stitching concept, we just illustrate the side view of the light-ray path schematic diagram of the proposed InIm display as shown in Fig. 2. As illustrated in Fig. 2, one LED marked with deep yellow and another LED marked with light yellow, which both belong to the LED-unit 2, are turned on simultaneously to project 2 parallel light beams with different propagating directions onto the compound lens-array. These 2 parallel light beams passing through any two longitudinally adjacent elemental lenses in the compound lens-array can converge to the same PLS in the space where the converge angles of defected light rays from those two adjacent elemental lenses are stitched seamlessly, and PLSs with a large emitting-ray divergence angle are obtained. That means the area where the elemental image can be displayed is expanded. Therefore, the viewing angle is obviously enlarged compared to the conventional PLSA-based InIm. As shown in Fig. 2, the viewing angle $\theta$ is corresponding to the inherent lens-pitch of the compound lens-array, while the enhanced viewing angle ${\theta }^{\prime }$ is corresponding to the stitched lens-pitch of the compound lens-array. Furthermore, triggered by the periodic time synchronization signal, the LED-units are turned on and turned off one by one to periodically generate uniformly arranged dense PLSs in the time-multiplexed way at predetermined locations in the front of the compound lens-array as shown in Fig. 2. The PLSs marked with yellow color and the PLSs marked with blue color periodically appear in turn at a high frequency. As a result, the number of the PLSs of the proposed InIm display method is 4 times that of the conventional PLSA-based InIm so that the viewing resolution of the displayed 3D image is enhanced up to 4 times.

 

Fig. 2 The side-view schematic diagram of time-multiplexed lens stitching.

Download Full Size | PDF

In order to obtain uniformly distributed dense PLSA, the parameters of the DTS-BL and the compound lens-array should be properly predetermined. The related parameters include the inherent lens-pitch of the compound lens-array and the focal length of the elemental compound lens in the array, the focal length of the compound CFL and the distance between adjacent LEDs in the array, which are respectively denoted as P, f, F and d as illustrated in the Fig. 2. According to geometric relationship, the mathematical expression between these parameters follows

With these parameters of the DTS-BL and the compound lens-array, the enhanced viewing angle of the proposed InIm display, denoted as $\theta \text{'}$, can be mathematically expressed as

To implement the time-multiplexed operation into the system, a dynamic time-synchronized controller (DTSC) is utilized to accurately determine the gating time of each LED-unit and the corresponding EIA loaded on the LCD panel. The arrangement of the LED in the 4 × 4 LED array and the time sequence diagram of the DTSC are illustrated and depicted in Fig. 3. In one time sequence cycle, triggered by the time synchronization signal, LED-units are respectively lighted up for 5 ms after the corresponding refreshed EIA is loaded on the LCD panel. The DTSC is operated in the PC. Indeed, a 15.6-inch twisted nematic (TN) LCD panel is designed and fabricated specifically for the demonstrated display prototype to meet the requirement of 1 ms response time for loading different EIAs. By the use of the PC, the frame rate of the LCD panel reaches 144 fps.

 

Fig. 3 (a) The arrangement of the LED in the 4 × 4 LED array and (b) the time sequence diagram of the DTSC.

Download Full Size | PDF

2.3 Optical optimization for the DTS-BL and the compound lens-array

As we all known, with the size of the lens-pitch increasing, the field angle of the light rays passing through lens increases, which will aggravate the aberrations, especially for the marginal light rays. In our proposed InIm display, the optical optimization for the design of the DTS-BL and the compound lens-array is implemented for suppressing the aberrations. In this way, a dense PLSA can be accurately produced with uniform arrangement at certain predesigned locations in space, and the high quality 3D image can be integrated by numbers of recovered light rays with corresponding spatial viewpoint information from the PLSA. The compound CLF of the DTS-BL and the compound lens-array are designed with two aspheric surfaces and two different refractive indices, and both of them are jointly optimized as a combination for optical design. The aspheric surface formula is given in Eq. (4)

 

Fig. 4 The schematic diagrams of designed structures of (a) the compound CLF and (b) the compound elemental lens.

Download Full Size | PDF

The spot diagrams for the imagery of the PLSA using the optimized aspheric lens combination, as well as the standard lens combination with the same focal lengths and the same diameters, are given in Fig. 5. Figure 5 shows that the biggest root mean square (RMS) radius of 5.625 mm imaging height of the optimized aspheric lens combination is 62.058 μm, while that of the standard lens combination is 657.328 μm. That is to say, the aberrations are well suppressed and the high imaging quality for the PLSA is achieved with the optimized aspheric lens combination compared to the standard lens combination.

 

Fig. 5 Comparisons of spot diagrams for the generated PLSAs using the standard lens combination and the optimized aspheric lens combination.

Download Full Size | PDF

2.4 Coding method for light-field reconstruction

The coding method based on backward ray-tracing technique for efficiently and concisely rendering EIAs is applied. It guarantees the reconstructed 3D image with a right 3D light-field perception and occlusion in the viewing zone, which geometrically maps the pixels of multiple off-axis recorded parallax images (PIs) captured by a camera array (CA) to the pixels of the scanned EIAs. It should be noted that, all of the scanned EIAs are rendered and obtained at the same time by the use of the coding method. According to the light-field pickup process and the designed viewing distance of the proposed display, the interval and field of view (FOV) of the camera in CA can be expressed as $d_c=\frac{W_E}{g}(L+g)$ and $fov=2\arctan \frac{P(n-1)}{4(L+g)}$ where L is the distance between the CA and the LCD panel, n is the lateral number of the PLSs and $W_E$ is the size of the pixel on the LCD panel. For the convenience of description, we simplify the proposed InIm display to illustrate the principle of the proposed coding method, which is shown in Fig. 6. In fact, the proposed coding method is the eye-based light-field pickup inversion to avoid pseudoscopic image, a most common inherent issue in autostereoscopic displays. As shown in Fig. 6, the position coordinate of arbitrary pixel of the EIA rendered at the time t is represented by $(xt,yt)$, and $(x_P^k,y_P^l)$ denotes the position coordinate of its corresponding k-th column and l-th row PLS in the PLSA. $(u,v)$ represents the position coordinate of the corresponding pixel on the PI captured by the i-th column and j-th row camera in the CA. The mathematical expression of the mapping relationship between pixels of the EIA and the PIs can be derived as Eq. (5), where $(u_C^i,v_C^j)$ denotes the position coordinate of the first pixel of the PI captured by the i-th column and j-th row camera.

 

Fig. 6 The code mapping relationship between pixels of EIAs and PIs based on backward ray-trace.

Download Full Size | PDF

2.5 Eliminating viewing blind areas based on wavefront recomposing

Viewing blind areas of the displayed 3D image appear in the viewing zone owing to the gap between PLSs in the PLSA, which become obvious when the observers are close to the reconstructed 3D image. In order to eliminate the viewing blind areas perceived by the viewers, the HFS is used to recompose the wavefronts of the light rays from the PLSA. The HFS is holographically printed with speckle patterns exposed on proper sensitive material and its diffusion angle is determined by the shape and size of speckles [9,19]. Considering the produced PLSA is vertically and horizontally symmetrical, the HFS is isotropous for diffusion angle. In our InIm prototype, the HFS is located in front of the LCD panel as shown in Fig. 7(a). Assuming that the diffusion angle of the HFS is $\varphi$ and the distance between the HFS and the PLSA is $d_H$, the mathematical expression of $\varphi$ can be derived as Eq. (9). Figure 7(b) shows that the viewing blind areas can be sufficiently alleviated with the HFS and the perceptive quality of the reconstructed 3D image is improved.

 

Fig. 7 (a) The schematic diagram of eliminating viewing blind areas with the HFS, and (b) comparison of perceptive effects for observed image with the HFS and not.

Download Full Size | PDF

3. Experimental results and discussions

In the demonstrated display prototype as illustrated in Fig. 8, a 15.6-inch TN LCD panel, a DTS-BL, a compound lens-array, a PC and a HFS are employed. It is worth mentioning that the LED array used in the demonstrated display prototype is designed and manufactured with the optimization of the uniformity of illuminance. The uniformity of illuminance, defined as the ratio of minimum illumination value to average illumination value within the illuminated area, of each parallel light beam produced by the DTS-BL is higher than 95%, which is acceptable for the viewers in practical application. The configuration of the display prototype is listed in Table. 1. The floating 3D image can be reconstructed with the acceptable viewing resolution in the viewing angle of 50°, where 84 × 84 viewpoints are presented. The floating focus depth of the reconstructed 3D image can reach 20 cm.

 

Fig. 8 (a) Experimental setup and (b) the compound lens-array.

Download Full Size | PDF

Table 1. Configuration of experiments

View Table

With the implementation of the optical optimization for the design of the DTS-BL and the compound lens-array, the imaging distortion for the PLSA can be obviously improved in the whole viewing zone, which facilitates the correct 3D light-field image reconstruction with the large viewing angle without crosstalk. The experimental results of illumination areas caused by the top left corner of the PLSA using optically optimized lenses and the standard lenses are experimentally shown in Fig. 9, and mesh distortion maps of the illumination area centers are measured to evaluate imaging distortion for the PLSA. As shown in Fig. 9, the distortion of the imagery of the PLSA with the optically optimized lenses is 0.23%, while the distortion of the imagery of the PLSA with the standard lenses is 5.80%. It is experimentally demonstrated that the imaging distortion with the optically optimized optics is sufficiently decreased compared to that of the PLSA with the standard lenses.

 

Fig. 9 Comparison of mesh distortion maps for the illumination area centers produced by the PLSAs using the standard lenses and the optically optimized lenses respectively.

Download Full Size | PDF

One of the prominent applications of the proposed InIm display is for medical analysis and diagnosis. The medical data of human skull is used to demonstrate feasibility and superiority of our proposed InIm display. Figure 10 shows the comparison of display effects for the conventional PLSA-based InIm method and the proposed method without the HFS. It is experimentally verified that the viewing resolution of the proposed method based on time-multiplexed lens stitching is 4 times as many as that of the conventional method.

 

Fig. 10 Comparison of display effects for (a) the conventional PLSA-based InIm method and (b) the proposed method without the HFS.

Download Full Size | PDF

A 3D image of human skull displayed with the prototype is captured from different directions as shown in Fig. 11 and Visualization 1. The captured pictures and video show that a natural 3D scene with smooth full-parallax and high viewing resolution in the 50° viewing angle is presented for the observers. In addition, a 3D image of city terrain, where some buildings with different heights are presented, is displayed and captured with a camera focusing the white building in the scene as shown in Fig. 12 and Visualization 2.

 

Fig. 11 3D image of human skull observed from different directions (see Visualization 1).

Download Full Size | PDF

 

Fig. 12 3D image of city terrain (see Visualization 2).

Download Full Size | PDF

Compared to the previously reported InIm displays, one main improvement of our proposed InIm display is to enlarge the viewing angle by 50° with increasing the viewing resolution by 4 times as shown in Figs. 11, 12 and 10 respectively. The effective enhancement of the viewing angle is realized by lens stitching implemented with directional backlight. Furthermore, the time-multiplexed operation is applied to the directional backlight, and the time-multiplexed backlight with 4K LCD display is then used to increase the display bandwidth. As a result, the number of voxels of the 3D image becomes 4 times as many as that of the conventional InIm display. Actually, our proposed method for the viewing angle and the viewing resolution enhancement could be helpful to the improvement for the light field displays implemented using refraction-based light-controlled optical components with periodic arrangement.

In addition, another contribution of our proposed method for the realization of the light field display is to apply optical optimization for principal optical components in order to suppress the aberrations. The joint optical optimization for the DTS-BL and the compound lens-array is implemented based on the damped least-squares method. As a consequence, the aberrations are well suppressed as demonstrated in Figs. 5 and 9. Besides, the viewing blind areas of the displayed 3D image, a common inherent issue for the PLSA-based InIm displays, are eliminated based on wavefront recomposing. In this way, the 3D image quality is obviously improved as shown in Fig. 7(b).

However, the proposed InIm display suffers from the limited viewing resolution enhancement. According to the principle of our method, the viewing resolution enhancement is due to increasing the number of the PLSs in the array and scanning the corresponding EIAs on the LCD panel in the time-multiplexed way. Theoretically, because of the sufficiently high refresh rate of the LED, it is easy to increase the number of the PLSs just by increasing the density of the LED in the LED array. However, increasing the refresh rate of scanning EIAs on the LCD panel is difficult because the limitation of the display bandwidth of the LCD display exists in the current stage of the liquid-crystal display technology. Hence, the viewing resolution enhancement of our proposed system is up to the limitation of the display bandwidth of the LCD display. The image flicking would be observed for the viewers if the refresh rate of the LCD panel is not enough high. Therefore, the requirement of the LCD panel with high performance parameters of display bandwidth and response time is an inherent primary drawback of our proposed InIm display. In addition, high display bandwidth and short response time need high accurate time-synchronized controller and complex drive circuitry supporting high frame rate, which will further lead to a significant increase in the cost of the display system. In the next research step, we will try to use multiple projectors with a large number of pixels like the previously reported research [20], instead of the LCD panel, to address the issue of the limited viewing resolution enhancement.

4. Conclusion

In summary, an effective method for enhancing the viewing-angle and the viewing-resolution of InIm is proposed, and the display prototype based on time-multiplexed lens stitching is experimentally demonstrated using the DTS-BL and the compound lens-array. In order to enlarge the viewing angle of InIm, the elemental lenses of the compound lens-array are continuously stitched to obtain a PLSA with a large emitting-ray divergence angles. To increase the viewing resolution of the perceived 3D image, time-multiplexed lens stitching for generating the PLSs is implemented to effectively increase the number of the PLSs in the array. To precisely generate uniformly arranged dense PLSA, the DTS-BL and the compound lens-array are optically optimized for suppressing the aberrations. As a result, the imaging distortion for the PLSA can be decreased to 0.23% from 5.80%, which ensures high quality of the reconstructed 3D image free from crosstalk between different viewpoint perspectives in a large viewing area. In order to efficiently render natural depth cues of complex 3D scene, a coding method for light-field reconstruction based on backward ray-tracing is employed. Additionally, the optimally designed HFS is used to recompose the light wavefronts for eliminating viewing blind areas of the displayed 3D image caused by the PLSA. In our demonstrated experiments, high quality 3D images of human skull and city terrain with natural depth cues are presented with 84 × 84 viewpoints and acceptable viewing resolution in the viewing angle of 50°.