Image quality improvement of multi-projection 3D display through tone mapping based optimization

Peng Wang; Xinzhu Sang; Yanhong Zhu; Songlin Xie; Duo Chen; Nan Guo; Chongxiu Yu

doi:10.1364/OE.25.020894

1. Introduction

In recent years, three-dimensional displays which can reproduce the 3D scenes without glasses earn lots of attention [1]. Multi-projection auto-stereoscopic 3D display, with the characteristics of easy scalability and high display resolution, is one of the most popular display technologies in research. Those displays are composed of different kinds of 3D screens and projector arrays, as shown in Fig. 1. According to the optical structure of the screen, those displays can be divided into parallax barrier or lenticular sheet based 3D display [2–7], and holographic optic elements (HOEs) based 3D display [8–11]. All those displays can be designed into rear-projection display and front-projection display, and the diagrams of the two types are shown in Fig. 1. The 3D screen of a front-projection 3D display usually consists of optical elements and a diffuse reflector, and the 3D screen of a rear-projection 3D display has only transparent optical elements. The optical element has the ability of splitting the incident light to different desired directions.

Fig. 1 The diagrams of multi-projection auto-stereoscopic 3D display. (a) A front projection 3D display, (b) a rear-projection 3D display.

Download Full Size | PDF

Ideally, a viewer observes the appropriate view at the corresponding viewing position and the captured light field is. $L_{d} = L_{d_s}$ However, an optical 3D screen shows a certain diffuse reflection or diffuse transmission [12], and the actually captured light field is $L_{d} = L_{d_s} + L_{d_r} + L_{a m b}$ , as shown in Fig. 2. $L_{d_s}$ is the light field of the observed view, $L_{d_r}$ is the diffuse reflective or diffuse transmitted light from all projected light field sequence with parallax, $L_{a m b}$ is the ambient light reflected from a display surface. The captured display images of a front-projection 3D display [6] and a rear-projection 3D display [9] are provided in Fig. 2(c), and the shadow parts show the existing of the diffuse reflection and the diffuse transmission. The crosstalk of the diffuse reflective or diffuse transmitted multi-projection contents and the ambient light decreases the definition and the local contrast of display light field, compared with the original 3D scene.

Fig. 2 The luminance models of the multi-projection 3D display. (a) A front-projection 3D display, (b) a rear-projection 3D display, (c) the diffuse reflective or diffuse transmitted content of single projector, the “Left” is the projected image, the “Center” is the displayed image of a front-projection 3D display [6], and the “Right” is the displayed image of a rear-projection 3D display [9].

Download Full Size | PDF

The crosstalk in stereoscopic displays is analyzed [13–15]. The crosstalk in time sequential 3D display with shutter eyewear was modeled based on psycho-visual responses, and the stereoscopic video was adjusted according to the model [14]. The crosstalk in color anaglyph was analyzed, which was reduced by introducing a red lightness to compensate for the red band loss due to eyewear devices [15]. A few works were also demonstrated for solving the crosstalk existing in the auto-stereoscopic 3D display by improving the optical design [16,17]. Those methods are effective, but either the methods were designed for stereoscopic 3D display or the optical structures of the 3D screens should be specially designed for autostereoscopic 3D display. Those methods cannot be used to reduce the crosstalk due to the diffuse reflective or diffuse transmitted multi-projection contents as demonstrated in this paper. In 2D image processing and display, a series of appearance improvement and contrast enhancement methods were demonstrated [18], and the tone mapping operator (TMO) can be used to adjust the tone of the image to preserver or enhance the scene contrast appearance [19–21]. Inspired by this, a TMO based contrast optimization method is firstly proposed to improve the image quality of the multi-projection 3D display, including suppressing the crosstalk of the multi-projection contents and improving the display local contrast in a computational way. The image quality of 3D display is improved by adjusting the details of the projected images with parallax from multiple projectors through TMO.

In the proposed method, the display model describing the luminance of the 3D display is analyzed, and the ambient light is also taken into consideration. Based on the display model, a linear piece-wise TMO is proposed to remap the light field sequence of multiple projectors, and the visible contrast distortions between the combined display light field and the desired reference light field without crosstalk or even with enhanced contrast are minimized by optimizing the tone mapping curve. Such a TMO is naturally formulated as an optimization problem, where the error function is weighted with the human visual system (HVS) model and constrains are dictated by the multi-projection 3D display model. The framework of our proposed display contrast optimization method is shown in Fig. 3. The validity of the proposed method is evaluated in both objective and subjective investigations.

Fig. 3 The framework of proposed display contrast optimizing method. Display model #1 denotes the luminance model of single projector, display model #2 denotes the multi-projection light field combination.

Download Full Size | PDF

2. Display contrast optimization method

The framework of our proposed tone mapping based optimization method is shown in Fig. 3. The display model #1 denotes the luminance model of single-projection display, and the display model #2 is the process of multi-projection light field combination. The original light fields are remapped according to the tone mapping curve, and the tone mapped light fields are combined to the final display light field. In our proposed framework, the original light field without crosstalk is used as the reference light field. A light field enhancement operator is carried to modify the reference light field to improve its appearance, and an enhancement of contrast by 15% is proposed [22]. Then visible contrast distortions between the final display light field and the reference light field with contrast enhancement are minimized by optimizing the tone mapping curve.

2.1 Display model for multi-projection display

A normal display model primarily accounts for the limited capabilities of a display device, such as maximum brightness, dynamic range. The multi-projection 3D display model will characterize the luminance model of the single-projection display and the multi-projection light field combination.

The luminance diagrams of the multi-projection 3D display are shown in Fig. 2. In a front-projection 3D display, the diffuse reflection on the optical elements surface takes responsibility for the decrease of the display contrast. While in a rear-projection 3D display, both the diffuse reflection and diffuse transmission should take responsibility for the decrease of the display contrast. Both the diffuse parts can be approximately modeled using a lambertian diffuse model [20]. The single-projection light field can be denoted as $L_{p}$ . Assuming a single-projection display with a lambertian diffuse screen which has a solid angle of 2π and reflectivity of 1, the available range of luminance can be modeled using the standard Gamma-Gain-Offset display model [23]. To keep the following TMO problem analytically tractable, the ambient light illuminance is averaged to the single-projection model. As shown by Eq. (1),

L_{p} (V) = \frac{1}{π} \cdot [{(V)}^{γ} \cdot (E_{\max} - E_{b l a c k}) + E_{b l a c k} + \frac{1}{N} E_{a m b}]

where

L_{p}

is the display luminance with the units [cd/m²], V is the pixel value ranging among [0, 1], and

^{γ}

is display gamma, usually 2.2 in sRGB space.

E_{m a x}

is the maximum illuminance projected by the projector, denoted in [lux], and

E_{b l a c k}

is the black level illuminance.

E_{a m b}

is the ambient illuminance, N is the number of projectors. For a lambertian reflector with a solid angle of 2π, E = πL. The luminance of the displayed view apart from the diffuse reflection or the diffuse transmission part can be denoted as L_{d_s}, the luminance of the diffuse reflected or the diffuse transmitted part by the optical elements can be denoted as L_{d_r}, which is also the crosstalk part. The luminance models can be defined as Eqs. (2) and (3).

L_{d_s} (V) = ρ_{s} \cdot L_{p} (V)

L_{d_r} (V) = ρ_{r} \cdot L_{p} (V)

In a front-projection 3D display,

ρ_{s}

is the reflection coefficient of the 3D screen,

ρ_{r}

is the diffuse reflection coefficient of the optical elements. While for a rear-projection 3D display,

ρ_{s}

is the transmission coefficient of the 3D screen,

ρ_{r}

is the diffuse transmission coefficient of the optical elements. All the possible light field redistributions introduced by the optical elements are taken into consideration in the coefficients [6]. The final displayed multi-projection light field is a combination of the multi-projection light field sequence as shown in Fig. 2. So the luminance model of the combined multi-projection light field is given as Eq. (4),

L_{d} (V_{k}) = L_{d_s} (V_{k}) + \sum_{i = 1}^{N} L_{d_r} (V_{i})

where N is the number of the views,

V_{k}

is the pixel value of the view k, ranges among [0, 1]. The display model of multi-projection 3D display expounds the captured luminance at any view k corresponding to the k-th projector, the first part at right side of the equal sign means the luminance of view k, and the second part means the luminance of multi-projection crosstalk.

Although the model from Eqs. (1) to (4) can be employed separately for each trichromatic primary (red, green, blue), this model is only used for luminance values since the color issues are not in the scope of this work. To retain the color information from the reference image after a tone mapping operator, we employ the desaturated color-to-luminance ratios as Eq. (5), C denotes the trichromatic value, L is the pixel luminance and in/out subscripts denote pixels before and after tone mapping. All colors are given in linear color space (no gamma-correct). s_a controls color saturation, s_a = 0.6 is used for our results, and more details are given in [24].

C_{o u t} = {(\frac{C_{i n}}{L_{i n}})}^{s_{a}} L_{o u t}

2.2 Human visual system model

The model of HVS processes input luminance data to produce the estimated response. To estimate the response of HVS to the contrast stimulus of the tone mapped light field, the HVS based contrast visible predicting model proposed by Wilson [25] is introduced. The model is a transducer function of contrast W and sensitivity S,

R = T (W, S)

where the resulted value R is a HVS response given in JND (Just Noticeable Difference) units. W is the Weber contrast, and the sensitivity S is the inverse of the detection threshold, which is modeled with Contrast Sensitivity Fucntion (CSF) [26],

S = C S F (ρ, L_{a}, v_{d})

, f is the frequency given in cycle per degree [cpd], L_a is the adapting luminance with the unit [cd/m²], and v_d is the viewing distance. From the above model, the response of the HVS for a given contrast value W can be obtained. To find a set of contrast values of the light field, the Laplacian pyramid [27] is employed. Firstly, the logarithm of luminance values is computed,

I = \log_{10} (L)

. Then the Gaussian pyramid I_l is computed from I, the contrast in the logarithm ratio unit for the l-th frequency band is G_l = I_l – I_{l + 1}, with I₁ = I. Finally, the Weber contrast W is computed from G,

W = 1 0^{| G |} - 1

.

2.3 Tone mapping as the minimum visible distortion problem

As shown in Fig. 3, TMO will operate on the multi-projection light field sequence, and the tone mapped light fields are combined to generate the display light field. Then an objective function based on HVS predicting the visual contrast distortions between the display light field and the reference light field is adopted to optimize the tone mapping curve.

TMO is always used to map HDR luminance values to the display’s luminance range, and the tone mapping curve is always continuous and non-decreasing. According to the Web-Fechner law [28], the sensitivity of the HVS to light is proportional to the logarithm of luminance. Thus our TMO will operate on the logarithm value of the luminance. To keep the problem analytically tractable, the tone mapping curve is designed as a linear piece-wise and non-decreasing function with the node $(x_{m}, y_{m})$ as shown in Fig. 4.

Fig. 4 Parameterization of linear piece-wise tone mapping curve.

Download Full Size | PDF

Each segment m between two nodes $(x_{m}, y_{m + 1})$ and $(x_{m + 1}, y_{m + 1})$ has a constant width δ. The tone mapping curve can be specified by a set of slopes,

d_{m} = \frac{y_{m + 1} - y_{m}}{δ}

and the forward tone mapping function is,

\begin{matrix} y (x, d_{1}, \cdot \cdot \cdot, d_{M}) = (x - (m - 1) \cdot δ) \cdot d_{m} + \sum_{i = 1}^{m - 1} d_{i} & w i t h & 1 \leq m \leq M \end{matrix}

where x locates in the m-th segment, m is a maximum integer which is no larger than (x/δ + 1). The luminance of the combined light field at the k-th view after TMO can be denoted as,

L_{d} (V_{k}, d_{1}, \cdot \cdot \cdot, d_{M}) = ρ_{s} \cdot 10^{y (\log_{10} (L_{p} (V_{k})), d_{1}, \cdot \cdot \cdot, d_{M})} + ρ_{r} \cdot \sum_{i = 1}^{N} 10^{y (\log_{10} (L_{p} (V_{i})), d_{1}, \cdot \cdot \cdot, d_{M})}

_{$V_{k}$} is the pixel value of the k-th view. The minimum log-luminance value that can be shown on a single projection display is fixed at 0, and the maximum value varies with the dynamic range r,

r = \log_{10} (L_{p} (1) / L_{p} (0))

The most relevant distortion due to TMO is the change of light field contrast, and to minimize the visual contrast distortions to the reference, the objective function measuring contrast distortions with HVS model is defined as Eq. (11). The visible contrast distortions between the combined display light field and the desired reference light field without crosstalk or even with enhanced contrast are minimized by optimizing the tone mapping curve, which is denoted as a set of slopes [d₁, d₂, …, d_M ].

\begin{matrix} E (d_{1}, \cdot \cdot \cdot, d_{M}) = \arg \min \sum_{k = 1}^{N} ‖ T (P_{L} (\log_{10} (L_{d_k}^{s} (d_{1}, \cdot \cdot \cdot, d_{M}))), S_{d}) \\ {- T (e \cdot P_{L} (\log_{10} (L_{d_k}^{r} (d_{1}, \cdot \cdot \cdot, d_{M}))), S_{r}) ‖}_{2}^{2} \end{matrix}

subject to:

\begin{matrix} d_{m} \geq 0 & f o r & m = 1... M \end{matrix}

\sum_{m = 1}^{M} d_{m} \cdot δ \leq r

where N is the number of views, M is the number of the segments of tone mapping curve. The first constraint ensures that the tone curve has a non-negative slope and the second one ensures that the tone mapped light field do not exceed the display’s dynamic range.

P_{L} (L)

denotes the operation of Laplacian pyramid decomposition, and the resulted value is converted into Weber contrast, L is the luminance map.

L_{d_k}^{s} (d_{1}, \cdot \cdot \cdot, d_{M})

is the luminance map of combined light field for the k-th view after tone mapping.

L_{d_k}^{r} (d_{1}, \cdot \cdot \cdot, d_{M})

is the referential luminance map for the k-th view, and the original light field of the k-th view without crosstalk is adopted as the reference. e is the contrast enhancement factor. Due to the difference in the viewing conditions, sensitivity can be different for a displayed light field and a reference light field.

S_{d} = C S F (f_{l}, L_{d_k}^{s}, v_{d})

for the tone mapped combined light field

L_{d_k}^{s}

, f_l is the frequency of the l-th frequency band in Laplacian pyramid, and

S_{r} = C S F (f_{l}, 1000, v_{d})

for reference light field

L_{d_k}^{r}

, assuming a luminance of adaptation L_a = 1000 cd/m² as proposed by [20].

2.4 Effective solution

The optimized tone mapping curve is achieved by minimizing the objective function Eq. (11). However the objective function is a high order nonlinear function, which is time-consuming in optimization, so an effective solution is proposed. The Eq. (4) can be modified into matrix notation as,

\begin{matrix} L_{d_k} = ρ_{s} \cdot L_{p_k} + ρ_{r} \cdot {L^{'}}_{p} \\ = (ρ_{s} + ρ_{r} \cdot A) \cdot L_{p_k} \end{matrix}

with

{L^{'}}_{p} = \sum_{i = 1}^{N} L_{p_i}

where L_{p_k} is the luminance map of the observed k_th view, L_{p_i} is luminance map of the view projected by the i-th projector.

A = {L^{'}}_{p} / L_{p_k}

. If the reference light field is assumed to be

L_{d_k}^{r} = (ρ_{s} + N \cdot ρ_{r}) \cdot L_{p_k}

, the objective function can be modified as Eq. (13), more details are provided in Appendix section.

\begin{matrix} E (d) = argmin \sum_{k = 1}^{N} ‖ T (P_{L} (\log_{10} (ρ_{s} + ρ_{r} \cdot A)) + B (P_{L} (\log_{10} (L_{p_k}))) \cdot δ \cdot d, S_{d}) \\ {- T (P_{L} (\log_{10} (ρ_{s} + N \cdot ρ_{r})) + B (P_{L} (\log_{10} (L_{p_k}))) \cdot e \cdot (r / M), S_{r}) ‖}_{2}^{2} \end{matrix}

with

d = {[d_{1}, d_{2}, \cdot \cdot \cdot, d_{M}]}^{T}

subject to:

\begin{matrix} d_{m} \geq 0 & f o r & m = 1... M \end{matrix}

\sum_{m = 1}^{M} d_{m} \cdot δ \leq r

The TMO in Eq. (9) is modified into a matrix multiplication operation as the second part in the transducer function, and the first transducer function in Eq. (13) can be written into a matrix notation as,

T (s + δ \cdot B d, S_{d}) \approx K B d

where s is the column vector of each resulting value of

P_{L} (\log_{10} (ρ_{s} + ρ_{r} \cdot A))

, and this part in the transducer function is free of TMO. d is the column vector of d_m (m = 1, …, M), B is a 0/1 matrix with M columns, where each row represents which segments in tone mapping curve are remapped according to each resulting value of

P_{L} (\log_{10} (L_{p_i}))

. K is a diagonal matrix,

K_{i i} = \frac{T ({[s + δ \cdot B d]}_{i}, S_{d})}{{[B d]}_{i}}

_{$K_{i i} = 0$}for

{[B d]}_{i}

= 0. The second transducer function in Eq. (13) does not depend on d_m, which can be pre-computed as column vector C, and the Eq. (13) can be modified as,

E (d) = \arg \min {‖ K B d - C ‖}_{2}^{2}

The modified objective function is a standard quadratic problem, can be easily and fast optimized [29]. Our method is implemented in C + + code, on a 3.4GHZ CPU, with a RAM of 16G. To optimize a parallax image sequence with resolution of 1280 × 720, it takes about 10 seconds for one iteration, and the objective function can usually be solved in 2-7 iteration.

3. Experiment and results

3.1 Experimental system

In the following experiments, we validate that our tone mapping based optimizing method can suppress the crosstalk and improve the light field contrast compared with the original light field with a front-projection 3D display. Two different kinds of experiments are designed, one is the display image quality improvement without ambient light (E_amb = 0 lux, dark room), and the other one is with the ambient light (E_amb = 400lux, office). A multi-projection 3D display with 24 projectors in front-projection mode is adopted as the experiment display [6], as shown in Fig. 5, the diagram of the structure is provided in Fig. 1(a). The parameters of the display system are listed in Table 1. The lenticular sheet with a lens pitch of 1.5 mm and a focal length of 20 mm is specially optimized for the 3D screen to achieve a good image quality with a viewing angle of 45 degrees [7]. The projector array is assigned 3.2 m in front of the 3D screen, with a projector area of 1.8 $\times$ 1.2 m². The viewing distance is 3.2 m, and a viewing pitch of 240 mm is got. The display response is measured with the Minolta T10A illuminance meter and LS-150 luminance meter, which is used as a display model to generate images used in our method. The maximum illuminance level of single projector E_max is 100 lux with a projector area of 1.5 m².

Fig. 5 The multi-projection 3D display used in experiment. (a) The front-projection 3D screen, (b) projector array used in the display system, (c) the luminance angular distribution of the 3D display.

Download Full Size | PDF

Table 1. Parameters of the front-projection 3D display

View Table | View all tables in this article

The luminance angular distribution of the 3D display in the viewing zone is measured with the luminance meter, as provided in Fig. 5(c). The maximum luminance of the reflected crosstalk part L_{d_r_max} is about 61 cd/m², and the maximum luminance of the signal part L_{d_s_max} is about 342 cd/m², the ambient luminance is about 67 cd/m². The display shows a good uniform luminance distribution in the field of view (45 degrees). So in our 3D display, the changes of the luminance according to different viewing positions can be ignored to simply the display model, the ρ_s is defined as $ρ_{s} = π \cdot L_{d_s_\max} / E_{\max}$ , and the ρ_r is defined as $ρ_{r} = π \cdot L_{d_r_\max} / (N \cdot E_{\max})$ .

3.2 Experimental results

In this section, we demonstrate the crosstalk suppression and display contrast improvement capabilities of our method. All the provided images in the results are reconstructed according to the metered maxinum display luminance, and they are converted to the linear trichromatic values assuming the RGB color space. To measure the contrast and the definition performance of the reconstructed light field, the local contrast G is defined as the gradient of the log-luminance map for a simplified analysis [30]. The PSNR in terms of local contrast is used to evaluate the reconstructed local contrast performance compared with the reference light field, as defiend in Eq. (17).

P S N R = 10 \cdot \log_{10} (\frac{G_{M A X}^{2}}{M S E})

with

M S E = \frac{1}{m n} \sum_{i = 0}^{m - 1} \sum_{j = 0}^{n - 1} {‖ G_{s} (i, j) - G_{r} (i, j) ‖}_{2}^{2}

where G_MAX is the maximum local contrast value in the reference light field. G_s is the contrast map of the display light field, and G_r is the contrast map of the reference light field. The PSNR can measure the difference of the local contrast between the display light field and reference light field.

A. Objective investigation

The experiment with a simple 3D object without ambient light is designed to show the validity of our proposed method, it help to distinguish the crosstalk reduction and the contrast enhancement, the results are provided in Fig. 6. The “Target” images show the reference light field, the PSNRs are calculated respectively for the “No change” and “With our method” light fields compared with the target one in our experiments. The crosstalk is suppressed and the local contrast is enhanced with our method compared with the original light field (No Change). The luminance distribution and local contrast distribution also prove the validity of our method and the PSNR is enhanced from 27.1dB to 29.8dB.

Fig. 6 Results with a simple 3D object. The luminance distribution and the local contrast distribution of the pixels on the yellow dash line in the images are provided. The images depict image appearance on a display, which however does not convey actual contrast or brightness due to print limitations.

Download Full Size | PDF

The scenes in real world are usually complex with abundant color (luminance). So two more complicated and natural scenes are demonstrated and fully analyzed in this section, one is “Desert” and the other is “Football”. Results of the experiments without ambient light and with ambient light are respectively shown in Figs. 7 and 8. And a subjective study is also carried out to evaluate the validity of the proposed method with those nature scenes.

Fig. 7 Results without ambient light. (a) Results of the scene “Desert”, (b) results of the scene “Football”. Both the scenes are enhanced in contrast with a factor e = 1.15. The images are recommended to be shown in a zoomed view. Notice the difference in detail visibility. The results of different views are provided in Visualization 1 (see Visualization 1).

Download Full Size | PDF

Fig. 8 Results with ambient light. (a) Results of the scene “Desert”, (b) results of the scene “Football”. Both the scenes are enhanced with a factor e = 1.45. The results of different views are provided in Visualization 1 (see Visualization 1).

Download Full Size | PDF

As shown in Fig. 7, the “Target” images show the reference light field with a contrast enhancement with a factor of e = 1.15, the “No Change” images show the original display light field with the decreased contrast due to the crosstalk of multi-projection contents. The original light field shows an apparently decreased contrast and definition compared with the reference light field. After the contrast optimized operator, the “With Our Method” images are provided, which show an apparent local contrast improvement compared with the original light field. The definition is improved and the crosstalk is apparently suppressed. Such as the “Desert”, the PSNR of the original light field to the reference light field is 29.7dB, which is improved to 31.4 dB with our method. The difference in visibility is shown in close-ups. As the log-luminance and the local contrast distributions of the pixels on the yellow dash line (in the yellow box) show, even though the crosstalk can be well suppressed, it cannot be eliminated. Although the local contrast of the optimized light field is higher than the original one, it is lower compared with the targeted one. The tone mapping curves for the light field sequence are also given, the blue line shows the optimized tone mapping curve with our method, and the red dash line shows the original tone curve without change.

Our proposed method can also be used to suppress the decrease of contrast due to ambient light. Results of the experiment with ambient light are shown in Fig. 8. A contrast enhancement factor of e = 1.45 is adopted to apparently enhance the degenerated color appearance due to the ambient light, which is shown in the Fig. 8(b). With the ambient light, the displayed combined light fields show an apparently decreased contrast. With our proposed method, the display contrast of the original light field is apparently improved. The PSNRs of the two “No Change” scenes are 23.4dB and 24.3 dB respectively and are improved to 25.3 dB and 27.4 dB respectively with our method. Even the definition and the local contrast of the optimized light field are lower than the targeted one, but they are significantly higher than those of the original one. The close-ups of both scenes, the luminance distributions, and the local contrast distribution of details are provided. The results of different views of both scenes are also provided in Visualization 1 as a supplementary material, the results of different views show good spatial coincidence after contrast optimizing operation. Human visual system is more sensitive to brightness contrast than absolute luminance [31], so the local contrast is mainly used to be restored in our method. As we can see from the luminance and local contrast distributions in Figs. 7 and 8, the local contrasts of the remapped light fields with our proposed method are improved compared with the original light fields at the cost of changing the original luminance distributions. Our proposed method is also verified with other four scenes, and one of the scenes is provided in Fig. 9. Other results are provided in the supplementary material.

Fig. 9 Reuslts of the scene “Animals”. (a) The results without ambient light, the contrast enhancement factor is 1.15, (b) the results with ambinet light, the contrast enhancement factor is 1.45. The results of different views are provided in Visualization 1 (see Visualization 1).

Download Full Size | PDF

The HVS based visible contrast distortions of the scenes “Desert” and “Football” are measured with the HDR-VDP-2 method [32]. The threshold normalized contrast maps of the visible contrast different between the original light field and remapped light field with our method are provided as Fig. 10. The image quality value Q which quantify the visual distortion with a single value of quality score is also provided for each scene, a value of 100 means no visible contrast difference. Our proposed method improves the local contrast and suppresses the crosstalk at the cost of changing the luminance distribution of the light field. There exists trade-off between the amount of crosstalk and visible image artifacts, and the trade-off can be qualitatively analyzed with relationship between the contrast enhancement factor and visible image artifacts, because a higher local contrast enhancement means a better crosstalk suppression. The results are shown in Table 2, as we increase the contrast enhancement factor e, the image quality Q becomes lower in both scenes.

Fig. 10 The HVS based visible contrast difference between the tone mapped light field and original light field. The contrast distortions are demonstrated as the threshold normalized contrast maps, and the image quality values Q is also calculated.

Download Full Size | PDF

Table 2. The trade-off between contrast enhancement factor and image quality

View Table | View all tables in this article

B. Subjective investigation

A subjective study is also carried out to evaluate the validity of the proposed method. Nine participants attend, who are naive about the purpose of the experiment. Each participant is asked to choose the light field which has better contrast and definition from a pair of display light fields. One is the original light field without change, and the other is the contrast optimized light field. In the experiment, 6 scenes are evaluated with 3 repetitions (the others are provided in the supplement), and the two light fields in one pair are demonstrated in random order. The scores (the times of the light field is selected) are listed in Table 3. The probability of choosing the optimized scene is 0.981 in Dark room experiment (without ambient light), and standard error is 0.136. The probability of choosing the optimized scene is 0.957 in the Office experiment (with ambient light), and standard error is 0. 208. Results show that the light fields optimized with our method are preferred than the original light fields. All the above experiments suggest that the tone mapping based contrast optimizing method can suppress the crosstalk, and the contrast of the display light field of multi-projection 3D display is improved.

Table 3. Result of subjective study

View Table | View all tables in this article

The captured images of the display light fields of the experiments are shown in Fig. 11. The images are captured with an EOS 60D camera, the parameters are given in the caption. Due to the limited dynamic range of the camera and the print limitation, the real perceived image by HVS cannot be conveyed actually, but the contrast changes can be conveyed. As we can see from Fig. 11(a), with our proposed method, the local contrast of the optimized light field is apparently higher than the original one, and the crosstalk is well suppressed. When there is ambient light, the global contrast is seriously decreased as shown in Fig. 11(b). But with our proposed method, the perceived contrast is apparently improved. These displayed results of different views are provided in Visualization 1 as a supplementary material.

Fig. 11 The captured images of the display light fields of the multi-projection 3D display. The images are all captured with an EOS 60D camera, (a) results captured with an f-number of 14, an exposure time of 1/15s, (b) results captured with an f-number of 13, an exposure time of 1/15s. The display results of different views are provided in Visualization 1 (see Visualization 1).

Download Full Size | PDF

4. Conclusion

A tone mapping based optimizing method is presented to suppress the crosstalk of multi-projection contents and improve the display contrast of multi-projection 3D display by minimizing the visible contrast distortions. The common display model depicting the display luminance of multi-projection 3D display is proposed, and the contrast reduction due to the crosstalk of multiple projectors and ambient light is analyzed. A linear piece-wise TMO is adopted to remap the projected light field sequence of multiple projectors, and the distortions between displayed combined light field and desired light field with enhanced contrast are penalized with HVS based contrast perception model. The minimizing of the distortion is achieved by optimizing the tone mapping curve. Both the objective and subjective experiment results prove that the proposed method can suppress crosstalk of the diffuse reflection or diffuse transmission of multi-projection contents. The image quality of multi-projection 3D display can be apparently improved with our proposed method.

Our method is the first method to reduce the crosstalk of the multi-view autostereoscopic 3D display in a computational way. The method try to restore the local contrast of 3D light field to suppress the perceived crosstalk, and this is different from those existing global intensity mapping or local intensity mapping approaches which aim at reproducing or improving the contrast performance in 2D displays or 2D images [18]. This method can be used to any kinds of 3D displays only if we have got the display models of the displays.

Appendix

In the effective solution, according to the Eq. (12), the Eq. (9) can be rewritten into a matrix notation as,

L_{d_k}^{s} = ρ_{s} \cdot 10^{y (\log_{10} (L_{p_k}))} + ρ_{r} \cdot \sum_{i = 1}^{N} 10^{y (\log_{10} (L_{p_i}))}

we define

A_{i}, A_{i} = L_{p_i} / L_{p_k}, t h e n A = \sum_{i = 1}^{N} A_{i},

\begin{matrix} L_{d_k}^{s} = ρ_{s} \cdot 10^{y (\log_{10} (L_{p_k}))} + ρ_{r} \cdot \sum_{i = 1}^{N} 10^{\log (A_{i}) \cdot y (\log_{10} (L_{p_k}))} \\ = ρ_{s} \cdot 10^{y (\log_{10} (L_{p_k}))} + ρ_{r} \cdot \sum_{i = 1}^{N} A_{i} \cdot 10^{y (\log_{10} (L_{p_k}))} \\ = ρ_{s} \cdot 10^{y (\log_{10} (L_{p_k}))} + ρ_{r} \cdot A \cdot 10^{y (\log_{10} (L_{p_k}))} \\ = (ρ_{s} + ρ_{r} \cdot A) \cdot 10^{y (\log_{10} (L_{p_k}))} \end{matrix}

In the same way, the reference light field can be defined as,

L_{d_k}^{r} = (ρ_{s} + N \cdot ρ_{r}) \cdot L_{p_k}

N is the number of projectors. Substituting the Eq. (19), (20) into Eq. (11), the Eq. (13) can be derived.

Funding

National Natural Science Foundation of China (NSFC) (61575025); “863” Program (2015AA015902); Fund of the State Key Laboratory of Information Photonics and Optical Communications.

References and links

1. N. S. Holliman, N. A. Dodgson, G. E. Favalora, and L. Pockett, “Three-Dimensional Displays: A Review and Applications Analysis,” IEEE Trans. Broadcast 57(2), 362–371 (2011). [CrossRef]

2. H. E. Ives, “The Projection of Parallax Panoramagrams,” J. Opt. Soc. Am. 21(7), 397–403 (1931). [CrossRef]

3. W. Matusik and P. Hanspeter, “3D TV: a scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes,” ACM Trans. Graph. 23(3), 814–824 (2004). [CrossRef]

4. Y. Kim, K. Hong, J. Yeom, J. Hong, J. H. Jung, Y. W. Lee, J. H. Park, and B. Lee, “A frontal projection-type three-dimensional display,” Opt. Express 20(18), 20130–20138 (2012). [CrossRef] [PubMed]

5. X. Gao, X. Sang, X. Yu, P. Wang, X. Cao, L. Sun, B. Yan, J. Yuan, K. Wang, C. Yu, and W. Dou, “Aberration analyses for improving the frontal projection three-dimensional display,” Opt. Express 22(19), 23496–23511 (2014). [CrossRef] [PubMed]

6. P. Wang, S. L. Xie, X. Z. Sang, D. Chen, C. Y. Li, X. Gao, X. B. Yu, C. X. Yu, B. B. Yan, W. H. Dou, and L. Q. Xiao, “A large depth of field frontal multi-projection three-dimensional display with uniform light field distribution,” Opt. Commun. 354, 321–329 (2015). [CrossRef]

7. S. F. Zhang, Q. H. Wang, W. X. Zhao, J. Zhang, and J. L. Liang, “A frontal multi-projection autostereoscopic 3D display based on a 3D-image-guided screen,” J. Disp. Technol. 10(10), 882–886 (2014). [CrossRef]

8. T. Agocs, T. Balogh, T. Gorgacs, F. Bettio, E. Gobbetti, and G. Zanetti, “A Large Scale Interactive Holographic Display,” in Proceedings of IEEE Conference on Virtual Reality (IEEE 2006), pp. 311.

9. X. Sang, F. C. Fan, C. C. Jiang, S. Choi, W. Dou, C. Yu, and D. Xu, “Demonstration of a large-size real-time full-color three-dimensional display,” Opt. Lett. 34(24), 3803–3805 (2009). [CrossRef] [PubMed]

10. X. Xia, X. Liu, H. Li, Z. Zheng, H. Wang, Y. Peng, and W. Shen, “A 360-degree floating 3D display based on light field regeneration,” Opt. Express 21(9), 11237–11247 (2013). [CrossRef] [PubMed]

11. K. Nagano, A. Jones, J. Liu, J. Busch, X. M. Yu, M. Bolas, and P. Debevec, “An autostereoscopic projector array optimized for 3D facial display,” in Proceedings of ACM SIGGRAPH(ACM 2013), pp. 1. [CrossRef]

12. Gigahertz-Optik, “Tutorials on measuring light,” http://light-measurement.com/reflection-absorption.

13. A. J. Woods, “Crosstalk in stereoscopic displays: a review,” J. Electron. Imaging 21(4), 040902 (2012). [CrossRef]

14. J. Konrad, B. Lacotte, and E. Dubois, “Cancellation of image crosstalk in time-sequential displays of stereoscopic video,” IEEE Trans. Image Process. 9(5), 897–908 (2000). [CrossRef] [PubMed]

15. A. J. Chang, H. J. Kim, J. W. Choi, and K. Y. Yu, “Ghosting reduction method for color anaglyphs,” Proc. SPIE 6803, 68031G (2008). [CrossRef]

16. K. H. Lee, Y. Park, H. Lee, S. K. Yoon, and S. K. Kim, “Crosstalk reduction in auto-stereoscopic projection 3D display system,” Opt. Express 20(18), 19757–19768 (2012). [CrossRef] [PubMed]

17. C. Lee, G. Seo, J. Lee, T. H. Han, and J. G. Park, “Auto-stereoscopic 3D displays with reduced crosstalk,” Opt. Express 19(24), 24762–24774 (2011). [CrossRef] [PubMed]

18. F. Banterle, A. Artusi, T. Aydin, P. Didyk, E. Eisemann, D. Gutierrez, R. Mantiuk, and K. Myszkowski, “Multidimensional image retargeting,” inProceedings of ACM SIGGRAPH Asia2011 (ACM 2011), course.

19. J. Tumblin and H. E. Rushmeier, “Tone reproduction for realistic computer generated images,” Georgia Institute of Technology, 1991.

20. R. Mantiuk, S. Daly, and L. Kerofsky, “Display adaptive tone mapping,” ACM Trans. Graph. 27(3), 68 (2008). [CrossRef]

21. M. Cadikm, M. Wimmer, L. Neumann, and A. Artusi, “Image attributes and quality for evaluation of tone mapping operators,” InProceedings of the 14th Pacific Conference on Computer and Graph and Applications. (2006), pp 35–44.

22. R. Hunt, The Reproduction of Colour in Photography, Printing and Television: 6th Edition (John Wiley & Sons, 2004).

23. R. S. Berns, “Methods for characterizing CRT displays,” Displays 16(4), 173–182 (1996). [CrossRef]

24. R. Mantiuk, A. Tomaszewska, and W. Heidrich, “Color correction for tone mapping,” Comput. Graph. Forum 28(2), 193–202 (2009). [CrossRef]

25. H. R. Wilson, “A transducer function for threshold and suprathreshold human vision,” Biol. Cybern. 38(3), 171–178 (1980). [CrossRef] [PubMed]

26. S. Daly, “The visible differences predictor: an algorithm for the assessment of image fidelity,” in Digital Image and Human Vision, A. Watson, ed. (MIT, 1993).

27. P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Commun. 31(4), 532–540 (1983). [CrossRef]

28. S. Hecht, “The visual discrimination of intensity and the Weber-Fechner law,” J. Gen. Physiol. 7(2), 235–267 (1924). [CrossRef] [PubMed]

29. S. Boyd and L. Vandenberghe, Convex optimization (Cambridge University, 2004).

30. G. Eilertsen, R. Mantiuk, and J. Unger, “Real-time noise-aware tone mapping,” ACM Trans. Graph. 34(6), 1–15 (2015). [CrossRef]

31. E. Peli, “Contrast in complex images,” J. Opt. Soc. Am. A 7(10), 2032–2040 (1990). [CrossRef] [PubMed]

32. R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, “Hdr-vdp-2:a calibrated visual metric for visibility and quality predictions in all luminance conditions,” ACM Trans. Graph. 30(4), 1–14 (2011). [CrossRef]

Parameter	Value
Screen size	1800 mm × 1200mm
Viewing distance	3200 mm
Projecting distance	3200 mm
Projector pitch	10 mm
Number of projectors (N)	24
Viewing angle	45 degrees
Viewing pitch	240 mm
Reflective coefficient of the screen (ρ_s)	10.75
Reflectivity of the optical elements (ρ_r)	0.08
Maximum illuminance level of projector (E_max)	100 lux
Minimum illuminance level of projector (E_black)	0.01 lux
Ambient illuminance level (E_amb)	400 lux
Resolution	1200 × 720

Contrast Enhancement Factor (e)	Image Quality (Q)
Without Ambient Light
	Desert	Football
1.0	70.5149	63.6487
1.05	67.8853	62.5449
1.10	65.5277	62.4553
1.15	63.1082	60.3950
With Ambient Light
1.0	74.4331	72.2201
1.15	73.4436	71.5880
1.30	72.1609	68.1949
1.45	69.0172	63.0168

Dark room(0 lux)		Office (400 lux)
No Contrast Change	With Contrast Optimizing	No Contrast Change	With Contrast Optimizing
3	159	7	155

Parameter	Value
Screen size	1800 mm × 1200mm
Viewing distance	3200 mm
Projecting distance	3200 mm
Projector pitch	10 mm
Number of projectors (N)	24
Viewing angle	45 degrees
Viewing pitch	240 mm
Reflective coefficient of the screen (ρ_s)	10.75
Reflectivity of the optical elements (ρ_r)	0.08
Maximum illuminance level of projector (E_max)	100 lux
Minimum illuminance level of projector (E_black)	0.01 lux
Ambient illuminance level (E_amb)	400 lux
Resolution	1200 × 720

Contrast Enhancement Factor (e)	Image Quality (Q)
Without Ambient Light
	Desert	Football
1.0	70.5149	63.6487
1.05	67.8853	62.5449
1.10	65.5277	62.4553
1.15	63.1082	60.3950
With Ambient Light
1.0	74.4331	72.2201
1.15	73.4436	71.5880
1.30	72.1609	68.1949
1.45	69.0172	63.0168

Image quality improvement of multi-projection 3D display through tone mapping based optimization

Abstract

1. Introduction

2. Display contrast optimization method

2.1 Display model for multi-projection display

2.2 Human visual system model

2.3 Tone mapping as the minimum visible distortion problem

2.4 Effective solution

3. Experiment and results

3.1 Experimental system

3.2 Experimental results

A. Objective investigation

B. Subjective investigation

4. Conclusion

Appendix

Funding

References and links

Supplementary Material (1)

Cited By

Figures (11)

Tables (3)

Equations (27)

Optics Express