1. Introduction

In traditional photography, it is often necessary to modify a picture to enhance its visual quality, or to create special effects that are absent in the image acquisition process. Amongst different techniques, relighting is perhaps one of the most popularly practiced methods as it allows the optical properties (such as illumination), which may be difficult to control in the real world environment, to be synthesized. Nowadays, relighting can now be conducted by both professional and novice users through a comprehensive choice of commodity photo-editing software. In brief, a picture is relit by modifying the value of each pixel according to a given criteria, an operation commonly referred to as the 'point' process. For example, the effect of a spotlight can be simulated by modulating the luminance of each pixel with the spatial distribution of the illumination. Apparently, it will be desirable if the relighting mechanism can be applied to digital holograms to enhance their impact to the observers. The problem is, rendering a digital hologram with the point process is erroneous, as each pixel is representing the holistic information from the entire object scene. Until now, research on hologram relighting has not been investigated. A straightforward solution is to render the original object scene, if it is still available, whenever a relighting task is required, and then regenerate the hologram afterwards. Despite the effectiveness and simplicity of this method, the process is time-consuming as the numerical generation of a digital hologram involves enormous amount of arithmetic operations. Although there are quite a number of fast algorithms, such as [1–5] which attempt to alleviate the problem, they are not capable of generating holograms in real time (e.g., at the video frame rate) if there are large number of object points. Besides, if the digital hologram is captured with optical means, the original scene may not be available afterwards. If that is the case, theoretically we can apply some sort of inverse mapping [6–8] to reconstruct, and then relight the scene image. Subsequently, the rendered scene image can be converted into a hologram. However, as reported in some literatures, the inverse process itself is complicated. Besides, so far only the reconstruction of holograms representing sparse images (i.e., images that contain few number of object points) had been successfully demonstrated. In this paper, we propose a fast method for relighting digital hologram without the presence, or the reconstruction of the original object scene. Our scheme is based on the wavefront recording plane (WRP) which is originally employed in fast generation of digital holograms [1] [9]. To our knowledge, it is the first time we have applied the WRP concept to achieve real-time relighting of holograms. In our proposed method, a digital hologram is projected onto a virtual wavefront recording plane (WRP) which is placed sufficiently near to the object points in the scene. At close proximity, each object will only cast its optical wave on a small region on the WRP. Hence, relighting the intensity of a pixel in the WRP is equivalent to modifying the intensities of a small cluster of object points that is contributing to the pixel of interest. On this basis, we apply relighting to the WRP, and subsequently expand it to a full hologram. The entire process mainly involves 4 Fast Fourier Transform (FFT) operations which can be realized with the graphical processing unit (GPU) in less than 20ms for a hologram comprising of 2048x2048 pixels. Experimental results demonstrated that the target relighting effects are correctly synthesized in the reconstructed images of holograms that are relit with our proposed method.

2. Hologram relighting

The concept of the proposed hologram relighting can be illustrated with Fig. 1 . To start with, we insert a hypothetical diffraction plane, known as the wavefront recording plane (WRP), between the digital hologram and the scene. Given an arbitrary object point, its optical wave will propagate by diffraction to the entire hologram. Other object points in the scene are contributed to the hologram in a similar manner. Hence, modifying a hologram pixel will affect the diffracted waves contributed by the entire scene image, instead of localizing in the region around the pixel of interest. However, as shown in the diagram, an object point will only cover a small area on the WRP (the dotted region). The closer the distance between the object point and the WRP, the smaller will be the coverage (a.k.a. the support) of the diffraction pattern on the latter. As such, relighting a pixel in the WRP will only affect the diffraction pattern of a small cluster of object points that share the same support. Our proposed relighting method is realized in 3 stages. First, we derive the WRP based on the mathematical framework in [1] that described the relationship between the object points in a 3D scene, the field distribution on WRP $u_w\left(x,y\right)$, and the hologram $u\left(x,y\right)$. These three entities are assumed to have the same horizontal, as well as vertical extents of X and Y units.-The complex wavefront contributed by the object points on the WRP is given by (Eq. (1).

 

Fig. 1 Spatial relation between the object point, the WRP, and the hologram.

Download Full Size | PDF

Next, the WRP is expanded to a hologram $u\left(x,y\right)$ given by (Eq. (3).

In the second stage, the WRP obtained in Eq. (4) is modulated with the relighting image (RI) $G\left(x,y\right)$ that simulates a given relighting condition. For example, the RI in Fig. 2 emphasizes the intensity within a circular region around the center of the scene, creating the effect of a spotlight. The relit WRP is given by (Eq. (5).

 

Fig. 2 Relighting image simulating a spotlight effect.

Download Full Size | PDF

Subsequently, $u_w^L\left(x,y\right)$ is expanded to a hologram as (Eq. (6)

The complete relighting process of our proposed method involves 4 FFT operations (2 in Eq. (4) and 2 in Eq. (6)), which constitutes to the majority of arithmetic operations. The pair of terms, $\frac{1}{F\left[h\left(x,y\right)\right]}$and $F\left[h\left(x,y\right)\right]$ can be pre-calculated in advance and stored in a look up table (LUT), and hence not count towards the computation load. With a typical PC and a GPU, the 4 FFTs can be executed in less than 20ms. The computation time on the rest of the process, comprising of multiplication between pairs of 2D arrays (e.g. Equation (5)), is negligible.

3. Experimental results

Our proposed hologram relighting method is demonstrated with the double depth image shown in Fig. 3(a) . The image is evenly partitioned into a left side and a right side, located at 0.55m and 0.6m from the hologram, respectively. A digital Fresnel hologram is generated with the 'point-light' method described in [10]. The wavelength of the optical beam and the pixel size of the hologram are $650nm$ and $7um$, respectively. A relighting image $G\left(x,y\right)$ shown in Fig. 3(b) is employed to simulate the directional illumination emerging from the upper right corner. $G\left(x,y\right)$ is divided into an illuminated (the white area) and a shadow (the grey area) regions that are separated by a sharp boundary. When relighting is not applied, the numerical reconstructed images at the two depth planes are shown in Figs. 3(c) and 3(d). When either side of the reconstructed image is in focus, it is a good recovery of the original content.

 

Fig. 3 (a) Scene image evenly divided into a left and a right sections, positioned at 0.55m and 0.6m from the hologram plane, respectively. (b) Relighting image simulating the directional illumination emerging from the upper right corner. (c),(d) Numerical reconstructed image of the digital hologram representing the image in Fig. 3(a), at a distance of 0.55m and 0.6m, respectively.

Download Full Size | PDF

Next we relight the hologram directly with $G\left(x,y\right)$ by multiplying the two images on a pixel by pixel basis. A hologram $u\left(x,y\right)$, after direct relighting with an image $G\left(x,y\right)$, is given by (Eq. (7)

The reconstructed images of $u_D^L\left(x,y\right)$ at the two depth plane are shown in Fig. 4(a) and 4(b). We observe that the relighting effect is not totally in line with the relighting image. Notably, the boundary between the illuminated and the shadow regions are fuzzy, and the area around it is heavily contaminated with slanting bars. The defects exhibited in Figs. 4(a) and 4(b) are expected, as direct modification of a small part of the hologram will change the diffraction waves contributed by the entire object scene instead of localizing in the neighborhood of the modified area. To overcome this problem, we applied our proposed method to relight the digital hologram. The latter is first converted to the WRP $u_w\left(x,y\right)$ based on Eq. (4), and multiplied with the relighting image$G\left(x,y\right)$. Subsequently, the result is expanded into a relit hologram based on Eq. (6). The reconstructed images of the relit hologram at the two depth planes are shown in Figs. 5(a) and 5(b). We observe that the relighting effect is in good agreement with the relighting image, with a clear boundary between the illuminated and the shadow regions.

 

Fig. 4 Numerical reconstructed image of the digital hologram that has been directly relit with the image in Fig. 3(b), at a focal distance of 0.55m and 0.6m, respectively.

Download Full Size | PDF

 

Fig. 5 Numerical reconstructed image of the digital hologram (relit with our proposed method based on the relighting image in Fig. 3(b) representing the image in Fig. 3(a), at a focal distance of 0.55m and 0.6m, respectively.

Download Full Size | PDF

To further illustrate our proposed method, we generated the digital hologram $u\left(x,y\right)$ of a hemisphere which is rendered with the texture of the earth image as shown in Fig. 6(a) . The hemisphere has a radius of 0.005m, with the tip located at 0.001m and 0.3m from the WRP and the hologram, respectively. A real, off-axis hologram $H\left(x,y\right)$ is generated by adding a planar reference wave $R\left(y\right)$ (illuminating at an inclined angle ${1.2}^o$ on the hologram) to $u\left(x,y\right)$, and taking the real part of the result to give (Eq. (8)

 

Fig. 6 (a) Hemisphere rendered with the texture of the earth image. (b) Optical reconstructed image of the hologram representing the hemisphere shown in (a). (c) Single frame excerpt of the optical reconstructed image of the hologram representing the hemisphere in (a), which has been relit with the spotlight image shown in Fig. 2.

Download Full Size | PDF

4. Conclusion

This paper reports a fast method for relighting a digital hologram without the need of re-generating the latter from the original object scene. We note that traditional photographic relighting techniques based on the 'point' process is not applicable on a hologram. To overcome this problem, we project the digital hologram to a hypothetic WRP that is inserted near to the object scene. Due to the close proximity, the wavefront of each object point is occupying a small neighborhood on the WRP. As such, relighting can be conducted by modulating the WRP with a relighting function, and expanded the modified WRP to a full hologram. Our proposed method is realized with the GPU, and the time taken to relight a 2048x2048 hologram is less than 20ms, equivalent to a rate of over 50 frames per second. Experimental results reveal that the reconstructed images of holograms that are processed with our proposed method are correctly relit with the target optical effects. In our evaluation, we have only adopted some simple relighting effects for synthesizing different kinds of illumination, mainly to demonstrate the effectiveness of our approach. However, it is evident that the current framework can be easily applied to handle more sophisticated effects, such as the image-based and the geometry-based relighting.