Open Access
Add to BibSonomy
Abstract
In this paper, the phase-only holograms with clear first diffraction orders have been generated based on the U-Net and the two-dimensional phase grating. Firstly, we proved the modulation effect of two-dimensional phase grating on diffraction field, and came to a conclusion that it could move the diffraction pattern of the hologram to the odd-numbered diffraction orders’ center of that. Then we changed the generation process of phase-only holograms and the training strategy for U-Net according to this conclusion, which converted the optimization target of the U-Net from the zeroth diffraction order in the center of diffraction field to the first diffraction order in the edge of that. And we also used a method called “phase recombination” to improve the structure of U-Net for less memory footprint and faster generating speed. Finally, the holograms with the 4K resolution have been generated in 0.05s, and the average peak signal to noise ratio (PSNR) of the reconstructed images is about 37.2 dB in DIV2K-valid-HR dataset.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Holography was first proposed by British physicist Gabor in 1948, which can reconstruct complete light field information including amplitude and phase [1]. With the development of technology, the research of holography has entered a new stage [2], and various branches have appeared. Common holography includes off-axis holography [3,4], Fourier transform holograms [5,6], phase holograms [7,8], color holograms [9,10] and so on. In recent years, as the computing speed of computers greatly increases, researchers can use computers to implement the simulation of large-scale light field. The computer-generated hologram (CGH) is the combination of computer technology and traditional holography, which has been widely used in three-dimensional display [11,12], interferometry [13,14], holographic lenses [15,16] and other fields.
The generation process of CGH is mainly divided into two parts: digital recording and optical reconstruction [17,18]. During optical reconstruction, the spatial light modulator (SLM) modulates the coherent light emitted by the laser to reconstruct the wavefront of the target object [19]. At present, the phase-only SLM is most widely used [20], so phase-only hologram has become the predominant form of CGH [21]. However, how to convert complex-amplitude information into pure-phase information is still the main problem for generating holograms. This problem is ill-conditioned and needs to be optimized by solving nonlinear and nonconvex inverse problems [22]. The common generation algorithms of phase-only holograms include Gerchberg-Saxton (GS) algorithm based on iterative idea [23], stochastic gradient descent (SGD) method based on explicit loss function [24], the Wirtinger holography (WH) [25], the double-phase hologram (DPH) encoding the diffraction field directly [26], the encoding method based on the neural network [27,28] and so on.
Compared with traditional methods, it is more balanced in generating speed and reconstruction quality to use neural network for generation of phase-only holograms, because there are plenty of nodes used to fit the deep relationship between diffraction field and phase-only hologram in the network. Shi et al. used MIT-CGH-4K (a large-scale CGH dataset) as the training dataset of the deep neural networks (DNN), and the photorealistic color 3D holograms with a high reconstruction quality can be generated from a single RGB-depth image in real time [29]. Moon et al. reconstructs the Gabor hologram in computers and inputs the phase part of the hologram into the conditional generative adversarial network (GAN), which can remove the superimposed twin-image noise in the Gabor hologram [30]. Wu et al. proposed an autoencoder-based neural network (holoencoder) in an unsupervised manner, which could fast generate phase-only holograms but the quality of the reconstructed image still needed to be improved [27]. Sun et al. proposed a dual-task convolutional neural network (CNN) based on the combination of the U-Net, which solved the dual tasks of the amplitude reconstruction and phase smoothing, but the time of hologram generation is too long [31]. Chang et al. designed an end-to-end CNN framework, which can rapidly generate holograms with clear depth cues from the directly recorded images of real-world scenes [32].
The generation process of phase-only holograms with U-Net was introduced in the paper, including the structure of the U-Net, the numerical calculation of diffraction field and the selected loss function. We analyzed the interference of DC component on reconstructed images and the errors caused by the introduction of the blazed grating for translating the zeroth diffraction order. Section 3 shows how we used a two-dimensional phase grating to resolve these problems reducing the optical reconstruction quality. We mathematically proved that the two-dimensional phase grating can translate the diffraction pattern of the hologram to the odd-numbered diffraction orders’ center of that, and swap the zeroth diffraction order and the first diffraction order in the diffraction field. Thus, we changed the training strategy for U-Net and obtained the holograms which reconstructed images in the first diffraction order of the diffraction field. The proposed method can protect reconstructed images from the interference of DC component, and do not reduce the reconstruction quality since no blazed grating is introduced. In the training process for U-Net, only one diffraction order’s degrees of freedom is constrained and others are not constrained, so there are diverse encoding methods to appear in generated holograms. We have introduced several common encoding methods and corresponding diffraction patterns. In this work, we chose the encoding method similar to DPH, because the zeroth diffraction order and the first diffraction order are easily separable in the diffraction pattern of that. Before the formal training, a pre-training is used to generate holograms which have the above encoding method. Besides, we also improved the structure of U-Net to reduce the memory footprint and promote the computing speed. The experiment result shows our method has great advantage compared with the conventional methods.
2. Generation of phase-only holograms based on U-Net
2.1 Structure of the U-Net
In the traditional full convolutional neural network (FCN), the feature fusion method of calculating the sum of corresponding points in convolutional layers is usually used, which means that a huge training dataset and a quite long training time are needed. U-Net is also a kind of FCN and the main difference between that and traditional FCNs is that the concatenation of feature maps from the contracting path and the expanding path is used as the feature fusion method of U-Net [33]. Thus, a significant training effect can be achieved in the small training dataset in a short time when U-Net is used as the network model for generation of phase-only holograms. The process of generating phase-only holograms with U-Net is shown in Fig. 1. According to the angular spectrum method (ASM) [34], the inverse diffraction field of the target image is
In Eq. (1) and Eq. (2), $L({{f_x},{f_y}} )$ is the band-limited function, which is used to filter out some high-frequency components for avoiding the aliasing of the spectrum. It is expressed as
The structure of the used U-Net is shown in Fig. 2, which consists of the input layer, the output layer, the down-sampling layers, the up-sampling layers and the skip connection layers. There are two input channels in the input layer, which are respectively the real and imaginary parts of the diffraction field. The output layer has only one channel, which is the encoded pure-phase information. The imaging distance is set to 0.01 m, and the depth of the U-Net is set two.
The left side of the network is a contracting path, which encodes the input information with two 3 × 3 convolutional network (padded convolutions) and a rectified linear unit (ReLU). The upper layer is downsampled with a 2 × 2 max pooling operation for reducing the size of the input information and extracting some shallow features. The right side of the network is an expansive path for decoding, the structure of which is the same as the left side and also consists of two 3 × 3 convolutional network (padded convolutions) and a ReLU. The lower layer is upsampled with a 2 × 2 convolution (“up-convolution”) for restoring the size of data and extracting some deep features. The U-Net is a quite typical “Encoder-Decoder” structure, the left side and right side of which are symmetrical. This symmetrical structure of each layer links the feature map obtained in the encoding process with feature map obtained in the decoding process through skip-connection, which can combine the shallow features and deep features to refine images. The resolution of data will halve after down-samping and double after up-sampling, eventually the resolution of the holograms will be the same as that of the diffraction field.
The U-Net is the neural network based on supervised learning, which needs the “label” in the training process. However, the phase recovery is a kind of ill-condition inverse problem, which usually does not have the optimal solution [22,35]. Thus, there are not standard phase-only holograms to be as the “label” of training dataset, and we can only calculate the errors between the target image and reconstructed image to update network nodes instead of the errors between generated holograms and other holograms. The process of reconstructing images from holograms is not encapsulated in the designed U-Net in pursuit of lightweight model. And it is also unnecessary to encapsulate the process of reconstruction into the model, because we only need the phase-only holograms in optical experiments and the target images are reconstructed by SLM.
The loss function used for backpropagation consists of the perceptual loss and L1 loss. The perceptual loss is about the comparison of the feature from the convolution of the target image’s amplitude $|{{U_{\textrm{tar}}}({x,y} )} |$ and the feature from the convolution of the reconstructed image’s amplitude $|{{U_{\textrm{rec}}}({x,y} )} |$, which can make the high-dimensional information (content and global structure) of those closer. L1 loss is the per-pixel loss, which represents the mean absolute value of the errors about all the corresponding pixels’ amplitude in these two light fields and can be expressed as
L1 loss can avoid errors’ cancellation and accurately reflects the actual forecast errors. It has stable gradients for any input data and will not cause exploding gradients. Reconstructed image can be optimized in semantics and pixel value when combining perceptual loss and L1 loss.
2.2 Interference of DC components to reconstructed image
At the present stage, the diffraction efficiency of the SLM can’t reach 100% due to the technical issues, which means that SLM can’t modulate the phase of all the incident light. Thus, the emergent light consists of the modulated light with the phase information and unmodulated DC components without the phase information. Suppose there is a phase-only hologram D loaded onto the SLM with the diffraction efficiency $\gamma $. When the SLM is illuminated vertically by the monochromatic plane wave whose amplitude is one, the light field of that is expressed as
where ${t_\textrm{D}}({x_{0},y_{0}} )= \textrm{exp} [{\textrm{j} \cdot {\phi_\textrm{D}}({x_{0},y_{0}} )} ]$ is the transmittance function of hologram D and ${\phi _D}$ is the phase distribution of that. The initial phase of the monochromatic plane wave is ignored here. The frequency spectrum of the light field on the SLM is expressed as
Fig. 3. The corresponding zeroth-diffraction-order reconstructions when the diffraction efficiency of the SLM is respectively (a) 0%, (b) 20%, (c) 40%, (d) 60%, (e) 80% and (f) 100%.
2.3 Offsetting the zeroth-diffraction-order reconstruction with a blazed grating
Suppose there is a phase-only hologram E whose phase distribution is ${\phi _\textrm{E}}({x_{0},y_{0}} )= \mu x_{0}$ and $\mu $ is the offset factor. After the above hologram D and E are superimposed, the new phase distribution is ${\phi _{\textrm{D + E}}}({x_{0},y_{0}} )= {\phi _\textrm{D}}({x_{0},y_{0}} )+ \mu x_{0}$. The new hologram is loaded onto SLM. When the SLM is illuminated vertically by the monochromatic plane whose amplitude is one, the light field of that is expressed as
In Eq. (8), ${t_\textrm{E}}({x_{0},y_{0}} )= \textrm{exp}({\textrm{j}\mu x_{0}} )$ is the transmittance function of the phase-only hologram E. The frequency spectrum of the light field is
Equation (9) shows that the center of frequency spectrum of the hologram D will be moved to $\left( {\frac{\mu }{{2\mathrm{\pi }}},0} \right)$ when superimposed with a hologram whose phase distribution is ${\phi _\textrm{E}}({x_{0},y_{0}} )= \mu x_{0}$. However, the frequency spectrum of the DC components has not been moved. The corresponding manifestation in spatial domain is that the propagation direction of the reconstructed image of hologram D will be offset by an angle $\sigma = \arcsin \left( {\frac{{\mu \lambda }}{{2\mathrm{\pi }}}} \right)$, but the DC components has not been changed. When $\tan (\sigma )= \frac{{{N_x}\triangle p}}{z}$, the zeroth-diffraction-order reconstruction of hologram D is exactly separated from the DC component. This condition can be simplified to
Figure 4 shows diffraction patterns of the hologram D, which is respectively superimposed by holograms with different offset factors $\mu $. The diffraction efficiency of the SLM is 80%. The diffraction pattern without offset is shown in Fig. 3. (e).
Fig. 4. The offset diffraction patterns when (a) $\mu = {{{\mu _0}} / 2}$, (b) $\mu = {\mu _0}$, and (c) $\mu = {{3{\mu _0}} / 2}$.
The phase distribution of the hologram E is modulo $2\mathrm{\pi }$ and is shown in Fig. 5. It can be found that the phase distribution is the same as that of the blazed grating.
It proved that the diffraction pattern can be moved when the hologram superimposes a hologram (blazed grating) with a linear phase distribution. The zeroth-diffraction-order reconstruction can be clearly observed after that is separated from the DC component. The above discussion is based on the assumption that the blazed grating has a continuous phase distribution, however actually the phase distribution on SLM is discontinuous. The reason is SLM has a minimum modulation area in which the phase is the same, and there is a jump at the edge of that. Besides, the phase range of commonly used SLMs is divided into 256 parts and has a minimum phase interval. Thus, the real phase distribution of blazed grating on the SLM is not smooth and like a “ladder”. As shown in Fig. 6, the smaller the blazed angle $\sigma $, the smaller the jump at the edges. Because the offset factor $\mu$ is proportional to the blazed angle $\sigma $, smaller blazed angel means smaller offset factor. According the Eq. (10), a larger diffraction distance is required to separate the reconstructed image and DC component when the offset factor $\mu$ decreases. However, as the diffraction distance increases, the complexity of the optical path will increase. Besides, this will also lose some high frequency information and reduce the reconstruction quality of phase-only holograms.
Fig. 6. The phase distribution on the SLM which is loaded blazed gratings with different blazed angles.
Above analysis show that the DC component can cause distortion of the reconstructed image, and the translation of diffraction pattern through the blazed grating will also cause errors due to the discontinuous phase distribution of the SLM. In the following research, a new method is proposed about generating phase-only holograms which reconstruct image in first diffraction order. The proposed method can solve the above problems well and obtain a high-quality reconstruction in optical experiments.
3. Optimization for the first diffraction order of phase-only holograms
3.1 Modulation effect of two-dimensional phase grating
As shown in Fig. 7, the fringe width of the two-dimensional phase grating is a, the period of the grating is $2a$, and the phase delay of the two parts in each cycle are ${\phi _1}$ and ${\phi _2}$, respectively. The phase distribution of the grating is
The transmittance function of the grating is expressed as
The frequency spectrum of two-dimensional phase grating is
The lengths of CGH in x-axis and in y-axis are respectively ${L_x}$ and ${L_y}$, the transmittance function of which is ${t_{\textrm{CGH}}}({x_{0},y_{0}} )$. The hologram is loaded onto SLM. When SLM is illuminated vertically by the monochromatic plane wave whose amplitude is one, the complex amplitude of Fresnel diffraction pattern on the receiving screen at the distance z from SLM is
After the hologram is superimposed with a two-dimensional phase grating, the transmittance function of that becomes
When the hologram superimposed with a phase grating is illuminated vertically by the monochromatic plane wave whose amplitude is one, the complex amplitude of Fresnel diffraction pattern on the receiving screen with the distance z from SLM is
The intensity is
In Eq. (17) and Eq. (18), n and m cannot be zero at the same time. Because the interval $\xi = \frac{{\lambda z}}{{2a}}$ between adjacent spectral points in term $\textrm{sinc}\left[ {\frac{{{L_x}}}{{\lambda z}}\left( {x - \frac{{n\lambda z}}{{2a}}} \right),\frac{{{L_y}}}{{\lambda z}}\left( {y - \frac{{m\lambda z}}{{2a}}} \right)} \right]$ is usually large enough, the overlap between diffraction terms can be ignored. SLM can be regarded as a grating with a slit pitch of $2a$, which can realize the function of light splitting through loading different phase distributions. The slit width of two-dimensional phase grating is a, and the slit spacing of that is $2a$. According to the grating equation
the center of $c\textrm{ - th}$ diffraction order is at distance $\frac{{c\lambda z}}{{2a}}$ from the center of the diffraction pattern under the paraxial approximation, which is the same as the translation distance $\frac{{n\lambda z}}{{2a}}$ and $\frac{{m\lambda z}}{{2a}}$ in term ${I_{\textrm{CGH}}}\left( {x - \frac{{n\lambda z}}{{2a}},y - \frac{{m\lambda z}}{{2a}}} \right)$ of Eq. (18). When n or m is even, $\textrm{sin}{\textrm{c}^2}\left( {\frac{n}{2},\frac{m}{2}} \right) = 0$. Therefore, the modulation effect of the two-dimensional phase grating on the hologram is translating the center of previous diffraction pattern to the center of the $({n,m} )\textrm{th}$ ($n$ and m are both odd) diffraction order and then superimposed the previous diffraction pattern. The intensity coefficient of both depends on the phase difference of the two-dimensional phase grating. There is a special case that the intensity coefficient of the second term in Eq. (18) is zero when the phase difference is $\mathrm{\pi }$. In this case, the new diffraction pattern only consists of translational pattern, and the previous pattern will disappear completely.Assuming that the diffraction efficiency of the SLM is 100%, the interference of the DC component can be ignored. As shown in Fig. 8(a), there is a clear zeroth-diffraction-order reconstruction in the center of the diffraction pattern of WH. After superimposing a two-dimensional phase grating, we can also observe clear images in odd-numbered diffraction orders in Fig. 8(b).
Fig. 8. The diffraction patterns of (a) WH, (b) WH superimposed a two-dimensional phase grating with the phase difference of ${{3\mathrm{\pi }} / 5}$, and (c) WH superimposed a two-dimensional phase grating with the phase difference of $\mathrm{\pi }$. The intersection points of the orange line segment are the center of the zeroth diffraction order, and the intersection points of the green line segment are the center of the odd-numbered diffraction orders.
As shown in Fig. 8(c), when the phase difference of the superimposed two-dimensional phase grating is $\mathrm{\pi }$, it can be found that the diffraction order in the center of the receiving screen is missing. These simulation results also prove the above conclusion.
3.2 Analysis about the diffraction pattern of DPH
We regard DPH as a superposition of an initial hologram and a two-dimensional phase grating with a phase difference $\mathrm{\pi }$, the phase distribution of which is expressed as
The transmittance function is
In the previous section, it was proved that the two-dimensional phase grating can translate the diffraction pattern to the center of the odd-numbered diffraction orders. When the phase difference of the grating is $\mathrm{\pi }$, the previous diffraction pattern will disappear completely. Figure 9(a) shows the diffraction pattern of the initial hologram. After superimposing a two-dimensional phase grating with a phase difference $\mathrm{\pi }$ (hereinafter referred to as the two-dimensional phase grating), we think that the diffraction pattern will be translated to the center of the odd-numbered diffraction orders. For example, the zeroth diffraction order in the center of the pattern will be translated the distance $({\xi ,\xi } )$ to the upper right and reach the location of the $({\textrm{1st,1st}} )$ diffraction order. The $({\textrm{ - 1st, - 1st}} )$ diffraction order will be also translated the same distance $({\xi ,\xi } )$ to the upper right and reach the location of the zeroth diffraction order. Finally, the zeroth diffraction order will be translated to the location of the odd-numbered diffraction orders, and the odd-numbered diffraction orders will be translated to the location of the even-numbered diffraction orders. Because the slit pitch of the two-dimensional phase grating is twice the slit width of that, the phenomenon of missing orders will cause the even-numbered diffraction orders to disappear except zeroth diffraction order. The diffraction pattern of DPH is shown in Fig. 9(b). DPH is regard as a superposition of an initial hologram and a two-dimensional phase grating, whose diffraction pattern is exactly in line with our inferences above.
In brief, the zeroth diffraction image of DPH is the same as the odd-numbered diffraction orders of the initial hologram, and the odd-numbered diffraction image of DPH is the same as the zeroth diffraction order of the initial hologram. Therefore, through calculating the zeroth diffraction order of the initial hologram with ASM, we can obtain the precise first-diffraction-order reconstruction of DPH whose resolution is the same as the hologram.
3.3 Generation of holograms with clear first diffraction order
In Section 2.1, because ASM is used for generation of holograms, only the degrees of freedom in the zeroth diffraction order are constrained, so the holograms with different encoding methods are generated during multiple training processes. In the trained model, there is only one encoding method in generated holograms, but this encoding method is uncertain and uncontrolled. It depends on the gradient direction of the optimization at the beginning of training the model, and the gradient direction is random. Four common encoding methods are shown in Fig. 10. In the diffraction pattern of the hologram encoded by these methods, the energy is mainly concentrated in the zeroth-diffraction-order reconstruction, while the energy distributed in the other diffraction orders is not the same. Except zeroth diffraction order, the energy of the light field shown in Fig. 10(a) is mainly distributed in the $({\textrm{ - 1st},\textrm{0th}} )$ diffraction order and the $({\textrm{1st},\textrm{0th}} )$ diffraction order, while the energy shown in Fig. 10(b) is mainly distributed in the $({\textrm{0th, - 1st}} )$ diffraction order and the $({\textrm{0th,1st}} )$ diffraction order. In the diffraction pattern of holograms with the encoding method shown in Fig. 10(c), the distribution of the energy is the same as the diffraction pattern of DPH. However, there is a certain energy in all eight diffraction orders around the zeroth diffraction order shown in Fig. 10(d).
According to the above modulation effect of the two-dimensional phase grating on DPH, the encoding method of generated hologram should be similar to that shown in Fig. 10(c). Therefore, a pre-training method is used to control the encoding method. As shown in Fig. 11, the input data is the diffraction field of target image, and the output data is the corresponding pure-phase information. After pre-training, the model will be used for formal training process. Thus, the structure of U-Net used in pre-training process is the same to that in formal training process. The difference between pre-training and formal training is that the loss function used in pre-training only contains loss L1, and the loss function used in formal training contains loss L1 and perceptual loss. The purpose of pre-training is only to make the encoding method of holograms generated by U-Net close to the encoding method of DPH, instead of generating holograms with high-quality reconstruction. Thus, it is unnecessary to use perceptual loss with a huge memory footprint in pre-training process. The desired effect can be obtained through with short training process. Usually, we used any 100 images in DIV2K-train-HR as the dataset of pre-training. Besides, the errors in formal training process are obtained by comparing reconstructed images and target images, but the errors in pre-training process are obtained by comparing generated holograms and DPH. In Fig. 11 and Fig. 12, green arrows indicate the flow of input data of U-Net, purple arrows indicate the flow of output data of U-Net, and orange arrows indicate the flow of errors. The left side of U-Net is the input layer and the right side of U-Net is the output layer. Because the errors are used to update the parameters of U-Net instead of input or output of U-Net, we make errors point to the upper or lower side of U-Net.
Fig. 11. The pre-training process of generating phase-only holograms whose encoding method and diffraction pattern are similar to DPH.
We start the formal training process after pre-training process. The left side of Fig. 12 is our previous training strategy shown in Section 2.1, in which the input data is the inverse diffraction field at the distance z from the target images, and the output data is the corresponding phase-only holograms with an encoding method similar to DPH. The zeroth-diffraction-order reconstruction is obtained by ASM, and the loss function is obtained by the comparison of the zeroth-diffraction-order reconstruction and target image. The holograms with a zeroth-diffraction-order reconstruction will be generated after training. However, this reconstruction will be disturbed by the DC component.
Therefore, combined with the two-dimensional phase grating, we proposed a new training strategy to resolve this problem. The hologram can be regarded as a superposition of an initial hologram and a two-dimensional phase grating with a phase difference $\mathrm{\pi }$. The initial hologram can be obtained by subtracting a two-dimensional phase grating from holograms generated by U-Net, and the errors are obtained through the comparison of the initial hologram’s zeroth-diffraction-order reconstruction and target image. After training, the initial hologram will have a clear zero-diffraction-order reconstruction. According to the conclusion in Section 3.2, the zeroth-diffraction-order reconstruction of the initial hologram and the first-diffraction-order reconstruction of the hologram generated by U-Net are both the same, and the optimization of the former is also the optimization of the latter. Therefore, we obtained phase-only holograms which reconstructs target images in first diffraction order through U-Net. The proposed method not only avoids the interference of DC component, but also does not introduce a blazed grating whose phase distribution in the SLM is discontinuous.
4. Experiment and discussion
The resolution of holograms is 2160 × 3840 (4 K), because there are obvious fringes at the edge of the diffraction pattern, we increase the resolution to 2304 × 4096 by zero-padding. The pixel interval of the SLM is 3.74 µm, and the imaging distance is set to 0.01 m. The U-Net have two downsampling layers, the number of features at highest level is 32, and the maximum number of channels is 1024. The mode of the upsampling is transposed convolution. The dropout probability of the node is 0.1 and the output layer is a linear layer. During the training process, we use NVIDIA GV100 and NVIDIA RTX 3090Ti to store the network model and complete the backpropagation of gradients, respectively. The trained model will be loaded in NVIDIA RTX 3090Ti for fast generation of holograms. The training process of U-Net model is based on Python v3.8.13, PyTorch v1.11.0 and CUDA v11.6. It is remarkable that the purpose of training the U-Net model is to establish a universal relationship between the diffraction field and the phase-only hologram. The trained model can generate phase-only holograms of any image with the set resolution, not just one image. In this paper, the set resolution is the above 4 K. The previous model will fail only when the resolution of input image changes. In this case, a new model needs to be trained by the proposed method to adapt to the new resolution.
It is important to choose a reliable training dataset for the neural network. DIV2K dataset is a commonly used super-resolution reconstruction dataset, which consists of DIV2K-train-HR dataset with 800 images for training and DIV2K-valid-HR with 100 images for validation. There are many different kinds of pictures in DIV2K dataset, which contain many combinations of different frequency components and can effectively reduce the overfitting of neural network. In this paper, we used DIV2K-train-HR as the training dataset of U-Net and DIV2K-valid-HR as the test dataset of U-Net. Because images in DIV2K dataset do not have the same resolution, we need to change the resolution of input image to 4 K by the way of bilinear interpolation before that used to train model. In the training process, an epoch means that all 800 images in DIV2K-train-HR dataset have been trained. The U-Net model was trained for a total of 50 epochs in five hours.
Figure 13 shows the configuration of the experimental setup. A non-polarizing semiconductor laser with the wavelength of 638 (± 8) nm was used as the reconstruction light source. The laser was combined with a single-mode fiber whose core diameter is 4 µm. The laser emitted from the fiber end can be considered as a point source. The output end of the fiber was positioned at the focal point of collimated lens for obtaining the plane wave, and the focal length f of the collimated lens was 100 mm. A neural density filter was inserted as an attenuator to adjust the light intensity. A polarizer was used to obtain a linearly polarized light, and then a half-wave-plate (HWP) was used to rotate the polarization of the light for matching the optimal polarization direction of the LCoS. A rectangular aperture was positioned after the half-wave-plate to reduce the size of plane wave and obtain a rectangular profile. A 50%/50% non-polarization beam splitter was used to split the laser. The phase-only hologram is loaded onto the nematic twisted liquid crystal LCoS with the resolution 4094 × 2400 and pixel interval 3.74 µm. The incident laser was modulated and reflected by using the LCoS and propagated through the beam splitter again. A Fourier Lens with the focal length of f = 100 mm is used for image magnification and a spatial filter is used to allow the first diffraction order pass through and block the other diffraction orders. A Canon EOS 5D Mark III camera equipped with an EF 100-mm f/2.8 macro lens was used to capture the magnified reconstructed image.
We also proposed a method named “phase recombination” to reduce memory footprint and promote generating speed, which is shown in Fig. 14. The number of the input channels is changed from two (previous) to eight (present) by splitting adjacent four pixels in the diffraction field, and the resolution of input data is changed to a quarter at the same time. Now the number of the output channels is also set to four, and the real phase-only hologram is obtained by combining four pixels in the corresponding positions of four channels. There is a reciprocal relation in the way to split pixels and the way to combine pixels.
As shown in Table 1, the size of convolutional layers also becomes a quarter through phase recombination, so the memory footprint of the model is significantly reduced and the graphics cards with less memory can be used to train the network model. This method reduces the cost of hardware equipment. At the same time, the calculation amount is also greatly reduced and the generating speed of holograms greatly increases due to the less size of convolutional layers. And we find that the reconstruction quality changes little although the size of convolutional layers decreases, we think it is because that there are many network nodes with little effect in previous structure and the present structure still have enough nodes for fitting the mapping between the diffraction field and phase-only holograms.
Table 1. Complexity of two model structures
Figure 15 shows the comparison of various methods in computation time and reconstruction quality. Our method (1) is using the U-Net with the standard structure to generate holograms, and our method (2) is using the U-Net with the phase recombination to generate holograms. We found that our method (2) is about 280 times faster than the WH algorithm while the reconstruction quality is similar. Compared with DPH algorithm, the generating speed of our method (2) decreases a little, while the quality improves a lot. Thus, the method (2) is used for all experiments below. The test images used in the next numerical and optical experiments are shown in Fig. 16.
Fig. 16. Test images used in numerical and optical experiments. (a) No. 0805 in DIV2K-valid-HR (b) No. 0882 in DIV2K-valid-HR.
Figure 17 shows the phase-only holograms of test images and corresponding numerical reconstruction, which are respectively obtained by DPH, WH and our method. When the diffraction distance is quite small, the wavefront of pixels cannot reach all positions of the diffraction field because of fixed diffraction angle, which will cause the large difference between amplitudes of different pixels. When the amplitudes in an area is about zero, the difference between two adjacent phases used to express a complex amplitude is very large, which will enlarge the spatial shifting noises [36,37]. In Fig. 17, we can observe these spatial shifting noises in the nose of the wolf and the tail of the butterfly, where the intensity is very low. In current reports, researchers alleviated spatial shifting noises appearing in reconstructed images of DPHs to a certain extent [38–40], but these methods wasted the space-bandwidth product or greatly reduced generating speed of DPH.
Fig. 17. The comparison of holograms and their numerical reconstructions generated by three different methods.
The method used in WH belongs iterative method, which unavoidably introduces the speckle noise. In our method, the U-Net will continuously optimize the encoding method of phases during the training process for generating holograms with smoother phase distribution. At the same time, the hologram generated by our method has few speckle noises because there is no random phase during the generation process. Thus, the reconstructed image of the hologram generated by our method has better details compared the other conventional methods.
Figure 18 shows the optical reconstructions of phase-only holograms generated respectively by different encoding methods. Spatial shifting noises in the numerical reconstruction of DPH also appeared in the optical reconstruction of DPH. It proved that the DPH’s encoding method will cause serious spatial shifting noises in the reconstructed image when imaging distance is quite small. It can be found that there is lots of speckle noises in the reconstructed image of WH, which lead to the loss of much detail information in the target image. Compared with above methods, the reconstructed images of holograms generated by our method have few spatial shifting noises and speckle noises because of the smooth phase distribution in holograms, which means that there are more details in the reconstructed image and it is more similar to target image.
Figure 19 shows the optical reconstruction of two standard test patterns (USAF-1951 and ISO 12233). It can be found that spatial shifting noises fill the entire reconstructed image of DPH, which greatly reduce the reconstruction quality and the definition. It is difficult to observe the details of reconstructed image. There are many speckle noises in the reconstructed image of WH, which also reduce the definition to a certain extent. Compared with DPH and WH, the phase-only holograms generated by our method have a stronger modulation effect of the light, which can concentrate more energy into the bright area in the standard test patterns. Thus, the reconstructed images have a higher display quality. At the same time, it also has a very high definition because there is no interference from spatial shifting noises and speckle noises.
Fig. 19. (a) Standard test patterns (USAF-1951 and ISO 12233) and their optical reconstruction generated by the method of (b) DPH (c) WH and (d) our method. USAF-1951 is on the left, and its resolution is 1200 × 1200. ISO 12233 is on the right, and its resolution is 2160 × 3840.
5. Conclusion
In this study, we described the generation process of phase-only holograms and analyzed the modulation effect of the two-dimensional phase grating on the diffraction pattern. We combined U-Net and two-dimensional phase grating to generate phase-only holograms which reconstruct images in the first diffraction order. Because the DC component does not interfere the first diffraction order, a high-quality reconstruction can be obtained without the use of blazed grating. We also improved the structure of U-Net, and the new structure called “phase recombination” greatly reduced the memory footprint and promoted the generation speed of phase-only holograms. Finally, the 4 K phase-only holograms with a mean reconstruction PSNR of 37.2 dB have been generated in 0.05s. Optical results proved that the reconstructed images of holograms generated by the proposed method have a higher definition compared with conventional methods.
Funding
National Key Research and Development Program of China (2017YFB1104500); National Natural Science Foundation of China (61775240).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. D. Gabor, “A new microscopic principle,” Nature 161(4098), 777–778 (1948). [CrossRef]
2. E. N. Leith and J. Upatneiks, “Reconstructed wavefronts and communication theory,” J. Opt. Soc. Am. 52(10), 1123–1130 (1962). [CrossRef]
3. D. R. MacQuigg, “Modulated-grating hologram,” Appl. Opt. 16(5), 1380–1389 (1977). [CrossRef]
4. J. Politch and M. Segal, “Holographic measurements of refractive-index changes,” Opt. Lett. 3(1), 33–35 (1978). [CrossRef]
5. W. A. Dyes, P. F. Kellen, and E. C. Klaubert, “Velocity synchronized Fourier transform hologram camera system,” Appl. Opt. 9(5), 1105–1112 (1970). [CrossRef]
6. C. B. Burckhardt, “Use of a random phase mask for the recording of Fourier transform holograms of data masks,” Appl. Opt. 9(3), 695–700 (1970). [CrossRef]
7. L. Larmore, H. M. A. El-Sum, and A. F. Metherell, “Acoustical holograms using phase information only,” Appl. Opt. 8(8), 1533–1536 (1969). [CrossRef]
8. N. C. Gallagher and J. A. Bucklew, “Nondetour phase digital holograms: an analysis,” Appl. Opt. 19(22), 3856–3862 (1980). [CrossRef]
9. M. T. Gale and K. Knop, “Color-encoded focused image holograms,” Appl. Opt. 15(9), 2189–2198 (1976). [CrossRef]
10. K. Ohnuma and F. Iwata, “Color rainbow hologram and color reproduction,” Appl. Opt. 27(18), 3859–3863 (1988). [CrossRef]
11. C. Slinger, C. Cameron, and M. Stanley, “Computer-generated holography as a generic display technology,” Computer 38(8), 46–53 (2005). [CrossRef]
12. Z. Hao, N. Collings, C. Jing, B. Crossland, D. Chu, and J. Xie, “Full parallax three-dimensional display with occlusion effect using computer generated hologram,” J. Phys.: Conf. Ser. 415(7), 012048 (2013). [CrossRef]
13. J. Dong, C. Jiang, and S. H. Jia, “Digital holographic metrology based on multi-angle interferometry,” Opt. Lett. 41(18), 4301–4304 (2016). [CrossRef]
14. M. Paturzo, V. Pagliarulo, V. Bianco, P. Memmolo, L. Miccio, F. Merola, and P. Ferraro, “Digital Holography, a metrological tool for quantitative analysis: Trends and future applications,” OPT. LASER ENG. 104(May), 32–47 (2018). [CrossRef]
15. J. W. Kang and C. K. Hong, “Three dimensional shape measurement of a micro Fresnel lens with in-line phase-shifting digital holographic microscopy,” J. Opt. Soc. Ko. 10(4), 178–183 (2006). [CrossRef]
16. C. Joenathan, A. Bernal, and R. S. Sirohi, “Spatially multiplexed X-Y lateral shear interferometer with varying shears using holographic lens and spatial Fourier transform,” Appl. Opt. 52(22), 5570–5576 (2013). [CrossRef]
17. A. W. Lohmann and D. P. Paris, “Binary Fraunhofer holograms, generated by computer,” Appl. Opt. 6(10), 1739–1748 (1967). [CrossRef]
18. R. E. Haskell and B. C. Culver, “New coding technique for computer generated holograms,” Appl. Opt. 11(11), 2712–2714 (1972). [CrossRef]
19. C. P. Li, S. H. Wang, and C. L. Wang, “Novel low-complexity SLM schemes for PAPR reduction in OFDM systems,” IEEE T Signal Process. 58(5), 1–5 (2010). [CrossRef]
20. P. T. Samsheerali, K. Khare, and J. Joseph, “Three-dimensional display system using near on-axis, phase-only digital holography,” Appl. Opt. 54(3), 451–457 (2015). [CrossRef]
21. Y. Z. Liu, J. W. Dong, Y. Y. Pu, B. C. Chen, H. X. He, and H. Z. Wang, “High-speed full analytical holographic computations for true-life scenes,” Opt. Express 18(4), 3345–3351 (2010). [CrossRef]
22. T. Yu, S. J. Zhang, W. Chen, J. Liu, X. Y. Zhang, and Z. J. Tian, “Phase dual-resolution networks for a computer-generated hologram,” Opt. Express 30(2), 2378–2389 (2022). [CrossRef]
23. R. W. Gerchberg and W. O. Saxton, “A practical algorithm for the determination of phase from image and diffraction plane pictures,” Optik 35, 237–250 (1972).
24. Y. Peng, S. Choi, N. Padmanaban, and G. Wetzstein, “Neural holography with camera-in-the-loop training,” ACM Trans. Graph. 39(6), 1–14 (2020). [CrossRef]
25. P. Chakravarthula, Y. Peng, J. Kollin, H. Fuchs, and F. Heide, “Wirtinger holography for near-eye displays,” ACM T GRAPHIC 38(6), 1–13 (2019). [CrossRef]
26. C. K. Hsueh and A. A. Sawchuk, “Computer-generated double-phase holograms,” Appl. Opt. 17(24), 3874–3883 (1978). [CrossRef]
27. J. C. Wu, K. X. Liu, X. M. Sui, and L. C. Cao, “High-speed computer-generated holography using an autoencoder-based deep neural network,” Opt. Lett. 46(12), 2908–2911 (2021). [CrossRef]
28. J. Lee, J. Jeong, J. Cho, D. Yoo, B. Lee, and B. Lee, “Deep neural network for multi-depth hologram generation and its training strategy,” Opt. Express 28(18), 27137–27154 (2020). [CrossRef]
29. L. Shi, B. Li, C. Kim, P. Kellnhofer, and W. Matusik, “Towards real-time photorealistic 3D holography with deep neural networks,” Nature 591(7849), 234–239 (2021). [CrossRef]
30. I. Moon, K. Jaferzadeh, Y. Y. Kim, and B. Javidi, “Noise-free quantitative phase imaging in Gabor holography with conditional generative adversarial network,” Opt. Express 28(18), 26284–26301 (2020). [CrossRef]
31. X. H. Sun, X. Y. Mu, C. Xu, H. Pang, Q. L. Deng, K. Zhang, H. B. Jiang, J. L. Du, S. Y. Yin, and C. L. Du, “Dual-task convolutional neural network based on the combination of the U-Net and a diffraction propagation model for phase hologram design with suppressed speckle noise,” Opt. Express 30(2), 2646–2658 (2022). [CrossRef]
32. C. L. Chang, D. Wang, D. C. Zhu, J. M. Li, J. Xia, and X. L. Zhang, “Deep-learning-based computer-generated hologram from a stereo image pair,” Opt. Lett. 47(6), 1482–1485 (2022). [CrossRef]
33. Y. Nagahama, “Phase retrieval using hologram transformation with U-Net in digital holography,” Opt. Continuum 1(7), 1506–1515 (2022). [CrossRef]
34. K. Matsushima and T. Shimobaba, “Band-Limited Angular Spectrum Method for Numerical Simulation of Free-Space Propagation in Far and Near Fields,” Opt. Express 17(22), 19662–19673 (2009). [CrossRef]
35. Z. Y. Yuan and H. X. Wang, “Phase retrieval via reweighted Wirtinger flow,” Appl. Opt. 56(9), 2418–2427 (2017). [CrossRef]
36. D. Mendlovic, G. Shabtay, U. Levi, Z. Zalevsky, and E. Marom, “Encoding technique for design of zero-order (on-axis) Fraunhofer computer-generated holograms,” Appl. Opt. 36(32), 8427–8434 (1997). [CrossRef]
37. V. Arrizón and D. Sánchez-de-la-Llave, “Double-phase holograms implemented with phase-only spatial light modulators: performance evaluation and improvement,” Appl. Opt. 41(17), 3436–3447 (2002). [CrossRef]
38. Y. K. Kim, J. S. Lee, and Y. H. Won, “Low-noise high-efficiency double-phase hologram by multiplying a weight factor,” Opt. Lett. 44(15), 3649–3652 (2019). [CrossRef]
39. X. M. Sui, Z. H. He, H. Zhang, L. C. Cao, D. P. Chu, and G. F. Jin, “Spatiotemporal double-phase hologram for complex-amplitude holographic displays,” Chin. Opt. Lett. 18(10), 100901 (2020). [CrossRef]
40. X. M. Sui, Z. H. He, G. F. Jin, D. P. Chu, and L. C. Cao, “Band-limited double-phase method for enhancing image sharpness in complex modulated computer-generated holograms,” Opt. Express 29(2), 2597–2612 (2021). [CrossRef]
