Open Access
Add to BibSonomy
Abstract
We present a deep learning (DL) framework based on a conditional generative adversarial network (CGAN) to perform compressive optical image hiding (COIH) with a single-pixel detector. An end-to-end compressive sensing generative adversarial network (eCSGAN) is developed, achieving the approximate equivalent model of an inverse system of a nonlinear COIH model, to reconstruct two-dimensional secret images directly from real acquired one-dimensional compressive sampling signals without the need of any security keys of the COIH system such as the sequence of illumination patterns, the host image, etc. Moreover, detailed comparisons between the image reconstructed using eCSGAN and compressive sensing (CS) shows that the proposed method can remarkably increase the quality in image reconstruction with a lower sampling rate. The feasibility and security of the proposed method are demonstrated by the numerical simulations and optical experiment results.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
As the most common way to express information, the security and efficiency of digital image in the process of storage and transmission attract more and more attention with the rapid development of cloud computing, Internet of things (IOT) and digital communication technology [1–4]. Optical image security technology has unique advantages in data transmission and protection due to its characteristics of large capacity, high-speed storage, and multi-dimensional parallel processing [5,6]. Although the image encryption technology can encrypt the image into unavailable information, even if the encrypted image information is stolen, the real information cannot be obtained without the correct key. However, the encrypted image is a noise-like ciphertext, which is easy to attract the attention of attackers [7,8]. On the other hand, a secret image can be embedded in a host image for image hiding and watermarking, so that the hidden image can look like the original host image in some domains, so as to achieve the purpose of hiding secret information and avoid the attention of eavesdroppers [9,10]. Phase-shifting interferometry (PSI) based optical image hiding technology [11–13] realizes secret image hiding into host image in optical Fresnel domain with the theory of light interference, which not only achieve the hidden image resemble the host image in Fresnel diffraction domain (real image data recorded as the interferograms), but also makes full use of the advantages of multi-dimensional parallel processing of light and the spatial bandwidth product of CCD to record the interferogram information. However, it needs to get multiple phase-shifting interferograms and the usage of various optical parameters to realize image reconstruction. The encrypted data will be multiplied compared with the original data, which greatly increases the amount of data and affects the efficiency of data transmission. Further, two-step phase-shifting method and compressive sensing (CS) are combined in previous work to further reduce the amount of data [10,14,15]. However, the quality of decrypted image is easily affected by the collimation degree of optical system, external vibration, and nonlinear noise in experiment, which is difficult to be applied in practice.
In this paper, we propose a new framework of compressive optical image hiding (COIH) for high quality image reconstruction under lower sampling rate. Deep learning (DL) can obtain the approximation of the optimal model of one system through large amount of prior information. Based on this principle, DL has been applied to solve various inverse problems in optical imaging, such as computational ghost imaging [16,17], single-pixel imaging [18,19], and Lensless computational imaging [20]. It also shows its advantages in other fields, such as holographic image reconstruction [21,22], image encryption transmission [23] and Cryptanalysis [24–26], etc. Among these methods, Convolutional networks [27] can be used to reconstruct the object wavefront directly from a single-shot in-line digital hologram [23]. Recently, the linear regression-based method has been proved to be able to use a small number of training samples to achieve complex object reconstruction [28]. However, the DL-based nonlinear method has more powerful processing ability than linear regression in dealing with nonlinear system and high-frequency detail information reconstruction of image [29]. By using a conditional generative adversarial network (CGAN) [29], an end to end compressive sensing generative adversarial network (eCSGAN) is developed to achieve the approximation of the inverse system of nonlinear model of COIH, and the two-dimensional images can be reconstructed directly from real acquired one-dimensional compressive sampling signals. Simulation and experimental results show that the proposed method can achieve high quality image reconstruction at low sampling rate in compressive image decryption.
2. Method analysis and network training
In the previous work about COIH [10,14], it is necessary to decompress and reconstruct the hidden interferogram first, and then the optical key is used for decryption. Therefore, the noise and deviation of the experimental system easily lead to the degradation of the quality in reconstructed image. Based on this, this paper proposes an end-to-end network, which directly uses the obtained one-dimensional compressed data to reproduce the original image. The detailed flow chart for the end to end deep learning-based compressive optical image hiding (DL-COIH) is shown in Fig. 1, and the whole process consists of three parts: Preparation of training data, Training process and Prediction process. Firstly, a modified Mach-Zehnder interferometer is used to realize optical image hiding, then by projecting the measurement matrix with the hidden experimental interferogram on the digital micromirror device (DMD), the compressed one-dimensional data is obtained, and then it is trained together with the corresponding label image. Finally, different from the traditional decryption method, only a set of compressed one-dimensional data is needed to reconstruct the two-dimensional image to be hidden with high quality at the receiving end. The detailed reconstruction process and network training will be discussed in the following analysis.
Fig. 1. Detailed flow diagram for the end to end deep learning-based compressive optical image hiding. Where preparation of training data is completed in the compressive optical image hiding system. (a) Compressive optical image hiding system. M: mirror, BE: beam expander, BS: beam splitter, NDF: neutral density filter, SLM: spatial light modulator, Oh: host image. DMD: digital micromirror device. (b) Training process and (c) prediction process.
2.1 Principle analysis on deep learning-based image reconstruction in compressive optical image hiding
In this section, the experimental system and basic principle of COIH will be introduced briefly. The schematic of COIH is showed in Fig. 1(a). The laser beam emitted from a He-Ne laser (REO/30989) with the wavelength of 633 nm is divided into an object beam and a reference beam. The object beam first irradiates the target image on the spatial light modulator (SLM) (FSLM07U-A); the host image is placed in the reference beam to achieve the modulation of the reference beam. When the DMD is not loaded with measurement matrix, the complex amplitude of object light wave on DMD is assumed to $U(x,y) = A(x,y)\exp [i\phi (x,y)],$ in which $A(x,y)$ and $\phi (x,y)$ are amplitude and phase information of the original image after diffraction. At the same time, the complex amplitude distribution of the host image on the DMD plane can be expressed as
where ${A_h}(x,y)$ and ${\phi _h}(x,y)$ are amplitude and phase information of host image diffracted on DMD plane. In this case, the interferogram obtained on DMD can be expressed as whereHere, $a(x,y)$ and $b(x,y)$ present the background and modulation amplitude of the interferogram. ${I_n}$ and I are the interferograms with multiplicative noise and without noise, respectively. ${e_1}\sim G({\mu _1},{\sigma _1})$ is the multiplicative noise, where $G()$ is the Gaussian noise, ${\mu _1}$ and ${\sigma _1}$ denote the mean value and variance of noise, respectively. The obtained interferogram is similar to the Fresnel diffraction fringe pattern of the host image. In order to realize the compressive sampling, the measurement matrix is loaded on the DMD to realize the inner product operation of the hidden hologram and the measurement matrix, and then the new measurement values are obtained and collected by a photodetector, which can be expressed as follows:
Here, $\Phi \in {R^{M \times N}}$ is the measurement matrix; M is the measurements of the compressive sampling and $\left\langle \cdot \right\rangle$ indicates inner product operation. Similarly, ${e_2}\sim G({\mu _2},{\sigma _2})$ is the additive Gaussian white noise, in which ${\mu _2}$ and ${\sigma _\textrm{2}}$ denote the mean value and variance of noise, respectively. Then, we can transmit the compressive sampling data to the receiver through the traditional communication channel.
The method proposed in this paper first employs eCSGAN model to learn the reconstruction of two-dimensional object image using one-dimensional sampling data. The process can be described as
where $O^{\prime}$ is the reconstructed object image in the training process, ${\varTheta _{eCSGAN\_G}}({\cdot} )$ is a nonlinear function mapping based on the generator network of eCSGAN model, which projects the one-dimensional sampling data s into the two-dimensional object image space through the convolutional neural network.Then, we employ N pairs of different labeled training data to learn the nonlinear function mapping, each pair has a known object image ${Y_n}$ and the sampling signal ${s_n}$, where $n = 1,2,\ldots ,N$. This learning process is essentially an optimization process and can be expressed simply as
2.2 Network training
Inspired by CGAN, an eCSGAN is constructed for COIH and the network structure is shown in Fig. 2. The proposed network structure consists of two parts: the generator network and the discriminator network. The U-net structure [31] is adopted in the generator network. The input of the network is a one-dimensional measurement signal with the length of M obtained by compressive sampling. First, two layers of full connection layer to the input of generator network and discriminator network are added to extract the features of one-dimensional signal. After the initial recovery of the sampling signal through two layers of full connection layer with the size of 4096×1 and 16384×1, it is reshaped to a feature map with the size of 128×128. At last, the original image can be recovered through 7 downsampling convolutional layers and 7 upsampling convolutional layers. PatchGAN structure [29] is adopted in the discriminator network, which is used to guide and optimize the generator network. The discriminator network has two groups of inputs. We concatenate ${X_1}$ with Y as group one. ${X_1}$ is a feature map generated by s through two fully connected layers. ${X_1}$ and object image $O^{\prime}$ are concatenated as group two. Each group is convoluted four times to obtain a 30×30 size feature map, which are used to calculate the loss of the discriminator and optimize the parameters of the generator network. Dropout (P = 0.5) in the network is used to prevent over fitting in model. The detailed parameters of these two networks can refer to the Fig. 2.
Fig. 2. Framework of the eCSGAN for COIH. The network structure consists of two parts: the generator network and the discriminator network. The U-net structure and PatchGAN structure are adopted in the generator network and discriminator network, respectively. Where the input of the network is a one-dimensional measurement signal and the output of the network is the reconstructed original image.
In the training process, we define the loss functions ${L_G}$ and ${L_D}$ for generator network model and discriminator network model in turn:
3. Results and discussions
3.1 Simulation results and robustness analysis
In order to prove the feasibility and superiority of our method in improving the quality of restoration, a set of one-dimensional compressive signals embedded by handwritten-digit patterns from the MNIST database [34] are used for training firstly, instead of making great effort to retrieve the object light wave by using compressive sensing reconstruction algorithm and multi-step phase-shifting method with various security keys, and then the original image can be retrieved directly by using a trained DL model. 3000 images with the size of 128 ×128 pixels are loaded on the SLM to perform hiding and a capital letter ‘S’ is used as the host image to modulate the reference light. In addition, a series of random binary measurement matrices are loaded on DMD to realize compressive sampling simultaneously. In the process of training, 2700 set of the corresponding one-dimensional compressive sampling data are used as the training data and the rest 300 as the testing data, the retrieved results are showed in Fig. 3(a). For the purpose of comparison, the decryption using traditional approach from COIH is performed, and the corresponding results are present in Fig. 3(b). Where the first row is the ground truth, the reconstructed images are showed in the second to sixth row, respectively. Evidently, only one-dimensional compressive sampling data is needed without the usage of various optical encryption keys and measurement matrix to retrieve the object images directly, and the retrieved images are of higher quality compared with the images retrieved using traditional approach. When the sampling rate is 1%, the original image recovered by the traditional method can hardly be distinguished. However, the proposed method can still get high-quality decryption results, and the average peak signal to noise ratio (PSNR) value of 300 groups of results with 1% sampling rate is 18.0637 dB. Moreover, when the sampling rate is as low as 0.09%, the recovered results using the proposed method can still be recognized. The above results prove that proposed DL- COIH has a better image reconstruction performance than COIH at lower sampling rate.
Fig. 3. Simulation results of MNIST test data using (a) DL- COIH and (b) COIH. Top row: Ground truth. Row 2 to 6: Reconstruction results at different sampling rate.
In order to further verify the feasibility of the method, 2700 set fashion-MNIST database [35] are used for training, in which the image content is more complex and the details are more abundant, and the retrieved results are distributed as shown in Fig. 4(a). Also, for comparison, the results of decryption using traditional method are shown in Fig. 4(b). The ground truths are showed in the first row, and the second row to sixth row are given the reconstructed images under different sampling rate. For more complex images, the decrypted image using traditional method can hardly be recognized when the sampling rate is 20%. However, when the sampling rate is as low as 0.09%, the retrieved images using DL can be visualized and the high-frequency features are still retained, in which the corresponding average PSNR value of 300 groups is 18.3177 dB. This further verifies the feasibility and superiority of our method in reconstruction of complex image.
Fig. 4. Simulation results of fashion-MNIST test data using (a) DL-COIH and (b) COIH. Top row: Ground truth. Row 2 to 6: Reconstruction results at different sampling rates.
Owing to only one-dimensional data is needed to be transmitted, there is no common attacks such as shearing, which solves the problem of robustness. Therefore, in this section, we only discuss the attack of different types of noise on method. Figures 5(a) and 5(b) show the images reconstructed using DL-COIH under different levels of additive noise and multiplicative noise at the sampling rate of 5%. When the additive noise with different variances is added, the quality of reconstruction results is almost unchanged, which can be seen from the average PSNR curve in Fig. 6(a), and the range of average PSNR value changes from 21.55 to 21.10 dB. Compared with the additive noise, the multiplicative noise has more influence on the quality of the reconstructed image. When the variance of noise changes from 0.01 to 0.04, the quality of image reconstruction is declining, and the curve of PSNR value in Fig. 6(b) is also declining steeply. It can be seen from Fig. 5, some images can't be distinguished when the variance of multiplicative noise is 0.04, but when the variance is less than 0.02, the results are basically recognizable.
Fig. 5. The performance of the proposed DL- COIH method under the noise attack with different variances.
Fig. 6. PSNR values under the noise attacks with different variances. (a) Additive noise and (b) Multiplicative noise.
In addition, to examine the generalization of the eCSGAN model, some object images that do not belong to the training set are used for reconstruction. For convenience, we test it with the English letter and the reconstructed images are shown in Fig. 7, in which the English letters are not included in the training set. The reference images are showed in the first row, and the second row are the corresponding reconstructed images, respectively. Although the network was trained by using the MNIST test data, it is clearly seen that it can be used to reconstruct the different English letter with high-quality, which shows that eCSGAN can learn the corresponding relationship between the compressive sampling data and the original image well, and different images hidden in the COIH system can be reconstructed according to the optimized reconstruction network. However, since this type of image data do not belong to the training set and the existing training data is limited, the details of the reconstruction results are still a little different from the reference images. Better results can be obtained by further training and optimizing the proposed network.
3.2 Analysis and discussion of experimental results
In the experiments, we used the same training set and testing set as in the simulation. 2700 different images in MNIST data and fashion-MNIST data are used to train the network. The experimental results for different sampling rates are present in Fig. 8. In Fig. 8, the images in the first row are the ground truth of different dataset in the testing set. And the second row to sixth row give the reconstruction images at different sampling rates. In addition, for quantitative analysis of experimental results, the average PSNR values and structural similarity index (SSIM) values between the reconstructed images and the ground truth under different sampling rates are plotted in Fig. 9. And the SSIM can defined as:
Fig. 8. Experimental results under different sampling rates. The images in the first row are the ground truth, the second to sixth row show the images reconstructed using DL- COIH under different sampling rate: (a) MNIST test data, (b) fashion-MNIST test data.
At last, the security of the proposed method is analyzed in this section. Keeping the image reconstruction network unchanged (the letter ‘S’ is used as the host image), the host images of the COIH system are changed to ‘C’ and ‘U’, and the recorded one dimensional sampling data embedded by MNIST test data and fashion-MNIST test data are input into the reconstruction network. The reconstruction results are shown in Fig. 10, in which the first row is the ground truth, the reconstructed images with the host image of ‘C’ under different sampling rates are showed in the second and third row in Fig. 10(a), respectively. Similarly, when the host image is changed as ‘U’, the corresponding reconstructed images under different sampling rates are present in Fig. 10(b). The results indicate that when the host image information is changed, the correct object information cannot be obtained completely, which proves that the proposed method has high security and the host image is the same import secret key in the DL-COIH system as in the COIH system. In addition, further research shows that the original optical parameters and random measurement matrices are also important keys, and any small changes of these keys will produce the wrong decryption result. So, the DL-COIH system not only significantly improves image-decryption performance but also remains the good security and the key space in the COIH system.
Fig. 10. Retrieved images at different sampling rates with the incorrect host information: (a) capital letter ‘C’ and (b) capital letter ‘U’.
4. Conclusion
Because of the wide range of applications, people have paid more attention to image security for image management, storing and processing in recent decades. Large data volume and the reconstruction quality are the important factors limiting the application of security technology. Based on this, an end to end compressive sensing generative adversarial network (eCSGAN) is developed to perform compressive optical image hiding (COIH) with a single-pixel detector. In this method, two-dimensional images can be reconstructed directly from experimentally acquired one-dimensional compressive signals, without the need of the sequence of illumination patterns. In addition, the performance of conventional COIH and the proposed method under different sampling rate conditions are analyzed, in which reasonable reconstructions can be obtained with a sampling ratio of only 0.2% using the proposed method and the corresponding average PSNR value of 300 groups is 17.6877 dB. Compared with the decryption method using the classical four step phase-shifting method, only the data amount of 0.05% is needed for correct decryption in the proposed DL-COIH method. Therefore, the proposed study can provide solutions to the security problems such as in remote sensing and cloud computing.
Funding
National Natural Science Foundation of China (61805086); China Postdoctoral Science Foundation (2018M643114).
Disclosures
The authors declare no conflicts of interest
References
1. P. Li, J. Li, Z. Huang, T. Li, C. Gao, S. Yiu, and K. Chen, “Multi-key privacy-preserving deep learning in cloud computing,” Future Gener. Comp. Sy 74, 76–85 (2017). [CrossRef]
2. S. Jiao, C. Zhou, Y. Shi, W. Zou, and X. Li, “Review on optical image hiding and watermarking techniques,” Opt. Laser Technol. 109, 370–380 (2019). [CrossRef]
3. S. Baluja, “Hiding Images within Images,” IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1685–1697 (2020). [CrossRef]
4. F. Hu, C. Pu, H. Gao, M. Tang, and L. Li, “An image compression and encryption scheme based on deep learning,” arXiv preprint arXiv:1608.05001 (2016).
5. P. Refregier and B. Javidi, “Optical image encryption based on input plane and Fourier plane random encoding,” Opt. Lett. 20(7), 767–769 (1995). [CrossRef]
6. W. Chen, B. Javidi, and X. Chen, “Advances in optical security systems,” Adv. Opt. Photonics 6(2), 120–155 (2014). [CrossRef]
7. Q. Wang, D. Xiong, A. Alfalou, and C. Brosseauc, “Optical image encryption method based on incoherent imaging and polarized light encoding,” Opt. Commun. 415, 56–63 (2018). [CrossRef]
8. X. He, H. Tao, Z. Jiang, Y. Kong, S. Wang, and C. Liu, “Single-shot optical multiple-image encryption by jointly using wavelength multiplexing and position multiplexing,” Appl. Opt. 59(1), 9–15 (2020). [CrossRef]
9. F. A. Petitcolas, R. J. Anderson, and M. G. Kuhn, “Information hiding-a survey,” Proc. IEEE 87(7), 1062–1078 (1999). [CrossRef]
10. J. Li, L. Zhong, Q. Zhang, Y. Zhou, J. Xiong, J. Tian, and X. Lu, “Optical image hiding based on dual-channel simultaneous phase-shifting interferometry and compressive sensing,” Appl. Phys. B: Lasers Opt. 123(1), 4–9 (2017). [CrossRef]
11. L. Cai, M. He, Q. Liu, and X. Yang, “Digital image encryption and watermarking by phase-shifting interferometry,” Appl. Opt. 43(15), 3078–3084 (2004). [CrossRef]
12. J. Li, J. Li, Y. Pan, and R. Li, “Optical image hiding with a modified Mach-Zehnder interferometer,” Opt. Laser. Eng. 55, 258–261 (2014). [CrossRef]
13. J. Li, J. Xiong, Q. Zhang, L. Zhong, Y. Zhou, J. Li, and X. Lu, “A one-time pad encryption method combining full-phase image encryption and hiding,” J. Opt. 19(8), 085701 (2017). [CrossRef]
14. J. Li, J. Li, Y. Pang, and R. Li, “Compressive optical image encryption,” Sci. Rep. 5(1), 10374 (2015). [CrossRef]
15. J. Li, M. Lei, X. Dai, J. Li, T. Zhong, H. Li, B. Jia, C. Yang, S. Wang, and C. Wang, Advanced secure optical image processing for communications, Chapter 11. A. A. Falou, ed. (IOP Publishing Ltd., 2018), pp. 1–24.
16. M. Lyu, W. Wang, H. Wang, H. Wang, G. Li, N. Chen, and G. Situ, “Deep-learning-based ghost imaging,” Sci. Rep. 7(1), 17865 (2017). [CrossRef]
17. F. Wang, H. Wang, H. Wang, G. Li, and G. Situ, “Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging,” Opt. Express 27(18), 25560–25572 (2019). [CrossRef]
18. S. Rizvi, J. Cao, K. Zhang, and Q. Hao, “Improving imaging quality of real-time fourier single-pixel imaging via deep learning,” Sensors 19(19), 4190 (2019). [CrossRef]
19. A. L. Mur, B. Montcel, F. Peyrin, and N. Ducros, “Deep neural networks for single-pixel compressive video reconstruction,” Proc. SPIE 11351, 27 (2020). [CrossRef]
20. A. Sinha, G. Barbastathis, J. Lee, and S. Li, “Lensless computational imaging through deep learning,” Optica 4(9), 1117–1125 (2017). [CrossRef]
21. Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light: Sci. Appl. 7(2), 17141 (2018). [CrossRef]
22. H. Wang, M. Lyu, and G. Situ, “eHoloNet: A learning-based end-to-end approach for in-line digital holographic reconstruction,” Opt. Express 26(18), 22603–22614 (2018). [CrossRef]
23. L. Zhang, R. Xiong, J. Chen, and D. Zhang, “Optical image compression and encryption transmission-based on deep learning and ghost imaging,” Appl. Phys. B: Lasers Opt. 126(1), 18–27 (2020). [CrossRef]
24. L. Zhou, Y. Xiao, and W. Chen, “Machine-learning attacks on interference-based optical encryption: experimental demonstration,” Opt. Express 27(18), 26143–26154 (2019). [CrossRef]
25. H. Hai, S. Pan, M. Liao, D. Lu, W. He, and X. Peng, “Cryptanalysis of random-phase-encoding-based optical cryptosystem via deep learning,” Opt. Express 27(15), 21204–21213 (2019). [CrossRef]
26. Y. Qin, Y. Wan, and Q. Gong, “Learning-based chosen-plaintext attack on diffractive-imaging-based encryption scheme,” Opt. Laser. Eng. 127, 105979 (2020). [CrossRef]
27. O. Ronneberger, P. Fischer, and T. Brox, “ U-net: Convolutional networks for biomedical image segmentation,” medical image computing and computer assisted intervention (2015), 234–241.
28. S. Jiao, Y. Gao, J. Feng, T. Lei, and X. Yuan, “Does deep learning always outperform simple linear regression in optical imaging,” Opt. Express 28(3), 3717–3731 (2020). [CrossRef]
29. P. Isola, J. Zhu, T. Zhou, and A. Efros, “Image-to-image translation with conditional adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition. (IEEE, 2017), 1125–1134.
30. C. B. Li, “An efficient algorithm for total variation regularization with applications to the single pixel camera and compressive sensing,” Rice University, Houston, (2010).
31. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in international conference on medical image computing and computer-assisted intervention (Springer2015), 234–241.
32. T. S. Ferguson, “An inconsistent maximum likelihood estimate,” J. Am. Stat. Assoc. 77(380), 831–834 (1982). [CrossRef]
33. J. Kingma and Ba, “Adam: A method for stochastic optimization,” arXiv e-prints 1412.6980 (2014).
34. L. Deng, “The MNIST database of handwritten digit images for machine learning research [best of the web],” IEEE Signal Process. Mag. 29(6), 141–142 (2012). [CrossRef]
35. H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, “ arXiv preprint arXiv:1708.07747 (2017).
