Abstract
We proposed a digital-optical co-design that can effectively improve the image quality of refractive-diffractive hybrid imaging systems over a wide ambient temperature range. Diffraction theory was used to establish the degradation model and blind deconvolution image recovery algorithm was used to perform recovery for simulated images. The peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) were used to evaluate the algorithm performance. A cooled athermalization dual-band infrared optical system with a double-layer diffractive optical element (DLDOE) was designed, and the results show an overall improvement for both PSNR and SSIM overall the whole ambient temperature range. This demonstrates the effectiveness of the proposed method for the image quality improvement of hybrid optical systems.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Diffractive optical elements (DOEs) have become increasingly popular in modern optical systems due to their distinctive achromatic and thermal characteristics. Opto-electronic instruments such as infrared optical systems and high-end commercial instruments, in particular, benefit from the capacity of DOEs in effectively improving the image quality of such systems while also achieving athermalization [1]. Traditional optical lens optimization [2], optical material matching [3], and wavefront encoding [4] provide some common methods for optical system athermalization, but these approaches are often based on aberration theory and principle of optical focal length distribution, making the optical systems complex, heavy, and bulky.
A hybrid optical system composed of DOEs and traditional lenses offers a promising solution, reducing complexity, material dependence, and simplifying the design optimization process [5]. Further, multi-layer DOEs (MLDOEs) can improve the diffraction efficiency over broad to multi-wavebands. However, they are highly sensitive to temperature alteration, which degrades the imaging quality of hybrid optical systems [6]. As a result, researchers seek for optimization designs of DOEs for hybrid optical system to ensure high diffraction efficiency across wide temperature range, yet this technology hasn’t been achieved.
In recent years, modern advances in computational imaging technology have made great progress in the field of DOEs. Y Peng proposed a diffraction achromatic imaging method which allowed for full-spectrum imaging using computationally optimized diffractive lenses [7]. F Heide created a coded DOE design for full-spectrum computational imaging which combined a coded DOE and self-calibrating imaging [8]. J Yang proposed an adaptive wiener filtering (AWN), which significantly improved the details of restored images and aided with the imaging of dark, weak targets [9]. X Dun created a learning rotationally symmetric diffractive achromatic lens design with concentric ring decomposition, reducing computational complexity and memory demands tenfold when compared with traditional end-to-end optimization [10]. Y Hu proposed an optical-digital joint design, centering the central wavelength for single-layer DOE in mid-wave infrared (MWIR) and imaging reconstruction with a constructed point spread function (PSF) model and then added to long-wave infrared (LWIR) [11]. P Wang proposed a chromatic aberration corrected diffractive lens for ultra-broadband focusing [12]. K Liu used networks to complete image restoration [13]. We combined optical design with image restoration, designing a dual-band infrared optical system with a MLDOE that utilized image restoration to improve the imaging quality of hybrid optical systems over wide ambient temperature range [14].
Although there have been researches concerning single-layer DOE, a digital-optical imaging method combining optical system optimization design and blind deconvolution restoration algorithms for hybrid optical systems with MLDOEs has yet to be explored, especially for athermalization need of hybrid optical system. Such a method can reduce the effect on diffraction efficiency reduction that occurs with ambient temperatures changing and improve imaging quality across a broader range of conditions. In this paper, we propose such a method and declare the potential importance for its applications for hybrid imaging systems athermalization realization that operate at wide temperature ranges, which can ultimately reduce the design difficulty and overall cost for athermalization optical system.
2. Design principle and method
2.1 Effect on diffraction efficiency of ambient temperature for MLDOE in dual-band
DLDOE is the most typical MLDOE consisting of two harmonic DOEs of substrate materials with different dispersion properties, and it is with the same period width but different micro-structure heights, illustrated in Fig. 1.
For imaging optical systems, since surface micro-structure parameters DOEs are much larger than the incident wavelength, scalar diffraction theory can satisfy the requirements of design and accuracy. So, at room temperature of 20°C, diffraction efficiency and band-integral average diffraction efficiency (BIADE) of DLDOE based on scalar diffraction theory can be expressed as [15]
To ensure high optical transform function (OTF) for the dual-band DLDOE, the design wavelength pair must be selected carefully to ensure high diffraction efficiency, where, it is ensured by two design wavelengths λ1 and λ2, as
When the maximum comprehensive BIADE (CBIADE) is obtained [16], the corresponding calculation can be performed and then the result is used for subsequent optical system design. The specific steps are as follows:
- (1) set the diffraction order to 1for the dual-band and search the entire waveband range for the maximum $\overline {{{\overline \eta }_1}({\lambda _1},{\lambda _2})}$;
- (2) select the design wavelength pair λ1 and λ2 when the CBIADE obtains to the maximum, and the two optimal micro-structure heights can be calculated;
- (3) calculated the two diffraction orders corresponding to the DLDOE.
Considering environment temperature T into account, its real BIADE can be expressed as
Then, CBIADE and diffraction efficiency can be expressed as
2.2 Image degradation model establishment
The image degradation is expressed as
where, f(x,y) stands for the original image, h(x,y) is the degradation function, n(x,y) is the noise, and g(x,y) is degraded image. For infrared optical systems, its PSF acts as a degradation function and is affected by temperature, expressed asFor hybrid optical systems with a DLDOE, diffraction orders of degraded images are imaged on different planes. And the images are then superimposed on image surface of the 1st order degraded image, resulting in blurred images. Therefore, the total degradation image for optical system is expressed as the product of degraded image at each diffraction order in primary plane and diffraction efficiency at each order. Secondly, the corresponding diffraction efficiency of each order is expressed as the average diffraction efficiency of each order. Additionally, the degraded image at each order during superposition is represented by PSF convolution corresponding to the original image and the diffraction order.
Based on the above, the image degradation model can be represented as
Specifically, we first obtain the restored image affected by environmental temperature, then calculate the spatially invariant PSF affected by temperature at each level, and finally obtain the synthesized PSF affected by temperature by superposition. When replacing ηm(x,y,T) in Eq. (10) by ${\overline \eta _m}(T)$, the image degradation model can be expressed as
The degenerate image of each order is represented as
Substituting Eq. (11) into Eq. (10) yields to
Equation (13) is the image degradation model affected by temperature used for simulation. When BIADE is used for the calculation, the degraded image can be expressed as
2.3 Establishment of image restoration model
From Eq. (13), it can be seen that two important factors affecting the degraded image, PSF and diffraction efficiency. They are both temperature related, and result in the degraded images being a comprehensive output by temperature. Image degradation is still a process of first convolution, then multiplication and superimposing spatially variable PSFs for all orders. Because of the complex nature of the deconvolution process of spatial changing, it is necessary to consider transforming the convolution of spatial change into a spatially invariant convolution.
The substitution in Eq. (12) indicates the temperature-affected restored image. It is replaced by a spatially invariant PSF representing the different orders affected by temperature. For superposition, it subsequently represents the composite PSF affected by temperature. Then, the above method is followed to obtain the restoration results from the temperature-affected images, modeled and then presented as
Equation (15) is the expression of the integrated PSF influenced by temperature, when BIADE is used for the calculation, it is presented as
Then, combining Eqs. (15) and (16) yields to
Equation (20) is the image degradation model used for recovery affected by temperature.
2.4 Image quality evaluation
Image quality evaluation methods consist of subjective evaluation and objective evaluation. Subjective evaluation is based on people’s subjective judgment to directly evaluate the image quality. It mainly evaluates the clarity, contrast of images, and won’t require complex calculations. The evaluation result will be affected by multiple factors, such as the color and grayscale of the image, which may cause visual illusions and lead to incorrect judgments. The evaluation result will also be affected by the subjective factors of the evaluator, such as the evaluator's evaluation criteria, mental state, mood, hobbies, etc., so subjective evaluation cannot achieve absolute consistency. Objective evaluation method uses the error value between the obtained image and the original image to judge the quality of the image. The image quality evaluation is a function that calculates the image with a certain functional relationship, which reflects the image quality. On the basis of the original clear image, the restored image is operated with the original clear image by a certain function, and the function value is used as an evaluation criterion to evaluate the quality of the restored image.
(1) PSNR [17]
PSNR is based on the mean squared error (MSE) definition. Given a raw image of size I and noise image K after noise is added, its MSE can be defined as
In Eq. (19), MAXI is the maximum pixel value of the image, PSNR is calculated in dB. If each pixel is represented by 8-bit binary, its value is 28-1 = 255. However, note that this is a calculation method for grayscale images. For a color image, it can usually be calculated by the following steps: calculate the MSE value for each of the three channels of RGB images and then average it, finally search PSNR, and the larger PSNR value means the better image quality.
(2) SSIM [18]
The basic idea of SSIM is to evaluate the similarity of two images through three aspects, namely brightness l(x,y), contrast c(x,y), and structure s(x,y), and expressed as
3. Dual-band infrared optical system design and image simulation
3.1 Cooled dual-band infrared optical system design
We optimally design a cooled dual-band infrared optical system with a DLDOE for realizing athermation overall the wide temperature range. For simplification, here we use only two optical materials for system design, germanium (Ge) and zinc selenide (ZnSe), and the number of lenses is controlled within 5 pieces. The specific indicators of the optical system and its detector are presented in Table 1.
The solid model of the optical system is shown in Fig. 2, which consists of 5 lenses and 1 flat glass. It is structured from left to right with the 1st, 4th, and 5th lenses made of ZnSe, the 2nd and 3rd lenses made of Ge. In addition, there are two diffractive surfaces on the rear of the 3rd and front of the 4rd lens, and 8 aspherical surfaces. The flat glass is the protective glass of the Dewar flask with material of Ge. The cold stop is implemented as the aperture stop, which is placed between the imaging surface and optical system, with the exit pupil placed at the cold stop, and the exit pupil size is the same as the cold stop, so the optical system can achieve 100% cold stop efficiency. The focal plane of the cooled detector is located inside the Dewar. The structural parameters of the optical system are given in Table 2.
The aspheric polynomial coefficients are determined by ai, represented as
where, z is the sagittal height of aspheric surface, c is the curvature, k is the conic constant, and r is the radial coordinate.The phase delay of diffractive surface can be represented as
where, M is the diffraction order, ρ is the radial radius of the DLDOE substrate, and Ai is the polynomial coefficient.The aspheric and DOE parameters are presented in Table 3 and Table 4, respectively.
MTFs for the infrared optical system at -40°C, 20°C and 60°C is shown in Fig. 3, and they are greater than 0.48 and 0.40 for MWIR and LWIR at the detector cut-off frequency of 17lp/mm, respectively. The GEO and RMS radii of the system for all fields at different temperatures are presented in Table 5. At -40°C, the maximum GEO radii of 27,500µm and 18,867µm for MWIR and LWIR, respectively, larger than the pixel size of the detector. At room temperature, the maximum GEO radii for MWIR and LWIR are 14.656µm and 13.631µm, respectively. At -40°C, the maximum RMS radii for MWIR and LWIR are 14.720µm and 11.402µm, respectively, smaller than the pixel size of the detector.
We can see that, at room temperature, the optical system exhibits good imaging quality in both dual-bands. However, when the temperature deviates from room temperature, the GEO radius changes drastically and the CBIADE decreases substantially. Additionally, the GEO radius of the non-1st-order diffraction plane is much larger than the pixel size, resulting in a blurred image and deterioration of imaging quality. Consequently, the imaging quality of the designed optical system is greatly affected by the change in ambient temperature. The DLDOE design result, based on CBIADE method, is presented in Table 6.
The CBIADE versus ambient temperature for the combination of Ge and ZnSe substrates and the BIADE versus ambient temperature for MWIR and LWIR is shown in Fig. 4.
3.2 Dual-band infrared image simulation
According to the previous analysis, the image degradation model of the dual-band infrared optical system can be calculated based on BIADE under the influence of ambient temperature, where the degradation image is obtained by the image simulation function with ZemaxOpticStudio software.
The image simulation process is clarified as follows: first, calculate the BIADE for each diffraction order for MWIR and LWIR at different temperatures; second, input a clear infrared image into the optical system as a light source bitmap, and each diffraction order image is output at certain temperature; third, multiply the images at each diffraction order with its corresponding BIADE and superimposing, the resulting image is approximately a simulated image at the certain temperature.
The relationship between BIADE and ambient temperature at diffraction orders for MWIR and LWIR is shown in Fig. 5, with only 5 diffraction orders of BIADE being plotted here due to its diffraction efficiency distribution.
The BIADE of MWIR and LWIR at different temperature are listed in Table 7. After calculation, the average of total BIADE at each temperature for MWIR and LWIR are 95.359% and 97.717%, respectively, which are relatively high and can be used for image simulation.
Then, we simulate the imaging by inputting a clear infrared image to the optical system, the size of the light source bitmap is set to 320*240 pixels, shown in Fig. 6.
Images at diffraction orders are output for MWIR and LWIR, and comparisons are shown in Table 8, where the sub-graphs (a1) ∼(a6) represent 1st, 2nd, 3rd, 0, -1st diffraction order and simulation image, respectively.
It can be seen that the diffraction images at the 1st diffraction order are relatively clear at each temperature, indicating that the athermal effect on the optical system is the best. The images at other diffraction orders, however, are quite blurred because the non-primary diffraction images are strongly scattered in primary images. The LWIR non-primary diffraction images are in particular more blurred than MWIR images, because LWIR non-primary diffraction images are more diffused, which is also due to the fact that the absolute values of the diffraction orders of DLDOE are smaller in LWIR than in MWIR. Interestingly, despite this, the diffraction efficiency of the 1st order for LWIR is more severe at each temperature, making the LWIR simulation image clearer than the MWIR image. As the ambient temperature further deviates from the normal temperature, the blurriness of the image becomes more severe.
4. Image restoration and analysis
For PSF establishment of dual-band infrared optical system, taking working temperature of -20°C as an example, the PSF at different diffraction orders (${\widehat {PSF}_m}(x,y,T)$) and synthetic PSF ($\widehat {PSF}(x,y,T)$) outputs for MWIR and LWIR is shown in Fig. 7, where (a1)∼(a6) and (b1)∼(b6) stand for the 1st, 2nd, 3rd, 0, -1st and the comprehensive PSF, respectively.
Since the PSF for each pixel is small and the actual display is dark, Fig. 7 is magnified results of multiplying the actual value by a certain multiple, where the sub-graph Fig. 7(a1) is magnified 30 times, Figs. 7(a2) and (a4) are magnified 300 times, Figs. 7(a3) and (a5) are magnified 1000 times, and Fig. 7(a6) are magnified 100 times. In Fig. 7(b), Fig. 7(b1) is magnified by a factor of 2, Fig. 7(b2) and (b4) are magnified by a factor of 1000 and Fig. 7(b3), (b5) and (b6) are magnified by a factor of 2000. It can be seen that the 1st diffraction order has a large effect on the overall PSF, the 0th and 2nd diffraction order have a small effect on the overall PSF, and the -1st and 3rd order diffraction have a small effect on the overall PSF.
Then, without considering the image noise for image restoration, a blind deconvolution algorithm is used for image restoration. The simulated images of MWIR and LWIR at each temperature are restored and compared with the simulated images before restoration, shown in Fig. 8, where the sub-graphs (a1) and (a2) are simulation and restoration diagrams at -40°C; (a3) and (a4) are simulation and restoration diagrams at -20°C; (a5) and (a6) are simulation and restoration diagrams at 0°C; (a7) and (a8) are simulation and restoration diagrams at 40°C; (a9) and (a10) are simulation and restoration diagrams at 60°C. Additional, sub-graphs for LWIR is the same as the MWIR imaging.
The restored image and the simulation image are compared with light source bitmap using PSNR and SSIM for calculation, respectively, and the results are shown in Table 9. It can be seen that the image quality is effectively improved within the ambient temperature range.
5. Conclusion
We presented a digital-optical co-design for athermalization of a refractive-diffractive hybrid imaging system in a wide ambient temperature range with a DLDOE. An image degradation model of DLDOE was established by converting the spatially varying PSF into a spatially invariant PSF. An image degradation model for restoration was also put forward and presented. A cooled dual-band refractive infrared hybrid optical system with a DLDOE was then designed achieving athermalization with good satisfied image quality using only two common optical materials, Ge and ZnSe, for the wavebands of 3.7-4.8µm and 7.7-10µm, working temperature range of -40∼60°C, focal length of 105 mm, F-number of 1.5, and full field of view of 6°. The optical system was used for image simulation, with a blind reconstruction algorithm used for image restoration. Finally, PSNR and SSIM evaluation functions were used to evaluate its image quality. Results showed that the quality of the restored image was improved at each temperature in both dual-bands. For instance, for MWIR, when the ambient temperature was -20°C, the diffraction efficiency of the 1st order decreased to about 40%, with the restored image displaying the best quality improvement in terms of both the PSNR increased from 23.4926 to 27.2919, and the SSIM increased from 0.8154 to 0.9296. These results confirmed the effectiveness of the proposed method in improving the image quality over a wide ambient temperature range. Our method and result can help to promote the wide temperature application of DOE while enriching the application range of computational imaging technologies.
Funding
National Natural Science Foundation of China (61905195, 61927810); Northwestern Polytechnical University Central University Basic Research Business (310202011QD001); Key Laboratory of Spectral Imaging Technology, Chinese Academy of Sciences (LSIT202006W); Xi'an Young Talents Sponsorship Project (095920201316).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. T. H. Jamieson, “A thermalization of optical instruments from the optomechanical viewpoint,” Pro. SPIE 10265, 1026508 (1992). [CrossRef]
2. T. Y Lim and S. CPark, “Achromatic and athermal lens design by redistributing the element powers on an athermal glass map,” Opt. Express 24(16), 18049–18058 (2016). [CrossRef]
3. S. E. Ivanov and G. E. Romanova, “Optical material selection method for an apochromatic athermalized optical system,” J. Opt. Technol. 83(12), 729–733 (2016). [CrossRef]
4. B. Feng, Z. L. Shi, H. Z. Liu, and Y. H. Zhao, “Effect of PSF position bias in wavefront coding infrared imaging system,” J. Opt. 22(2), 025703 (2020). [CrossRef]
5. C. R. Bigwood, “New infrared optical systems using diffractive optics,” Pro. SPIE 4767, 1–12 (2002). [CrossRef]
6. M. X. Piao, Q. F. Cui, H. Zhu, C. X. Xue, and B. Zhang, “Diffraction efficiency change of multilayer diffractive optics with environmental temperature,” J. Opt. 16(3), 035707 (2014). [CrossRef]
7. Y. Peng, Q. Fu, F. Heide, and W. Heidrich, “The diffractive achromat full spectrum computational imaging with diffractive optics,” SIGGRAPH ASIA 2016 Virtual Reality meets Physical Reality: Modelling and Simulating Virtual Humans and Environments, 1–2 (2016).
8. F. Heide, Q. Fu, Y. Peng, and F. Heidrich, “Encoded diffractive optics for full-spectrum computational,” Sci. Rep. 6(1), 33543 (2016). [CrossRef]
9. J. Yang, S. Wang, L. Wen, P. Yang, W. Yang, C. Guan, and B. Xu, “Experimental study on imaging and image deconvolution of a diffractive telescope system,” Appl. Opt. 58(33), 9059–9068 (2019). [CrossRef]
10. X. Dun, H. Ikoma, G. Wetzstein, Z. Wang, X. Cheng, and Y. Peng, “Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging,” Optica 7(8), 913–922 (2020). [CrossRef]
11. Y. Hu, Q. F. Cui, L. D. Zhao, and M. X. Piao, “PSF model for diffractive optical elements with improved imaging performance in dual-waveband infrared systems,” Opt. Express 26(21), 26845–26857 (2018). [CrossRef]
12. P. Wang, N. Mohammad, and R. Menon, “Chromatic-aberration-corrected diffractive lenses for ultra-broadband focusing,” Sci. Rep. 6(1), 21545 (2016). [CrossRef]
13. K. Liu, X. Yu, Y. Xu, Y. Xu, Y. Yao, N. Di, Y. Wang, H. Wang, and H. Shen, “Computational imaging for simultaneous image restoration and super-resolution image reconstruction of single-lens diffractive optical system,” Appl. Sci. 12(9), 4753 (2022). [CrossRef]
14. S. Mao, H. L. Nie, T. Lai, and N. Xie, “Computational imaging in dual-band infrared hybrid optical system with wide temperature range,” Sensors 22(14), 5291 (2022). [CrossRef]
15. N. Takehiko and O. Hideki, “Research on multi-layer diffractive optical elements and their application to camera lenses,” Optical Society of America, (2002).
16. C. X. Xue, Q. F. Cui, T. Liu, L. L. Yang, and B. Fei, “Optimal design of a multilayer diffractive optical element for dual wavebands,” Opt. Lett. 35(24), 4157–4159 (2010). [CrossRef]
17. P. Garg and R. R. Kishore, “Optimized color image watermarking through watermark strength optimization using particle swarm optimization technique,” J. Inform. Opt. Sci. 41(6), 1499–1512 (2020). [CrossRef]
18. Z. Wang, A. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. on Image Process. 13(4), 600–612 (2004). [CrossRef]