Open Access
Add to BibSonomy
Abstract
The rapid advancement of portable electronics has created enormous demand for compact optical imaging systems. Such systems often require folded optical systems with beam steering and shaping components to reduce sizes and minimize image aberration at the same time. In this study, we present a solution that utilizes an inverse-designed dielectric metasurface for arbitrary-angle image-relay with aberration correction. The metasurface phase response is optimized by a series of artificial neural networks to compensate for the severe aberrations in the deflected images and meet the requirements for device fabrication at the same time. We compare our results to the solutions found by the global optimization tool in Zemax OpticStudio and show that the proposed method can predict better point-spread functions and images with less distortion. Finally, we designed a metasurface to achieve the optimized phase profile.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The rising demand for portability in smartphones, near-eye displays, medical devices, etc. has necessitated the development of miniature optical system designs characterized by reduced sizes and weights. Metasurfaces are capable of manipulating light at a subwavelength scale through nanostructures [1] and have shown great potential to significantly shrink the overall footprint and achieve system miniaturization. In recent years, metasurfaces have been extensively investigated for achromatic focusing [2,3], beam shaping [4–6], filtering [7,8], polarization control [9,10], aberration correction [11,12], all-optical image processing [13–16] and more. In addition, multiple functions can be multiplexed onto one single metasurface device [17–19]. Among these studies, considerable attention has been devoted to enhancing the optical quality at paraxial imaging conditions with rotationally symmetric systems, where the performance degrades rapidly as the beam deviates from normal incidence. Metasurfaces with asymmetric phase functions for off-axis image relaying are insufficiently investigated. The latter represents a more general case in practice, where both the positions of the input and sensor are arbitrary and an image-relay path is to be determined. In this work, we present our solution for designing metasurfaces that address this problem. Specifically, we utilize a physical optics propagation model to evaluate the aberration in the deflected images and adopt a series of artificial neural networks to find the optimal phase function that ensures good image quality for a wide field of view.
Figure 1 shows a generalized imaging configuration. Light from the upstream optics encounters the input surface and is redirected to a tilted image plane. In practice, the input and image planes can be any two adjacent surfaces along the optical path of a system. In this paper, we use a metasurface to fulfill the role of beam shaping. The first function of the metasurface is to relay the image collected by previous optics to another image plane tilted by a large angle. The second and most important function of the metasurface is to perform optical aberration correction on the deflected image. Thus, the design of the metasurface is to optimize the phase profile to accommodate known inputs from the upstream optics. Traditionally, optimization of systems for aberration reduction has relied upon gradient-free approaches, such as evolutionary (or genetic) algorithms [20,21]. Evolutionary algorithms use an iterative process of mutation and selection based on solution fitness, and do not require the computation of a gradient, making them useful in optimization with complicated loss functions with discontinuities. Commercial optical design software packages like Zemax OpticStudio (Ansys, Inc.) have adopted these methods in their built-in optimization tools to predict starting point designs. Nonetheless, these algorithms take a great amount of time to converge and struggle to scale well as the number of parameters increases. Gradient-based methods such as adjoint optimization are also used in photonic inverse design and can handle a large-number of parameters such as in topology optimization [22]. However, these methods require a fully differentiable forward model and can be quite complex to implement. Recently, machine learning has become increasingly popular for the inverse design of photonic devices and imaging systems [23–28]. It has been demonstrated that artifical neural networks (ANNs) can be trained as a surrogate forward model for optimization in cases where the physical model is either unknown, non-differentiable, or complex to map between the input and output data.
Fig. 1. A schematic of a general image relay configuration, with two adjacent planes extracted from an optical train. The angle between the planes can be large. $D_{{ap}}$ is the clear aperture diameter of the metasurface and $Z_{{prop}}$ is the center-to-center distance of the planes. Right panel: Without correction optics, light from the upstream optics misses the image plane and forms aberrated images. After adding an optimized metasurface, light can be correctly deflected to the image plane and forms unaberrated images.
There are several common approaches for deep learning based inverse design in photonics. The problem of inverse design can be defined as the following: find the set of parameters $D$ that when passed through the physical model $f$ gives a certain response $R$, i.e., $f(D) = R$. The first and most simple approach is to train a neural network that tries to directly solve the inverse problem: $D = f^{-1}(R)$ given some desired response. However, in practice this approach often fails as the inverse problem is ill-posed due to non-uniqueness. In many problems there are multiple $D$ that produce the same $R$ and the network struggles to learn the inverse function.
One popular approach to overcome this is the tandem neural network approach where a surrogate forward model network is combined with an inverse design network [29]. In this method an ANN is first trained to approximate the forward model of a system and is then fixed. The model is then coupled with a second neural network which learns to generate solutions that produce the desire response when passed through the surrogate forward model. In this way physical consistency is enforced and the inverse design network is aided by the surrogate forward model.
Similarly, iterative surrogate methods have gained attention as an alternate to direct inversion approaches [30–32]. In these approaches the optimal design is solved for by evaluating the design parameters through the surrogate forward model and back-propagating the loss to update the parameters. These methods differ from the "one-shot" methods like tandem neural networks as they must re-run optimization for each desired design. Unlike direct inversion, they are similar to traditional iterative algorithms except that the forward model is replaced with a trained (and fixed) neural network. These methods can be useful when it is difficult to calculate the gradients of the forward model or when the speed of the optimization process needs to be increased. Building off these methods, we use an iterative surrogate approach combined with a deep generative prior to design the metasurface phase profile.
In the following work, we will first introduce the modeling and building of the physical optics propagator surrogate and neural network optimizer in section 2. The results of phase optimization are shown in section 3 and the design of the metasurface elements is introduced in section 4. Finally, we will discuss the results and conclude this work in sections 5.
2. Metasurface phase optimization process
To begin, we set the clear aperture diameter $D_{{ap}}$ to 6 mm and the distance $Z_{{prop}}$ to 20 mm, and assume a field of view (FOV) of 16° on the image deflector. These specifications are chosen arbitrarily. For the convenience of demonstration, we restrict the rotation to along one axis, such that we can speed up the simulation by taking advantage of the symmetry of our system. However, the method can be extended to cases with an arbitrary rotated image plane.
Our ANN phase optimization process is a two step framework where two neural networks work in tandem to produce a phase distribution that results in the highest image quality. We assess image quality by evaluating the point spread functions (PSFs) of the imaging system. One network serves as a surrogate model for the off-axis field propagation and can be used repeatedly for new design problems once training is complete. This network is trained first on a set of paired data mapping phase profiles to PSFs. The other network takes the place of the Zernike coefficient parameterization and is retrained for each new design problem. Here, Zernike polynomials are chosen to represent the phase profile due to their good mathematical properties in defining functions in a circular domain (see Supplement 1) and to prevent high spatial frequency solutions.
As shown in Fig. 2, a randomly initialized and fixed tensor is the input to a generative neural network. The generative network produces a set of Zernike coefficients that represent a phase profile. Then, a physical optics propagator (POP) evaluates the PSFs produced by this phase profile for a series of angles covering the image plane field of view. We evaluate PSFs in terms of root-mean-square (RMS) radius and eccentricity. The RMS radius is defined as:
Fig. 2. Optimization pipeline of the NN inverse design process. Step 1 trains a NN POP surrogate to predict the qualities of the PSFs with a set of Zernike coefficients ($\bf {Z_1}$ to $\bf {Z_n}$). Step 2 uses a generative neural network and the trained POP surrogate to find a set of Zernike coefficients that maximize the quality of the PSFs.
In this work, we trained an ANN POP surrogate to replace the slow and non-differentiable off-axis POP implemented in MATLAB. To generate the training data, a free-space angular spectrum method (ASM) for scalar diffraction between non-parallel planes is constructed. To simulate the beam aberration correctly, the ASM POP will first transform the spatial frequency spectrum (K-spectrum) of the aperture in the source coordinate into the rotated coordinate of the image plane, as shown in Fig. 3.
Fig. 3. Illustration of K-spectrum rotation in the off-axis ASM POP. The tilt phase deflects the image to an off-axis angle and the aberration correction phase reduces the aberration introduced during the deflection ASM POP first simulates the off-axis image aberration by transforming the K-spectrum at the input coordinate in the rotated output coordinate. During this process, the spectrum is shifted to the low-K region. Then, a regular paraxial ASM can be applied to simulate the final image.
Numerically, the rotation operation has two effects: 1) the frequency components of the source field is shifted when observed in the rotated coordinate, and 2) the rotation distorts the equidistant-sampled spatial frequencies and requires an interpolation to form a regular grid of frequencies for simulation. To ensure fidelity in the interpolated spectrum and avoid aliasing in the real space, non-parallel ASM requires fine sampling of the field and intensive computation. In this work, the non-uniform fast Fourier transform (NUFFT) algorithm is adopted to combine the FFT and interpolation to accelerate the computation [34,35]. At the time of this study, there is no stable integration of NUFFT with deep-learning frameworks. Hence, we use a learned ANN POP surrogate, which is differentiable and fast. Once the training in step 1 is complete, we freeze the weights of the trained NN POP and use it for step 2.
Taking advantage of the system’s vertical symmetry, we evaluate half of the FOV at six field angles, i.e., (0°, 8°), (0°, 0°), (0°, −8°), (4°, 4°), (4°, −4°), and (8°, 0°), expressed in a Cartesian coordinate. For the same reason, only the first eight Zernike polynomials with vertical axis symmetry are used. The piston phase ($\bf {Z_0}$) is excluded.
As a reference, a model with the same geometry is also optimized in Zemax OpticStudio with the provided global optimization tool. The metasurface phase element is modeled as a Zernike standard phase surface with the same Zernike polynomials as the NN phase optimization. The process includes a 48-hour-long global optimization followed by a 10-hour-long Hammer optimization. Details of the non-parallel ASM POP, neural networks, and Zemax optimization can be found in the supplementary material.
3. Results and comparison
Figure 4 compares the six PSFs of Zemax optimization (Zemax PSFs, first column) with those of the neural network (NN PSFs, second column). The Zemax PSFs are simulated with Zemax POP. The NN PSFs are simulated with our custom-built non-parallel ASM POP. The third column shows the NN PSFs simulated with the Zemax POP as a verification of our POP substitute. Details of the Zemax POP for PSF simulations are included in the supplementary material. Overall, the Zemax PSFs have smoother intensity distributions with cleaner backgrounds. However, the NN PSFs show a rounder intensity distribution, albeit with halos, around a tightly focused PSF cores. Statistically, The RMS radii of the NN PSFs are relatively constant at all field angles, while the Zemax PSFs grow larger as the field angle increases, as indicated in Table 1. In terms of the shape of the PSF, the NN PSFs are more rotationally symmetric compared to the Zemax PSFs, as indicated by the eccentricities.
Fig. 4. PSF comparison: Zemax PSFs (first column) and NN PSFs simulated with non-parallel ASM POP (second column) and Zemax POP (third column). Scale bar: 300 µm.
Table 1. RMS Radii and Eccentricities
Lastly, we simulate full field-of-view images to visually assess the imaging quality on the off-axis sensor. As the PSF varies across the FOV, with a given phase profile, the Zemax Image Simulation tool simulates the PSFs of 19 (horizontal) × 25 (vertical) incident angles within the 16°×16° FOV, and then generates images via a pixel-interpolated spatially varying convolution algorithm. The results are shown in Fig. 5. Among them, Fig. 5(a)-(c) span the FOV of 16°. (d)-(f) are the corresponding images formed with the NN PSFs and (g)-(i) are those with the Zemax PSFs. Asymmetric pincushion distortion and stretching are observed in these images, which is the combined outcome of imaging with a flat sensor and the existence of a strong tilt phase in the metasurface. The phase tilt deflects the image to 41° and causes the ray angle to rotate non-linearly to the incident angle, resulting in different magnification as field height varies. As is shown in Fig. 5, the images from the Zemax PSFs have good contrast. However, the vertical resolution degrades quickly at large field angles, rendering the fine details in this direction unrecognizable. In comparison, the fine features, such as texts and lines, can still be recognized across the full FOV in the images from the NN PSFs, as a result of each PSF possessing a bright center. In terms of the halos, the large pixel size averages the intensity quickly and forms a rather uniform background behind the details. This background can be easily removed with simple digital image processing algorithms. To summarize, our NN optimized phase profile leads to images with better resolution and less distortion. The final optimized radii and eccentricities of the NN-PSFs are shown in Table 1.
Fig. 5. Simulated images with the optimized PSFs. Left panel: (a)-(c) Objects spanning 16° FOV. (d)-(f) Images simulated with the NN PSFs. (g)-(i) Images simulated with the Zemax PSFs. Right panel: zoom-in images of crop regions on (d)-(i) (yellow dotted boxes). Among them, Fig. (1)–(3) are corresponding to (d)-(f), respectively. So are Fig. (4)–(6) to figures (g-i). Logos of UCSD Geisel Library and Triton Athletics are used with permission from the University of California San Diego.
4. Optimized phase profile and metasurface design
In this study, a dielectric metasurface with high transmission efficiency is designed. The metasurface is composed of silicon ($n=3.5$) nanoposts on a fused silica substrate ($n=1.44$). The lattice period ($P$) is 660 nm and the height ($H$) of the nanoposts is 1550 nm, as shown in 6. The metasurface has similar phase responses in both TE and TM polarization, as indicated by Fig. 6(c). The simulation of the metasurface unit cell response is carried out with the wave optics module on COMSOL Multiphysics. Periodic boundary conditions are applied to four lateral sides of the simulation domain. Light is injected from a port at the top of the domain at an angle of 41°. The transmission coefficient is calculated with a receiving port at the bottom.
Fig. 6. Metasurface design. (a) The geometry of the nanoposts. (b) The transmittance and phase as functions of the diameter (D) and height (H) of the nanopost. The solid background color denotes the transmittance and overlaid are the phase contours. (c) Phase response as functions of the nanopost diameter with TE and TM polarized light incident at 41°.
The optimized phase profile is a sum of a strong linear phase for image deflection and a phase with a trefoil shape for aberration correction, as shown in Fig. 7(a). To ensure phase fidelity, in the design we separate the linear phase from the sum. The former can be realized with a transmission grating with a groove density of 0.43 linepairs per micron. As a result, the residual aberration-correction phase has a much slower phase variation with a maximum phase gradient of 0.14 waves per micron, as shown in (b), corresponding to at least 10 phase steps per $2\pi$ phase change with the above mentioned metasurface unit cell design. During fabrication, the grating and metasurface can be made on both sides of the same substrate with a negligible gap.
Fig. 7. (a) Aberration phase on the metasurface. The contours are the projection of the phase function. (b) The spatial derivative of the aberration correction phase. The maximum phase gradient is 0.14 waves per micron, which is compatible with the common high-efficiency metasurface designs.
5. Discussion and conclusion
In this work, we demonstrate a large-angle off-axis image relaying method based on a metasurface. This method describes the image relay between the input and the image planes shown in Fig. 1, and can be applied to systems with more complicated designs. In those designs, the fields can be traced with the appropriate propagator and every element becomes the image plane and input plane alternatively, such that the starting-point design of a system can be rid of laborious hand-tuning and be optimized in a highly automated style.
The proposed method can be improved in several ways. First, the system becomes more asymmetric rotationally as the off-axis angle increases and more aberration is expected. With the Zernike basis and the current data-driven model, more polynomials should be added to reduce the fitting errors, which increases the size of the dataset needed for the POP surrogate training. To reduce the data preparation time, one can use basis that represent the aberration more efficiently with fewer orders. Second, the weighting parameters in the training loss function are chosen empirically. In future work, a fully differentiable physical model could be used to simulate image quality and achieve an end-to-end optimization of the optics without user input.
To conclude, we present a solution for aberration correction in arbitrary-angle image-relay systems by utilizing an inverse-designed dielectric metasurface. The aberrations in the relayed image at large angles are reduced noticeably. Compared to Zemax optimization, the ANN predicts PSFs with smaller sizes and similar resolutions in both direction. The use of the metasurface has enabled us to overcome the limitations posed by traditional optics and achieve a lightweight and small system. With the ongoing interdisciplinary studies of nanophotonics and computational optical design, we believe that this work will inspire further development of novel optical system design in a wide range of applications.
Funding
Leonardo DRS; National Science Foundation Graduate Research Fellowship (DGE-2038238).
Acknowledgement
We thank Marco Lopez from Leonardo DRS for valuable discussion and suggestions. This project is partially supported by Leonardo DRS. Zachary Burns is funded by the National Science Foundation Graduate Research Fellowship (DGE-2038238).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Supplemental document
See Supplement 1 for supporting content.
References
1. N. Yu, P. Genevet, M. A. Kats, et al., “Light propagation with phase discontinuities: generalized laws of reflection and refraction,” Science 334(6054), 333–337 (2011). [CrossRef]
2. S. Wang, P. C. Wu, V.-C. Su, et al., “A broadband achromatic metalens in the visible,” Nat. Nanotechnol. 13(3), 227–232 (2018). [CrossRef]
3. W. T. Chen, A. Y. Zhu, J. Sisler, et al., “Broadband achromatic metasurface-refractive optics,” Nano Lett. 18(12), 7801–7808 (2018). [CrossRef]
4. Y. Deng, C. Wu, C. Meng, et al., “Functional metasurface quarter-wave plates for simultaneous polarization conversion and beam steering,” ACS Nano 15(11), 18532–18540 (2021). [CrossRef]
5. J. Scheuer, “Metasurfaces-based holography and beam shaping: engineering the phase profile of light,” Nanophotonics 6(1), 137–152 (2017). [CrossRef]
6. J. Zhou and Z. Liu, “Photonic spin-dependent wave shaping with metasurfaces: applications in edge detection,” in Plasmonic Materials and Metastructures, (Elsevier, 2024), pp. 227–243.
7. F. Tian, J. Zhou, Q. Wang, et al., “Tunable topological phase transition in the telecommunication wavelength,” Opt. Mater. Express 13(6), 1571–1578 (2023). [CrossRef]
8. Y. Horie, A. Arbabi, E. Arbabi, et al., “Wide bandwidth and high resolution planar filter array based on dbr-metasurface-dbr structures,” Opt. Express 24(11), 11677–11682 (2016). [CrossRef]
9. Y. Ren, S. Guo, W. Zhu, et al., “Full-stokes polarimetry for visible light enabled by an all-dielectric metasurface,” Adv. Photonics Res. 3(7), 2100373 (2022). [CrossRef]
10. A. Zaidi, N. A. Rubin, M. L. Meretska, et al., “Metasurface-enabled compact, single-shot and complete mueller matrix imaging,” arXivarXiv:2305.08704 (2023). [CrossRef]
11. W. Liu, D. Ma, Z. Li, et al., “Aberration-corrected three-dimensional positioning with a single-shot metalens array,” Optica 7(12), 1706–1713 (2020). [CrossRef]
12. C. Kim, S.-J. Kim, and B. Lee, “Doublet metalens design for high numerical aperture and simultaneous correction of chromatic and monochromatic aberrations,” Opt. Express 28(12), 18059–18076 (2020). [CrossRef]
13. Q. Wu, J. Zhou, X. Chen, et al., “Single-shot quantitative amplitude and phase imaging based on a pair of all-dielectric metasurfaces,” Optica 10(5), 619–625 (2023). [CrossRef]
14. G. Chen, J. Zhou, S. E. Bopp, et al., “Visible and near-infrared dual band switchable metasurface edge imaging,” Opt. Lett. 47(16), 4040–4043 (2022). [CrossRef]
15. J. Zhou, H. Qian, C.-F. Chen, et al., “Optical edge detection based on high-efficiency dielectric metasurface,” Proc. Natl. Acad. Sci. 116(23), 11137–11140 (2019). [CrossRef]
16. Y. Ni, C. Chen, S. Wen, et al., “Computational spectropolarimetry with a tunable liquid crystal metasurface,” eLight 2(1), 23 (2022). [CrossRef]
17. Y. Kim, G.-Y. Lee, J. Sung, et al., “Spiral metalens for phase contrast imaging,” Adv. Funct. Mater. 32(5), 2106050 (2022). [CrossRef]
18. J. Zhou, J. Zhao, Q. Wu, et al., “Nonlinear computational edge detection metalens,” Adv. Funct. Mater. 32(34), 2204734 (2022). [CrossRef]
19. B. Born, S.-H. Lee, J.-H. Song, et al., “Off-axis metasurfaces for folded flat optics,” Nat. Commun. 14(1), 5602 (2023). [CrossRef]
20. S. Molesky, Z. Lin, A. Y. Piggott, et al., “Inverse design in nanophotonics,” Nat. Photonics 12(11), 659–670 (2018). [CrossRef]
21. S. D. Campbell, D. Sell, R. P. Jenkins, et al., “Review of numerical optimization techniques for meta-device design,” Opt. Mater. Express 9(4), 1842–1863 (2019). [CrossRef]
22. J. S. Jensen and O. Sigmund, “Topology optimization for nano-photonics,” Laser Photonics Rev. 5(2), 308–321 (2011). [CrossRef]
23. W. Ma, Z. Liu, Z. A. Kudyshev, et al., “Deep learning for the design of photonic structures,” Nat. Photonics 15(2), 77–90 (2021). [CrossRef]
24. P. R. Wiecha, A. Arbouet, C. Girard, et al., “Deep learning in nano-photonics: inverse design and beyond,” Photonics Res. 9(5), B182–B200 (2021). [CrossRef]
25. G. Côté, J.-F. Lalonde, and S. Thibault, “Deep learning-enabled framework for automatic lens design starting point generation,” Opt. Express 29(3), 3841–3854 (2021). [CrossRef]
26. X. Yang, Q. Fu, and W. Heidrich, “Curriculum learning for ab initio deep learned refractive optics,” arXiv, arXiv:2302.01089 (2023). [CrossRef]
27. J. He, Z. Guo, Y. Zhang, et al., “Physics-model-based neural networks for inverse design of binary phase planar diffractive lenses,” Opt. Lett. 48(6), 1474–1477 (2023). [CrossRef]
28. E. Tseng, S. Colburn, J. Whitehead, et al., “Neural nano-optics for high-quality thin lens imaging,” Nat. Commun. 12(1), 6493 (2021). [CrossRef]
29. D. Liu, Y. Tan, E. Khoram, et al., “Training deep neural networks for the inverse design of nanophotonic structures,” ACS Photonics 5(4), 1365–1369 (2018). [CrossRef]
30. A. Khaireh-Walieh, D. Langevin, P. Bennet, et al., “A newcomer’s guide to deep learning for inverse design in nano-photonics,” Nanophotonics 12(24), 4387–4414 (2023). [CrossRef]
31. Y. Deng, S. Ren, K. Fan, et al., “Neural-adjoint method for the inverse design of all-dielectric metasurfaces,” Opt. Express 29(5), 7526–7534 (2021). [CrossRef]
32. Y. Augenstein, T. Repan, C. Rockstuhl, et al., “Neural operator-based surrogate solver for free-form electromagnetic inverse design,” ACS Photonics (2023).
33. E. Bostan, R. Heckel, M. Chen, et al., “Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network,” Optica 7(6), 559–562 (2020). [CrossRef]
34. A. H. Barnett, J. Magland, and L. af Klinteberg, “A parallel nonuniform fast fourier transform library based on an “exponential of semicircle” kernel,” SIAM J. Sci. Comput. 41(5), C479–C504 (2019). [CrossRef]
35. A. H. Barnett, “Aliasing error of the exp $(\beta \sqrt {1-z^ 2})$ kernel in the nonuniform fast fourier transform,” Appl. Comput. Harmon. Anal. 51, 1–16 (2021). [CrossRef]
