Fig. 1. Deep-learning-based forward solvers for ultra-fast physics predictions. (a) Simultaneous electric and magnetic dipole resonance prediction and inverse design in multi-layer nano-spheres. Adapted with permission from [56], copyright (2019) American Chemical Society. (b) Nano-optics solver network, which predicts the optical response of a grating based on multiple Lorentz oscillators. As shown in the right panel, the physics-based data representation allows the network to generalize well outside the range of the training data (blue points). Adapted with permission from [57], copyright (2020) Optical Society of America. (c) Internal electric polarization density predictor network. The results can be used in a coupled dipole approximation framework to calculate a large number of secondary near- and far-field effects. Adapted with permission from [58], copyright (2020) American Chemical Society.

Fig. 2. Examples of devices inverse designed by ML algorithms. (a) Encoder–decoder type tandem inverse network used to design perturbation patterns for $3\times 3$ MMIs as arbitrary transmission matrix elements. The light routing behavior of the second and the third input channels is interchanged between cases (i) and (ii), while the first input channel keeps routing light to the second output. Adapted with permission from [84], copyright (2021) American Chemical Society. (b) Double-focus flat lens designed by a conditional WGAN inverse network. (i) shows the dielectric metasurface, (ii) the corresponding amplitude, and (iii) the phase mask. (iv) shows a numerical simulation of the field intensity to test the ANN design. Adapted with permission from [73].

Fig. 3. Concepts to improve common shortcomings of inverse design ANNs. (a) Iterative training data generation, in which a network learns from its own errors, here applied to the inverse design of an invisibility cloak device. Adapted from [89], copyright (2021) Optical Society of America. (b) Comparison of the $Q$-factors for photonic crystal cavities in a random dataset (left) and in an iteratively generated dataset after the first iteration (right). Adapted from [94], copyright (2019) de Gruyter. (c) Together with the training data, the network complexity can be progressively growing, allowing even better performance by successive learning of smaller features. Reprinted with permission from [95], copyright (2020) American Chemical Society. (d) Mixture density ANN which represents multiple solutions with Gaussian probability distributions to find several non-unique solutions to ambiguous problems. The shown example deals with the spectral design of a multi-layer stack. Adapted with permission from [100], copyright (2020) American Chemical Society. (e) De-noising inverse ANN as robust approach for training on noisy data (noise parameter $a$ increasing from top to bottom). Adapted from [101], copyright (2019) Optical Society of America. (f) “GLOnet”: inverse design ANN using a transfer-matrix model loss for reflectivity and transmission spectra optimization of multi-layer stacks. Adapted from [97], copyright (2020) de Gruyter.

Fig. 4. Examples of input data pre-processing for optimized physics domain representation. (a),(b) Deep learning on irregular grids via coordinate transform (a) which is implemented within the deep learning toolkit to allow fast gradient calculations through the coordinate system transformation. (b) The transformation allows to efficiently train networks on complex-shaped physical domains. Adapted with permission from [109], copyright (2020) Elsevier. (c) Data encoding and compression using a topology description based on low-frequency Fourier components, which allows data-efficient treatment of complex shapes, here for example a free-form metagrating. Adapted from [110], copyright (2020) Optical Society of America.

Fig. 5. Physics-informed neural networks (PINNs) for nano-optics. (a) PINN for solving the wave equation in the time domain. Adapted with permission from [116]. (b) Top: solving the Helmholtz equation (frequency domain); bottom: using the PINN for inverse design of the permittivity distribution in domain ${\mathrm{\Omega }}_1$ for an invisibility cloak application. Adapted with permission from [102], copyright (2020) IEEE.

Fig. 6. Examples of “knowledge discovery” through machine learning. (a) The feasibility of a physical response by a defined geometric model can be assessed by a dimensionality reduction through an autoencoder neural network and subsequent non-convex hull determination. Adapted from [121], copyright (2019) the authors. (b) Study of the impact of the number of bottleneck neurons $N$ (left spectra) as well as of nano-structure design variations on the activation of the bottleneck neurons (W1–W4 in case $N=4$, yellow neurons in the top right panel). This analysis allows to assess the physical importance of individual design parameters and reveals information about the complexity of the optical response. Adapted with permission from [117], copyright (2019) John Wiley and Sons. (c), (d) By mimicking the human approach of interpreting and modeling physical observations (c), a conditional encoder–decoder network (d) can be used to discover implicit physics concepts from data. Reprinted with permission from [120], copyright (2020) APS. (e) Exploiting the high speed of a physics predictor network permits a systematic analysis of the achievable phase and intensity variations in metasurface constituent design. Adapted from [122], copyright (2020) Optical Society of America.

Fig. 7. Examples of ML applications in experimental data interpretation. (a)–(c) ANN used to decode information from optical information storage via a spectral scattering analysis from sub-diffraction small nano-structures. (a) Each bit sequence is encoded by a specific geometry which is designed such that it possesses a unique scattering spectrum. (b) A neural network is trained on a large amount of spectra such that it learns to decode noisy spectra of formerly not seen structures. (c) Even if only few wavelengths are probed, the readout accuracy of the network is excellent. Adapted with permission from [130], copyright (2019) Springer Nature. (d), (e) Holographic anthrax spore classification via holography microscopy. A machine learning algorithm is trained on phase images of different spore species, as depicted in (d). The neural network is capable to classify five different anthrax species with a very high accuracy. Adapted from [131], copyright (2017) the authors. (f) Microscopy force field calibration (top left, green line: trapping potential; dots: reconstructed potential). Evaluation of $U(x)$ via ANN-based analysis of Brownian motion from undersampled statistical data (top right). Comparison of reconstruction fidelity of ANN (bottom left) and conventional method (bottom right). Ground truth is indicated by a black dashed line. Adapted from [134], copyright (2020) the authors. (g) ANN enabled real-time hyper-spectral image reconstruction from speckle patterns produced by a multi-core multi-mode fiber bundle (MCMMF). The technique exploits the wavelength dependence of the speckle patterns. Adapted from [152], copyright (2019) Optical Society of America. (h) Scheme depicting the use of machine learning for statistics reconstruction of few-shot data acquisitions. Reprinted from [160], with the permission of AIP Publishing.