1. Introduction

With the continuing development of science and technology, the requirements that are being placed on imaging optics are gradually increasing. Based on the premise of achieving improved imaging performance, imaging optical systems development is moving in the direction of systems with a large field-of-view (FOV), large apertures, small F-numbers (F#), low volume, and reduced numbers of optical elements to meet advanced application requirements. In addition, to satisfy the specific requirements of system structures while also avoiding light obscuration, nonrotationally symmetric system configurations are generally used. Traditional spherical and aspheric surfaces have low degrees of design freedom and it is difficult to achieve high-level design requirements while also correcting the unconventional aberrations induced by breaking of the rotational symmetry. To address these problems, freeform optical surfaces, which can be characterized as nonrotationally symmetric surfaces, have been introduced into optical design and have been used successfully in several areas. These surfaces can correct the aberrations that occurs in nonrotationally symmetric systems well and thus help to achieve better imaging performance, improved system parameters, more compact system configuration, and fewer elements [1–11]. The use of freeform optical surfaces in imaging systems can be regarded as a revolution in the optical design field [12].

The traditional design method for imaging optics is to begin by finding a system that is close to the design requirements to act as a starting point, and then perform subsequent optimization of that design. For the design of traditional rotationally symmetric systems, there are numerous design theories and considerable design experience exists, along with sufficient design examples for use as starting points. However, because freeform imaging system are generally designed to achieve advanced system parameters, nonrotationally symmetric system configurations, and/or special functions, appropriate starting points are very rare. If the design process begins from an inappropriate starting point, the designer may then need to devote extensive amounts of time and effort to achieving the final design or may even fail. In recent years, some new methods to generate the required starting points have been proposed, e.g., the partial differential equations (PDEs) method [13,14], the Simultaneous Multiple Surface (SMS) method [15], the design methods based on aberration theory [16,17], the Construction-Iteration (CI) [18] method and its related automated point-by-point design framework [19,20], and a deterministic direct design method based on differential equations derived from Fermat’s principle and then solved using power series [21]. The freeform surfaces in these systems are constructed or calculated based on given system parameters and configurations. Most of these methods offer a specific solution for a single system design task and specified system parameters, which can be regarded as a lack of ability to draw inferences. When new system parameters and configurations are input, the designer will then have to repeat the entire design process, which is a waste of the designer’s time and effort.

Deep neural network (DNN)-based deep learning (DL) has enabled important achievements in classification, recognition, prediction, and decision-making. DL also has been used widely in optics-related research areas including computational imaging [22,23], nanostructures and metasurface design [24,25], quantum optics [26], fiber optics [27], and laser optics [28]. There have also been some researches on the combination of DL with optical design. G. Côté et al. used DL to obtain lens design databases that provided high-quality starting points from various optical specifications using both supervised training and unsupervised training approaches [29]. For freeform optics, C. Gannon and R. Liang used an artificial neural network to generalize the relationships between performance parameters and lens shapes for freeform illumination design tasks [30]. In our previous work on freeform imaging optics, we demonstrated that neural network-based machine learning can be applied successfully to starting point design of freeform surface imaging systems [31]. However, the existing design framework is only suitable for a narrow range of system parameters and for freeform three-mirror systems, thus meaning that its applications are highly limited. If DNNs can be obtained for wider ranges of system parameters and different system types, they can then meet wider ranges of design requirements and applications. These DNNs can be also integrated into optical design software for ease of use.

In this paper, we propose a generalized framework to generate starting points for the design of freeform imaging optics based on DL. This framework can be used to design nonrotationally symmetric freeform refractive, reflective, and catadioptric systems with various applications, and can support a wide range of system parameters. For each given type of system design task, different system parameter combinations are selected within the allowable range, and the corresponding surface data are then obtained via a special system evolution method. A judgment and re-optimization/skipping mechanism is used to overcome problems of ray-tracing errors and bad local minima, thus leading to fully automatic dataset generation. Then, using this primary dataset, secondary dataset is generated in parallel using a K-nearest neighbor method. The full dataset consisting of the system and surface data is then used to train the DNN (or can be regarded as a multilayer perceptron (MLP)), followed by sufficient testing. When the DNN is trained, the surface data and the corresponding freeform system for a given system parameter combination can be output immediately as the starting point, which can then be used for further optimization. The feasibility of this design framework is demonstrated by using it to design three types of freeform optical system with advanced system parameters: an off-axis three-mirror imaging system, an off-axis four-mirror afocal telescope, and a prism system for use in near-eye displays. The design framework significantly reduces the time and effort required from the designer and also reduces their dependence on advanced design experience. Additionally, the design framework can be integrated into optical design software and cloud servers for the convenience of more designers.

2. Method

For a given type of freeform imaging system with a specific system configuration, the entire framework for generation of the starting points is shown in Fig. 1, and this framework can be divided into the following main steps. First, the representative system parameters (such as FOV, F#, and focal length) and the surface parameters (which can fully describe the location and shape information of surfaces) are determined. The system parameter space (SPS) can then be determined based on the design requirements and the range of system parameters involved. Next, the primary dataset is obtained: large numbers of system parameter combinations are sampled in the SPS, and the corresponding surface parameters are then obtained through a system evolution method. However, system evolution to generate large training datasets and thus obtain a good training result for the DNN is time-consuming. Therefore, the next step is to obtain the secondary dataset: system parameter combinations are selected at random in the SPS and a weighted K-nearest neighbor method is then used to obtain the corresponding surface parameters; these processes can be performed in parallel. The primary dataset and the secondary dataset are then integrated together as the final training dataset. The system parameters and the surface parameters are taken to be the input and output parts of the dataset, respectively, and the DNN is trained using this dataset. When the network has been tested and is found to be good, then, for a given system parameter combination in the SPS acting as the input, the DNN can output the corresponding surface parameters directly. The corresponding system can be regarded as a good starting point for further optimization. The following sections provide a detailed description of the proposed framework.

 

Fig. 1. Illustration of the generalized design framework

Download Full Size | PDF

2.1 Generation of the large training dataset

2.1.1 System parameters selection

One important step within the whole design framework is to find a series of systems with good imaging performance and obtain the input and output parts of the training dataset. The imaging performance of the starting points generated by the DNN depends heavily on the performance of the systems in the training dataset (both the primary and secondary datasets discussed in the following). Using optimization, each system in the dataset should have as good imaging performance as possible using given number of design variables and design steps, and can be actually used for imaging to some extent. Of course, the optimization should not be too careful considering the time cost for the dataset generation. Before this step, it is necessary to determine the representative parameters of the system. Common system parameters include effective focal length (EFL), the reduction ratio, the magnification (Γ), the back focal length, the F-number (F#), the numerical aperture, the FOV in the x and y directions, the entrance pupil diameter (ENPD), the exit pupil diameter, and the pupil location. However, for a specific type of imaging system, not all the above system parameters will be needed or even exist. In addition, certain parameters are linked together, which means that some of the system parameters can be calculated using other parameters. For example, for an imaging system for remote sensing, the required selection of key and independent system parameters can be EFL, FOV and ENPD; for an afocal telescope, this selection of key and independent system parameters can be FOV, ENPD, Γ. Additionally, to make it easy for the designers to use the framework, some system parameters are sometimes fixed. Therefore, in our design framework, only a set number of independent representative system parameters g₁, g₂…, g_t, …, g_T are selected, where T represents the total number of selected system parameters, and g_t represents the tth (1≤t ≤ T) system parameter. The output data in the training dataset are the surface parameters that correspond to the different system parameter combinations and contain the position and shape information for each surface. The position information for each surface can be described using the global decenter and tilt values relative to a predefined global coordinate system. With regard to the shape of the freeform surface, there are many different types of freeform surface, including the XY polynomial surface, the Zernike polynomial surface, the Q-type polynomial surface, and the radial basis function surface. The design framework proposed in this paper can work for different types of freeform surfaces. The differences between them do not affect the core of the whole design framework. Currently we do not find specific freeform surface types yielding superior starting points for the general cases. The performance of the starting points depends on both the dataset and the network (network complexity and the training process). When designing some specific kinds of imaging systems or to realizing specific functions, some specific surface types may lead to better results and should be used in the dataset generation. Choosing the best freeform surface type is a complex and systematic work and remains to be explored in future research. The surface coefficients can be used as the parameters to describe the surface shape. We use h₁, h₂,…, h_q,…, h_Q to denote the surface parameters for each system, where h_q represents the qth (1≤q ≤ Q) surface parameter, and Q represents the total number of different surface parameters.

2.1.2 Determination of the SPS

When the representative system parameters have been determined, the next step is to define the SPS. The SPS is a high-dimensional space that is defined based on the representative system parameters. The boundaries of the SPS are determined by the constraints that act between the system parameters, depending on their ranges and other actual design needs (see Section 3 for examples). If M is the total number of constraints, then the μth constraint can be written as

 

Fig. 2. An example of SPS determined by the representative system parameters and constraints between the system parameters.

Download Full Size | PDF

2.1.3 Generation of the primary dataset

After the appropriate SPS has been determined, the next step is to generate the dataset required to perform network training. This dataset can be divided into a primary dataset and a secondary dataset. For the primary dataset, a series of systems should be generated. Different combinations of the system parameter values SYSP_p,i = [g_1,i,…, g_t,i,…, g_T,i] (1≤i ≤ N_p) should be determined within the SPS to act as the input part of the dataset, where N_p represents the total number of system parameter combinations in the primary dataset. For each system parameter g_t (1≤t ≤ T), g_t,i represents the specific sampled value of g_t in the ith combination. Typical sampling methods used in this process include uniform sampling and random sampling, as illustrated in Fig. 3. For uniform sampling of the single parameter g_t, the values of g_t,i are sampled at equal intervals (but the intervals used for the different parameters can be different). The full range of combinations of the different values of g_t,i for different g_t can be obtained, but only the combinations that lie inside the SPS are selected. In the random sampling method, N_p system parameter combinations are randomly sampled in the SPS. For each SYSP_p,i, a corresponding base system called BaseSys_p,i with good imaging quality must be generated to obtain the surface data. Therefore, every system should be optimized carefully; otherwise, the optimization process will be difficult or could easily fall into local minima or provide other unexpected results.

 

Fig. 3. The primary dataset in the SPS. (a) The uniform sampling method. (b) The random sampling method.

Download Full Size | PDF

We use a special system evolution method to obtain the primary dataset. The main concept of this method is described as follows. An initial base system (corresponding to non-advanced system parameter combination that does not necessarily have to be BaseSys_p,1) must be generated first. This system can be designed by non-specialists or beginners of optical design. Designers can find a corresponding starting system from patents, or generate a starting system using some other methods such as PDEs method, SMS method, and CI method based on design requirements. Then optimization is conducted to get good imaging performance. Note that the basic constraints for optimization (used in primary and secondary dataset generation) should have been established properly to control the system parameters (such as focal length, F#, etc.), system structures (avoiding light obscuration and achieving specific requirements), aberrations (such as distortion) and so on. The remaining base systems are then generated in sequence (regardless of the number i). Each base system (not the initial base system) is optimized starting from the specific system that has the most similar system parameters to current base system among the base systems that have already been obtained. For base system BaseSys_p,i, if it is the ϖth system generated in the time sequence, the weighted distances of SYSP_p,i to N_p−ϖ remaining system parameter combinations are calculated. The weighted distance D between two system parameter combinations SYSP* and SYSP** can be calculated as ||υ⊗(SYSP* −SYSP**)||₂, where ||·||₂ is the 2-norm and υ=[υ₁,…, υ_t,…, υ_T] is the weight vector that determines the individual weight for each system parameter g_t (1≤t ≤ T). The T scalar weight elements should therefore be selected to approximately balance the contribution to the overall optical design difficulty that results from the same quantity of data change for each system parameter. ⊗ is the element-wise multiplication, i.e., an operation in which the vectors are multiplied element by element. Find the smallest weighted distance and the corresponding system parameter combination, and this corresponding system is the next one ((ϖ+1)th system in time sequence) to be optimized. This system should be evolved from the system which has the smallest weighted distance among the base systems that have been already obtained (may be not BaseSys_p,i). Repeat the above process until all base systems have been obtained. Then the surface data of all the systems can be obtained and they are taken as the output data of the dataset. See Ref. 31 for further details. Here, the surface data of BaseSys_p,i can be written as SURP_p,i = [h_1,i,…, h_q,i,…, h_Q,i] (1≤i ≤ N_p), where h_q,i represents the specific value of the surface parameter h_q (1≤q ≤ Q). The evolution approach above works well for generation of the starting points for a freeform three-mirror imaging system with a narrow range of system parameters [31]. However, for some other systems types, when the system parameters are advanced and the ranges of these system parameters are wide, ray tracing errors (or other unexpected fatal errors) may occur and/or the system may become stuck in a very bad local minima during system optimization. Because the base systems are generated in a sequence, the errors described above will lead to dataset generation failure. The system evolution method is therefore improved in this paper to address this problem. We begin by evaluating the imaging performance after optimization of each base system, starting from the “nearest” base system that has system parameters that are most similar to those of the current base system. If the current base system has fatal errors (e.g., a fatal ray tracing error) and/or poor imaging performance that is far from the design expectations or the normal performance level, then the current base system must be re-optimized, beginning with the specific system that has the next most similar set of system parameters to the current system one among the rest of the base systems that have already been obtained. Another approach is to skip and drop this bad system from the dataset generation process directly. In most cases, the bad systems may appear sporadically in entire range of the system parameter space, and will not appear continuously at some local regions in the system parameter space in most cases. Therefore, removing the bad systems will not introduce severe bias and will not affect the overall performance of the network in most cases. If bias actually exists, the surface parameters of the corresponding systems can be learned by the network using other nearby systems in the dataset with a high probability. The entire primary dataset generation process can be performed automatically.

2.1.4 Generation of the secondary dataset

Although the method presented in Section 2.1.3 can generate the dataset required for network training, the generation process is sequential and thus only one base system can be generated at a time. For training of a DNN corresponding to a wide range of system parameters, the dataset must be large enough to obtain good training results, generation of the dataset will be very time-consuming. Therefore, only part of the dataset (i.e., the primary dataset) should be generated using the method described above, and the remainder of the dataset (i.e., the secondary dataset) can be generated using the following K-nearest neighbor method. N_s different system parameter combinations SYSP_s,j (1≤j ≤ N_s) are generated randomly within the SPS. For each system parameter combination SYSP_s,j, the K different system parameter combinations denoted by ${\boldsymbol {SYSP}}_{\textrm{p},m}^{{\ast} ,j}$ (1≤m ≤ K) that are most similar to SYSP_s,j are selected from the primary dataset, as illustrated in Fig. 4. This means that the K most similar systems that have the smallest weighted distance (D_j,m) with respect to SYSP_s,j are selected. The D_j,m between SYSP_s,j and ${\boldsymbol {SYSP}}_{\textrm{p},m}^{{\ast} ,j}$ can be calculated as follows:

The weighted average of the surface parameters of K systems is calculated as the initial surface parameters ${\boldsymbol {SURP}}_{\textrm{s},j}^{\textrm{initial}}$ of the system corresponding to SYSP_s,j

 

Fig. 4. Generation of the secondary dataset. The yellow points represent the system parameter combinations in the primary dataset, the blue points represent the random parameters combinations, whose corresponding initial surface data are obtained through a K-nearest neighbor method (in this illustration K=3). The irregular polyhedron represents the SPS. (a) and (b) shows the uniform and random sampling method of primary dataset respectively.

Download Full Size | PDF

When the initial surface parameters ${\boldsymbol {SURP}}_{\textrm{s},j}^{\textrm{initial}}$ are obtained for all N_s systems, optimizations of these systems are then performed to obtain the final surface parameters SURP_s,j; these optimizations can be carried out in parallel to reduce the time cost. The optimization constraint types are the same as those used during generation of the primary dataset. The entire secondary dataset generation process can also be performed automatically. Then, any systems with fatal errors or very poor imaging performances are removed from the dataset. This secondary dataset and the previous primary dataset form the full training dataset (input part: SYSP_p,i (1≤i ≤ N_p) and SYSP_s,j (1≤j ≤ N_s), output part: SURP_p,i (1≤i ≤ N_p) and SURP_s,j (1≤j ≤ N_s)) for training of the DNN. Each system in these primary and secondary datasets is designed by starting from another system nearby. In addition, systems with very poor imaging performance are not included in the dataset in general. Therefore, the case in which the surface parameters are considerably different for systems with similar parameters can be avoided in general. Of course, this case may occur with low probability. However, a few such “large discontinuities” will not greatly affect the interpolation or training of the network. In addition, because we are focusing on the generation of starting points (and not the final design results) for the system design, the performances of systems generated using an imperfect network are also acceptable. In addition, designers can also add a third dataset to the training data, taking the primary and secondary dataset as the new “primary dataset” and apply the method given above. This will increase the number of different samples in the training dataset, which will then improve the performance of the trained DNN in general. The generalization ability of the DNN will increase.

2.2 Training and testing of the DNN

2.2.1 Architecture of the DNN

Neural networks can be simply understood as generalized function approximation tools. For any given training set, a neural network will attempt to generate a computational architecture that maps all the inputs to the outputs in the training set accurately [32]. The computational structure of a neural network is shown in Fig. 5. The goal of a neural network is to learn and implement mappings that relate an input vector to an output vector. A standard neural network is composed of multiple layers, which include the input layer, hidden layers, and the output layer. The network usually contains multiple hidden layers to enhance the performance of the entire neural network. Each layer is formed by specific activation nodes that are connected to the corresponding activation nodes of the preceding and succeeding layers. The number of activation nodes per layer constitutes the layer width. The number of hidden layers indicates the network depth. The number of nodes varies within the different layers. The DNN used in this paper is the same as a multilayer perceptron (MLP). In general, a more complex DNN has a stronger expressive ability. To provide more complex optical systems with more surface parameters and increased design difficulty, the numbers of layers and nodes of the corresponding network should also be increased accordingly.

 

Fig. 5. Simplified schematics of DNN.

Download Full Size | PDF

Each node in the layer expresses a type of nonlinear function that is typically the same for all nodes and is denoted by A(·). With the exception of the first layer, the inputs to the nodes are weighted sums of the outputs from the nodes in the previous layer. S^l_k is the output of the kth node in the lth (2≤l ≤ L) layer, which can be represented as:

2.2.2 Training the DNN

The dimensions of the different system or surface parameters will vary, and there may be large differences among the values of these different system or surface parameters. This may therefore affect the convergence of the training process. The purpose of preprocessing the training data is to scale the data values to within [−1,1] or another specific range. Common data preprocessing methods include min-max scaling, the extreme value method, z-score standardization, logarithm methods, and the inverse tangent method. In this paper, because the data in the dataset are distributed randomly or uniformly within the data range and these data do not follow a normal distribution in general, we use the simple extreme value method given in Eq. (6) (a linear preprocessing method similar to min-max scaling) to preprocess the training data. For example, for each surface parameter, the preprocessing method can be expressed as follows:

The training parameters, including the loss function, the optimization algorithm, and the learning rate, must be chosen carefully. The loss function is a function that reflects the DNN’s performance on a particular task. The training of the DNN is intended to reduce the loss function. Different types of loss function are used that depend on the specific problem to be handled. Using supervised training as an example, the loss function in this case is a function of the difference between the predicted values from the DNN and the network output values in the dataset. The selection of the optimization algorithm will affect the training and the convergence speed of the DNN significantly. The choice of the learning rate is another crucial factor. When the learning rate is too great, the loss function value of the DNN fluctuates greatly and even becomes difficult to converge. When the learning rate is too low, the loss function value of the DNN will oscillate in a relatively small manner in the training process, but it will greatly increase the training time required. At the beginning of DNN training, the weights and biases should be initialized randomly. Obviously, these initialized parameters will not yield good results. In the training process, underfitting and overfitting problems often occur. However, in the initial stage, the model is often underfitted, which is the reason why the optimization process is necessary. The values of the weights and biases are adjusted constantly during the training process. When the DNN is overfitted, common methods to address the problem include regularization, the dropout method, the early stopping method, and increasing the number of samples included in the training dataset.

2.2.3 Testing the performance of the DNN

After the DNN has been trained, it then becomes necessary to test the performance of the DNN. First, the DNN that has been trained must be restored, and then the test system parameter combinations, which are selected at random in the SPS, should be preprocessed in the same manner as the training data. The network then outputs the predicted results directly. Because the surface data in the dataset have been processed before the training process, the values that are output directly by the network then need to be reverse processed (i.e., the reverse process to the preprocessing method) into actual surface parameter values to generate an imaging system. For example, for the τth surface parameter, the reverse processing method corresponding to Eq. (6) can be expressed as follows:

3. Example demonstrations

In this section, we use three specific types of freeform imaging system with wide and advanced system parameter ranges (including reflective and refractive systems) to verify the feasibility of our proposed design framework.

3.1 Freeform off-axis three-mirror imaging system

The first type of freeform imaging system is a freeform off-axis three-mirror imaging system. For this system, we have selected half of the FOV in the x direction (half-XFOV), half of the FOV in the y direction (half-YFOV), and the ENPD as the parameters to be used to describe the system parameters. The focal length for each base system is set at 1 mm. In this way, the output surface data from the DNN can be used to form a system with about 1 mm focal length, and the starting points for other focal length values can be obtained via scaling. The system has a traditional Wetherell configuration [33]. The second surface is selected as the aperture stop. The freeform surface type for each surface is the XY polynomial surface and is used up to the sixth order with no base conic. Because the optical system is symmetric about the YOZ plane, only the even components of x are used:

Using or not using the latter three constraints does not affect the core of the whole design framework. For this design example, the dataset can be also generated automatically in a larger design space if these constraints are removed. The starting points for challenging designs can be generated using the proposed method. However, we add these constraints considering the actual design cases, as it is generally difficult to achieve very large FOV in x and y direction as well as large entrance pupil diameter (small F-number) simultaneously while maintaining excellent imaging performance for actual final designs (the focal length is much longer, not 1 mm). The FOV is given in units of degrees and the ENPD is given in mm. The system parameters have a wide range of values and can be comparatively advanced. For the primary dataset, different combinations of these system parameter values were generated using uniform sampling. When only the first three constraints are considered, the number of different half-XFOVs is 40, the number of different half-YFOVs is 40, and the number of different ENPDs is 15. After full combinations are determined, 8457 different system parameter combinations are selected inside the SPS. The parameters of the initial base system are [half-XFOV, half-YFOV, ENPD] = [1°, 1°, 0.125 mm]. This means that the system has the smallest FOV and ENPD values within the SPS, which can be designed easily using a variety of methods. Because the system is symmetric about the YOZ plane, only the field points in the half-FOV are considered during the design process. Some typical fields are sampled in the optimization process, including the central field, the edge fields, and some other fields. For the initial base system with the 1° half-XFOV and half-YFOV values, we select the following 10 fields: (0°, 0°), (0°, −1°), (0°, −1°), (1°, 0°), (1°, −1°), (1°, −1°), (1°,0.5°), (1°, −0.5°), (0.5°, −0.5°), and (0.5°, −0.5°). Similar field sampling methods are then applied to the other systems in the dataset.

The optimization process is conducted in optical design software CODE V. The default transverse ray aberration type error function is used. Step optimization mode is used. The derivative increments are computed using finite differences. The above optimization setups are the same for three design examples in this paper. The global α-tilt, global y-decenter, global z-decenter of three mirrors and image plane with respect to a global coordinate system (with the exceptions of the global y-decenter and global z-decenter of stop surface), as well as the freeform surface coefficients of three freeform mirrors are used as design variables. It is necessary to control the EFL, the usage areas for the freeform surfaces, the relative distortion, and the light obscuration. The EFLs of the system in the x and y directions can be calculated using the ABCD matrix and can then be controlled. The central region of the freeform surface is used for all mirrors, i.e., the intersection point of the chief ray of the central field (0°, 0°) with each freeform surface is controlled at the mathematical vertex. When the value of system parameters increases, the difficulty of distortion control gradually increases. The relative distortion of the five edge fields (including the fields with their maximum field angles in the x and/or y directions) is controlled within a certain range. In the design process, the maximum allowable relative distortions in the x and y directions for each system are determined using the following equations:

In addition, the system structure must be controlled. The five distances designated L₁ to L₅ that are marked in Fig. 6 must be controlled to eliminate light obscuration and prevent surface interference.

 

Fig. 6. Structural constraints for the off-axis three-mirror freeform system.

Download Full Size | PDF

The entire primary dataset formed from the system and surface parameters of N_p=8457 systems with good imaging performance is then obtained automatically using the method presented in Section 2.1.3. The time cost for generating the primary dataset is 5.37 h. The next step in the process is to generate the secondary dataset. The value of K in this case is 3. Of course, the value of K is not limited to 3 and other different values can be used. But it is not recommended to be small than 3, otherwise the initial surface parameters for the secondary dataset may be far from the final values and it may be not beneficial for the system optimization of the secondary dataset. In addition, the value of K should not be too great, as some systems “far from” the current system may be used to calculate the initial parameters, which has little contribution and no practical meaning. So, in the design examples given in Section 3, K=3 is used as a balance of the above two issues. The number of secondary systems is N_s=8414. In other words, for each randomly generated system parameter combination SYSP_s,j (1≤j≤8414), the weighted average of the three nearest neighbor systems’ surface parameters are used to obtain the initial surface parameters ${\boldsymbol {SURP}}_{\textrm{s},j}^{\textrm{initial}}$ of the system and a subsequent optimization is then performed. A larger number of N_s will increase the number of different samples in the training dataset, which will then improve the performance of the trained DNN in general. For the design examples given in the paper, the number of N_s is chosen as a demonstration and a balance of the network performance and the time cost of dataset generation. The numbers of N_p and N_s are not restricted to the values given in this paper. N_s can be increased to several times of N_p. The optimization process (including the optimization setup and constraints) of the systems in the secondary dataset is the same with the primary dataset. Finally, the secondary dataset is generated automatically and the entire training dataset, which comprises the system and surface data of 16871 base systems, is thus obtained.

When the dataset has been obtained, the next step is to train the DNN. For the design examples given in this paper, the number of surface parameters is large (including the decenters and tilts as well as the surface coefficients of multiple freeform surfaces) and the ranges of system parameters are wide. As a result, a complex network should be used. In this three-mirror design, the DNN has 12 hidden layers and uses a symmetric structure. This means that the hidden layers are symmetric about the central hidden layer. For example, the first hidden layer has the same number of nodes as the last hidden layer. The number of nodes in the first six hidden layers of the network decreases gradually, from 150 nodes in the 1st hidden layer to 100 nodes in the 6th hidden layer, with a reduction of 10 nodes in each successive layer. The numbers of nodes in the subsequent hidden layers increase from 100 nodes in the 7th hidden layer to 150 nodes in the 12th hidden layer, with 10 nodes being added to each successive layer. We have tried to train networks with less nodes and hidden layers. But the training results are not good (the loss is relatively large) and the systems generated by the networks are not good (the overall imaging performance is worse and many systems have light obscurations and ray tracing error). For the activation function, in our previous research [34], we conducted a network parameters exploration and comparison process for a training dataset of designing freeform three-mirror system starting points with small range of system parameters. A simpler network is used. In that example, we found that using ReLU as activation function will lead to inferior training result than using tanh function. Based on this result, we also use tanh as the activation function in this paper. However, the choice of activation function is not fixed and may vary with different design tasks. The mathematical expression for the tanh function is given as follows:

The Adam algorithm and the mean square error (MSE) are selected as the optimization algorithm and the error function, respectively. The value of learning rate is chosen to be 0.0006, which is a balance of training time and convergence ability of the training process. The total number of training epochs is 0.7 million. The final loss function value is 0.225, and the training time is approximately 14 h on a computer using an Intel Core i9-10900 K central processing unit @ 3.7 GHz (maximum: 5.3 GHz after Turbo Boost) and 128 GB of internal memory. Other designs are also produced on this computer.

After the DNN is trained, the performance of the DNN must be tested. The test system data are not used in network training and does not affect the updating of network parameters. In this paper, the number of systems in the dataset of the three examples are all around 15000 (see Sections 3.2 and 3.3), and we randomly generate 2000 system parameter combinations in the SPS to test the performance of the DNN for each example. In future work, we will choose a consistent percentage of the number of training systems for testing, e.g., 25%. The predicted surface data can be obtained immediately when the test system parameters are input into the DNN. The next step is to evaluate the performance of the systems that correspond to these output surface data. For each system, it is necessary to evaluate whether or not there are ray tracing errors and/or light obscuration problems, and then evaluate the system imaging performance. There are 28 systems with ray tracing errors and four systems with light obscuration among all the test systems (three out of four of the systems with obscuration in fact have low levels of light obscuration, and they can still be regarded as good starting points). Some typical systems predicted using the well-trained DNN in the test dataset are shown in Fig. 7. The average RMS spot diameter of the sample fields and the maximum absolute distortion among the sample fields for each test system without ray tracing errors, as well as the minimum weighted distances between each test system and all the systems in training dataset, are plotted in Fig. 8. The test system number (number on the abscissa axis) of the random test systems is rearranged according to the value of the minimum weighted distance (from small to large). Figures 9 shows the average RMS spot diameter and diameter of Airy disk versus the half-XFOV, half-YFOV as well as ENPD of each test system. Figure 10 shows the maximum relative distortion in x and y directions for the marginal field points. Most of the systems produced by the DNN can serve as good starting points for further optimization. When the design framework is used for a specific system design task, if another surface is added in the system, the time cost for dataset generation and network training may increase, as the design is more complex and more surface parameters are used. However, if the network has been obtained, the starting points can be also generated approximately immediately when the system parameters are input.

 

Fig. 7. Some of the typical freeform off-axis three-mirror systems predicted by the DNN. The target system parameters are listed below the system.

Download Full Size | PDF

 

Fig. 8. (a) The average RMS spot diameter of the sampled fields of each freeform three-mirror off-axis system. (b) The maximum absolute distortion among the sample fields of each system. The minimum weighted distances between each test system and all the systems in training dataset are plotted. The test system number (number on the abscissa axis) of the random test systems is rearranged according to the value of the minimum weighted distance (from small to large).

Download Full Size | PDF

 

Fig. 9. (a) The average RMS spot diameter of each three-mirror test system versus the half-XFOV, half-YFOV as well as ENPD. (b) The diameter of Airy disk (λ=587.6 nm). The size of the circle represents the value of ENPD (varying linearly based on the scale circles given in the inset) and the color represents the value of diameter.

Download Full Size | PDF

 

Fig. 10. (a) The maximum relative distortion in x direction for the marginal field points of each three-mirror test system versus the half-XFOV, half-YFOV and ENPD. (b) The maximum relative distortion in y direction. The size of the circle represents the value of ENPD (varying linearly based on the scale circles given in the inset) and the color represents the relative distortion.

Download Full Size | PDF

3.2 Freeform off-axis four-mirror afocal telescope system

The second freeform imaging system is the freeform off-axis four-mirror afocal telescope system, which has the configuration shown in Fig. 11. The aperture stop is located at the end of the system. The incoming rays from the object space and the outgoing rays from the aperture stop are oriented perpendicular to each other in the central field. We select the half-XFOV, the half-YFOV, and the magnification (Γ) of the FOV as the system parameters. The ENPD of each base system is set at 1 mm. In this way, starting points with other ENPD values can be obtained after scaling. The freeform surface type is the XY polynomial surface and ranges up to the sixth order. The surface parameters for each surface include the global α-tilt, the global y-decenter, the global z-decenter, and 14 surface coefficients. In this way, a total of 68 surface parameters are selected as the outputs for each system. The SPS is determined using the specific constraints that apply between the system parameters, as follows:

The latter two constraints mean that the three different system parameters should not all have large values simultaneously, based on consideration of the design difficulty of actual systems. When only the first three constraints are considered, the number of different half-XFOVs is 26, the number of different half-YFOVs is 26, and the number of different Γ is 15. After full combinations are determined, 7056 different system parameter combinations are selected inside the SPS. The parameters of the initial base system are [half-XFOV, half-YFOV, Γ] = [0.5°, 0.5°, 2]. Because the system is symmetric about the YOZ plane, only the field points in the half-FOV are considered in the design process.

 

Fig. 11. Structural constraints for freeform off-axis four-mirror afocal telescope system.

Download Full Size | PDF

In the optimization process, the global α-tilt, the global y-decenter, the global z-decenter of four mirrors with respect to a global coordinate system, and the freeform surface coefficients of four freeform mirrors are used as design variables. An ideal lens with a focal length of 0.5 mm is added at the end of the afocal system to aid in the design process (other methods, e.g., setting the system to the true afocal mode, are also feasible). The Γ of the system is controlled via strict control of the difference between the actual image height and the ideal image height of a paraxial field (e.g., the value of 0.01° used in this design) using real ray tracing data. The central region of the freeform surface is used for all mirrors. The eight distances designated L₁ to L₈ that are marked in Fig. 11 must be controlled to eliminate light obscuration and prevent surface interference. In addition, to control the system volume, the diagonal length of the system in the YOZ plane is also controlled. Furthermore, to reduce the difficulty of the system assembly process, the four mirrors are expected to be fabricated on a monolithic substrate using an advanced ultra-precision machining process [35]. To realize this fabrication process, the ratio of the system sizes in the z direction and the y direction is constrained to the range between 0.8 and 1.2, and the difference between the locations of the primary and tertiary mirrors in the z direction is also controlled. The relative distortions of the edge fields in the x and y directions are controlled within a specified range. In the design process, the maximum allowable relative distortions in the x and y directions for each system are determined using the following equations:

The entire primary dataset formed from the system and surface parameters of N_p=7056 systems with good imaging performance is then obtained automatically using the method described in Section 2.1.3. The time cost for generating the primary dataset is 5.63 h. The next step is to generate the entire secondary dataset. The optimization process (including the optimization setup and constraints) of the systems in the secondary dataset is the same with the primary dataset. The value of K is 3 in this case. The number of secondary systems is N_s=7314. The entire training dataset comprises the system and surface parameters from 14370 base systems.

In this design, the DNN has 15 hidden layers with a symmetric structure. The number of nodes changes from 170 in the 1st hidden layer to 100 in the 8th hidden layer, and back to 170 nodes in the final hidden layer. The number of training epochs is 0.7 million, the training time is approximately 20 h, and the final loss function is 0.77. To test the DNN, we generate 2000 system parameter combinations randomly in the SPS and then evaluate the performances of the systems corresponding to the output surface data. There are 14 systems with ray tracing errors and 71 systems showing light obscuration among all the test systems. The obscuration in 45 systems from the total of 71 systems with obscuration is low (≤0.1 mm) and can be eliminated through either minor optimization or a slight reduction of the ENPD. These systems can still be selected as good starting points. Some of the typical systems predicted by the well-trained DNN using the test dataset are shown in Fig. 12. The average RMS spot diameters of the sample fields and the maximum absolute distortion (where an ideal lens is added) among the sample fields for each test system without ray tracing errors, as well as the minimum weighted distances between each test system and all the systems in training dataset, are shown in Fig. 13. Figures 14 and 15 show the average RMS spot diameter, diameter of Airy disk, and the maximum relative distortion in x and y directions for the marginal field points of each test system versus the half-XFOV, half-YFOV as well as Γ. The systems with light obscuration generally demonstrate worse imaging performance. Most of the systems produced by the DNN can serve as good starting points for further optimization.

 

Fig. 12. Some typical freeform four-mirror off-axis afocal telescope systems predicted by the well-trained DNN in the test dataset. The target system parameters are listed below the system.

Download Full Size | PDF

 

Fig. 13. (a) The average RMS spot diameter of the sample fields of each freeform four-mirror off-axis afocal telescope system. (b) The maximum absolute distortion among the sample fields of each system. The minimum weighted distances between each test system and all the systems in training dataset are plotted. The test system number (number on the abscissa axis) of the random test systems is rearranged according to the value of the minimum weighted distance (from small to large).

Download Full Size | PDF

 

Fig. 14. (a) The average RMS spot diameter of each four-mirror test system versus the half-XFOV, half-YFOV as well as Γ. (b) The diameter of Airy disk (λ=587.6 nm). The size of the circle represents the value of Γ (varying linearly based on the scale circles given in the inset) and the color represents the value of diameter.

Download Full Size | PDF

 

Fig. 15. (a) The maximum relative distortion in x direction for the marginal field points of each four-mirror test system versus the half-XFOV, half-YFOV as well as Γ. (b) The maximum relative distortion in y direction. The size of the circle represents the value of Γ (varying linearly based on the scale circles given in the inset) and the color represents the relative distortion.

Download Full Size | PDF

3.3 Freeform prism system

The third system type is the freeform prism used for near-eye display applications [36]. As shown in Fig. 16, the ray emitted from the microdisplay (on the image plane) is initially refracted by surface 4. After two consecutive reflections by surfaces 3 and 2, the ray is then transmitted through surface 1 (the same surface with surface 3) and reaches the exit pupil of the system (the human eye). Surface 3 should thus satisfy the total internal reflection (TIR) condition for rays reflected by this surface. Surface 2 of the prism is coated as a half-mirror to enable an optical see-through capability. An auxiliary element attached to the prism can be used to maintain a clear and undistorted see-through view of a real-world scene [36]. We select half-XFOV, half-YFOV, and F# as the system parameters in this case. The exit pupil diameter is 4 mm. The freeform surface type is the XY polynomial surface and ranges up to the sixth order. The specific constraints of the SPS between the system parameters are given as follows:

The latter three constraints mean that the three different system parameters should not all have large values simultaneously, based on consideration of the design difficulty. The parameters for the initial base system are [half-XFOV, half-YFOV, F#] = [5°, 5°, 5]. During the optimization process, the global α-tilt, global y-decenter, global z-decenter of three freeform surfaces and image plane with respect to a global coordinate system as well as the freeform surface coefficients of three freeform surfaces are used as design variables. Reasonable eye clearance can be maintained by controlling the distance L₅ between the entrance pupil and surface 1. The TIR constraint at surface 3 is ensured by controlling the smallest angle of incidence for all rays on surface 3 to be greater than the critical angle. The central region of the freeform surface is used for surfaces 2, 3 and 4. In addition, to guarantee both a feasible system structure and a freeform surface shape, distances L₁ to L₄ and L₆, as marked in Fig. 16, and the surface slopes and local curvatures of surfaces 1, 3, and 4 should all be controlled. The maximum allowable relative distortions in the x and y directions are determined using the following equations:

 

Fig. 16. Structural constraints for freeform prism system.

Download Full Size | PDF

The entire primary dataset formed from N_p=7451 systems with good imaging performance is then obtained automatically using the method presented in Section 2.1.3. The time cost for generating the primary dataset is 17.97 h (the design is much more complex than the three-mirror and four-mirror examples). For the secondary dataset, the optimization process (including the optimization setup and constraints) of the systems in the secondary dataset is the same with the primary dataset. The value of K is 3 in this case. The number of secondary systems is N_s=8852. A DNN with a symmetric structure using 26 hidden layers is used. The number of nodes changes from 180 (in the 1st hidden layer) to 60 (13th hidden layer), and back to 180 nodes in the final hidden layer. The number of training epochs is 0.6 million and the learning rate is 0.0006. All other training parameters were the same as those in the DNN used for the freeform off-axis three-mirror system. Of the three system types described in Section 3, the numbers of network nodes and layers are highest for the prism system (with its large number of surface parameters and the highest design difficulty), and the numbers of network nodes and layers are lowest for the three-mirror system (with the smallest number of surface parameters and the lowest design difficulty). In addition, the specific numbers of layers and nodes for each network are also selected to achieve a balance between the network’s expressive ability and the training complexity. The overall network structure and the numbers of nodes and layers are not fixed. The process for selection of the best network parameters will be explored in future research. The training time was approximately 25 h. The final loss function obtained is 0.35. To test the performance of the DNN, we randomly generated 2000 system parameter combinations in the SPS and evaluated the resulting system performance. There were 83 systems with ray tracing errors and 12 systems showing light obscuration among all the test systems. Among the systems with ray tracing errors, many of these systems have correct structures and can be changed into normal systems using minor optimization processes. Some of the typical systems predicted by the well-trained DNN from the test dataset are shown in Fig. 17. The average RMS spot diameter of the sample fields and the maximum absolute distortion among the sample fields for each test system without ray tracing errors, as well as the minimum weighted distances between each test system and all the systems in training dataset, are shown in Fig. 18. Figures 19 and 20 show the average RMS spot diameter, diameter of Airy disk, and the maximum relative distortion in x and y directions for the marginal field points of each test system versus the half-XFOV, half-YFOV as well as F#. Most of the systems produced by the DNN in this case can serve as good starting points.

 

Fig. 17. Some typical freeform prism system predicted by the well-trained DNN in the test dataset. The target system parameters are listed below the system.

Download Full Size | PDF

 

Fig. 18. (a) The average RMS spot diameter of the sample fields of each freeform prism system. (b) The maximum absolute distortion among the sample fields of each system. The minimum weighted distances between each test system and all the systems in training dataset are plotted. The test system number (number on the abscissa axis) of the random test systems is rearranged according to the value of the minimum weighted distance (from small to large).

Download Full Size | PDF

 

Fig. 19. (a) The average RMS spot diameter of each prism test system versus the half-XFOV, half-YFOV as well as F#. (b) The diameter of Airy disk (λ=587.6 nm). The size of the circle represents the value of F# (varying linearly based on the scale circles given in the inset) and the color represents the value of diameter.

Download Full Size | PDF

 

Fig. 20. (a) The maximum relative distortion in x direction for the marginal field points of each prism test system versus the half-XFOV, half-YFOV as well as F#. (b) The maximum relative distortion in y direction. The size of the circle represents the value of F# (varying linearly based on the scale circles given in the inset) and the color represents the relative distortion.

Download Full Size | PDF

In this paper, the system parameter combinations used for testing are generated randomly within the SPS, and are all different from the system parameters used in the dataset for network training. As the design examples given in Section 3 show, good starting points (corresponding to random testing system parameters) can be generated by the network in most cases, which prove that the network generalizes. New systems are “learned” from the dataset under the design framework, and are not “memorized” as they do not exist in the training dataset.

Table 1 shows the imaging performances of the systems on the primary datasets, secondary datasets, and test datasets (without the systems with fatal ray tracing error) for the three system types. As the table shows, the imaging performances of the systems on the primary dataset and the secondary dataset are similar because the optimization processes of all these systems are similar (the difference is that the initial system used for each single system design process is obtained using different methods, but this is not the predominant factor). It should also be noted that the systems with very poor imaging performances or fatal errors are not included in the dataset in general. All systems in the dataset can thus be seen as “good” solutions. In addition, all systems in the dataset have the potential to improve their imaging performances (especially for the prism system). However, this would require longer optimization times and may also require more variables to be included. Longer optimization times would increase the time cost for dataset generation. Addition of further variables would increase the complexity of both the network and the training process. Therefore, the current approach shows balance between system performance and time cost/design complexity. In addition, because we are focusing on the generation of starting points for system design (and not on the final design results), the systems in the dataset are “good enough” for use in the starting point generation framework. The systems in the dataset have been optimized and have good performance in general. The performance of starting points output by the DNN will generally have lower performance (some systems may be closer to the final design, and some may be far from the final design). The systems predicted by the DNN cannot be used for imaging directly in general. For most cases, the systems output by DNN can serve as good starting points for further optimization. If the starting points output by the DNN are further optimized in optical design software, the imaging performance of the final designs will at least reach the performance of the systems in the dataset. Of course, the systems after optimization have the potential to be better than the systems in the dataset, if more variables are further used and more careful/longer optimization is conducted. For the three examples given in Section 3, the cases are similar. It should be noted that more complex networks are used for the more difficult and complex design tasks (such as prism system), in order to maintain the performance of the output starting points. Furthermore, as has been demonstrated in the design examples in Section 3, limited by the network training process, the generalization ability of the DNN and imperfect training dataset, a few systems output by the network may have fatal ray tracing error or structure error. Therefore, currently it is not guaranteed that the DNN can output a good result every time after system parameters are input. In future work, we will focus on improving the stability of the design framework.

Table 1. Imaging performance of the systems

View Table

The design framework proposed in this paper is a generalized framework for generating starting points. The framework can be used for highly nonrotationally symmetric freeform refractive, reflective and catadioptric systems. The system parameters can be advanced and the range of system parameters can be wide. For the partial differential equations (PDEs) method [13], only a single field point is considered in the design process of one or two surfaces for most cases and good imaging performance of this field point can be obtained (better than the method proposed in our paper). However, this does not meet the imaging system design requirement that using rays from multiple fields and different pupil coordinates. The SMS method [15] can generate freeform starting points considering the rays from multiple fields and different pupil coordinates. Almost ideal imaging performance can be achieved for the sample field points used in the design, which is better than the method proposed in our paper. However, the number of field points and system structure considered in the design process are often limited. In this aspect, the method proposed in our paper is much more generalized. The CI method [18] can generate freeform starting points considering the rays from multiple fields and different pupil coordinates. The imaging performance of the resulting systems may be superior, similar or inferior to the systems generated by our proposed method depending on the specific system parameters and system structure. The deterministic design method given in Ref. [21] may achieve better performance than our method in general. The design methods based on nodal aberration theory proposed in [17] uses spherical surfaces in the early design stages and then the freeform surface terms are added and optimized step by step. As a result, the imaging performance of the resulting systems at different design stages may be superior, similar or inferior to the systems generated by our proposed method.

For the time spent of generating the starting points, all other methods will take about tens of seconds or several minutes (maybe much longer depending on the complexity of the design tasks). This will be faster than the whole design framework proposed in this paper as it takes more time to generate the dataset and train the network. But the dataset generation and training processes can be conducted automatically and nonstop on computers or cloud servers, and the networks can be prepared before actual use. If the networks are ready, our method will be much faster than any other methods to the best of the authors’ knowledge, as the starting points generation process of the proposed framework can be done almost immediately (about 0.5s for a system in this paper). Therefore, we can just store the networks as databases in servers or in optical design software for easy and fast use. The networks can be also shared, exchanged and improved in the optical design community.

4. Conclusions and discussion

A generalized framework is proposed for generation of starting points for the design of freeform imaging optics. This DL-based framework can be used to design nonrotationally symmetric freeform refractive and reflective systems for various applications using a wide range of system parameters. For each given type of system design task, different system parameter combinations are selected within an allowable range, and the corresponding surface data are then obtained through a system evolution method. A judgment and re-optimization/skipping mechanism is used to overcome the problems of ray-tracing errors and bad local minima, which leads to fully automatic dataset generation. A secondary dataset is used and is generated using a K-nearest neighbor method in a parallel manner to reduce the process time cost. A DNN is then trained using the dataset that combines the primary and secondary parts. When the DNN is fully trained, both the surface data and the corresponding freeform system can be output immediately for a given system parameter combination to provide a starting point that can be used for further optimization. The design framework significantly reduces the designer’s time and effort and their dependence on advanced design experience. Even beginners in optical design can use this work easily to handle freeform imaging system design tasks. The design framework and the neural networks can also be integrated into optical design software and cloud servers for convenient use by designers. The dataset generation and network training process can be run nonstop on these servers to generate complex networks for different system types for use in various applications.

The examples given in this paper are systems that have plane-symmetry with the YOZ plane (which is the most common case for freeform systems). For systems with no plane symmetry, the surface data in the dataset should include the surface coefficients with odd powers of x in the polynomial surface types (or coefficients that represent the non-plane-symmetry for other freeform surface types). The number of nodes in the final layer should also be increased. All the above differences only add complexity to the proposed method, but the core idea remains unchanged.

The current design framework proposed in this paper still has some limitations and can be improved further using other techniques. For example, the time costs for dataset generation and network training can be reduced further using advanced programming skills, system design and optimization methods, network structures, and training algorithms. Nodal aberration theory can also be used to guide rapid design of the freeform systems in the dataset using an analytical manner. In addition, the systems in the dataset and those output by the network may be not the best structure considering surface locations and power distributions for the corresponding system parameters (for example, a positive primary mirror in the small FOV three-mirror system may result in better systems compared to the results output by the DNN in Section 3.1), and one network only works for a single folding geometry. For the current design framework, new dataset and new DNN should be generated when the number of elements changes or the design constraints change significantly. It should be noted that minor changes in the design constraints will generally not lead to rerun the dataset generation and network training. For example, the original systems in the dataset may be not exactly or strictly limited by the structure constraints and some allowances exist. If the structure constraints are changed a little in the design requirements, new constraints may be still satisfied. If the new constraints are not completely satisfied and small obscurations or other structure problems exist in the starting points output by the DNN, they can be eliminated through minor optimization. Future work will focus on the design framework to enable exploration and generation of systems with different or best system structures and folding geometries by combining more complex training data set (having different structures, folding geometries and number of elements), the generative adversarial network (GAN), reinforcement learning design framework and transfer learning. The proposed design framework also has major potential for use in starting point generation for the design of imaging systems using phase elements such as metasurface elements and holographic optical elements (HOEs).