Open Access
Add to BibSonomy
Abstract
There is a trend in optical system design toward explicitly considering real-world industrial demands in the metrics to be optimized, from which emerges a cost-performance trade-off. Another relevant recent tendency is the so-called end-to-end design, where the design metric is an expected quality index of the final image, after digital restoration. We propose an integrated approach for analyzing the cost-performance trade-off in end-to-end designs. We exemplify it with a simple optical model where the cost is determined by the inclusion of an aspherical surface. We show that the resulting optimal trade-off configurations when applying an end-to-end design are substantially different from a conventional design. Such differences, as well as the increase in performance, are especially significant for lower-cost configurations.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Traditional optical system design is based on optimizing design parameters with respect to a quality metric without explicitly considering a priori real-world industrial demands, such as manufacturing, optical inspection, assembly limitations, and their economic costs. However, currently, there is a trend toward explicitly considering those industrial aspects in the design metrics [1–4]. The crux of this realistic design strategy is to take into account, in the design process itself, the impact of the differences between the designed system and the final as-built system on the performance, and their economical cost. These discrepancies may be due to tolerance and uncertainties in manufacturing, assembling, and inspection. Achieving a desired final image performance and meeting the industrial requirements with a reduced cost are competing goals, hence posing a design trade-off.
A conventional approach to tackle this trade-off is the so-called tolerance analysis, which is typically performed a posteriori of the quality metric optimization. It involves an iterative process with feedback refinements provided by the tolerance analysis, which involves a serious computational workload. A more efficient, purely feed-forward, alternative consists in setting a cost-performance trade-off right at the early stages of the design process (see [3] and references there). Because the extent of the discrepancy between nominal and final design values is known to be correlated with certain features of the optical elements, a single cost index summarizing those features can be used as a penalty term during the system design. This offers a more convenient a priori integrated design strategy.
The inclusion of aspherical surfaces, in an optical system, is an archetypal example for illustrating the cost-performance trade-off in the design of imaging systems. Although in many applications asphericity is essential to reduce dimensions and increase the nominal optical quality performance, departing from spherical surfaces has some counterparts. First, surface inspection uncertainties in aspherical surfaces are higher than in spherical ones. Second, aspherical surfaces are more sensitive to optical aberrations caused by manufacturing, polishing, and assembly errors, as compared to systems comprising only spherical surfaces; a typical example is that of camera mobile system design [5]. Thus, it is not surprising that some recent proposals for considering cost-performance trade-offs involve asphericity. In particular, the so-called slope departure (difference in slopes between an aspherical surface and a reference sphere) has been used as a cost index. Recently, Ma et al. have demonstrated, particularly in lithographic [2] and mobile camera lenses [6], that constraining the slope departure in the optical design process leads to final geometries more robust against non-axis-symmetric aberrations, caused by tilt and off-centering introduced during system assembly.
So far we have only discussed conventional optical design. However, nowadays many optical systems make use of digital image post-processing of the sensor image for improving the final image quality. The conventional way to apply image processing in digital imaging systems is sequential, i.e., optics are designed without considering any digital post-processing, and subsequently, a digital processing step is included in the camera pipeline. However, the ability of this sequential design scheme for optimizing the final image quality is quite limited. In order to go beyond it, some integrated digital-optical approaches have been proposed in recent years that jointly consider the optical and the digital processing steps [7–14]; a strategy that has been termed end-to-end design. The basic idea is to search for the physical parameters that optimize the expected value of an image quality index after digital restoration, instead of merely optimizing an optical quality metric on the sensor plane. This novel approach seems especially suited for highly constrained optical designs, with non-negligible aberration levels in the final design. Additionally, a plethora of deep-learning techniques may be used for optimizing more elaborated metrics [9,12].
In this work, we provide a method for studying and optimizing the cost-performance trade-off in one aspherical surface end-to-end design (however, the underlying concepts can be applied to systems comprising several aspherical surfaces, as discussed in section 5). This allows us to show, through simulations, how an end-to-end imaging system design approach may significantly boost the aforementioned trade-off as compared with a conventional procedure.
We illustrate our proposal with a simple model: a thin lens comprising an anterior spherical surface and a posterior conic one [15]. Contrary to a previous work of ours where only on-axis quality was taken into account [15], here we consider the complete third-order monochromatic aberrations of the thin lens model. This provides a rich scenario for studying the cost-performance trade-off. In addition, using such a simple model allows us to deal with just two degrees of freedom in the design (namely, the lens shape factor and the conic constant), which, in turn, allows for an exhaustive evaluation of all studied configurations. Furthermore, avoiding design complexity helps us to better illustrate the involved concepts and results. It is also worth mentioning that singlet camera designs have themselves become a research hot-spot (see [12,14] and references there).
This article is organized as follows. In section 2, we explain the optical and image model used, the design scenarios with their associated quality metrics and cost, and the cost-performance trade-off. The parameters of the optical system, including the sensor and the noise-related aspects, are described in section 3. Finally, the results of the simulations and their discussion are presented at sections 4 and 5, respectively.
2. Methods
2.1 Optics and observation model
We considered a thin lens comprising a spherical anterior surface and a posterior conic surface (conic constant $Q$) with a focal length denoted by $f$. We analyzed different possible shape factors of the lens. The shape factor is defined by:
being $r_1$ and $r_2$ the radii of curvature of the lens. For convenience, we chose half the angle of view ($\alpha$) as the object-dependent variable. Then, it is not difficult to prove that the ray coordinates at the image plane is $h(\alpha )=\frac {2f}{p-1} tan(\alpha )$, where $p$ is the so-called position factor, i.e., $p = 1 - \frac {2f}{t}$, being $t$ the distance from the lens posterior surface to the image plane [16]. Using this relation, the third-order wave aberration function can be written as:
Fig. 1. (a) Scheme the geometrical parameters of the thin lens model. (b) Different lenses for different values of the shape factor: $s=-2$, $s=-1$, $s=0$, and $s=1$.
From the wave aberration function of Eq. (2), we computed the PSF as a function of $\alpha$ through Fourier optics applying the Fraunhofer diffraction approximation. The spatially variant PSF field, i.e., the PSF as a function of its reference $x_0$ and $y_0$ coordinates at the image plane, can be described mathematically as a matrix ${\mathbf {H}}$. Each column of ${\mathbf {H}}$ is a vector-reshaped version of the $PSF(x_0,y_0)$, which corresponds to each discrete location in the image plane. $PSF(x_0,y_0)$ is easily obtained from $PSF(\alpha )$ by simply shifting to $(x_0,y_0)$ and rotating the latter. Then, we can write our observation model describing the discrete image at the output of the sensor as ${\mathbf {y}} = {\mathbf {H}} {\mathbf {x}} + {\mathbf {w}}$, where the vector-reshaped ${\mathbf {x}}$ represents the ideal (paraxial and noise-free) image, and ${\mathbf {w}}$ represents the noise. Under favorable conditions, the noise is predominantly due to photon counting (Poisson statistics). As usual, we have assumed it to be white (spectrally flat) and uncorrelated to the ideal image.
2.2 Design scenarios, quality metrics, and cost
We have considered three design scenarios: (1) a conventional optical design, i.e., optical design parameters are chosen to minimize a purely optical metric; (2) a design aimed at minimizing an error metric on the digital image registered by the sensor; (3) an end-to-end design, in which optical parameters are chosen to minimize an error metric on a digitally restored version of the sensor’s image [7–13]. For the conventional optical design, we minimized the commonly used root-mean-square of the wavefront aberration function ($RMS_w$), obtained from Eq. (2). For characterizing the image quality, we have used the expected mean square error, either on the sensor image ($MSE_s$) or at the output (restored) image ($MSE_r$). For the last design scenario, we made the same assumptions as in [15] that allowed us to estimate the expected mean square error at the output image. Namely, (1) an optimally tuned local linear (Wiener) filter is applied to the sensor image; (2) the noise relative level is known; and (3) ideal images obey a given power spectral model. For the cost, we chose a metric explained hereafter. Let’s define the profile of the asphere as $z(\rho )$. The slope of $z$ with respect of $\rho$ is $z'(\rho ) \equiv \tan (\alpha )$. First, we compute the slopes of the conic surface in a set of equally spaced $n$ points in the variable $\rho$ (black dashed lines in Fig. 2). Second, we fit a sphere whose slopes ($\tan (\beta _{i})$ in Fig. 2) minimize the root mean square of the slope departure defined by:
Fig. 2. Scheme showing the conic profile (black line) and best-fitted sphere (red line) in terms of slope departure.
This metric essentially fits the sphere of least slope departure from the conic and measures such departure. This measure is especially suited for being optimized using the so-called Forbes polynomials [1,17].
2.3 Computing the expected image quality index
We computed $MSE_s$ and $MSE_r$ by assuming a certain statistical model for the uncorrupted image (described below), and a certain noise level; the role of the noise and its features are discussed in section 3.2. For the $MSE_r$ case, we assumed a mean-square optimal linear filter (Wiener restoration) [8,15], in a spatially variant version (see, e.g., [18]), applied to the sensor image. Note that, in a real optical camera embedding digital restoration, it will be preferable to use a state-of-the-art non-linear restoration method instead of a linear one (see, e.g., [18,19]). However, for designing purposes, assuming a linear restoration has the key advantage over non-linear models of providing closed-form expressions of the expected mean square error [7]. For this purpose, although our image formation model is spatially variant, we may nevertheless apply a convolutional (spatially invariant) local model for the PSF of each image location. This simplification allows us to compute the expected MSE at each location of the image from the corresponding optical transfer functions (obtained as the Fourier transform of the local PSFs), and the power spectral densities (PSDs) of the ideal image and the noise [15].
We denote $H_{\alpha }({\mathbf {f}})$ the OTF for each angle $\alpha$, where ${\mathbf {f}}$ denotes the spatial frequency vector. Denoting $P_x({\mathbf {f}})=\mathbb {E} \{|X({\mathbf {f}})|^2\}$ and $P_w({\mathbf {f}})=\mathbb {E} \{|W({\mathbf {f}})|^2\}$ the PSDs of the ideal image and noise, respectively, the corresponding Wiener filter that tries to compensate for the effect of the PSF blur and the noise is [20]:
Then, the PSD of the error on the sensor image is: $P^{\alpha }_{e-s}({\mathbf {f}}) = \mathbb {E} \{|Y_\alpha ({\mathbf {f}}) - X({\mathbf {f}})|^2\}$, where $Y_\alpha ({\mathbf {f}}) = H_{\alpha }({\mathbf {f}})X({\mathbf {f}}) + W({\mathbf {f}})$ is the blurred and noisy sensor image in the Fourier domain, for a given $\alpha$. And the PSD of the error at the output of the Wiener filter is: $P^{\alpha }_{e-r}({\mathbf {f}}) = \mathbb {E} \{|H^{\alpha }_W({\mathbf {f}})Y_\alpha ({\mathbf {f}}) - X({\mathbf {f}})|^2\}$. Operating we obtain [8,15]:
2.4 Cost-performance trade-off analysis
The general procedure consists of two major sequential stages. First, an evaluation of the involved quality metrics $RMS_W$, $MSE_S$, $MSE_R$, and the cost index $C_a$, as functions of our two design parameters $(Q,s)$. Second, the computation of the optimal cost-performance trade-off solutions for each design scenario. The optimal cost-performance trade-off solutions are the curves, in the design parameters space ($Q,s$), providing optimal quality for a given cost, or, equivalently, minimal cost for a given quality. We have computed these curves by a two-steps procedure: (1) obtaining a family of iso-level $C_a$ cost curves in the design parameter space ($Q,s$) at regular increments covering the intervals of interest; (2) for each iso-level curve, we find the optimal solution, according to the design strategy (minimal $RMS_W$, $MSE_S$, or $MSE_R$).
3. Simulations
3.1 Parameters
In our simulations, we chose a lens of focal length $f = 25$ mm, a pupil radius of 2 mm, and a half field-of-view of $14^{o}$. The punctual sources are located at infinity, hence the position factor is $p = -1$. The physical parameters of our thin lens were: 1) refractive index $n= 1.5$ for a wavelength $\lambda = 0.5$ $\mu m$; 2) $Q$ ranging from −10 to 1.2, in 0.1 steps; 3) shape factor $s$ ranging from −2 to 1.025, in 0.025 steps. The field-of-view was sampled in $1^{o}$ steps.
Overall, we computed 13,786 lens configurations and 206,790 PSFs. Finally, for the cost-performance trade-off analysis, we have swept the cost $C_a$ from 0 to $2\times 10^{-3}$, in 30 uniformly distributed values.
3.2 Sensor grid and noise
To make our analysis as simple as possible, we used a detector size equal to $\Delta = \lambda f \# / 2$, ($f \#$: f-number), i.e., the size used for computing the PSF from the wave aberration function via Fourier transform. This is the largest value ensuring an aliasing-free PSF representation. For our optical parameters (see above), we obtained $\Delta = 1.5625$ $\mu m$, a relatively small detector size, but not uncommon nowadays (see, e.g., [22]). Taking as a guide the practical analysis in [23] based on real sensors, we have estimated a full-well capacity ($FWC$) of 3,200 for this detector’s size, which corresponds to a likely linear SNR (signal-to-noise ratio) of 24 ($\approx 27.6$ dB), for an average charge of 18% of the FWC [23]. In addition, for studying the sensitivity of the design results to the relative amount of noise, we have also considered a poor illumination/exposure scenario (400 ISO, only 4.5% of detectors’ FWC, i.e., SNR $\approx 21.6$ dB), as well as a higher SNR scenario (virtual 25 ISO, SNR $\approx 33.6$ dB). These SNRs are per pixel. For typical distributions of luminance on the sensor (with a standard deviation around 20% of the peak value) the average SNR for the whole image increases by approximately 3.5 dB.
3.3 Experiments
We evaluate the dependence of the industrial cost index $C_a$ on the design parameters by means of a contour graph of the cost landscape, as shown in Fig. 3. After that, we computed the three targeted quality metrics ($RMS_W$, $MSE_S$, and $MSE_R$) on the same range of design parameters and represented them in Fig. 4. Based on the previous data, and following the procedure explained in subsection 2.4, we computed the optimal cost-performance trade-off curves for each quality metric, always within the same intervals of the lens parameters (see Fig. 5). Note that for the end-to-end design case, we have considered three different signal-to-noise ratios (see subsection 3.2). Then, we computed the final performance (after Wiener restoration under the expected SNR) for the previous optimal curves and plotted the corresponding cost-performance trade-off signatures (see Fig. 6).
Fig. 3. Contour plot of the industrial cost ($C_a$) as a function of conicity and shape factor of the thin lens model of subsection 2.1. White lines are iso-cost contour lines.
Fig. 4. Contour plots of the three considered quality metrics as a function of conicity and shape factor. From left to right: Wave aberration RMS ($RMS_W$); Sensor MSE ($MSE_S$); End-to-end MSE ($MSE_R$). The two latter for the expected (medium) noise level. Thin white lines represent contour lines of equal quality.
Fig. 5. Lens configurations providing optimal cost-performance trade-off according to two quality criteria: $MSE_S$ (red crosses) and $MSE_R$, the latter under three noise levels: low (upward magenta triangles), medium (blue squares), and high (downward green triangles). Dotted curves are fitted polynomials in the two variables. Encircled are the absolute best-quality solutions for each design ($MSE_R$ under medium noise).
Fig. 6. $RMS_W$ and $MSE_S$ (blue curves, left vertical axis) and expected output MSE on the local-Wiener-restored image (orange curves, right vertical axis) as a function of $C_a$. Solid lines denote designs optimizing $RMS_W$ and $MSE_S$. Dashed orange lines denote $MSE_R$ (end-to-end design). Absolute conicity was upper-bounded to $10$.
As pointed out in section 1, digital cameras inherently contain the potential for achieving more favorable cost-performance trade-offs by substituting the sequential optical-digital design (which includes digital post-processing) with an end-to-end digital-optical design. We quantified this potential by following the procedure described in subsection 2.4, with the parameters specified in subsection 3.1 and with the noise levels described in subsection 3.2.
To validate and evaluate the actual impact of our design approach on the final results, we have included a set of simulations of restored images. Given the size of our virtual detector and the field-of-view of our optical model, we have chosen real photographs spanning the closest nominal 14$^{\circ }$ half field-of-view; particularly, two (a portrait and an outdoor scene) 24 MP $6000\times 4000$ photographs (obtained with a Canon EOS R8) from Chris Niccolls [24], a format that, when applied to our virtual camera provides a field-of-view of 12.7$^{\circ }$. To make these images closer to ideal, we first applied a gentle denoising [25], ensuring no significant loss of high spatial frequencies. Then, these images were converted from gamma-corrected into the linear space, where both the optical blur and Poisson models hold. For simulating the optically-blurred sensor images we performed local convolutions in $512\times 512$ overlapping windows. These were shifted 32 pixels each time; the central $64\times 64$ regions were linearly averaged in their overlapping areas. We have tested a very high accuracy level for this simulation procedure: over 70 dB SNR with respect to exactly processing a single pixel at a time. Noise is then produced by simulating a Poisson process on the blurred image. We obtained the three desired SNR levels by first scaling the blurred images to the maximum value of the estimated full-well capacity of our detector (3,200), and divide/multiply this value times 4, for the low/high light scenarios (400 and 25 equivalent ISO, respectively). For the spatially variant restoration, we applied an adaptive Wiener local deconvolution scheme, similar to that in [13], but keeping the same PSD image model of $P_x(\mathbf {f})$, and for which not only the PSF changes for one image location to another, but also the local signal and noise variance. Finally, we converted back the image to the original (non-linear) space.
4. Results
Figure 3 reveals that, in general terms, cost increases for rising values of the modulus of the conic constant and more negative values of the shape factor. In addition, our lens model implies that there is one zero-cost contour line within the cost landscape. Specifically, this curve is obtained when the surface is spherical ($Q=0$) or flat ($s = 1$).
When considering the dependence of the quality metrics on the design parameters, generally speaking, the digital performance landscapes are strikingly different from the pure optical ones. Whereas the optical performance keeps improving when going to larger conicity absolute values (with optimal shape factors close to −0.75), the end-to-end error index achieves its lowest values for moderate absolute conicity, but larger absolute shape factors (see Fig. 4).
The cost-performance trade-offs are analyzed in Fig. 5 and Fig. 6. Figure 5 shows some curves over a contour map of the cost. Thin colored lines represent iso-cost curves. Within each iso-cost curve, we computed the optimal performance point for $MSE_S$ (depicted with red crosses) and for $MSE_R$ with three noise levels: high (downwards green triangles), low (upwards purple triangles) and medium (blue squares). In this figure, we did not include the $RMS_W$ optima because they occur for extremely high absolute conicity values, so including them would have obscured the visualization of the other plotted points. The most relevant point to remark on Fig. 5 is the difference in the optimal points of $MSE_S$ and $MSE_R$. In particular, these differences are larger for the lower-cost contour lines. Implicit in this figure is the huge difference in the optimal configurations for the case of minimizing the pure optical metric ($RMS_W$), outside the showed range. This figure also reveals the optimized trade-off curves for the three noise levels (magenta, blue and green colors) do not substantially differ.
Figure 6 shows the performance indices as a function of the cost index, using the optimized trade-off curves previously shown in Fig. 5. Here we compare also the end-to-end design MSE ($MSE_R$, in dashed lines) with the MSE at the output (continuous dark orange lines) for the conventional designs, for the case of minimizing the $RMS_W$ (left panel Fig. 6) and for minimizing the $MSE_S$ (right panel Fig. 6). The MSE improvement is moderate when optimizing $MSE_S$ ($\approx 7\%$), but much stronger when optimizing $RMS_W$ ($\approx 37\%$). In addition, the cost indices for the optimal-quality sequential designs increased very strongly, as shown in Table 1.
Table 1. Expected relative MSE at the camera output and the industrial cost index $C_a$ for the three studied designs, each optimizing its own quality criterion.
With respect to simulating realistic examples of the camera output, and evaluating their image quality, Fig. 7 shows the two original (ideal) images and the simulated output of our camera model for our low-cost end-to-end design, assuming a typical noise level (100 ISO). For facilitating a visual evaluation, Fig. 8 shows high-resolution crops of the original and restored images for six $128\times 128$-pixel locations. Figure 8 also includes the (noiseless) simulated sensor images, both for the optimal cost-performance $RMSE_W$ and $RMSE_R$ designs. Restoration results include the most favorable noise scenario (25 ISO equivalent) for the $RMS_W$ design, and also the three noise scenarios (400, 100, and 25 ISO equivalent) for the $MSE_R$ end-to-end design ($MSE_S$-optimized results, less than 1 dB worse than the end-to-end results, were omitted here for brevity). From the previous results, we observe a general agreement with the model predictions: the end-to-end design performance is neatly superior to the optical aberration-based one and slightly superior to the sensor-based design. We emphasize that the visual quality using the end-to-end design is good in our examples (in objective terms too: between 27 and 32 dB PSNR), even when zooming in on small details, especially for incidence angles lower than $5^o$ (FOV $\leq 10$) and medium-low noise levels.
Fig. 7. Top: Ideal (gently denoised originals) 24 MP images; Bottom: results of the end-to-end designs (100 ISO). Original photographs by Chris Niccolls [24]. Differences are only distinguishable using a strong zoom (see cropped details in Fig. 8)
Fig. 8. First three rows correspond to crops from the portrait image and the rest from the outdoors image. Columns from left to right: (1) ideal; (2) simulated sensor image when minimizing $RMS_W$ ($s=-0.725,Q_2=-10$); (3) previous column restoration (low noise, 25 ISO); (4) simulated sensor image for the end-to-end design ($s=-0.775,Q_2=-2.4$); (5)-(7) previous column restoration: end-to-end results (400 ISO, 100 ISO, 25 ISO). Incidence angles ($\alpha$) of each crop are (top to bottom): $1.3^o$, $2.1^o$, $7.1^o$, $3.4^o$, $4.8^o$, $8.8^o$.
5. Discussion
The first conclusion to draw is that when considering the cost-performance trade-off, the design strategy determines strong differences in the resulting optical configurations. These differences are especially large for low-cost systems, precisely the ones for which the end-to-end strategy seems most appropriated [8,11,12]. Not only this but also the relative gains in restored image quality, when comparing end-to-end to conventional design strategies, are considerably higher for low-cost optical designs. In addition, the observed robustness of the optimized trade-off curves for different noise levels is very positive, as reveals that noise-independent end-to-end designs may not depart much from being optimal, a result with important practical consequences. On the other hand, the observation that there is a single optimal cost in the final image quality beyond which any further cost increase is counterproductive (as depicted in Fig. 6), implies that the end-to-end designer has to deal with a much-reduced range of design parameters. This is in strong contrast with conventional designs, where, within large margins, the lowest $RMS_W$ aberration is obtained for very high absolute values of conicity. However, as it is by now well-known (and clearly shown in Fig. 4), lower aberration does not necessarily imply a better digital compensation of the optical blur. The latter depends on the MTF being well above the noise level in all but the very high frequencies, a situation that depends on both the type and amount of the aberrations, not merely on their combined amount (see, e.g., [7,8]). In addition, we have shown that the particular end-to-end strategy proposed here not only has better optimal performance but also lower industrial cost, as compared to sequential designs (see Table 1). Thus, it satisfies our aim of obtaining more favorable cost-performance trade-offs by exploiting the potential of end-to-end designs. To demonstrate the practical applicability of our approach, we have performed realistic simulations of the camera pipeline, and shown visual results for high-resolution (24 MP) images.
In this work, we have illustrated our conceptual framework with a model that depends on a single optical surface and two design parameters. However, the method can be generalized to optical systems comprising several aspherical surfaces. To deal with the higher dimensionality of the problem without requiring an exhaustive evaluation of the merit and cost functions we propose (1) defining a single industrial cost metric for all the optical surfaces (e.g., the sum of the costs of each surface); (2) generating a set of (low-quality) initial solutions spanning the useful range of costs; (3) optimizing the quality metric for each of this configurations without changing their cost. This results in a family of iso-cost curves that converge to the optimal quality each, jointly converging to the optimal cost-performance trade-off curve. When the merit and cost functions’ gradients are computable, the third step can be done by projecting the gradient of the quality metric on the local subspace orthogonal to the cost gradient. Otherwise, and assuming the merit function can be efficiently computed numerically, one can still apply alternating marginal optimizations of a single surface, keeping the others fixed, and compute at each step the marginal optimal quality within the iso-cost curve passing by the parameters being marginally optimized.
We note that, in practice, these types of end-to-end designs should still be subject to tolerance analysis. The image restoration quality may be affected by errors in the estimation of the actual PSF, which may be caused, in turn, by uncertainties in manufacturing and assembly. Fortunately, in this case, the absolute optima of the lens configurations, for the end-to-end designs, are located at regions of lower cost, as compared to that for traditional sequential (optical, then digital) design. As already mentioned in section 1, it has been shown that constraining the slope departure, hence the cost, in the optical design process, leads to final geometries being more robust against optical degradation [2,6]. Therefore, the end-to-end design may offer an indirect way to increase the quality tolerance to discrepancies between nominal and final designs.
Funding
Ministerio de Ciencia e Innovación (PID2020-113596GB-I00).
Acknowledgments
We thank Chris Niccolls for kindly permitting us to use his photographs [24]. This work was supported by grant PID2020-113596GB-I00 from Spanish Ministerio de Ciencia e Innovacion.
Disclosures
The authors declare no conflicts of interest.
Data availability
No data were generated or analyzed in the presented research.
References
1. G. W. Forbes, “Manufacturability estimates for optical asphere,” Opt. Express 19(10), 9923–9942 (2011). [CrossRef]
2. B. Ma, L. Li, K. P. Thompson, and J. P. Rolland, “Applying slope constrained Q-type aspheres to develop higher performance lenses,” Opt. Express 19(22), 21174–21179 (2011). [CrossRef]
3. R. N. Youngworth, “Maximizing tolerancing benefits using optical design tools,” Opt. Rev. 20(2), 118–119 (2013). [CrossRef]
4. N. Takaki, A. Bauer, and J. P. Rolland, “On-the-fly surface manufacturability constraints for freeform optical design enabled by orthogonal polynomials,” Opt. Express 27(5), 6129–6146 (2019). [CrossRef]
5. J. P. McGuire, “Manufacturable mobile phone optics: higher-order aspheres are not always better,” in International Optical Design Conference (2010).
6. B. Ma, K. Sharma, K. P. Thompson, and J. P. Rolland, “Mobile device camera design with Q-type polynomials to achieve higher production yield,” Opt. Express 21(15), 17454–17463 (2013). [CrossRef]
7. D. G. Stork and D. G. Robinson, “Theoretical foundations for joint digital-optical analysis of electro- optical imaging systems,” Appl. Opt. 47(10), B64–B75 (2008). [CrossRef]
8. J. Portilla and S. Barbero, “Comparing optical to digital metrics: What is the optimal defocus in a rotationally symmetric system?” in Proc. IEEE International Conference on Image Processing, WA-L2.6, 1–5 (2017).
9. V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein, “End-to-end Optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging,” ACM Trans. Graph. 37(4), 1–13 (2018). [CrossRef]
10. E. Tseng, A. Mosleh, F. Mannan, K. St-Arnaud, A. Sharma, Y. Peng, A. Braun, D. Nowrouzezahrai, J. F. Lalonde, and F. Heide, “Differentiable compound optics and processing pipeline optimization for end-to-end camera design,” ACM Trans. Graph. 40(2), 1–19 (2021). [CrossRef]
11. Y. Liu, C. Zhang, T. Kou, Y. Li, and J. Shen, “End-to-end computational optics with a singlet lens for large depth-of-field imaging,” Opt. Express 29(18), 28530–28548 (2021). [CrossRef]
12. Z. Li, Q. Hou, Z. Wang, F. Tan, J. Liu, and W. Zhang, “End-to-end learned single lens design using fast differentiable ray tracing,” Opt. Lett. 46(21), 5453–5456 (2021). [CrossRef]
13. A. Fontbonne, H. Sauer, and F. Goudail, “Improved performance of a hybrid optical/digital imaging system with fast piecewise Wiener deconvolution,” Opt. Express 30(19), 34343–34361 (2022). [CrossRef]
14. R. Zhang, F. Tan, Q. Hou, Z. Li, Z. Sun, C. Yang, and X. Gao, “End-to-end learned single lens design using improved Wiener deconvolution,” Opt. Lett. 48(3), 522–525 (2023). [CrossRef]
15. J. Portilla and S. Barbero, “Hybrid digital-optical imaging design for reducing surface asphericity cost while keeping high performance,” in Proc. SPIE 10694, Computational Optics II, 1069406 (2018).
16. V. N. Mahajan, Optical Imaging and Aberrations. Part I: Ray Geometrical Optics (SPIE Opt. Engineering Press, 1988).
17. G. W. Forbes, “Robust, efficient computational methods for axially symmetric optical aspheres,” Opt. Express 18(19), 19700–19712 (2010). [CrossRef]
18. D. Miraut and J. Portilla, “Efficient shift-variant image restoration using deformable filtering (Part I),” EURASIP J. Adv. Signal Process. 2012(1), 100 (2012). [CrossRef]
19. J. Portilla, A. Tristán-Vega, and I. Selesnick, “Efficient and robust image restoration using multiple-feature L2-relaxed sparse analysis priors,” IEEE Trans. on Image Process. 24(12), 5046–5059 (2015). [CrossRef]
20. N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series (MIT U. Press, 1964).
21. W. Mauersberger, “Generalised correlation model for designing 2-dimensional image coders,” Electron. Lett. 15(20), 664–665 (1979). [CrossRef]
22. M. Riechert, “All cameras,” https://letmaik.github.io/pixelpitch/.
23. R. N. Clark, “Digital camera reviews and sensor performance summary,” http://www.clarkvision.com/articles/digital.sensor.performance.summary/.
24. C. Niccolls, “https://www.dpreview.com/sample-galleries/1916332853/canon-eos-r8-sample-gallery/1235333711,” and “https://www.dpreview.com/sample-galleries/1916332853/canon-eos-r8-sample-gallery/2841961449”.
25. J. Portilla, V. Strela, M. J. Wainwright, and E. P. Simoncelli, “Image denoising using scale mixtures of Gaussians in the wavelet domain,” IEEE Trans. on Image Process. 12(11), 1338–1351 (2003). [CrossRef]
