Open Access
Add to BibSonomyAbstract
We propose a resolution enhancement method for mobile small f-number compact imaging systems based on wavefront coding and super-resolution image processing. Wavefront coding increases the focus depth of an optical system and produces point spread functions (PSFs) with similar characteristics at different field and defocus positions. The designed target wavefront is realized as a combination of wavefront errors of each rotationally symmetric lens, without including an additional phase plate. Finally, using one deconvolution filter containing all the characteristics of the PSFs, we achieve high resolution, breaking the diffraction limit of small f-number and the resolution limit of the image sensor by super-resolution image processing.
©2008 Optical Society of America
1. Introduction
Wavefront coding can increase the focus depth of an optical system. Image quality is improved through an image restoration process in the spatial frequency domain called “super-resolution.” Wavefront-coded optical system uses a deconvolution filter that is designed based on point spread function (PSF) characteristics, which remain nearly constant at different field and defocus positions. In general, an asymmetric phase plate is placed in the optical system to realize the coded wavefront [1, 2]. In a compact imaging system, however, dimensions are fixed within the mobile devices, leaving no space for additional optical components. In addition, the image resolution of compact imaging systems is limited by their dimensions and by limitations on available lenses. Image resolution can be improved by decreasing the f-number that determines the resolving power of optics. However, the depth of focus becomes shorter with the f-number, limiting the possible decrease. Wavefront coding extends the depth of focus while maintaining a constant f-number. Finally, super-resolution enhances the resolution of the images over the diffraction limit by using a deconvolution filter containing the PSF characteristics of the optical system. The diffraction limit can be broken with the help of the calculation power of modern image processing and a high-resolution image sensor with a pixel size less than 1 µm.
In conventional optical design, small pixel sizes exceeding the Rayleigh resolution limit cannot be resolved even when the optical system exhibits no aberrations. As optics and digital image processing are separately developed, image resolution cannot break the limit of each part. Modern optical design utilizes several research fields simultaneously to break through the current limit of optical systems [3]. By optimizing optics and digital image processing simultaneously, such limits can be broken over the diffraction limit of optics and the spatial resolution limit of the image sensor.
However, commercial optical design software does not provide for simultaneous optics and digital image processing design. We developed an algorithm that optimizes the optics and digital image processing simultaneously, giving a final lens design that satisfies the given high-resolution specification. A wavefront-coded f-2.0 compact imaging system with four plastic lenses whose resolution is much higher than the conventional optical design is given as an example.
2. Overview of design methodology
Creating a wavefront-coded optical system involves optimizing the initial lens, wavefront, final lens, and deconvolution filter under various design constraints. Lack of phase plates is one such constraint. Designing the target wavefront to properly combine wavefront errors for each lens in the optical system is another. A schematic diagram of our design procedure is shown in Fig. 1.
Initial lenses are designed by conventional methods using commercial software, and the system is optimized to eliminate wavefront errors and ensure maximum spatial resolution at the best focus position.
The target wavefront, covering the whole field of view (over 60°) and defocus range (over 30 µm) of the optical system, must be constructed such that the PSFs of each position have similar shapes and characteristics. The wavefront has a rotationally symmetric shape, allowing it to be handled as a combination of wavefront errors for each lens without any additional optical components. Optimization is performed as a routine of PSF and modulation transfer function (MTF) calculations that finally satisfy the target of through focus MTF.
Using a limited number of lenses from the initial lens design, the final optical system is designed for a target wavefront at the exit pupil; that is, the wavefront error of the final optical system matches the designed target wavefront.
Finally, a deconvolution filter is constructed as a weighted sum of PSFs over each field and defocus position. Using this deconvolution filter and a simulated image from the wavefront-coded optical system, the final image is restored in the spatial frequency domain and the spatial frequency response (SFR) is calculated. Optimization is performed as a calculation routine of filter and SFR for higher SFR values that satisfy the target specifications of resolution through the wide range of spatial frequency.
3. Initial lens design
We designed an f-2.0 compact imaging system for a mobile device using four plastic lenses. Design specifications are listed in Table 1. Figure 2 shows the diagram of lens design and spherical aberration. In conventional optical design and manufacturing processes, four plastic lenses barely manage to achieve resolutions corresponding to a 3 megapixel image sensor at the best focus position. SFR is calculated at 0.5 field position and the results of calculations are shown in Fig. 3. Results for the initial lens, designed by conventional methods, show degradation of resolution performance from the best focus position to the defocused position. This occurs because the PSFs of each defocus position have different characteristics, and the SFR is calculated by the Fourier transform of the image, which is simulated as a convolution of the PSFs and the original image.
As defined by the Rayleigh criterion, the small pixel size (under 1µm) of a high-resolution image sensor (over 10 megapixels) cannot be resolved by optics even when the system exhibits no aberrations. However, a high-resolution image sensor helps resolution enhancement in the deconvolution process with the calculation power of current hardware and memory.
Table 1. Specification of the f-2.0 compact imaging system
4. Wavefront coding (target wavefront design)
To compensate for the insufficient performance of the initial lens design and obtain suitable optical characteristics for image restoration, the wavefront of the initial lens design is modified and optimized. The target wavefront extends the depth of focus and gives the PSFs of all fields of view similar characteristics.
4.1 Calculation of Target Specification
The depth of focus (δ) is defined by Eq. (1) using the Rayleigh criterion, which defines the resolution limit with the Airy Disc size. It gives only 8.6 µm for the f-2.0 camera. The short depth of focus of the small f-number camera is the main factor to be modified by wavefront coding. The conventional optics cannot generate enough depth of focus to match the large tilt of the auto-focus actuator, causing image blurring. The target specification of wavefront coding is calculated by the increase in the depth of focus desired to compensate for the maximum tilt of the auto-focus actuator.
Figure 4 displays the schematic diagram of the target specification calculation. From the tilt (α) specification of the actuator, the target depth of focus is calculated, which will be the wavefront coding specification. Using Eq. (2), we find 30 µm of the target depth of focus.
4.2 Extending Depth of Focus Using Wavefront Coding
Infinite numbers of wavefronts are possible for the modified depth of focus. The pupil is divided into several zones and each zone is controlled for particular optical path differences (OPDs), as shown in Fig. 5. The wavefront is coded using several design parameters such as number of zones, sizes of each zone, and maximum OPDs of each zone.
4.3 Calculation of PSF and MTF
After the wavefront is coded, it is fitted with Zernike polynomials and implemented in the initial design, permitting calculations of the PSF and MTF. The PSF is calculated at each field of view and defocus position and the MTF is obtained as the Fourier transform of the PSF.
4.4 Optimization of the Wavefront
Optimization is performed as a calculation routine of MTFs at each field and defocus position using the design parameters (number of zones, sizes of each zone, and maximum OPDs of each zone) as variables. Finally, we generate a target wavefront with an extended depth of focus and nearly constant PSF characteristics through the broad range of defocus positions as shown in Fig. 6. Figure 6 represents the PSFs of conventional optical system (a) and wavefront-coded optical system (b) at 0.0 and 0.5 field position.
5. Final lens design
The final lens is designed using the optimized wavefront. The optimized target wavefront is realized as a combination of wavefront errors of each lens, without any additional optical plate, as the compact camera is small (under 5mm) and is fixed within a mobile device, which uses the camera module as a capturing device.
Final lens design, particularly the construction of a target wavefront with few lenses, can be another research area in the development of wavefront-coded optical systems. In this paper, Zernike coefficients of target wavefront are used for the final lens design. Using the Zernike coefficients as design constraints, the lenses from the initial design are modified such that the target wavefront is realized at the exit pupil of the final optical system without the use of additional optical components such as wave plates. Individual lenses are easily optimized by commercial software; furthermore, specific lenses can be chosen to be modified in order to relax the tolerance sensitivities of the optical system. The final lens design and its spherical aberration are shown in Fig. 7, and the spatial frequency response is shown in Fig. 8.
6. Deconvolution Filter Design and Image Restoration
6.1 Deconvolution Filter Design
Using super-resolution image processing, the captured image is restored by deconvolving the image with a spatial filter called the “deconvolution filter.” Breaking the diffraction limit is possible with the help of recent developments in digital sensors and memory technologies. The deconvolution filter is designed as a weighted sum of PSFs that best describes the response of point source as an optical transfer function.
6.2 Image Restoration and Calculation of Spatial Frequency Response
Images are restored in the spatial frequency domain for real-time implementation in mobile devices. Images are simulated at several defocus positions from the final lens design and the images have correspondence to the tilt angle of AF actuator. All the images are restored by only one deconvolution filter and SFRs are calculated to optimize the deconvolution filters. Figure 9 shows the process of image restoration and filter optimization.
6.3 Optimization of the Deconvolution Filter and Final Result
Optimizing the deconvolution filter is one of the most important parts in design procedures with the optimization of target wavefront. Optimization is also performed as a calculation routine of filter and SFR as shown in Fig. 9.
The final result of SFR is calculated from the restored images (Fig. 11 and Fig. 12) at 0.5 field position. And it is overcoming the diffraction limit of f-2.0 camera at best focus position as shown in Fig. 10. The resolution data shows outstanding performance of the wavefront-coded optical system through the super-resolution image processing, which has SFR over 60% in the high-frequency region even when the system is defocused (within 30 µm) by the tilt of the AF actuator.
Figures 11 and 12 are images of the conventional optical system and wavefront-coded optical system. Images are magnified for effective comparison from the central and outer regions. Final images represent the extended depth of focus of wavefront coding and the resolution enhancement of super-resolution image processing.
Fig. 11. Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the central region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.
Fig. 12. Image comparison of conventional optical system and wavefront-coded optical system (restored image). Images are magnified from the outer region for effective comparison. (a) Image at −15 µm defocused position, (b) best focus position and (c) 15 µm defocused position.
7. Conclusions
Traditional optical design was focused on minimizing the wavefront error of optical system with a proper combination of spherical and aspherical lenses. In this paper, we generate certain amount of wavefront error which extends the focus depth of optical system and enables the final images to be restored compensating the tilt of auto-focus actuator. It is possible because the wavefront-coded optical systems display similar PSF shapes in a broad range of focus and wide field positions. The designed target wavefront is realized as a combination of the wavefront errors of each lens without any additional optical components. This means that compact imaging systems can be designed with special coded wavefront shapes while maintaining their dimensions and f-number. Using super-resolution image processing with only one deconvolution filter, the final image has a spatial resolution that breaks the diffraction limit. We successfully designed a small f-number optical system composed of only four plastic lenses whose resolution breaks the diffraction limit and meets the resolution specification of 10 megapixel image sensor. SFR is enhanced over 65% at the best focus position in the high-frequency region.
Acknowledgments
This work was supported by the Korea Science and Engineering Foundation (KOSEF) grant funded by the Korea government(MEST) (No. R17-2008-040-01001-0).
References and links
1. K. Kubala, E. R. Dowski, and W. T. Cathey, “Reducing complexity in computational imaging systems,” Opt. Express 11, 2102–2108 (2003). [CrossRef] [PubMed]
2. E. R. Dowski and W. T. Cathey, “Extended depth of field through wavefront coding,” Appl. Opt. 34, 1859–1866 (1995). [CrossRef] [PubMed]
3. W. T. Cathey and E. R. Dowski, “New paradigm for imaging systems,” Appl. Opt.41, 6080–6092 (2002).
