Diffraction theory reveals that an optical imaging system acts as a low-bandpass filter, sparing the low spatial frequencies of an object’s spectrum but cutting off high frequency information. Some of the high spatial-frequency signal information will be lost almost completely, due to the finite size of the lens apertures. Moreover, for digital imaging, the finite number of detectors in a sensor array (e.g., a CCD) is another factor that limits an imaging system’s spatial bandwidth. How to obtain a full spatial-frequency image is an important and fundamental research topic. Many attempts have been made to improve the resolving power of optical imaging systems, typically requiring large size lenses and complicated configurations that are costly and not always practical. However, recent advances in imaging technology have shown that combining the design of optics and optical electronics with digital image processing can lead to more effective imaging systems [1]. For instance, the well-known synthetic aperture (SA) concept has been tested as a promising approach using computational imaging to increase resolving power in many applications [2], which can extend the spatial frequency of an imaging system beyond its normal limitations.

The optical aperture synthesis technique is widely used in various imaging systems for holographic, microscopic, and astronomical imaging with different modes. In the imaging system of digital holography, a high resolution image can be achieved from two or more real or virtual subapertures via a combination of low resolution images through postprocessing, on condition that the object’s complex amplitude is available and the CCD is shifted to different off-axis positions [3–5]. Then the SA will improve the system resolution. As another promising method, distributed aperture or multiaperture synthesis [6] is able to generate high-resolution images from an array of small apertures but with a more complex setup and at higher cost. However, most of the above-mentioned methods are usually restricted to coherent imaging within in a laboratory setting, requiring complex configuration. Furthermore, the speckle noise of coherent imaging is often unavoidable and results in a serious negative impact on image quality.

For both spatially and temporally incoherent illumination, a high-resolution image can also be generated from a SA by superposing, on a single photographic plate, successive exposures [7]. In [7], it has been experimentally demonstrated that the synthesis of a high-resolution image could be achieved in a laboratory. In this experiment, however, a set of aperture “masks” were placed in front of a large size lens to avoid the problem of focus plane array position detection. We do not think that such an approach can fully recover the spatial resolution of the large aperture lens, but it has a hidden advantage: the smaller aperture enables greater depth-of-field while reconstructing higher resolution images.

Triggered by the above SA experiments, we present a computational imaging approach for super-resolution reconstruction (SRR) by synthesizing a virtual large aperture based on an incoherent imaging system. The key idea here is to consider a sequence of low resolution images of the same scene captured by one lens but with minor shifts, as images taken separately by an array of identical lenses or a phase shifted CCD camera with successive exposures. Then, a high spatial resolution image will be produced by exploiting temporal freedom of the imaging system according to the information capacity theory of optical systems [8]. In this SA mode, only a series of intensity images is required but benefiting from an identical lens and a flexible setup. The reconstruction image will have the higher spatial resolution of the synthesized larger aperture lens but meanwhile have the greater depth of field offered by the small aperture lens.

Superposition of the multiviews is one of the key steps for SA-based SRR, requiring accurate image registration, which has been studied in [9] with a secondary camera to determine the CCD position. The accuracy of the displacement estimation was close to $1/7$th pixel. In this Letter, however, the superposition is based on the displacement matrix with accuracy of $1/30$th pixel, and then sparse a priori constraints deconvolution is applied for SRR. The approach can efficiently overcome degradation due to the limited size of the lens as well as the size of the CCD pixel. The final result of a SA generation extends the cutoff frequency of the imaging system much beyond that of an individual small lens aperture and thus yields a super-resolution image.

When the object illumination is incoherent, the imaging system can be expressed as

Figure 1 illustrates a simplified scheme of the idea of this incoherent optical SA-based SRR method from a sequence of low resolution images, which are taken by a conventional imaging system under both spatially and temporally incoherent illumination. A camera with a small lens is assumed to shift over a relatively short range, with random subpixel increments, capturing consecutive images. The convolution operation in Eq. (2), a spatial integration effect, makes the high spatial frequency components largely unresolvable. The aim of this study is thus to find an optimal estimation of the SRR image $\widehat{f}$ when a set of images $g_i(x,y)$ is available from the shift operation, each of which only contains partial information. Superposition based on pixel-wise phase-correlation and sparse a priori constraints deconvolution is performed for the incoherent SA-based SRR.

 

Fig. 1. Observation and SA reconstruction for conventional imaging system.

Download Full Size | PDF

The CCD relative positions between the multiviews are calculated using a pixel-wise phase-correlation method [10], which produces a subpixel accuracy displacement matrix. The phase correlation function is defined as

Then, the a priori constraints deconvolution is used to retrieve the high-frequency components by alleviating aliasing. Finding an optimization approximation $\widehat{f}$ only from a set of low spatial observations $g_i$ is an ill-posed linear inverse problem, which generally leads to a convex minimizing problem. Mathematically,

$R(f)$ is a regularization term that incorporates a priori knowledge of the desired high resolution image. For large underdetermined systems of linear equations, there are various regularization operators. However, in this Letter, the $L1$ norm sparse prior regularization term including a first order differential operator as well as a second order differential operator are taken into account in this convex minimizing problem, formulated as

$|Df|$ is a first-order-differential regularization term. Without $|D^{2}f|$, Eq. (5) is equivalent to the total variation image restoration algorithm, which produces a sparse solution. It is especially effective on image deblurring with piecewise constant and piecewise linear image components, but otherwise staircase effects will be created, which is often the case for images of natural scenes. To alleviate the staircase effect, $|D^{2}f|$ for the second order differential operator is added as a smoothing regularity term in the proposed method, as shown in Eq. (5). This modification improves smoothness while keeping image structure/edges sharp and alleviating effectively the ringing artifacts generally resulted from deconvolution algorithm. Equation (5) is solved by the corresponding Euler–Lagrange equation with CG iteration algorithm. The viability of the proposed method has been demonstrated using standard simulation test images as well as real image datasets.

First, the 1951 United States Air Force (USAF) resolution target (AF target) was used for experimental validation and to quantify the improvement in image resolution. Figure 2(a) shows a sequence of 25 low resolution images each with vertical or horizontal 0.2 pixel displacement increments between consecutive images. They are generated from the ground-truth high-resolution image through blurring by a Gaussian function with delta 1.3, a down-sample factor 5, and Gaussian additive noise with noise level 0.005.

 

Fig. 2. AF target images for SA-based SRR: (a) low resolution image sequence; (b) the SRR reconstruction result with the proposed method; (c) and (d) present center parts ROIs of one low resolution image in (a); (e) and (f) present center parts of ROIs of the SRR result (b) corresponding to the ROIs (c) and (d).

Download Full Size | PDF

The bar elements ($\text{groups}/n\text{:}1$, $\text{elements}/m\text{:}2$) shown in Fig. 2(d), a boxed region of interest (ROI) boxed area in Fig. 2(a), are hardly resolvable. By contrast, the high-resolution image synthesized using the proposed SA-based SRR, shown in Fig. 2(b), is sharply enhanced. The texture is fairly sharp. Figures 2(e) and 2(f) showing the corresponding ROIs to Figs. 2(c) and 2(d) demonstrate the improvement of image spatial resolution. Figure 2(e) shows that the bar elements ($n:2$, $m\text{:}6$) are easy resolvable.

Figure 3 presents 1D column profiles of horizontal bar elements of group 2 before and after the SA-based SRR. Figure 3(a) shows five profiles ($n\text{:}2$, $m\text{:}2–6$) out of 25 low resolution images, which are completely unresolved. By contrast, the profiles after processing of superposition and SRR, shown, respectively, as blue dashed line and a solid gray line in Fig. 3(b), are clearly resolved with sharp edges. For the AF target, an improvement by ($n$ bars, $m$ elements) is equivalent to a resolution improvement by a linear factor of $2^{(n+m/6)}$. Note that the spatial resolution gain of the proposed method is about 2.8–3.2.

 

Fig. 3. 1D column profiles analysis of ROI (group 2, elements 2–6) in AF target.

Download Full Size | PDF

Figure 4 shows the normalized power spectrum of the 25th low resolution image, the red bottom line, out of the AF target sequence, the SRR image in black, and the ground truth image in the blue. The blue dashed line and the black line are closely matched and both are far away from the red line at higher frequencies. This indicates that the high frequency components are substantially recovered by the SA-based SRR approach.

 

Fig. 4. Images normalization power spectrum.

Download Full Size | PDF

A further test was carried out using aerial images captured with a vertical viewing mode in incoherent illumination. Figure 5 presents the consecutive aerial images and the SRR result. The camera moved along one direction, and the initial motion vector estimation is demonstrated in Fig. 6 with subpixel displacement. Figure 5(a) shows 50 low resolution subimages. The original grayscale images have a resolution of $480\times 512$ pixels with a very high overlapping rate($\cong 85\%$), which were assumed to be captured approximately in the same image plane. A high spatial resolution image was synthesized as shown in Fig. 5(b). Compared with Fig. 5(a), Fig. 5(b) shows that the spatial resolution has been effectively improved. The ROI, boxed area in Fig. 5(e), shows much finer structures than the corresponding ROI boxed area in Figs. 5(c) and 5(d), which are results of different interpolation methods. It indicates that more higher spatial frequency components, beyond what the interpolation and deconvolution algorithms can achieve from one single image, have been recovered from the aliased low resolution images without introducing ringing artifacts.

 

Fig. 5. Aerial image sequence and the SRR result.

Download Full Size | PDF

 

Fig. 6. Aerial camera initial motion vector estimation.

Download Full Size | PDF

Figure 7(a) presents 30 low-resolution images captured with a camera in a slow panning motion under incoherent illumination. The original grayscale images have a resolution of $360\times 288$ pixels with a very high overlapping rate. The initial estimation of camera motion vectors is shown in Fig. 8 with random and subpixel displacement for each image. Figure 7(b) shows the SA-based SRR result. Compared with Fig. 7(a), Fig. 7(b) is fairly sharp. For comparison of details, ROI 1 and ROI 2 in Fig. 7(a), two boxed areas in red, are presented in Figs. 7(c) and 7(d) and Figs. 7(f) and 7(g), which are results of different interpolation algorithms. The characters in ROI $1^{\prime }$ and building structures detail in ROI $2^{\prime }$ are clearly resolvable with significant resolution improvement, as shown in Figs. 7(e) and 7(h). It shows that the spatial resolution has been effectively improved with much finer structures and sharp edge without introducing ringing artifacts, what cannot be achieved with a interpolation or a deconvolution algorithm.

 

Fig. 7. Experiment with real video images: (a) presents 30 frames low images; (b) presents the SRR result; (c), (d), and (e) show, respectively, ROI 1 nearest interpolation image, bicubic interpolation, and SRR result; (f), (g), and (h) present, respectively, ROI two nearest interpolation image, bicubic interpolation, and SRR result.

Download Full Size | PDF

 

Fig. 8. Camera initial motion vectors estimation.

Download Full Size | PDF

In this Letter, an incoherent optical SA-based SRR approach is proposed experimentally and demonstrated for multiview low resolution images taken by incoherent imaging systems with simple and flexible configurations. Characterized by accurate superposition based on pixel-wise phase correlation and deconvolution constrained by a L1 norm regularization term involving a first order differential operator and a second order differential operator, the proposed method achieves a state-of-the-art result. It dramatically improves the image spatial resolution in experiments using controlled test images as well as two real image datasets. The spatial resolution gain factor is about 3.0 and displacement matrix estimation is as accurate as $1/30$th pixel.