Super-resolution imaging for infrared micro-scanning optical system

X. F. Zhang; W. Huang; M. F. Xu; S. Q. Jia; X. R. Xu; F. B. Li; Y. D. Zheng

doi:10.1364/OE.27.007719

1. Introduction

Infrared imaging is a technology used for detecting the infrared radiation energy of objects. Owing to the advantages of infrared imaging such as its real-time, non-contact, non-invasive, and simple operation, it has been an effective technology in military affairs, industrial inspection, monitoring systems, medical diagnosis, etc [1–4]. Because of the limitations of semiconductor materials and manufacturing techniques, there is a lower limit to the pixel size of the infrared focal plane array, which results in infrared images with low resolution, low contrast, less detail, and much noise. Therefore, it is necessary to develop an effective method for overcoming these disadvantages.

The super-resolution (SR) reconstruction method can be used to obtain a high-resolution (HR) image from one or more frames of low-resolution (LR) images of the same scene, and the resolution of the resulting image thus obtained can overcome the diffraction limit of the imaging sensor [5]. Furthermore, the multi-frame SR algorithm can perform better than the single-frame ones because the use of a greater number of frames implies, to some extent, that more information is being captured. SR reconstruction can be realized in both the frequency and spatial domains [6–9]. The SR algorithm in the frequency domain has limitations in applications owing to its global-translation property. While the spatial domain SR algorithm is more flexible because the spatial observation model used in the algorithm can take advantage of various types of previously obtained information [10,11]. Scholars have put forward various spatial-domain algorithms such as interpolation, iterative back-projection (IBP), and projection onto convex sets (POCS) algorithms [12], each with its own pros and cons. Furthermore, the super-resolution convolutional neural network (SRCNN) [13] has become a popular method in recent years. Our study is focused on the POCS-based method in the spatial domain.

Various SR techniques have been implemented to mitigate resolution loss by utilizing sub-pixel displacements in the imaging system, which is achieved, for example, by shifting the illumination source, sensor array, and/or sample, followed by a digital synthesis of a smaller effective pixel by merging these sub-pixel-shifted LR images [14]. A micro-scanning imaging system can acquire multiple images with sub-pixel displacements in the horizontal or vertical direction by controlling the micro-scanning platform, while just satisfying the low-resolution images requirements of multi-frame SR methods [15–17]. The micro-scanning imaging system enforces the sub-pixel displacements using an electric motor or piezoelectric actuator, and it can be operated in the controllable mode and non-controllable mode. Currently, the majority of the SR algorithms have great computation complexity. By using the controllable micro-scanning method, the image registration step can be simplified and the accuracy of the reconstruction results can be increased. Therefore, we design a medium-wave infrared micro-scanning lens and construct a controllable infrared micro-scanning optical system for infrared image SR reconstruction.

In this article, we propose a frequency domain phase-based projection onto convex sets (FPPOCS) SR algorithm, which takes advantage of the phase information in the frequency domain to obtain further details of the infrared image. The FPPOCS method can realize image SR reconstruction while performing denoising and retaining the details of the infrared image simultaneously. Accordingly, we design an infrared micro-scanning lens, construct an infrared micro-scanning optical system, and realize infrared image SR reconstruction using our proposed algorithm. The infrared micro-scanning optical lens comprises a decentered system in the medium-infrared range of 3–5 μm. It can guarantee both the displacements uniformity of the entire field of view (FOV) and the ideal image quality in the image plane when the micro-scanning system is in use. The infrared micro-scanning optical system realizes controllable sub-pixel micro-scanning of an arbitrary step size, which means that the registration procedure in the SR algorithm can be simplified. We also propose an image quality assessment (IQA) method based on the frequency domain phase spectrum. Using our proposed algorithm and our infrared micro-scanning optical system, we practically realize infrared SR imaging.

The remainder of this paper is organized as follows. In Section 2, the micro-scanning SR imaging theory is presented. Section 3 presents the simulation results and the discussions. In Section 4, we illustrate our infrared micro-scanning optical system and the results of our method for obtaining LR infrared images. Finally, Section 5 presents the conclusion of our study.

2. Micro-scanning SR imaging theory

Our infrared micro-scanning SR imaging theoretical system comprises two crucial technologies. The first is our presented FPPOCS approach, which takes advantage of the phase information in the frequency domain to realize image SR reconstruction. The second is the design of our infrared micro-scanning system, which can capture multiple frames of sub-pixel images for the image SR reconstruction.

2.1 FPPOCS algorithm

2.1.1 Conventional POCS theory

SR reconstruction is an ill-posed inverse problem, the process of which is based on the image degradation model as described in the following Eq. (1):

f_{k} = D_{k} B_{k} M_{k} X + n_{k}, k = 1, ..., p

where X is the ground truth HR image,

f_{k}

is the kth LR image, and

M_{k}

,

B_{k}

, and

D_{k}

denote the shift matrix, blurring matrix, and down-sampling matrix of the kth LR image, respectively.

n_{k}

is a Gaussian noise vector, and p denotes the number of frames.

The image degradation model is illustrated in Fig. 1.

Fig. 1 Image degradation model.

Download Full Size | PDF

The POCS-based SR algorithm regards prior knowledge as closed convex sets $C_{i}$ and incorporates it into the reconstruction process iteratively as follows:

f^{n + 1} = P_{m} P_{m - 1} ... P_{2} P_{1} f^{n}, n = 0, 1, 2, ..., n

where n is the number of iterations, and m is the number of prior knowledge sets.

f^{n}

denotes the result of the nth iteration.

P_{i}

denotes the mapping operator of the convex set

C_{i} (i = 1, 2, \dots, m)

.

As described above, the solution space of POCS is the intersection of all the prior knowledge sets. It can be represented graphically as shown in Fig. 2, where $f^{0}$ denotes the initial point, which can be an arbitrary vector, and sets A and B denote two prior knowledge convex sets.

Fig. 2 POCS theory.

Download Full Size | PDF

The POCS algorithm is intuitive and code implementation in it is simple. Moreover, it can build a complex image degradation model and introduce various prior knowledge. Therefore, we propose an improved FPPOCS SR algorithm based on the conventional POCS theory for performing denoising and obtaining details for the infrared image SR reconstruction.

2.1.2 FPPOCS theory

In the Fourier representation of the signals, the spectral amplitude and phase tend to play different roles [18]. We can reasonably consider that the amplitude spectrum contains information of the luminance contrast, and the phase spectrum contains the texture details of the image. It is generally plausible that the phase reflects the location of “events” such as lines and points. Furthermore, the phase spectrum can represent the texture and structure information despite the low illuminance and low contrast [19]. This is an important feature that can be utilized for the low-contrast infrared images.

The phase spectrum of an image in the frequency domain can be obtained using the discrete Fourier transform (DFT):

F (u, v) = \sum_{x = 0}^{M - 1} \sum_{y = 0}^{N - 1} f (x, y) e^{- j 2 π (u x / M + v y / N)}

F (u, v) = A (u, v) e^{j φ (u, v)}

where u = 0, 1, 2…, M-1 and v = 0, 1, 2…, N-1 denote the frequency variables. M and N denote the number of rows and columns of the image, respectively. F(u, v) is the Fourier transform for f(x, y), A(u, v) denotes the amplitude spectrum, and

φ (u, v)

denotes the phase spectrum of F(u, v).

Therefore, the proposed algorithm is an attempt to integrate the phase spectrum advantage into the process of the conventional POCS method. The phase information employed in the proposed algorithm takes full advantage of the phase-spectrum texture details and contrast-independent feature and can obtain more details of the infrared image in the SR reconstruction result.

In our method, we extract the frequency domain phase spectrum $φ (u, v)$ from the given LR images and then update the preliminary HR result with $φ (u, v)$ . When we obtain the DFT of an image, the image and the transformation are periodic. The convolution for the periodic function causes adjacent period crosstalk called a folding error [20]. Consequently, in the DFT process, we should perform an adding-zero operation to pad the image size in order to prevent the occurrence of the folding error. The padded zero DFT operation is abbreviated as pDFT in the rest of the paper. Furthermore, the phase spectrum information cannot only provide more details but also result in additional noise. Therefore, we should also perform denoising in the process. In the majority of the existing SR denoising methods, the SR process and denoising operation are separated—either the denoising is followed by the SR reconstruction, or the SR reconstruction is followed by image denoising [21]. This type of image denoising method may result in the loss of details from the image. Therefore, we perform Wiener denoising [22] and phase spectrum updating jointly in our algorithm. The Wiener filter is a famous denoising method, which can aid in reducing the ringing effect introduced by the Fourier transform [20]. Therefore, we use the Wiener filter as our denoising method.

The proposed FPPOCS method can be summarized as follows:

1) Reconstructed HR image initialization: For the purpose of simplification, we choose the first frame of the LR images $f_{1}$ as the reference image and then resize it to the ideal HR gridding size using linear interpolation to obtain the initial estimated HR image $X_{0}$ . $X_{0} = i n t e r p (f_{1})$
2) Motion estimation: We estimate the relative displacements $M_{k}$ between the LR frames and the reference image based on the affine transformation theory. $M_{k} = a f f i n e (f_{k}, f_{1}), k = 2, \dots, p$
where $f_{k}$ is the kth LR image, $M_{k}$ denotes the shift matrix, and p is the number of frames.
3) Phase spectrum extraction: In order to extract phase spectrum information from the LR frames, we perform image registration according to the motion estimation result and then perform bicubic interpolation for the LR frames. We can then obtain the frequency domain phase spectrum $φ (u, v)$ after performing the pDFT for the interpolation result. ${\begin{matrix} F (u, v) = p D F T (b i c u b i c (f_{1}, f_{2}, \dots, f_{p})) \\ F (u, v) = A (u, v) e^{j φ (u, v)} \end{matrix}$
4) HR image reconstruction: We define appropriate constraint convex sets and the corresponding mapping operators, and then correct the pixel value in the HR gridding $X_{0}$ iteratively based on the motion estimation and mapping operators to obtain the preliminary HR result $X_{h}$ . The iteration details are described in Eq. (2).
5) Wiener filtering: As the infrared images have a great amount of noise, and the phase spectrum updating process can retain more details but may also bring in some noise, we first perform Wiener filtering on the preliminary HR image $X_{h}$ . $X_{d} = w i e n e r (X_{h})$
where $X_{d}$ is the HR image obtained after the Wiener denoising.
6) HR image renewal based on phase spectrum: We perform the pDFT for the preliminary HR result $X_{d}$ and then update it with the phase spectrum information as in Eq. (10). Finally, we acquire the SR result $\tilde{X}$ after performing the inverse discrete Fourier transformation (IDFT) for the above result. ${\begin{matrix} F_{X_{d}} = p D F T (X_{d}) \\ F_{X_{d}} = A_{X_{d}} e^{i φ_{X_{d}}} \end{matrix}$ ${\begin{cases} Δ φ = | φ (u, v) - φ_{X_{d}} | \\ m = \frac{1}{M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} Δφ \\ F_{\tilde{X}} = A_{X_{d}} {\begin{cases} e^{i φ (u, v)} i f Δ φ > m \\ e^{i φ_{X_{d}}} e l s e \end{cases} \end{cases}$ $\tilde{X} = I D F T (F_{\tilde{X}})$
where $F_{X_{d}}$ is the pDFT result for $X_{d}$ , $A_{X_{d}}$ denotes the amplitude spectrum, and $φ_{X_{d}}$ denotes the phase spectrum of $F_{X_{d}}$ . $Δφ$ is the absolute difference of the phase spectrum between $φ (u, v)$ and $φ_{X_{d}}$ , and m is the mean value of $Δφ$ . $F_{\tilde{X}}$ is the pDFT for $\tilde{X}$ .

The aforementioned workflow of our method is illustrated in Fig. 3.

Fig. 3 Workflow of our FPPOCS algorithm.

Download Full Size | PDF

2.1.3 Image quality assessment

The IQA method can be divided into the subjective observation method and objective image quality index method. The objective method can further include full reference method [23] and no reference method [24].

In this paper, we propose an IQA metric based on the frequency domain phase spectrum. In order to make the information acquisition more intuitive, we calculate the phase spectrum error (PSE) between the SR result and the ground truth HR image as shown in Eq. (12).

PSE = \frac{1}{M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} [P_{\tilde{X}} (i, j) - P_{X} (i, j)]

where M and N denote the number of rows and columns of the image, respectively,

X (i, j)

is the ground truth HR image, and

\tilde{X} (i, j)

is the restored image. P denotes the phase spectrum. The smaller the PSE value, the better the image quality.

We also utilize the classical mean square error (MSE), peak signal-to-noise ratio (PSNR), and structure simulation (SSIM) as the full reference IQA [25,26]. The indices are defined in Eqs. (13)–(15).

MSE = \frac{1}{M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} {[\tilde{X} (i, j) - X (i, j)]}^{2}

PSNR = 10 \log_{10} \frac{1}{M S E} = 10 \log_{10} \frac{1}{\frac{1}{M N} \sum_{i = 1}^{M} \sum_{j = 1}^{N} {[\tilde{X} (i, j) - X (i, j)]}^{2}}

SSIM (X, \tilde{X}) = \frac{(2 u_{X} u_{\tilde{X}} + C_{1}) (2 σ_{X \tilde{X}} + C_{2})}{(u_{X}^{2} + u_{\tilde{X}}^{2} + C_{1}) (σ_{X}^{2} + σ_{\tilde{X}}^{2} + C_{2})}

where

u_{X}

,

σ_{X}

, and

σ_{X \tilde{X}}

denote the mean, standard deviation, and covariance, respectively.

C_{1} = {(K_{1} L)}^{2}

and

C_{2} = {(K_{2} L)}^{2}

, where

K_{1}

= 0.01,

K_{2}

= 0.03, and L = 255 for gray images of unit 8. The smaller the MSE value, the better the image quality, and the larger the PSNR and SSIM values, the better the image quality.

For the practical results obtained from the experiment without a reference, we use a “completely blind” IQA method called natural image quality evaluator that was proposed in [27]. The quality of the distorted image is expressed as the distance between the quality aware natural scene statistic feature model [28] and the multivariate Gaussian (MVG) fit to the features extracted from the distorted image:

D (υ_{1,} υ_{2,} Σ_{1,} Σ_{2}) = \sqrt{{(υ_{1} - υ_{2})}^{T} {(\frac{Σ_{1} + Σ_{2}}{2})}^{- 1} (υ_{1} - υ_{2})}

where

υ_{1}

,

υ_{2}

,

Σ_{1} and Σ_{2}

are the mean vectors and covariance matrices of the natural MVG model and the distorted image’s MVG model, respectively. The smaller the D value, the better the image quality.

2.2 Design of infrared micro-scanning optical system

2.2.1 Technical design principles

In this work, we design a medium-wave infrared micro-scanning lens and construct a controllable infrared micro-scanning optical system. To achieve the objective of realizing accurate micro-scanning in the horizontal and vertical directions, we designed a decentered infrared micro-scanning lens system.

A successful design of an optical system should take into consideration both the system performance and engineering facilities. In the design and optimization process, we have designed a program to simultaneously evaluate the micro-scanning displacements of the entire FOV and the diffraction modulation transfer function (MTF) of the image after performing the micro-scanning in the image plane, and we optimize our system according to the evaluation results. Thus, the infrared micro-scanning lens we designed can guarantee both displacements uniformity of the entire FOV and ideal image quality in the image plane using the micro-scanning system. The infrared micro-scanning optical system we constructed can be used to realize controllable sub-pixel micro-scanning of an arbitrary step size. Furthermore, we can realize sub-pixel LR image collection by just moving two light and compact pieces instead of moving the entire lens, sensor array, or sample as in the traditional method. Thus, the precision of the sub-pixel movements can be improved greatly.

The details of our technical design principles for realizing the function mentioned above are as follows:

1) Guarantee displacements uniformity of the entire FOV in the image plane: When using the micro-scanning system, the displacements of all the points in the image plane must be consistent with the center point. Therefore, we design a program to evaluate the micro-scanning displacements of the entire FOV in the image plane and optimize our system according to the evaluation results.
2) Guarantee ideal image quality in the image plane: When using the micro-scanning system, the image quality should be guaranteed every time the decentered lens is moving. Thus, in the evaluation program we mentioned in the first principle, we also evaluate the diffraction MTF of the image simultaneously with the evaluation of the displacements. Furthermore, by optimizing our system according to the evaluation results, we can guarantee the image quality when the micro-scanning is performed.
3) Ensure practical effectiveness: An idea or design should not be significative unless it can be realized in practice; therefore, our design results should be effective not only for the ideal designed system but also practical devices. Therefore, in order to ensure the practical effectiveness of our final result, we add a random disturbance tolerance and set the best focus at the beginning of the evaluation process.
4) Select appropriate decentered lens: The decentered lens used for realizing micro-scanning is especially crucial because it plays a decisive role in both the engineering facilities and sub-pixel micro-scanning accuracy of the system. First, the scanning of the FOV surface can be realized using either a tilted plate or motive lens. Considering the manufacture facility, we choose to realize micro-scanning by moving the decentered lens in the horizontal and vertical directions. Thus, it can be both handy and intuitive during the micro-scanning. Second, from the packaging point of view, the position we choose for the decentered lens should be convenient for maintaining the concentricity during the assembly. Lastly, the decentered lens connected with the micro-scanning platform should be easy to control when the micro-scanning system is being used. Thus, we can realize sub-pixel LR image collection by simply moving two light and compact pieces instead of moving the entire lens, sensor array, or sample as in the traditional method. Therefore, we finally choose the last two pieces from surface 12 to surface 15 as the decentered lens to ensure the accuracy of the sub-pixel micro-scanning displacements.

2.2.2 Design results

The infrared micro-scanning system we designed operates in a medium infrared band between 3 μm and 5 μm with a focal length of 223.5 mm and total length of 338 mm. The aperture of the system is f ∕2. There are two aspheric surfaces and four decentered surfaces in our system. As shown in Fig. 4, the third and fifth surfaces are aspheric, and the decentered lens is positioned at four surface positions from surface 12 to surface 15.

Fig. 4 Layout of the infrared micro-scanning lens.

Download Full Size | PDF

To ensure both the accuracy and conciseness of the results, we should sample suitable FOV points in the image plane for evaluation. As shown in Fig. 5, in our evaluation process we sample 11 × 12 points, which are uniformly distributed in the entire image plane. As we have added a random disturbance tolerance at the beginning of the process, the system may no longer be rotationally symmetric. For comprehensiveness and practicability, we simulate four integral half-pixel displacements of (0, −0.0075), (0, 0.0075), (−0.0075, 0), and (0.0075, 0) for the evaluation. As the infrared detector we used in our experiments is 640 × 512 pixles with a pixel size of 15 × 15 μm, the half-pixel displacement should be 7.5 μm. In addition, the default unit in CODE V is millimeter (mm), i.e., 0.0075 mm for a half-pixel displacement.

Fig. 5 Sampling points for the micro-scanning displacements evaluation.

Download Full Size | PDF

To clarify our results, we draw the graph of the displacements for all the 11 × 12 FOV points in Fig. 6, where Figs. 6(a)-6(d) shows the displacements of (0, −0.0075), (0, 0.0075), (−0.0075, 0), and (0.0075, 0), respectively. The black line denotes the ideal half-pixel micro-scanning displacement, and the red circle, blue upper triangle, purple lower triangle, and green diamond denote the sampling points after the integral half-pixel displacements of (0, −0.0075), (0, 0.0075), (−0.0075, 0), and (0.0075, 0), respectively. The abbreviation MS in the graph denotes micro-scanning.

Fig. 6 Displacements for all the 11 × 12 FOV points: (a) (0, −0.0075), (b) (0, 0.0075), (c) (−0.0075, 0), (d) (0.0075, 0).

Download Full Size | PDF

From the results in Fig. 6, it can be observed that the micro-scanning displacements of all the sampling points in the entire image plane are synchronized with an error of less than 0.01 μm, which is 0.13% as compared with the ideal displacements of 7.5 μm. Therefore, we can draw the conclusion that our system satisfies the uniformity of the image displacements of the entire FOV for a practical system. Furthermore, our algorithm has the correction ability for the displacement error in the registration step. Therefore, the effect of the displacement error on our reconstruction result is negligible, and the simulation and experiment results in sections 3 and 4 also demonstrate this.

Figure 7 shows the diffraction MTF of the image after the micro-scanning is performed. As mentioned above, there are 11 × 12 evaluation results for each micro-scanning step. In Fig. 7, we particularize the MTF of five representative FOVs for one micro-scanning step to illustrate the effectiveness of our work. As shown in Fig. 7, the MTF of the system is greater than 0.36 at 32 lp/mm. Therefore, we can conclude from the results that our system can still get good image quality when performing micro-scanning in a practical system.

Fig. 7 Diffraction MTF of the image after micro-scanning.

Download Full Size | PDF

Figure 8 shows the schematic of our infrared micro-scanning optical system. The system mainly comprises four parts: the infrared lens, micro-scanning controller, cooled infrared detector, and computer. The last two decentered pieces of the infrared lens is connected with the micro-scanning controller, and we can realize sub-pixel micro-scanning by operating the software on the computer.

Fig. 8 Schematic of the infrared micro-scanning optical system.

Download Full Size | PDF

3. Simulation results and discussion

In the simulation experiments, we choose the “baboon” and “monarch” from data set Set14 [29] as the ground truth HR images and then perform a sub-pixel shift, blurring, down-sampling, and adding of noise to obtain the degraded LR images.

To demonstrate the reconstruction effectiveness of our method, we use the IBP algorithm, bicubic interpolation algorithm, SRCNN, traditional POCS algorithm, and POCS and denoising separated method for the comparison. As for the POCS and denoising separated method, we have abbreviated the method in which the Wiener filter is applied first followed by the POCS method as wPOCS, and the method in which POCS is applied first followed by the Wiener filter as POCSw. In order to ensure that we are making an appropriate comparison of the results, we use the same parameters and prior knowledge for the traditional POCS algorithm, wPOCS, POCSw, and our FPPOCS method.

We discuss the results from the three perspectives of resolution performance, noise robustness, and the computation cost of our method.

3.1 Resolution performance

In this section, we have employed the 2 × 2, 3 × 3, and 4 × 4 micro-scanning modes to obtain multi-level sub-pixel LR frames. The noise added for the simulation is a Gaussian distribution with a mean value equal to 0 and variance equal to 0.01. We then perform an SR reconstruction for them using our algorithm and assess the final result using the IQA index.

Figure 9 shows the SR results of 2 × 2 magnification for “baboon”, where Fig. 9(a) shows the reference image, Fig. 9(b) shows the ground truth HR image, Figs. 9(c)–9(i) show the SR results of the IBP, bicubic interpolation, SRCNN, traditional POCS, wPOCS, POCSw, and FPPOCS algorithm, respectively. The ground truth HR image is 480 × 500 pixels and 256 grayscale. The simulated reference image is 240 × 250 pixels, and the SR result is 480 × 500 pixels. The value of m is equal to 1.078134 for “baboon”.

Fig. 9 SR results of 2 × 2 magnification for “baboon”: (a) reference image, (b) ground truth HR image, (c) IBP, (d) bicubic interpolation, (e) SRCNN, (f) POCS, (g) wPOCS, (h) POCSw, (i) FPPOCS.

Download Full Size | PDF

It can be observed from Fig. 9 that the IBP method brings in too much noise on the face, while the bicubic interpolation result shows less noise but also causes a loss of some details—the whiskers are nearly indistinguishable. The SRCNN can restore more details of the whiskers but also introduce more noise on the face. The traditional POCS result shows clearer whiskers but there still exists redundant noise on the face. The wPOCS and POCSw results have been over-smoothed because of the rough denoising process. The FPPOCS method can provide both a smooth face and distinguishable whiskers, which is a significantly improved result with less noise and more details.

The image quality of Fig. 9 can be assessed using Eqs. (12)–(16). The corresponding IQA results for “baboon” are shown in Table 1.

Table 1. IQA results for “baboon”.

View Table | View all tables in this article

The IQA results in Table 1 show that the FPPOCS method can provide the best assessment value as compared with the other algorithms, which means that the FPPOCS method can provide the SR result with less noise and more details, and the result has more similarities with the ground truth HR image.

Figure 10 shows the SR results of 2 × 2 magnification for ”monarch”, where Fig. 10(a) is the reference image, Fig. 10(b) is the ground truth HR image, Figs. 10(c)–10(i) show the SR results of IBP, bicubic interpolation, SRCNN, traditional POCS, wPOCS, POCSw, and FPPOCS algorithm, respectively. The ground truth HR image is 512 × 512 pixels and 256 grayscale. The simulated reference image is 256 × 256 pixels, and the SR result is 512 × 512 pixels. The value of m is equal to 1.146757 for “monarch”.

Fig. 10 SR results of 2 × 2 magnification for “monarch”: (a) reference image, (b) ground truth HR image, (c) IBP, (d) bicubic interpolation, (e) SRCNN, (f) POCS, (g) wPOCS, (h) POCSw, (i) FPPOCS.

Download Full Size | PDF

It can be observed from Fig. 10 that the IBP method brings in too much noise into the entire picture, while the bicubic interpolation result has less noise but also causes some loss in details, which makes the image fuzzy. The SRCNN can restore more details but also introduces a lot of noise in the background. The traditional POCS result is clearer for the edge information, but also has some redundant noise in the smooth area. The wPOCS and POCSw results are over-smoothed at the edges because of the rough denoising process. The FPPOCS method can provide a significantly improved result with less noise and more details, which conforms greatly to the ground truth HR image.

The corresponding IQA results for “monarch” are shown in Table 2.

Table 2. IQA results for “monarch”.

View Table | View all tables in this article

The IQA result in Table 2 shows that the FPPOCS method obtains the best assessment value as compared with the other algorithms, which is identical with the result shown in Fig. 10.

Figure 11 shows the IQA charts for “baboon” for various algorithms under various magnifications, where Figs. 11(a) and 11(b) represent the PSNR and SSIM results of “baboon,” respectively. Figure 12 shows the IQA charts of “monarch” for various algorithms under various magnifications, wherein Figs. 12(a) and 12(b) represent the PSNR and SSIM results of “monarch,” respectively. The horizontal axis represents the various magnifications, and the vertical axis represents the IQA. The red dot denotes the result of FPPOCS, the blue square denotes the result of SRCNN, the green diamond denotes the result of bicubic interpolation, the pink star denotes the result of wPOCS, the caramel upper triangle denotes the result of POCSw, the blue lower triangle denotes the result of POCS, and the purple dot denotes the result of IBP.

Fig. 11 IQA charts for “baboon” for various algorithms under various magnifications: (a) PSNR, (b) SSIM.

Download Full Size | PDF

Fig. 12 IQA charts for “monarch” for various algorithms under various magnifications: (a) PSNR, (b) SSIM.

Download Full Size | PDF

From Figs. 11 and 12, we can observe that our FPPOCS method can always obtain the highest PSNR and SSIM results under the magnifications of 2 × 2, 3 × 3, and 4 × 4. The SRCNN result can provide almost the same level of performance as that of our method, while it is sensitive to noise as shown in section 3.2, and its computation time is much longer as shown in section 3.3. The performances of the other algorithms show some irregularity as the IQA metrics or the scene changed, which may be because of their instability. It can also be observed that the IQA results decrease as the magnification increases. Theoretically, we can obtain ideal results for any magnification. However, our experimental results show that the 2 × 2 magnification can provide the best results, which may be because a greater number of moving steps would introduce more errors.

3.2 Noise robustness

We simulated the image down-sampling by 1/2 with noise variances of 0.01, 0.03, 0.05, and 0.1 for the purpose of comparison. The PSNR charts for the various algorithms under different noise levels for “baboon” and “monarch” have been illustrated in Fig. 13, where Figs. 13(a) and 13(b) represent the PSNR for “baboon” and “monarch,” respectively. The horizontal axis represents the noise level, while the vertical axis represents the PSNR. The red dot denotes the FPPOCS result, the blue square denotes the SRCNN result, the green diamond denotes the Bicubic interpolation result, the pink star denotes the wPOCS result, the caramel upper triangle denotes the POCSw result, the blue lower triangle denotes the POCS result, and the purple dot denotes the IBP result.

Fig. 13 PSNR chart for “baboon” and “monarch” for various algorithms under different noise levels: (a) “baboon”, (b) “monarch”.

Download Full Size | PDF

We can draw a conclusion from Fig. 13 that our FPPOCS method can always provide the highest PSNR under various noise levels, and the PSNR value exhibits a small steady decline as the noise level increases, which demonstrates that our method is robust to noise. Furthermore, in the SRCNN method, which exhibits a performance equal to that of our method under a low noise level, the PSNR decreases greatly as the noise level increases.

3.3 Computation cost

In this section, we present a comparison of the computation cost for obtaining the images with a 2 × 2 magnification mode and a noise variance equal to 0.01. Tables 3 and 4 show the computation time to reach convergence for different algorithms for “baboon” and “monarch,” respectively.

Table 3. Computation time for “baboon”.

View Table | View all tables in this article

Table 4. Computation time for “monarch”.

View Table | View all tables in this article

It can be observed from Tables 3 and 4 that our FPPOCS method requires the least time for convergence except in the case of the IBP and bicubic interpolation methods, while the performance of our method is far superior to those of IBP and bicubic interpolation as discussed in section 3.1. In terms of the comparison of the SRCNN and FPPOCS methods, our method requires half the time required by SRCNN for obtaining a similar performance level. As compared with the POCS, wPOCS, and POCSw methods, we can obtain both a better performance and lower computation time despite the fact that the frequency domain requires an extra DFT to be performed during our reconstructions. This is because our infrared micro-scanning optical system can obtain accurate sub-pixel LR images, which can greatly simplify the registration process and, thus, reduce the computation complexity of the FPPOCS method.

In summary, our FPPOCS method can commendably realize image denoising and SR reconstruction for the infrared image within a short computation time.

4. System construction and experiments

Based on the schematic of the system shown in Fig. 8, we construct an infrared micro-scanning optical system as shown in Fig. 14. The infrared detector is 640 × 512 with a pixel size of 15 × 15 μm. The scanning step is 5.54 μm per half-pixel as an optimization result of the optical system.

Fig. 14 Infrared micro-scanning optical system.

Download Full Size | PDF

We perform indoor experiments for an identification plat and outdoor experiments for a building, and we employed the 2 × 2, 3 × 3, and 4 × 4 micro-scanning modes to obtain multi-level sub-pixel LR frames using our optical system.

Figure 15 shows the indoor results of the identification plat of the 2 × 2 magnification. The size of the initial images we obtained is 640 × 512 pixels, and we eliminated the useless black edge pixels and retained the effective image as 256 × 256 pixels. The SR result is 512 × 512 pixels. Figure 15(a) shows the reference image, Figs. 15(b)–15(h) show the SR results of the IBP, bicubic interpolation, SRCNN, traditional POCS, wPOCS, POCSw, and FPPOCS algorithms, respectively. The value of m is equal to 1.022219 for the identification plat.

Fig. 15 Indoor results of the identification plat of 2 × 2 magnification: (a) reference image, (b) IBP, (c) bicubic interpolation (d) SRCNN, (e) POCS, (f) wPOCS, (g) POCSw, (h) FPPOCS.

Download Full Size | PDF

It can be clearly observed from Fig. 15 that the IBP method brings in too much noise and useless information, while the bicubic interpolation result causes less noise but is over-smooth. The SRCNN can restore more details but also introduces edge rings. The traditional POCS result is clearer than the IBP and bicubic interpolation results, but not sufficiently clear because the redundant noise causes the edge to become blurred with the small identification units. The wPOCS and POCSw results have a certain degree of smoothness because of the rough denoising process. The FPPOCS result shows a higher resolution image with relatively distinguishable edges.

Using Eq. (16), the image quality of Fig. 15 can be assessed. The corresponding IQA and computation time results for the identification plat are shown in Table 5.

Table 5. IQA and computation time results for identification plat.

View Table | View all tables in this article

The IQA result in Table 5 shows that the FPPOCS method can provide a smaller NR assessment value than the other algorithms, which means our FPPOCS result contains more ground truth information. The SRCNN result can provide almost the same level of performance as that of our method, while its computation time is twice that of our method. The FPPOCS method requires a little more time than the IBP and bicubic interpolation methods, while the performance of our method is far superior to theirs. As our infrared micro-scanning optical system can obtain accurate sub-pixel LR images, which can greatly simplify the registration process and thus reduce the computation complexity of the FPPOCS method, we can obtain both a better performance and low computation time as compared with the POCS, wPOCS, and POCSw algorithms.

Figure 16 shows the outdoor result of the building with a 2 × 2 magnification. The size of the initial images we obtained is 640 × 512 pixels; for simplification, we eliminate the pixels we do not want and just retain the image as 256 × 256 pixels. The SR result comprises 512 × 512 pixels. Figure 16(a) shows the reference image, while Figs. 16(b)–16(h) show the SR results of the IBP, bicubic interpolation, SRCNN, traditional POCS, wPOCS, POCSw, and FPPOCS algorithms, respectively. The value of m is equal to 0.859981 for the building.

Fig. 16 Outdoor result of the building of 2 × 2 magnification: (a) reference image, (b) IBP, (c) bicubic interpolation, (d) SRCNN, (e) POCS, (f) wPOCS, (g) POCSw, (h) FPPOCS.

Download Full Size | PDF

The outdoor images we used have low gray levels and low contrasts, and thus, we include a contrast enhancement algorithm in our method. It can be observed from Fig. 16 that the IBP result has aliased the information at the edge of the image and exhibits serious distortion, while the bicubic interpolation result exhibits over-smoothness. The SRCNN can restore more details but also introduces some blur. The traditional POCS result is clearer but still introduces redundant noise in the smooth area and background region. The wPOCS and POCSw results show a certain degree of smoothness at the edges because of the rough denoising process. The FPPOCS method can provide a much better result, not only with distinguishable details but also with a sufficiently smooth background area.

The corresponding IQA and computation time results for the building are shown in Table 6.

Table 6. IQA and computation time results for the building.

View Table | View all tables in this article

The IQA result in Table 6 shows that the FPPOCS method exhibits a better assessment value than the other algorithms. The SRCNN method almost exhibits a similar assessment value as our method, while its computation time is twice that of ours. Our FPPOCS method required a little more time than the IBP and bicubic interpolation methods, but the performance of our method is far superior to theirs. As our infrared micro-scanning optical system can obtain accurate sub-pixel LR images, which can greatly simplify the registration process and thus reduce the computation complexity of the FPPOCS method, we can obtain a better performance at a shorter computation time as compared with the POCS, wPOCS, and POCSw algorithms. Thus, we can conclude that our FPPOCS method can realize image denoising and SR reconstruction for infrared images commendably with little computation.

Figures 17(a) and 17(b) represent the IQA of the identification plat and the building, respectively. The horizontal axis represents various magnifications, while the vertical axis represents the IQA. The red dot denotes the result of the FPPOCS method, the blue square denotes the SRCNN result, the green diamond denotes the bicubic interpolation result, the pink star denotes the wPOCS result, the caramel upper triangle denotes the POCSw result, the blue lower triangle denotes the POCS result, and the purple dot denotes the IBP result.

Fig. 17 IQA chart of the identification plat and the building for different algorithms under different magnifications: (a) identification plat, (b) building.

Download Full Size | PDF

From Fig. 17, it can be observed that our FPPOCS method can always provide the smallest (which means the best) IQA results under the magnification of 2 × 2, 3 × 3, and 4 × 4. The SRCNN result can almost provide the same performance as that of our method, while its computation time is much longer as shown in the former discussion. The performance of the other algorithms show some irregularity as the IQA metrics or the scene changed, which may be because of their instability. We can also observe that the IQA results decrease as the magnification increases because of the errors introduced by the increase in the moving steps.

5. Conclusions

In conclusion, we propose a FPPOCS SR algorithm based on an infrared micro-scanning optical system. The FPPOCS SR algorithm takes full advantage of the texture details and contrast-independent feature of the phase information in the frequency domain and can reduce the noise in the image and obtain more details of the infrared image in the SR reconstruction result. We also propose the use of an IQA method based on the frequency domain phase spectrum, which can reflect the SR reconstruction effectiveness in the frequency domain. An infrared micro-scanning lens has been designed, which is used to construct an infrared micro-scanning optical system and realize the infrared image SR reconstruction using our proposed algorithm. The infrared micro-scanning optical system we constructed can guarantee both displacement uniformity of the entire FOV and ideal image quality in the image plane with the use of the micro-scanning system; also, it can realize target controllable sub-pixel micro-scanning of an arbitrary step size by moving two light and compact pieces instead of the entire lens, sensor array, or sample as in the traditional method. Using our proposed algorithm and our infrared micro-scanning optical system, we can realize image denoising and SR reconstruction for the infrared image commendably with little computation. In this paper, we obtain the best results when the magnification is 2 × 2. We focus our further studies on obtaining a better performance with our method in the case of other magnifications.

Funding

State Key Laboratory of Applied Optics.

References

1. R. E. Fischer, B. Tadic-Galeb, P. R. Yoder, R. Galeb, B. C. Kress, S. C. McClain, T. Baur, R. Plympton, B. Wiederhold, and B. G. Alastair J., Opt. Syst. Des., 61–95 (McGraw Hill: 2000).

2. S. Bagavathiappan, B. Lahiri, T. Saravanan, J. Philip, and T. Jayakumar, “Infrared thermography for condition monitoring–A review,” Infrared Phys. Technol. 60, 35–55 (2013). [CrossRef]

3. C. Balaras and A. Argiriou, “Infrared thermography for building diagnostics,” Energy Build. 34(2), 171–183 (2002). [CrossRef]

4. B. Lahiri, S. Bagavathiappan, T. Jayakumar, and J. Philip, “Medical applications of infrared thermography: a review,” Infrared Phys. Technol. 55(4), 221–235 (2012). [CrossRef]

5. J. A. Kennedy, O. Israel, A. Frenkel, R. Bar-Shalom, and H. Azhari, “Super-resolution in PET imaging,” IEEE Trans. Med. Imaging 25(2), 137–147 (2006). [CrossRef] [PubMed]

6. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20(3), 21–36 (2003). [CrossRef]

7. M. Protter, M. Elad, H. Takeda, and P. Milanfar, “Generalizing the nonlocal-means to super-resolution reconstruction,” IEEE Trans. Image Process. 18(1), 36–51 (2009). [CrossRef] [PubMed]

8. X. Hu, S. Peng, and W.-L. Hwang, “Learning adaptive interpolation kernels for fast single-image super resolution,” Signal Image Video Process. 8(6), 1077–1086 (2014). [CrossRef]

9. H. Chen, J. Xue, S. Zhang, Y. Lu, and C. Guo, “Image super-resolution based on adaptive cosparse regularisation,” Electron. Lett. 50(24), 1834–1836 (2014). [CrossRef]

10. A. V. Kanaev and C. W. Miller, “Multi-frame super-resolution algorithm for complex motion patterns,” Opt. Express 21(17), 19850–19866 (2013). [CrossRef] [PubMed]

11. H. Cui, W. Zhao, Y. Wang, Y. Fan, L. Qiu, and K. Zhu, “Improving spatial resolution of confocal Raman microscopy by super-resolution image restoration,” Opt. Express 24(10), 10767–10776 (2016). [CrossRef] [PubMed]

12. D. Keren, S. Peleg, and R. Brada, Image sequence enhancement using sub-pixel displacements. in Computer Vision and Pattern Recognition,1988. Proceedings CVPR'88,Computer Society Conference. 1988. IEEE.

13. C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016). [CrossRef] [PubMed]

14. W. Luo, Y. Zhang, A. Feizi, Z. Göröcs, and A. Ozcan, “Pixel super-resolution using wavelength scanning,” Light Sci. Appl. 5(4), e16060 (2016). [CrossRef] [PubMed]

15. A. J. Patti, M. I. Sezan, and A. Murat Tekalp, “Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time,” IEEE Trans. Image Process. 6(8), 1064–1076 (1997). [CrossRef] [PubMed]

16. J. Fortin and P. C. Chevrette, “Realization of a fast microscanning device for infrared focal plane arrays,” in Infrared Imaging Systems: Design, Analysis, Modeling, and Testing VII, International Society for Optics and Photonics (1996).

17. X.-S. Wu, F.-Y. Deng, M. Chen, and Y. Cai, “Development of rotary infrared micro-scanner,” J. Infrared Millim. W. 30(3), 263–267 (2011). [CrossRef]

18. A. V. Oppenheim and J. S. Lim, “The importance of phase in signals,” Proc. IEEE 69(5), 529–541 (1981). [CrossRef]

19. P. Kovesi, “Image features from phase congruency,” Videre: J. Comp. Vision Res. 1(3), 1–26 (1999).

20. R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital image processing using MATLAB (Gatesmark Publishing, 2009).

21. J. Chen, W. Wang, T. Liu, Z. Zhang, and H. Gao, “A POCS super resolution restoration algorithm based on BM3D,” Sci. Rep. 7(1), 15049 (2017). [CrossRef] [PubMed]

22. R. C. Hardie and K. J. Barnard, “Fast super-resolution using an adaptive Wiener filter with robustness to local motion,” Opt. Express 20(19), 21053–21073 (2012). [CrossRef] [PubMed]

23. H. R. Sheikh, M. F. Sabir, and A. C. Bovik, “A statistical evaluation of recent full reference image quality assessment algorithms,” IEEE Trans. Image Process. 15(11), 3440–3451 (2006). [CrossRef] [PubMed]

24. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. Image Process. 21(12), 4695–4708 (2012). [CrossRef] [PubMed]

25. R. Dosselmann and X. D. Yang, “Existing and emerging image quality metrics,” in Electrical and Computer Engineering, 2005. Canadian Conference on, 1906–1913 (2005), IEEE. [CrossRef]

26. A. Medda and V. DeBrunner, “Color image quality index based on the UIQI,” in Image Analysis and Interpretation, 2006 IEEE Southwest Symposium on, 213–217 (2006), IEEE. [CrossRef]

27. A. Mittal, R. Soundararajan, and A. C. Bovik, ““Making a” Completely Blind” Image Quality Analyzer,” IEEE Signal Process. Lett. 20(3), 209–212 (2013). [CrossRef]

28. D. L. Ruderman, “Statistics of natural images,” Network-Comp. Neural 5(4), 517–548 (1994) [CrossRef] [PubMed]

29. R. Zeyde, M. Elad, and M. Protter, “On Single Image Scale-Up Using Sparse-Representations,” in International Conference on Curves and Surfaces. 2012. [CrossRef]

	IBP	Bicubic	SNCNN	POCS	wPOCS	POCSw	FPPOCS
PSE	1.844484	1.828542	1.838987	1.827867	1.845019	1.840654	1.819197
MSE	537.840322	288.015333	268.408306	287.979036	315.645202	298.4579	232.897521
PSNR	18.891484	21.241493	21.867169	21.251758	21.276992	21.048990	22.759829
SSIM	0.314292	0.561205	0.571784	0.572358	0.503604	0.544408	0.716952
D	11.8111	9.1973	7.0764	9.1762	6.1681	4.9314	4.4488

	IBP	Bicubic	SRCNN	POCS	wPOCS	POCSw	FPPOCS
PSE	1.862878	1.855213	1.678952	1.841352	1.802255	1.855750	1.829220
MSE	508.756645	227.214325	168.408306	217.562123	217.781099	279.334974	166.282285
PSNR	21.065703	23.566447	25.867169	24.815327	24.750602	23.669550	25.922344
SSIM	0.343766	0.591413	0.771784	0.623587	0.7803527	0.744293	0.837680
D	11.9162	10.1922	6.3591	9.9325	4.8002	4.3690	4.2684

	IBP	Bicubic	SRCNN	POCS	wPOCS	POCSw	FPPOCS
D	12.2081	9.5796	5.7742	10.3738	10.3182	6.4242	5.8462
Time	4.890047	4.845262	14.553802	10.103584	10.460739	10.763091	5.907446

	IBP	Bicubic	SRCNN	POCS	wPOCS	POCSw	FPPOCS
D	12.0011	9.1842	7.3498	11.8422	10.4899	9.3363	6.8241
Time	3.941580	4.310526	12.766742	8.799650	9.020304	8.696297	5.242737

	IBP	Bicubic	SNCNN	POCS	wPOCS	POCSw	FPPOCS
PSE	1.844484	1.828542	1.838987	1.827867	1.845019	1.840654	1.819197
MSE	537.840322	288.015333	268.408306	287.979036	315.645202	298.4579	232.897521
PSNR	18.891484	21.241493	21.867169	21.251758	21.276992	21.048990	22.759829
SSIM	0.314292	0.561205	0.571784	0.572358	0.503604	0.544408	0.716952
D	11.8111	9.1973	7.0764	9.1762	6.1681	4.9314	4.4488

Super-resolution imaging for infrared micro-scanning optical system

Abstract

1. Introduction

2. Micro-scanning SR imaging theory

2.1 FPPOCS algorithm

2.1.1 Conventional POCS theory

2.1.2 FPPOCS theory

2.1.3 Image quality assessment

2.2 Design of infrared micro-scanning optical system

2.2.1 Technical design principles

2.2.2 Design results

3. Simulation results and discussion

3.1 Resolution performance

3.2 Noise robustness

3.3 Computation cost

4. System construction and experiments

5. Conclusions

Funding

References

Cited By

Figures (17)

Tables (6)

Equations (16)

Optics Express