Open Access
Add to BibSonomy
Abstract
As a super-resolution imaging method, high-resolution medium wave infrared (MWIR) images can be obtained from a low-resolution focal plane array-based (FPA) sensor using compressive imaging (CI) technology. As a common problem in MWIR FPA imaging, the non-uniformity reduces image quality, which is turning worse in MWIR FPA CI. This paper investigates the source of the non-uniformity of MWIR FPA CI, both in the captured low-resolution MWIR images and in the reconstructed high-resolution ones. According to the system model and the image super-resolution computation process of FPA CI, we propose a calibration-based non-uniformity correction (NUC) method for MWIR FPA CI. Based on the actual MWIR FPA CI system, the effectiveness and practicability of the proposed NUC method are verified, obtaining better results than the traditional method. According to the theoretical analysis and experimental results, the particularities of the non-uniformity in MWIR FPA CI are discovered and discussed, which have certain great guiding significance and practical value.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
As a hotspot in the scientific research and engineering application field, imaging technology enables humans to see the world beyond their innate capability [1,2]. Although vision is one of the most important human senses, only a very small window, spanning 0.4∼0.7 µm in wavelength [3], is available for human eyes. Because objects at the temperature above absolute zero emit medium wave infrared (MWIR) radiation, MWIR focal plane array-based (FPA) imaging has the potential to enhance the cognitive ability of the world [4], especially in the darkness and smog [5]. Due to these outstanding properties, MWIR FPA imaging plays an important role in many engineering fields, such as environmental monitoring [6,7], astronomical observation [8,9], medical diagnostics [10,11] and target recognition [12]. As MWIR imaging system performance is closely related to the resolution of FPA sensor, to obtain more image details, a high-resolution MWIR FPA sensor is desired. However, the more detectors the MWIR FPA sensor contains, the higher price the purchaser pays [13–15], which exceeds the purchasing capacity of ordinary scientists and engineers.
Compressive sensing (CS) provides a method to capture a compressible signal at a rate which is significantly below the Nyquist rate [16–18]. Compressive imaging (CI), which is the combination of imaging technology and CS, makes it possible for a low-resolution FPA sensor to obtain high-resolution MWIR images. Mahalanobis of Lockheed Martin built the first MWIR FPA CI system [19], which is “merged-type”, and we have built a “separate-type” MWIR FPA CI system [20].
Based on the actual system, the feasibility of FPA CI for MWIR image super-resolution has been proved by the experimental results. However, compared with visible light, there are still some special problems left unresolved in MWIR FPA CI. For MWIR FPA sensor, each detector has its inherent radiation response function with different gain and offset, which leads to the non-uniformity in the acquired images [21]. Due to the non-uniformity, fixed pattern noise (FPN) is presented in the MWIR images, degrading the imaging quality and radiometric accuracy. In order to remove FPN from images, non-uniformity correction (NUC) must be applied to MWIR FPA imaging. For a MWIR FPA imaging system, NUC algorithms can be classified into two categories: the calibration-based method [22,23] and the scene-based method [24,25]. The calibration-based technique is the simplest and most accurate NUC method, so it is commonly used in MWIR FPA imaging systems [26], containing single point correction (SPC), two point correction (TPC) and multiple point correction (MPC). The scene-based methods have advantages over the calibration-based methods in some cases, but scene-based methods require numerous image sequences for the parameter estimation [21].
Unfortunately, these traditional NUC algorithms cannot be directly applied to MWIR FPA CI, because the final MWIR image is computed from multiple low-resolution images, rather than captured by a MWIR FPA sensor. As a result, the pattern noise of the final MWIR image caused by the non-uniformity of MWIR FPA CI is not fixed, because the non-uniformity relates not only with the inherent radiation response function of each detector, but also with the image super-resolution computation process.
This paper investigates the source of the non-uniformity of MWIR FPA CI, both in the captured low-resolution MWIR images and in the reconstructed high-resolution ones. The non-uniformity of the reconstructed high-resolution MWIR images is a combination of ${({\cos } )^4}$ vignetting and “blocky” structural artifacts (BSA). According to the image super-resolution computation process, a calibration-based NUC method is proposed for MWIR FPA CI. The choice of calibration-based technique in NUC is fueled by its simplicity and accuracy. Based on the actual system, a series of experiments are conducted, which shows that the proposed NUC algorithm can effectively correct the non-uniformity and suppress the BSA in the final high-resolution MWIR image, improving the imaging quality.
This paper is organized as follows. Section 2 describes the architecture of the MWIR FPA CI system, and then presents the non-uniformity of MWIR FPA CI. Section 3 analyzes the causes of the non-uniformity in MWIR FPA CI, and describes the proposed NUC algorithm in detail. Section 4 gives experimental results obtained by the actual MWIR FPA CI system and shows the improvement of imaging quality by the proposed NUC algorithm. Finally, Section 5 makes a conclusion for this work.
2. Non-uniformity in MWIR FPA CI
The schematic diagram of our MWIR FPA CI system is depicted in Fig. 1. Object is first imaged onto the digital micro-mirror device (DMD) through the imaging lens, then reflected onto the MWIR FPA sensor through the relay lens. We utilize the DMD (DLP9500, Texas Instruments, USA) as a spatial light modulator, which consists of 1920 × 1080 micro-mirrors. Each micro-mirror can be independently rotated to either +12° or −12° around the normal vector of DMD. When the rotation angle of a micro-mirror is −12°, the MWIR radiance incident onto the micro-mirror is discarded, which is absorbed by the cooling board. When the rotation angle is +12°, the corresponding MWIR radiance of the object is reflected and focused on the MWIR FPA sensor using the relay lens. To minimize the imaging noise, a cooled 640 × 512 MWIR FPA sensor (LEO MW, Sofradir, France) is used as the imaging device, acquiring low-resolution MWIR images for image super-resolution. In the view of system design, DMD is placed at the focal plane of the image lens, modulating the image of the object by controlling the rotation angles of the micro-mirrors. The modulated object image is captured by the MWIR FPA sensor. In this way, the DMD can be deemed as a programmable high-resolution binary mask. Based on the image reconstruction algorithm, the MWIR image resolution, which is the number of effective pixels, is improved to the number of the micro-mirrors in the DMD, from the number of detectors contained by the FPA sensor.
Fig. 1. (a) Schematic diagram of our MWIR FPA CI system, including an imaging lens, a DMD, a relay lens, a MWIR FPA sensor and a cooling board. (b) Photograph of the actual system.
Considering the influence of the lens aberrations, only the square central regions of DMD and MWIR FPA sensor are utilized for the imaging process. In our actual laboratory setup, the effective regions of DMD and MWIR FPA sensor are 1280 × 1024 and 320 × 256. Therefore, each block of 4 × 4 micro-mirrors on the DMD maps to one detector of the MWIR FPA sensor, realizing 16 times MWIR image super-resolution. From the perspective of system design, the term “block” refers to a group of micro-mirrors on the DMD that address a region of the image scene, and then are ideally mapped to a single detector on the FPA sensor. In addition, in FPA CI, the “block” can also refer to a group of pixels in the reconstructed high-resolution image, which reconstructed from a series of image compressive sampling values of the single detector, and to an area of the DMD masks, which is applied to the micro-mirror block.
A blackbody (CDS 100-4, EOI, USA) is placed in front of the imaging lens, acting as the imaged object. The blackbody is planar type, so the MWIR radiation intensity incident into the imaging lens is considered to be uniform. Ideally, every pixel has the same gray value in the reconstructed high-resolution image, which is also the goal of NUC for a MWIR imaging system.
In our CI process, all the blocks have the same patterns in the DMD masks, and revised SPL algorithm is chosen as the image reconstruction algorithm, which has been described in details in [20]. According to the number of the low-resolution images used for image reconstruction, the compression ratio $\alpha $ of a FPA CI system can be defined as
Fig. 2. The non-uniformity in the reconstructed high-resolution images of the MWIR FPA CI system. In (a)-(c), the compression ratios are 0.125, 0.25 and 0.375, respectively. The same area of all image are zoomed in to highlight the non-uniformity. The temperature of the blackbody is 100°C.
Not only in the reconstructed high-resolution images, but the non-uniformity also exists in the low-resolution images. For a traditional MWIR imaging system, vignetting can be observed, attenuating from the center to the corner. This is caused by the ${({\cos } )^4}$ variation in MWIR radiance incident onto the FPA sensor, which is relative to the optic axis of the imaging system [27]. For a MWIR FPA CI system, besides the vignetting, aperture interference phenomenon aggravates the non-uniformity, as the result of the size of micro-mirror (10.8µm×10.8µm) and the imaging band (3.7µm∼4.8µm) are in the same order. Compared with traditional MWIR imaging systems, the non-uniformity of MWIR FPA CI system is special due to the combination of ${({\cos } )^4}$ vignetting and aperture interference. Figure 3 shows the non-uniformity in the low-resolution images obtained directly from the MWIR FPA sensor. The DMD mask patterns are depicted on the left in the first row, where the micro-mirror is “opened” when the rotation angle is +12°, and is “closed” when the angle is −12°. The corresponding low-resolution MWIR images are placed on the right in the first row. To highlight the gray level distributions of these low-resolution MWIR images, different colors are used in the colormap. The gray level ranges are (90, 160) for Fig. 3(a) and (50, 100) for Figs. 3(b)–3(f). It is obvious that different DMD mask patterns lead to distinct distributions of light and dark areas in the MWIR images.
Fig. 3. The non-uniformity in the low-resolution images obtained directly from the MWIR FPA sensor. In the first row of (a) - (f), the DMD mask patterns are on the left, and the corresponding MWIR images are on the right. In the second row, color-mapped MWIR images are used to represent the distributions of light and dark areas in the original image. In the DMD mask patterns, white squares stand for “opened” and grey squares for “closed”. The temperature of the blackbody is 100°C.
3. NUC for MWIR FPA CI
In this section, we first analyze the sources of non-uniformity in MWIR FPA CI, including the low-resolution MWIR images and the high-resolution ones. From the system model, it is obvious that the non-uniformity in the reconstructed high-resolution MWIR images is caused by the non-uniformity of the low-resolution images. Then, aimed at the particularity of MWIR FPA CI, a calibration-based NUC method is proposed. Compared with the traditional NUC methods, the proposed method treats the low-resolution images obtained from different DMD masks as a whole and realizes the NUC of all low-resolution images comprehensively, instead of processing independently. At last, we discuss the reasonableness of NUC before the image reconstruction. Depending on the theoretical analysis, the method of NUC after the image reconstruction is proved to be undesirable.
3.1 Source of the non-uniformity
For MWIR imaging systems, the ${({\cos } )^4}$ vignetting is a common phenomenon. However, in our MWIR FPA CI system, object radiance is not directly projected onto the FPA sensors. The 12° between the DMD plane and the “opened” micro-mirrors causes the displacement between the brightest region of MWIR image and the center of FPA sensor, which is shown in Fig. 3(a). In addition, aperture interference phenomenon leads to different non-uniformities in the low-resolution MWIR images obtained by different DMD masks.
As a practical rule-of-thumb, Fraunhofer diffraction will occur at an aperture, if
where, b is the greatest width of the aperture, and $\lambda $ is the imaging wavelength. R is the smaller of the two distances, of which one is from light sources to the aperture, and the other is from the aperture to the imaging plane [28].For our MWIR FPA CI system, each micro-mirror can be regarded as an aperture, of which the width is 10.8µm, and the central wavelength of MWIR used for imaging is 4.3µm. As b and $\lambda $ are in the same order, aperture interference phenomenon is inevitable.
According to the relationship among DMD plane ${X_1}{O_1}{Y_1}$, FPA sensor plane $XOY$ and the relay lens, the MWIR radiance obtained by the FPA sensor at the point $({x, y} )$ can be calculated by the formula Fraunhofer diffraction [29]
According to Eq. (3), it is obvious that in low-resolution MWIR images, the aperture interference phenomenon depends mostly on the DMD mask patterns. In particular, multi-slit interference occurs when the “opened” micro-mirrors are aligned into parallel lines, as shown in Figs. 3(b)–3(e).
For the reconstructed high-resolution image, the gray values of all pixels are computed from the low-resolution MWIR images. Therefore, the non-uniformity of the high-resolution MWIR image comes mainly from the computation process of image reconstruction, instead of the intrinsic characteristics of MWIR FPA CI system. As FPA CI system can be viewed as an array of single pixel cameras [30–32], the system has the matrix notation as
where, for the ith detector of MWIR FPA sensor, ${y_i} \in {R^m}$ is the set of m compressed samples, ${x_i} \in {R^n}$ represents the high-resolution image block, and ${\Phi _i} \in {R^{m \times n}}$ is the measurement matrix for the block.In the practical computation, ${y_i}$ is consisted of the ith pixel of all low-resolution MWIR images, and each row of ${\Phi _i}$ is corresponding to the ith block pattern of the DMD mask. For the DMD mask modulation, if the micro-mirror is “opened”, the corresponding element is “1” in ${\Phi _i}$, otherwise “0” is used. For example, if the DMD masks of Figs. 3(a)–3(d) are used for image reconstruction, Eq. (5) can be rewritten as
Whatever image reconstruction algorithm is selected, the essence is solving the underdetermined equations, but the detailed process is different. If ${y_{i,1}} = 2 \cdot {y_{i,2}} = 2 \cdot {y_{i,3}} = 2 \cdot {y_{i,4}}$, all the pixels of the ith block in the reconstructed high-resolution have the same gray value, that is ${x_{i,1}} = {x_{i,2}} = \cdots = {x_{i,16}}$. However, for the ith block, as the non-uniformities of the low-resolution MWIR images are not exactly the same with each other, the values of the compressed samples ${y_i}$ may vary from one element to another. Not satisfied with the ideal numerical relationship among the compressed samples, BSA is generated in all block of the reconstructed high-resolution MWIR image, which can be observed in Fig. 2(b). This theoretical derivation can be extended easily to other compression ratios, as shown in other figures of Fig. 2.
What's more, the overall trend of ${({\cos } )^4}$ vignetting is also presented in the reconstructed high-resolution MWIR image. As a result, the non-uniformity of the reconstructed high-resolution MWIR image consists mainly of two parts: the BSA and the ${({\cos } )^4}$ vignetting. The ${({\cos } )^4}$ vignetting is universal for a traditional MWIR imaging system, but the BSA is unique for FPA CI system. In particular, the BSA fatally destroys the image details and seriously reduces the image quality of the reconstructed high-resolution MWIR images.
3.2 Calibration-based NUC method for MWIR FPA CI
Although the non-uniformities exist in both the captured low-resolution MWIR images and the reconstructed high-resolution ones, however, the motive of MWIR FPA CI is image super-resolution, so the non-uniformity of MWIR FPA CI is defined as the one of the reconstructed high-resolution MWIR image. As the most common NUC method used to correct the non-uniformity of MWIR imaging system, the calibration-based technique is the simplest and most accurate [26], so it is selected as the NUC method used for our MWIR FPA CI system.
If a planar type blackbody is used to correct the non-uniformity of the reconstructed high-resolution MWIR image, realizing all pixels have the same gray value, all the low-resolution MWIR images obtained from different DMD masks should be processed together as a group, not separately. The trick is, the MWIR FPA CI system with different DMD masks should be treated as a group of separate imaging systems, and the corrected gray value of each system is determined by the number of “opened” micro-mirrors of the DMD mask. Specifically, due to aperture interference phenomenon depends heavily on the pattern of the DMD masks, the MWIR FPA CI system presents varying forms of non-uniformities when different DMD masks are used. As the non-uniformities can be seen as the inherent characteristics of the MWIR FPA CI system with different DMD mask patterns, the system with one DMD mask pattern should be distinguished from that with another pattern. However, according to Eqs. (5) and (6), the pixels at the same position of all low-resolution MWIR images cannot be corrected to arbitrary values, if the identical gray value filled into the corresponding block of the high-resolution image is desired. The corrected gray values for the low-resolution MWIR images ought to be in proportion to the “opened” micro-mirrors of the used DMD masks.
As the most commonly and widely used method of the calibration-based technique [26,33], TPC is selected as the NUC method for our MWIR FPA CI system. The TPC method for MWIR FPA CI, which can be reduced to SPC or extend to MPC with little effort, is described as follows and can be performed in seven steps:
- (1) With different masks, capture the low-resolution MWIR images by the MWIR FPA CI system at temperature ${T_1}$. These low-resolution images are labeled as ${I_{T1,mask1}}$, ${I_{T1,mask2}}$,…, ${I_{T1,mask\textrm{n}}}$, where the first subscript means that these image are captured at temperature ${T_1}$ and the second subscript shows the mask number, from $mas{k_1}$ to $mas{k_n}$.
- (2) Calculate the mean values of ${I_{T1,mask1}}$,$${\bar{G}_{T1,mask1}} = \frac{1}{{N \times M}}\sum\limits_{i = 1}^N {\sum\limits_{j = 1}^M {{I_{T1,mask1}}({i,j} )} } $$where, ${\bar{G}_{T1,mask1}}$ is the mean gray value for the low-resolution MWIR image ${I_{T1,mask1}}$, and ${I_{T1,mask1}}({i,j} )$ is the gray value of the pixel $({i,j} )$ in ${I_{T1,mask1}}$. N and M are the numbers of rows and columns in each MWIR image.
- (3) Set ${\bar{G}_{T1,mask1}}$ as the reference point of gray value for temperature ${T_1}$. According to Eqs. (5) and (6), compensate the expected gray value for each images,$${{G_{T1,mask \ast }} = \frac{{O{P_{mask \ast }}}}{{O{P_{mask1}}}}{{\bar{G}}_{T1,mask1}}} \quad , \quad{ \ast = 1,\ldots ,n}$$where, ${G_{T1,mask \ast }}$ is the expected gray value for the low-resolution MWIR image ${I_{T1,mask \ast }}$, $O{P_{mask \ast }}$ is the number of the “opened” micro-mirrors in one block of the DMD mask $mas{k_ \ast }$, and $O{P_{mask1}}$ is just for $mas{k_1}$.
The reference point for temperature ${T_1}$ can be set as ${\bar{G}_{T1,mask \ast }}$, ${\ast}{=} 2,\ldots ,n$, as long as ${G_{T1,mask \ast }}$ does not exceed the limit of gray level in the MWIR image.
- (4) Complete the same work of step (1) to (3) at temperature ${T_2}$, and capture the low-resolution MWIR images ${I_{T2,mask \ast }}$, as well as calculate the expected gray value ${G_{T2,mask \ast }}$, where ${\ast}{=} 1,\ldots ,n$.
- (5) For the different DMD masks, calculate the corrected gain factors of all pixels,$$ {{g_{mask \ast }}(i,j) = \frac{{{G_{T1,mask \ast }} - {G_{T2,mask \ast }}}}{{{I_{T1,mask \ast }}({i,j} )- {I_{T2,mask \ast }}({i,j} )}}} \quad , \quad { \ast = 1,\ldots ,n}$$and corrected offset factors,$$ {{o_{mask \ast }}(i,j) = \frac{{{I_{T1,mask \ast }}({i,j} )\cdot {G_{T2,mask \ast }} - {I_{T2,mask \ast }}({i,j} )\cdot {G_{T1,mask \ast }}}}{{{I_{T1,mask \ast }}({i,j} )- {I_{T2,mask \ast }}({i,j} )}}} \quad , \quad { \ast = 1,\ldots ,n}$$where ${g_{mask \ast }}$ and ${o_{mask \ast }}$ are the corrected gain factor and corrected offset factor for $mas{k_ \ast }$ respectively. $(i,j)$ is the pixel coordinate of the MWIR image.
- (6) For an imaging scene, capture the original low-resolution MWIR images ${I_{ori,mask \ast }}$, ${\ast}{=} 1,\ldots ,n$. Based on the corrected gain factor and corrected offset factor of each DMD mask, get the final corrected low-resolution MWIR image,$${{I_{cor,mask \ast }}({i,j} )= {g_{mask \ast }}(i,j) \cdot {I_{ori,mask \ast }}(i,j) + {o_{mask \ast }}(i,j)} \quad , \quad { \ast= 1,\ldots ,n}$$where, ${I_{cor,mask \ast }}$ is the corrected low-resolution MWIR image, which is corresponded to ${I_{ori,mask \ast }}$.
- (7) Reconstruct the high-resolution MWIR image from the corrected low-resolution images, from ${I_{cor,mask1}}$ to ${I_{cor,maskn}}$, instead of the original ones, from ${I_{ori,mask1}}$ to ${I_{ori,maskn}}$.
3.3 NUC before or after the image reconstruction?
In our proposed NUC method for MWIR FPA CI, the non-uniformity of the low-resolution MWIR images is corrected, before the process of image reconstruction. The attendant question is, can we do the NUC for the reconstructed high-resolution MWIR image, after the image reconstruction? The answer may be negative. There are two main reasons:
First, the NUC after the image reconstruction offends against the physical meaning of NUC for MWIR imaging. The non-uniformity is mainly caused by the optical characteristics of the imaging system and the inherent radiation response functions of all detectors on the FPA sensors. But for CI, in the image reconstruction, the best algorithms inevitably introduce some computational errors in the reconstructed images [34]. Thus, the non-uniformity may vary along the image reconstruction computation, not determined only by the MWIR FPA CI system. The un-FPN caused by the varying non-uniformity make the NUC meaningless. By contrast, NUC before the image reconstruction corrected the non-uniformity of captured low-resolution MWIR images, which is determined only by the system and shows the FPN in the images.
Second, even if an ideal image reconstruction algorithm was proposed, no computational errors were introduced, the NUC after the image reconstruction would not be recommended for MWIR FPA CI. From the viewpoint of engineering application, computing costs and storage space can be significantly reduced by the NUC before the image reconstruction than after. In general computer systems, the corrected gain factor for each pixel, as well as the corrected offset factor, occupies one storage space of the same size. The computational complexity and time are also the same for the correction of all pixels. However, the number of corrected pixels significantly decreases in NUC before the image reconstruction than after, and the decreasing ratio is equal to the compression ratio $\alpha $.
Therefore, in MWIR FPA CI, only the NUC before the image reconstruction is feasible, whether from the view of theoretical analysis or engineering practice.
4. Experiment and discussion
In order to verify the proposed NUC method, we take experiments in the actual MWIR FPA CI system. A blackbody (CDS 100-4, EOI, USA) is used for NUC. In the calibration-based NUC, the temperature ${T_1}$ is set to 100°C, and ${T_2}$ to 60°C. To test the effect of NUC for MWIR FPA CI, the blackbody is set to 80°C.
As shown in Fig. 4, at compression ratio of 0.125, the non-uniformities of the reconstructed high-resolution images obtained by different NUC methods have significant differences. In Fig. 4(a), the reconstructed high-resolution image is computed from the original low-resolution images, obtained directly from the MWIR FPA sensor, without NUC. In the zoomed-in region, obvious BSA is observed. To highlight the non-uniformity of the reconstructed image, the corresponding color-mapped images are show in Fig. 4(d).
Fig. 4. The non-uniformity in the reconstructed high-resolution images, with the compression ratio of 0.125. In (a), the reconstructed image is computed from the original low-resolution images without NUC. One region is zoomed in to highlight the non-uniformity, and the corresponding color-mapped images are shown in (d). Similarly, the reconstructed high-resolution image with the traditional NUC method is depicted in (b) and (e). The reconstructed high-resolution image with the proposed NUC method is shown in (c) and (f).
As a contrast, we use traditional NUC method for the low-resolution images, then reconstruct the high-resolution image. That is, treat the low-resolution MWIR images obtained from different DMD masks separately and omit the third step of the proposed method. In the whole image of Fig. 4(b), the non-uniformity have some improvement than Fig. 4(a), as the brightness variations across the image is mitigated. However, in the zoomed-in regions, BSA can also be observed. The color-mapped images are show in Fig. 4(e).
In Figs. 4(c) and 4(f), the reconstructed high-resolution image obtained by the proposed NUC method is depicted. In the zoomed-in regions of Fig. 4(c), the BSA is remarkably suppressed compared with Figs. 4(a) and 4(b).
In addition, for the color-mapped images of Fig. 4, the gray level ranges are (50, 130) and (85, 120) for the whole images and the zoomed-in regions respectively.
Intuitively the effect of NUC in the reconstructed high-resolution images is best in our proposed method. In the traditional NUC method, the non-uniformities of the whole image is mitigated than that in the image without NUC, in spite of the negligible improvement in the zoomed-in regions. The differences be easily observed in the color-mapped images. Because the color mapping ranges are the same, the non-uniformities of the reconstructed images are directly reflected by the color distributions.
This trend is also reflected in the compression ratios of 0.25 and 0.375, which is shown in Figs. 5 and 6.
Fig. 5. The non-uniformity in the reconstructed high-resolution images, with the compression ratio of 0.25. In (a) and (d), the reconstructed image is computed from the original low-resolution images without NUC. (b) and (e) show the reconstructed high-resolution image with the traditional NUC method. (c) and (f) are the image obtained by the proposed NUC method.
Fig. 6. The non-uniformity in the reconstructed high-resolution images, with the compression ratio of 0.375. In (a) and (d), the reconstructed image is computed from the original low-resolution images without NUC. (b) and (e) are the image obtained by the traditional NUC method. (c) and (f) show the reconstructed high-resolution image with the proposed NUC method.
Figure 5(a) shows the reconstructed high-resolution image without NUC, in which BSA is obvious. With our proposed NUC method, BSA is effectively suppressed as shown in Fig. 5(c). In Fig. 5(b), with the traditional NUC method, the non-uniformity of the whole image is similar with that of Fig. 5(c), but in the zoomed-in region, BSA is also obvious. In the color-mapped images, the gray level ranges are (40, 140) and (80, 130) for the whole images and the zoomed-in regions respectively.
Figure 6 shows the reconstructed high-resolution image at the compression ratios of 0.375. The gaps of the non-uniformities among the different NUC methods are increasing, which is depicted in the zoomed-in regions of Figs. 6(a)–6(c). In the color-mapped images, the gray level ranges of the whole images and the zoomed-in regions are (30, 150) and (60, 140) respectively.
In order to objectively evaluate the non-uniformities in the reconstructed high-resolution MWIR images, the evaluation parameter —— non-uniformity $({NU} )$ [35] is used, which is defined as
In a traditional MWIR FPA imaging system, if the temperature sampling interval, that is $|{{T_1} - {T_2}} |$, is shortened, the non-uniformity of the captured MWIR images can get more improvement [33]. This trend is also reflected in MWIR FPA CI system. In order to prove that, based on our proposed NUC method, we test another temperature sampling interval, which is shorter than that for the previous experiments of Figs. 4–6. The temperature ${T_1}$ is set to 90°C and ${T_2}$ to 70°C, while the blackbody is still set at 80°C. The high-resolution images computed with the proposed NUC method are shown in Fig. 8.
Fig. 8. The non-uniformity in the reconstructed high-resolution images, with the temperature sampling interval of 20°C. In (a) and (d), the reconstructed image is computed at the compression ratio of 0.125. For (b) and (e), the compression ratio is 0.25, and for (c) and (f), that is 0.375. All of these high-resolution images are computed with the proposed NUC method.
In Fig. 8(a), the image is computed at the compression ratio of 0.125, and the corresponding color-mapped image is shown Fig. 8(d). The gray level ranges of the whole image and the zoomed-in region are (50, 130) and (85, 120), which are the same with that in Fig. 4. In Fig. 8(b), the image is computed at the compression ratio of 0.25, and the corresponding color-mapped image is shown in Fig. 8(e). The gray level ranges are (40, 140) and (80, 130) for the whole image and the zoomed-in region, which are consistent with Fig. 5. Similarly, Figs. 8(c) and 8(f) is the reconstructed image at the compression ratio of 0.375, and the gray level ranges are the same with Fig. 6.
By the comparison between Fig. 8 and the third columns of Figs. 4–6, we can observe that the improvement is not obvious in zoomed-in regions, but for the whole images, the color distribution is flatter. Thus, the non-uniformities of the reconstructed high-resolution MWIR images, computed at the compression ratio of 0,125, 0.25 and 0.375, have a small improvement when temperature sampling interval is shortened.
To objectively reflect the improvement of the reconstructed high-resolution images with the proposed NUC method at temperature sampling interval of 20°C than that of 40°C, we use non-uniformity of Eq. (12) to show the difference between these two different temperature sampling intervals. From Fig. 9, we can conclude that if the temperature sampling interval is shortened, the non-uniformity of the reconstructed high-resolution MWIR images can get more improvement. However, to correct the same temperature interval, more temperature sampling points for NUC are required when the temperature sampling interval is shortened. What is more, MPC is also required for the NUC instead of TPC, which significantly increases computing costs and storage space [26].
To prove the applicability of the proposed NUC method, a transmission type of United States Air Force (USAF) resolution test chart is used as the imaging object. The result of MWIR image super-resolution is shown in Fig. 10. The images in the first row are reconstructed at the compression ratio of 0.125, and those in the second row are at 0.25. In Figs. 10(a) and 10(d), the high-resolution MWIR images are obtained without NUC. The traditional NUC method is applied in Figs. 10(b) and 10(e), while the proposed NUC method is in Figs. 10(c) and 10(f). Although both of the traditional and the proposed NUC method help in improving the non-uniformity in the reconstructed high-resolution MWIR images, as depicted in the whole images, there is a clear difference between the two methods, which is depicted in the zoomed-in regions. In the whole images, these two NUC methods make the gray values almost equal in both marginal area and central area. However, in the zoomed-in region, the patterns are consecutive in the proposed NUC method, but those are divided into strips in the traditional NUC method. Therefore, the proposed NUC method can effectively correct the non-uniformity of MWIR FPA CI system, which is beyond the reach of the traditional NUC method.
Fig. 10. Image contrast of USAF resolution test chart among the image reconstructions without NUC, with the traditional NUC method and with the proposed NUC method. (a) and (d) are the reconstructed images without NUC. (b) and (e) are the images with the traditional NUC method. (c) and (f) are the images with the proposed NUC method. The compression ratio is 0.125 in the first row, and 0.25 in the second row.
A temperature-controlled electric iron is also used as an imaging object, which is set at 200°C, and the blackbody is used as the background, which is set at 0°C. The result of MWIR image super-resolution is shown in Fig. 11. The images in the first row are reconstructed at the compression ratio of 0.125, and those in the second row are at 0.25. In Figs. 11(a) and 11(d), the high-resolution MWIR images are obtained without NUC. The traditional NUC method is applied in Figs. 11(b) and 11(e), while the proposed NUC method is in Figs. 11(c) and 11(f). The images are similar when seeing as a whole; however, in the zoomed-in region, the image details of the electric iron are preserved in the proposed NUC method, which is consistent with the images of USAF resolution test chart shown in Fig. 10.
Fig. 11. Image contrast of electric iron among the image reconstructions without NUC, with the traditional NUC method and with the proposed NUC method. (a) and (d) are the reconstructed images without NUC. (b) and (e) are the images with the traditional NUC method. (c) and (f) are the images with the proposed NUC method. The compression ratio is 0.125 in the first row, and 0.25 in the second row.
With theoretical analysis and experimental verification, the particularities of the non-uniformity in MWIR FPA CI are discovered. These particularities are worth to be mentioned:
- (1) In the low-resolution MWIR images, the ${({\cos } )^4}$ vignetting is the main factor for the non-uniformity. However, besides the ${({\cos } )^4}$ vignetting, BSA also plays an important role in the non-uniformity of reconstructed high-resolution MWIR images.
- (2) BSA is generated in the process of image reconstruction, as the result of the variance of brightness distribution across the low-resolution images. Because of the limits of DMD, the aperture interference phenomenon is inevitable, so the brightness distribution of the low-resolution image varies depending largely on the pattern of the DMD mask. Consequently, BSA is presented in the reconstructed high-resolution MWIR images.
- (3) Traditional NUC method, treating the low-resolution MWIR images separately, can effectively eliminate the ${({\cos } )^4}$ vignetting, so the non-uniformity is improved. But for the proposed NUC method, treating the low-resolution images together as a group, not only is the ${({\cos } )^4}$ vignetting almost eliminated, but the BSA is also suppressed to a great extent. Thus, the non-uniformity is further improved for the proposed NUC method than the traditional one, which is tally with Fig. 7.
- (4) As the non-uniformity of the low-resolution MWIR image is temperature-related, BSA is difficult to be eliminated entirely, which contaminates the image details. The more low-resolution MWIR images are used in the image reconstruction, the more serious BSA will be presented in the reconstructed high-resolution MWIR images. Fortunately, less low-resolution images used for image reconstruction is advocated, which is consistent with the purpose of MWIR FPA CI.
- (5) If the temperature sampling interval is shortened, the non-uniformity of the reconstructed high-resolution MWIR images may get more improvement, at the expense of more computing costs and larger storage space. As a common topic in engineering application, the tradeoff between accuracy and cost also exists in MWIR FPA CI, so the selection of temperature sampling interval is determined by the actual application requirement.
5. Conclusions
In MWIR FPA imaging, the non-uniformity reduces the imaging quality, which gets worse in MWIR FPA CI. In this paper, we have analyzed the sources of the non-uniformity in the low-resolution MWIR image directly captured by the FPA sensor and in the reconstructed high-resolution images. For the low-resolution MWIR images, the non-uniformity is mainly caused by the ${({\cos } )^4}$ vignetting, but in the reconstructed high-resolution images, the non-uniformity is a combination of the ${({\cos } )^4}$ vignetting and the BSA. According to the image super-resolution computation process, we have pointed out the NUC algorithm should be applied to the low-resolution MWIR image, rather than the final high-resolution image. In addition, the MWIR FPA CI system with different DMD masks should be treated as a group of separate imaging systems, and the low-resolution MWIR images obtained from different DMD masks must be processed together rather than separately. From these observations, a calibration-based NUC method have been proposed for MWIR FPA CI, which has good expansibility.
Based on the experimental results obtained from the actual MWIR FPA CI system, we have verified the effectiveness and practicability of the proposed NUC method. Compared with the traditional NUC method, both the ${({\cos } )^4}$ vignetting and the BSA can be effectively eliminated. According to the theoretical analysis and experimental results, the particularities of the non-uniformity in MWIR FPA CI are discovered and discussed, which may provide guidance for researchers realizing the NUC for MWIR FPA CI system and leading to better image reconstruction quality.
Disclosures
The authors declare no conflicts of interest.
References
1. R. Weissleder and M. Nahrendorf, “Advancing biomedical imaging,” Proc. Natl. Acad. Sci. 112(47), 14424–14428 (2015). [CrossRef]
2. J. Grant, M. Kenney, Y. D. Shah, I. Escorcia-Carranza, and D. R. Cumming, “CMOS compatible metamaterial absorbers for hyperspectral medium wave infrared imaging and sensing applications,” Opt. Express 26(8), 10408–10420 (2018). [CrossRef]
3. A. Rogalski, “Progress in focal plane array technologies,” Prog. Quantum Electron. 36(2-3), 342–473 (2012). [CrossRef]
4. M. Razeghi and B. Nguyen, “Advances in mid-infrared detection and imaging: a key issues review,” Rep. Prog. Phys. 77(8), 082401 (2014). [CrossRef]
5. F. Abolmaali, A. Brettin, A. Green, N. I. Limberopoulos, A. M. Urbas, and V. N. Astratov, “Photonic jets for highly efficient mid-IR focal plane arrays with large angle-of-view,” Opt. Express 25(25), 31174–31185 (2017). [CrossRef]
6. M. S. de Cumis, S. Viciani, S. Borri, P. Patimisco, A. Sampaolo, G. Scamarcio, P. De Natale, F. D. Amato, and V. Spagnolo, “Widely-tunable mid-infrared fiber-coupled quartz-enhanced photoacoustic sensor for environmental monitoring,” Opt. Express 22(23), 28222–28231 (2014). [CrossRef]
7. B. Mizaikoff, “Infrared optical sensors for water quality monitoring,” Water Sci. Technol. 47(2), 35–42 (2003). [CrossRef]
8. M. Penn, S. Krucker, H. Hudson, M. Jhabvala, D. Jennings, A. Lunsford, and P. Kaufmann, “Spectral and imaging observations of a white-light solar flare in the Mid-infrared,” Astrophys. J., Lett. 819(2), L30 (2016). [CrossRef]
9. M. Honda, K. Okada, T. Miyata, G. D. Mulders, J. R. Swearingen, T. Kamizuka, R. Ohsawa, T. Fujiyoshi, H. Fujiwara, and M. Uchiyama, “Mid-infrared multi-wavelength imaging of Ophiuchus IRS 48 transitional disk,” Publ. Astron. Soc. Jpn. 70(3), 44 (2018). [CrossRef]
10. C. Hughes and M. J. Baker, “Can mid-infrared biomedical spectroscopy of cells, fluids and tissue aid improvements in cancer survival? A patient paradigm,” Analyst 141(2), 467–475 (2016). [CrossRef]
11. A. B. Seddon, B. Napier, I. Lindsay, S. Lamrini, P. M. Moselund, N. Stone, and O. Bang, “Mid-infrared Spectroscopy/Bioimaging: Moving toward MIR optical biopsy,” Laser Focus World 52(2), 50–53 (2016).
12. M. Kowalski and M. Kastek, “Comparative studies of passive imaging in terahertz and mid-wavelength infrared ranges for object detection,” IEEE Trans. Inf. Forensics Secur. 11(9), 2028–2035 (2016). [CrossRef]
13. Lynred by Sofradir & Ulis, “Widest range of advanced infrared detectors,” https://www.lynred.com/products.
14. Teledyne Scientific & Imaging, “Infrared and Visible FPAs,” http://www.teledyne-si.com/products-and-services/imaging-sensors/infrared-and-visible-fpas.
15. Leonardo, “Infrared cameras and detectors,” https://www.leonardocompany.com/en/land/optronics/infrared-cameras-and-detectors.
16. R. G. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007). [CrossRef]
17. R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde, “Model-based compressive sensing,” IEEE Trans. Inf. Theory 56(4), 1982–2001 (2010). [CrossRef]
18. X. Chen, Z. Yu, S. Hoyos, B. M. Sadler, and J. Silva-Martinez, “A sub-Nyquist rate sampling receiver exploiting compressive sensing,” IEEE Trans. Circuits Syst. I 58(3), 507–520 (2011). [CrossRef]
19. A. Mahalanobis, R. Shilling, R. Murphy, and R. Muise, “Recent results of medium wave infrared compressive sensing,” Appl. Opt. 53(34), 8060–8070 (2014). [CrossRef]
20. Z. Wu and X. Wang, “Focal plane array-based compressive imaging in medium wave infrared: modeling, implementation, and challenges,” Appl. Opt. 58(31), 8433–8441 (2019). [CrossRef]
21. X. Jian, R. Lu, Q. Guo, and G. Wang, “Single image non-uniformity correction using compressive sensing,” Infrared Phys. Technol. 76, 360–364 (2016). [CrossRef]
22. M. Sheng, J. Xie, and Z. Fu, “Calibration-based NUC method in real-time based on IRFPA,” Phys. Procedia 22, 372–380 (2011). [CrossRef]
23. Y. Cao and Y. Li, “Strip non-uniformity correction in uncooled long-wave infrared focal plane array based on noise source characterization,” Opt. Commun. 339, 236–242 (2015). [CrossRef]
24. R. Sheng-Hui, Z. Hui-Xin, Q. Han-Lin, L. Rui, and Q. Kun, “Guided filter and adaptive learning rate based non-uniformity correction algorithm for infrared focal plane array,” Infrared Phys. Technol. 76, 691–697 (2016). [CrossRef]
25. T. Toczek, F. Hamdi, B. Heyrman, J. Dubois, J. Miteran, and D. Ginhac, “Scene-based non-uniformity correction: from algorithm to implementation on a smart camera,” J. Syst. Architect. 59(10), 833–846 (2013). [CrossRef]
26. A. E. Mudau, C. J. Willers, D. Griffith, and F. P. le Roux, “Non-uniformity correction and bad pixel replacement on LWIR and MWIR images,” (IEEE, 2011), pp. 1–5.
27. Y. Zheng, S. Lin, C. Kambhamettu, J. Yu, and S. B. Kang, “Single-image vignetting correction,” IEEE Trans. Pattern Anal. 31(12), 2243–2256 (2009). [CrossRef]
28. E. Hecht, Optics (Pearson Education, 2016).
29. F. L. Pedrotti, L. M. Pedrotti, and L. S. Pedrotti, Introduction to optics (Cambridge University, 2017).
30. M. A. Herman, J. Tidman, D. Hewitt, T. Weston, and L. McMackin, “A higher-speed compressive sensing camera through multi-diode design,” Proc. SPIE 8717, 871706 (2013). [CrossRef]
31. M. Sun, W. Chen, T. Liu, and L. Li, “Image retrieval in spatial and temporal domains with a quadrant detector,” IEEE Photonics J. 9(5), 1–6 (2017). [CrossRef]
32. M. Sun, H. Wang, and J. Huang, “Improving the performance of computational ghost imaging by using a quadrant detector and digital micro-scanning,” Sci Rep-UK 9(1), 4105 (2019). [CrossRef]
33. M. Vollmer and K. Möllmann, Infrared thermal imaging: fundamentals, research and applications (John Wiley & Sons, 2017).
34. A. S. Unde and P. P. Deepthi, “Block compressive sensing: Individual and joint reconstruction of correlated images,” J. Vis. Commun. Image. R. 44, 187–197 (2017). [CrossRef]
35. Y. Sheng, X. Dun, W. Jin, F. Zhou, X. Wang, F. Mi, and S. Xiao, “The On-Orbit Non-Uniformity Correction Method with Modulated Internal Calibration Sources for Infrared Remote Sensing Systems,” Remote Sens. 10(6), 830 (2018). [CrossRef]
