High resolution snapshot imaging spectrometer using a fusion algorithm based on grouping principal component analysis

Shuaishuai Zhu; Yu Zhang; Jie Lin; Liangyu Zhao; Yanmei Shen; Peng Jin

doi:10.1364/OE.24.024624

1. Introduction

An imaging spectrometer is capable of capturing a three-dimensional (3D) datacube (x, y, λ) of an object scene. In recent decades, imaging spectrometers have been wildly applied in remote sensing and biomedical imaging [1–4]. In some specific applications, such as cellular dynamics research and tissue classification in vivo, imaging spectrometers are required to capture real-time scenes [5,6]. However, most traditional imaging spectrometers suffer from low temporal resolution caused by long scanning time which causes serious blur for dynamic scenes.

To overcome such drawbacks, snapshot imaging spectrometers, which can capture the 3D datacube in a single snapshot, were developed [7–14]. Gao et al. developed a snapshot Image Mapping Spectrometer (IMS) based on the dispersion of image zones by an image mapper, a lens array and a prism array [7]. A snapshot hyperspectral imaging Fourier transform (SHIFT) spectrometer was derived from a Multiple-image Fourier Transform Spectrometer (MFTS) [8,9]. The computed tomographic imaging spectrometer (CTIS) utilized a two-dimensional (2D) disperser to project the object scene onto a detector and the 3D datacube was recovered by tomographic reconstruction techniques [10]. The coded aperture snapshot spectral imaging (CASSI) uses a binary-coded mask to build a transmission pattern on the detector and estimated the 3D datacube during post-processing [11]. Image replicating imaging spectrometer (IRIS) is compact and available in real-time applications because its direct acquisition of a 3D datacube is based on a Lyot Filter [12].

In general, some of these techniques acquire all voxels of the 3D datacube simultaneously by dividing it into multiple 2D elements. However, limited by the number of detector pixels, this mechanism causes a serious trade-off problem between spatial resolution and spectral resolution. Hegyi et al developed a hyperspectral imaging system based on a liquid crystal polarization interferometer [15]. In this system a series of small images are acquired to reconstruct a 3D datacube. Trade-off among the spectral resolution, imaging speed, and spatial resolution can be implemented in software. However, the system only makes the trade-off selectable and it does not migrate it. The trade-off problem in CASSI was reduced by using compressive sensing theory and Wang et al. proposed a dual-camera CASSI [16], which utilized a grayscale camera to improve the reconstruction quality, though overall computational burden remains high.

Information fusion technique was first proposed in military survey field to produce an informative image from multiple sources [17]. By applying the information fusion technique, the measurement systems can achieve high resolution both in spatial and spectral domain by merging a multispectral (MS) image and a high spatial resolution image. Therefore, the trade-off problem is reduced, since the high resolution in spatial and spectral domain is not required simultaneously. Recently, the information fusion technique was applied to spectral imaging. Ma et al. achieved content-adaptive high resolution hyperspectral video system based on a trilateral interpolation filtering approach [18,19]. A hybrid-resolution spectral imaging (HRSI) was proposed to combine a high resolution RGB image and a low resolution spectral image by utilizing piecewise Wiener estimation [20]. For scenes with high sparsity in spectral domain, these techniques can achieve high spatial-spectral resolution. However, for the low sparsity situation, these techniques are unavailable because of their low sampling rate in the low spatial resolution hyperspectral image. Moreover, these techniques are very sensitive to metamerism, which is a phenomenon that different spectra appear as the same color to RGB cameras and human eyes.

Here, we report a high resolution snapshot imaging spectrometer (HR-SIS) and a matching fusion algorithm. In the proposed HR-SIS, a polarization beam splitter is used to separate the system into a spectral branch and an imaging branch. The spectral branch, which consists a spectral imager based on the SHIFT spectrometer, is compact and robust. A low spatial resolution MS image (which is also called 3D datacube. To make it explicit, we insisted on using “MS image” in the following sections.) can be acquired in the spectral branch while the imaging branch captures a high spatial resolution panchromatic (PAN) image. These images are merged by using of the fusion algorithm based on grouping principal component analysis (GPCA) with a high fusion accuracy. Meanwhile, the HR-SIS is insensitive to metamerism because there is no RGB camera in the system. However, it is worth noting that the fusion algorithm assumes that the object scene can be clustered into several groups per the correlation between each spectral band. In some extreme cases, such as a target with high frequency spatial variation and very different spectral lines for different spatial areas, the fusion algorithm will introduce significant errors.

2. Principle of HR-SIS

The layout of the proposed HR-SIS is shown in Fig. 1. It contains the imaging branch and the spectral branch based on a SHIFT spectrometer. The target is first imaged by the objective lens (OL) onto the field stop (FS). The polarizing beam splitter (PBS) behind the FS splits the object beam into two orthogonally polarized components. The reflected component is imaged by the imaging lens (IL) onto the first focal plane array (FPA₁). The polarization orientation of the transmitted component is rotated by the first half-wave plate (HWP₁). Then, the transmitted component enters into the spectral branch after collimation by the collimating lens (CL). In the spectral branch, the lenslet array (LA) forms the multiple images of the object scene. The Nomarski prisms (NP₁ & NP₂), the second half-wave plate (HWP₂) and the analyzer (A) compose the birefringent polarization interferometer (BPI). Compared to the SHIFT spectrometer, the HR-SIS contains several extra components, including a PBS, an IL, a half-wave plate (HWP₁) and a focal plane array (FPA₁). The FPA₁ can be much smaller than the FPA₂.

Fig. 1 (a) Schematic of HR-SIS. (b) Polarization optical elements with optical axes indicated by arrows. Acronyms: Objective Lens (OL), Field Stop (FS), Polarizing Beam Splitter (PBS), Imaging Lens (IL), Focal Plane Array (FPA), Half-Wave Plate (HWP), Collimating Lens (CL), Lenslet Array (LA), Birefringent Polarization Interferometer (BPI) Nomarski Prism (NP), Analyzer (A).

Download Full Size | PDF

In HR-SIS, the PBS acts as orthogonal polarizers for the reflected and transmitted beam. The corresponding polarization directions are 90° and 0° with respect to the x-axis. Their Jones matrices are denoted as $J_{R}$ and $J_{T}$ , respectively. Assuming that the Jones vector of incident light is $E_{I} = {[E_{x} E_{y}]}^{T}$ , the Jones vector of the transmitted beam and the reflected beam are calculated by,

E_{T} = J_{A} J_{B P I} J_{H W P_{1}} J_{T} E_{I}

Ε_{R} = J_{R} E_{I}

where

J_{A}

,

J_{B P I}

and

J_{H W P_{1}}

are the Jones matrices of the analyzer, BPI and first HWP, respectively. According to the optical axes of the polarization elements depicted in Fig. 1(b), Eq. (1) and (2) can be transformed to,

Ε_{Τ} = \frac{1}{2} [\begin{matrix} 1 & 1 \\ 1 & 1 \end{matrix}] \cdot [\begin{matrix} 1 & 0 \\ 0 & e^{- i 2 π σ Δ} \end{matrix}] \cdot \frac{\sqrt{2}}{2} [\begin{matrix} 1 & 1 \\ 1 & - 1 \end{matrix}] \cdot [\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}] \cdot [\begin{matrix} E_{x} \\ E_{y} \end{matrix}] = \frac{\sqrt{2}}{4} E_{x} (1 + e^{- i 2 π σ Δ}) [\begin{matrix} 1 \\ 1 \end{matrix}]

E_{R} = [\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix}] \cdot [\begin{matrix} E_{x} \\ E_{y} \end{matrix}] = [\begin{matrix} 0 \\ E_{y} \end{matrix}]

where Δ is the optical path difference (OPD) introduced by the BPI. In [8], specific calculation of Δ is discussed in detail. The detected intensity of each FPA can be derived as,

I_{F P A_{2}} = \frac{I_{x} (σ)}{2} [1 + \cos (2 π σ Δ)]

I_{F P A_{1}} = I_{y} (σ)

where

I_{x} (σ)

and

I_{y} (σ)

are the x- and y-component of the input spectra, respectively.

According to Eq. (5), an MS image can be reconstructed by the reconstruction method described in [8]. Meanwhile, a PAN image is obtained from the FPA₁. There is only 25% energy lost in the HR-SIS since only one polarizer is installed in the transmitted branch of the PBS. On the other hand, there are two polarizers which cause a 75% energy lost in the SHIFT spectrometer. Therefore, theoretical optical throughput of the HR-SIS is triple that of the SHIFT spectrometer. However, the signal-to-noise rate (SNR) of the system is a more complicated issue [21], which is outside the scope of this article.

3. Fusion algorithm based on GPCA

The spatial resolution of the unfused MS image is low due to the trade-off problem, while the PAN image processes high spatial resolution but no spectral information. To solve the dilemma, the fusion technique is applied to achieve high spatial-spectral resolution by merging the MS image and PAN image.

Information fusion technique can synthetize an informative image from images of multiple sources. It is widely used in the remote sensing field [22]. The image with a high spatial-spectral resolution can be generated by combining a high spectral resolution MS image and a high spatial resolution PAN image. This process, also called pansharpening, can reduce the required storage capacity on satellites and data transmission rates. Since 1980s, several effective image fusion techniques have been developed [22–24]. Component substitution (CS) and multiresolution analysis (MRA) are the two most popular classes in these techniques. CS techniques, which rely on the substitution of a component of the MS image by the PAN image, are fast and simple. However, serious spectral distortion is introduced when the MS image and the PAN image don’t share the same spectral bandwidth. MRA techniques can mitigate the spectral distortion and the computational burden depends on the method used to decompose the PAN image. In the snapshot spectral imaging field, the fusion process is required to be fast and accurate. CS techniques are suitable except for the spectral distortion issue. To settle the problem in CS techniques, several essential requirements are necessary [25],

(i) High correlation between the replacement component of the MS and the PAN image, where the replacement component is the component replaced by the PAN image.
(ii) High percentage of variance accounted by the replacement component.
(iii) Equal contribution from each MS band in the substitution component.

In the remote sensing field, the spectral responses of the MS image are not always perfectly overlapped with the bandwidth of the PAN image [25] which causes the loss of Requirement (i). In contrast, two visible light sensors, whose spectral responses cover the same range, are used in spectral branch and imaging branch of the HS-SIS. The arrangement ensures that the system can meet the Requirement (i). On the other hand, Requirement (ii) and (iii) can be met with high correlation among spectral bands of the MS image. The correlation is determined by the spectral complexity of the MS image. Higher spectral complexity lead to lower correlation. Fortunately, sparsity is widely present in nature. The spectral bands of the MS image always can be divided into several groups in which the correlation among group members is high.

Based on the property of the HR-SIS, we proposed a novel fusion algorithm based on grouping principle component analysis (GPCA) to fuse the MS and PAN image. The framework of the proposed fusion algorithm based on GPCA is shown in Fig. 2. The MS image is firstly clustered into several groups according to the factor loading matrix. Then a modified principle component analysis (PCA) fusion process is carried out in each group.

Fig. 2 Framework of proposed fusion algorithm based on GPCA.

Download Full Size | PDF

Firstly, a factor analysis based on PCA is operated to the MS image,

X = A F + ε

where

X^{T} = [X_{1}, \dots, X_{N}]

denotes the MS image. N is the number of spectral bands.

X_{k}

(k = 1,…,N), which indicates the k-th band of the MS image, contains m × n elements. m × n is the number of pixels in a single band of the MS image. A is the factor loading matrix with size of

N \times M

, where M is the number of common factors.

F^{T} = [F_{1}, \dots, F_{M}]

are the common factors of X. M is determined when the accumulated contribution ratio of the principal components of X exceed a threshold value,

\frac{\sum_{i = 1}^{M} λ_{i}}{\sum_{i = 1}^{N} λ_{i}} \geq t h r e s h o l d

where

λ_{i}

(

λ_{1} > λ_{2} > \dots > λ_{N}

) is the i-th eigenvalue of the covariance matrix of X.

During the factor analysis, A is rotated based on the varimax criterion [26]. This procedure, also called as varimax rotation, pushes the elements of A towards ± 1 and 0. According to the rotated factor loading matrix A, a clustering of X, $π (X)$ , can be stated as follows:

π (X) = {C_{1}, C_{2}, ..., C_{| π (X) |}}

where

C_{i}

(

1 \leq i \leq | π (X) |

) is a single cluster of

π (X)

, and

X = \cup_{i = 1}^{| π (X) |} C_{i}

Then, the MS image is interpolated at the scale of the PAN image. ${\tilde{X}}^{T} = [{\tilde{X}}_{1}, \dots, {\tilde{X}}_{N}]$ denotes the up-sampling MS image. ${\tilde{X}}_{k}$ (k = 1,…,N), which indicates the k-th band of the up-sampling MS image, contains r × r × m × n elements. r is the spatial resolution ratio between the PAN and MS image. Apparently, $\tilde{X}$ can be clustered in the same pattern with X, i.e,

π (\tilde{X}) = {{\tilde{C}}_{1}, {\tilde{C}}_{2}, ..., {\tilde{C}}_{| π (\tilde{X}) |}}

According to the property of varimax rotation, there is a strong correlation among elements in cluster ${\tilde{C}}_{i}$ while the correlation between ${\tilde{C}}_{i}$ and ${\tilde{C}}_{j}$ ( $i \neq j$ ) is weak. Therefor a simple PCA can extract the primary information in each cluster. $P \tilde{C} 1_{i} (i = 1, \dots, | π (\tilde{X}) |)$ denotes the first principal component of cluster ${\tilde{C}}_{i}$ . In the next step, the detailed information of PAN image is injected into $P \tilde{C} 1_{i}$ ,

P \hat{C} 1_{i} = P \tilde{C} 1_{i} + α (P - P_{L})

where

P \tilde{C} 1_{i}

is the pansharpened version of

P \tilde{C} 1_{i}

. P denotes the PAN image with r × r × m × n elements.

P_{L}

is a low-pass version of P.

α

is the gain factor calculated by,

α = \frac{σ (P \tilde{C} 1_{i})}{σ (P)} \cdot \frac{P \tilde{C} 1_{i}}{\sum_{j = 1}^{| π (\tilde{X}) |} P \tilde{C} 1_{j}}

where

σ (P \tilde{C} 1_{i})

and

σ (P)

are the standard deviation of

P \tilde{C} 1_{i}

and P respectively. The ratio of

P \tilde{C} 1_{i}

and

Σ_{_{j = 1}}^{^{| π (\tilde{X}) |}} P \tilde{C} 1_{j}

is based on the consideration of physical properties of the objective [27].

Finally, inverse PCA in every cluster ${\tilde{C}}_{i}$ ( $1 \leq i \leq | π (\tilde{Χ}) |$ ) with the pansharpened first principal component $P \tilde{C} 1_{i}$ yields the fused MS image, ${\hat{X}}^{T} = [{\hat{X}}_{1}, \dots, {\hat{X}}_{N}]$ , where ${\hat{X}}_{k}$ (k = 1,…, N), indicates the k-th band of the fused MS image.

Obviously, a precise image registration should be performed before the fusion process. A chess-board is viewed by the HR-SIS and a series of subpixel-level feature points can be obtained from the imaging branch and spectral branch by the combination of a quick detection algorithm [28] and detail refinement strategy [29]. Image registration coefficients can be calculated via the corresponding feature points by using a least-squares fitting. The registration coefficients can be considered as a systematic parameter. Therefore, the primary amount of computations, involving searching feature points and calculation of registration coefficients, is saved.

4. Results and Discussion

The experimental prototype of the proposed system is illustrated as Fig. 3. The cage system is designed to provide precision axial alignment. Two commercial lens (Canon 50mm f/1.4 USM) are utilized as the OL and CL. A broadband polarizing beam splitter (PBS) and an achromatic half wave-plate (HWP₁) are between these two lenses. A 1 × Telecentric Lens (TEC-M1065MP, Computar) is used as IL in the reflection optical path of the PBS. A 17 × 22 lenslet array (f = 10.9mm) made from silica glass with a pitch size of 1 mm × 1 mm is arranged after the CL. Meanwhile, the BPI, which is installed in front of the FPA₂, comprises an achromatic half wave-plate (HWP₂) with fast axis oriented at 45° with respect to the x-axis, two Nomarski prisms (γ = 86.1°, δ = 75°, which are identified in Fig. 1(b)) and an analyzer (A). Two FPAs, FPA₁ and FPA₂, are used in the system. Specifically, the FPA₁ in imaging branch is a 2456 × 2058 monochromatic camera (AM-500 GE, JAI), while the FPA₂ (BM-800 GE, JAI) in spectral branch processes 3296 × 2472 active pixels. Note that only part of the pixels are utilized in the FPA₁ which can be replaced by a smaller sensor.

Fig. 3 Experimental prototype of the proposed system. Acronyms: Objective Lens (OL), Field Stop (FS), Polarizing Beam Splitter (PBS), Half-Wave Plate (HWP), Imaging Lens (IL), Focal Plane Array (FPA), Collimating Lens (CL), Lenslet Array (LA), Birefringent Polarization Interferometer (BPI).

Download Full Size | PDF

To evaluate the performance of the HR-SIS, several experiments had been implemented. Firstly, the spectral accuracy was demonstrated by imaging a commercial color checker. The experiment can only illustrate the spectral accuracy of the HR-SIS since the color checker is not varying rapidly spatially. Secondly, the spectral resolution was measured by using a He-Ne laser as light source. Thirdly, a test chart was imaged by the HR-SIS to illustrate the spatial improving capability. Then, the property of insensitivity to metamerism was demonstrated by distinguishing between a green leaf and artifacts. Finally, to evaluate the performance of the proposed fusion algorithm several colorful targets were captured by the HR-SIS and comparison between several state-of-the-art fusion algorithms and the proposed algorithm was carried out.

4.1 Spectral accuracy

To evaluate the spectral accuracy, an Xrite color checker Passport consisting 24 different color blocks, which is shown in Fig. 4(a), was imaged and a halogen lamp (MI-150, Edmund) was used as illumination. After post-processing, an MS image with spatial size of 170 × 170 pixels was achieved from the original interferogram acquired by the spectral branch. The imaging branch captured a high spatial resolution PAN image with 680 × 680 pixels. The proposed fusion procedure, which is described in Sec. 3, was implemented. The threshold in Eq. (8) is 0.95. As a result, an MS image with a high spatial size of 680 × 680 pixels was produced in a single snapshot.

Fig. 4 (a) Xrite color checker. (b) Normalized root mean square error (NRMSE) in 24 color-block areas. (c) Spectral curves of color blocks from HR-SIS and AvaSpec (ground truth).

Download Full Size | PDF

The average spectra of 30 measurements by a commercial fiber spectrometer (AvaSpec-ULS2048-USB2, Avantes, spectral resolution 1.15nm) was utilized as ground truth. The integrate time of each measurement is 10 ms. The normalized root mean square error (NRMSE) of spectra in 24 color-block areas were calculated. The calculated area of color-block No. 21 is shown in Fig. 4(a) with a red dash square. Each area contains 65 × 65 pixels. The average and standard deviation of NRMSEs for each color-block are depicted in Fig. 4(b). The comparison between the ground truth and data of HR-SIS are further shown in Fig. 4(c).The data of HR-SIS is averaged from 9 × 9 pixels cross the color-block area. It is worth noting that the spectra of each color block are normalized individually. This is based on two factors. On one hand the measured light intensities by the fiber spectrometer are highly impacted by the orientation of the fiber probe. And the spectrum of each color block is measured successively. Therefore, we cannot ensure that all the measurements are under the same condition. On the other hand, the light source used in the experiment is not highly stable. While the measurements by the HR-SIS and fiber spectrometer are not simultaneous. Therefore, the HR-SIS and fiber spectrometer may measure the target under different illumination. Based on the above factors, a uniform normalization factor for all the color blocks cannot be obtained.

Obviously, the NRMSEs of most color blocks are smaller than 10%, expect for blocks with No. 8, No. 13, No. 18, No. 19 and No. 24 (the average NRMSEs are 18.7%, 16.5%, 9.3%, 10.4% and 9.3% respectively). The performance in blue blocks of No. 8, No. 13 and No. 18 is mainly caused by the low radiance of halogen lamp in short-wavelength range. To overcome this problem, an LED (IF803, IFIRE) was added as illumination. The results are shown in Fig. 5. One can see that the NRMSEs of blocks No. 8, 13, and 18 all fall below 10%. Saturated and insufficient reflectance of block No. 19 and No. 24 lead to high NRMSEs.

Fig. 5 Spectral curves of color blocks (a) No.8, (b) No. 13, and (c) No. 18 from HR-SIS and AvaSpec (ground truth). (d) NRMSEs in the three color-block areas.

Download Full Size | PDF

4.2 Spectral resolution

In the prototype, 13 × 18 sub-lenses were employed. According to the apex angle of the NPs (γ = 86.1°, which is identified in Fig. 1(b)), the sample interval of OPD between the adjacent sub-lenses is approximate to 0.2 μm. Therefore, the spectral resolution of the HR-SIS is better than 250 cm⁻¹, which is approximately 10 nm at 632.8 nm. To demonstrate the spectral resolution, an integrating sphere (GAF-030, NMERRY) irradiated by a He-Ne laser (25-LHP-925-230, Melles Griot) is used as an objective and the full width at half maximum (FWHM) of obtained spectrum was used to characterize the spectral resolution of the HR-SIS. Figure 6 is the spectrum of the center point in the field of view (FOV). The standard deviation of fitted Gaussian curve (the red dotted line) is 4.1 nm which yields a FWHM of 9.7 nm. The average FWHM of 140 × 140 points across the FOV is 9.6 nm.

Fig. 6 Spectrum of center point of the FOV with a fitted Gaussian curve (red dashed line).

Download Full Size | PDF

4.3 Spatial resolution

A test chart (Negative USAF1951 test target, Thorlabs) was captured by the proposed system to demonstrate the improvement of spatial resolution through the fusion algorithm. The composite images of the test chart before and after fusion are shown in Fig. 7. The composite image after image fusion contains much more spatial details. According to the resolving power lookup table of the test chart (as shown in Table 1) the spatial resolution was improved from 0.891 lines/mm (Group −1, element 6) to 3.56 lines/mm (Group 1, element 6).

Fig. 7 (a) Composite image from the MS image before fusion. (b) Composite image from the MS image after fusion.

Download Full Size | PDF

Table 1. Number of Line Pairs / mm in USAF Resolving Power Test Target 1951

View Table | View all tables in this article

4.4 Insensitivity to metamerism

Unlike other snapshot spectral imaging techniques based on information fusion [18–20], the proposed HR-SIS is insensitive to metamerism because of the monochrome sensors and high spectral resolution. As shown in Fig. 8(a), a leaf was attached to a green artificial background. Four different-sized green artifacts were cut into the word ‘HIT’ attached on the leaf, which is illuminated by a halogen lamp (MI-150, Edmund). As shown in Fig. 8(a), the artifacts are hardly distinguished from the RGB image obtained by a color digital camera since they appear the same color with the leaf, which is a typical example of metamerism. The spectra of two different points on the leaf and the artifact are shown in Fig. 8(b). The spectral difference is because of the absorbing property of chlorophyll. Two bands of the MS image at 655.2 nm and 713.2 nm are shown in Figs. 8(c) and 8(d), respectively. The artifacts can be clearly identified in the spectral band of 713.2 nm.

Fig. 8 (a) RGB picture of the scene (white balance was corrected in Photoshop). (b) Spectra of two points on the leaf and the green artifacts respectively. (c) Spectral band at 655.2nm. (d) Spectral band at 713.2nm.

Download Full Size | PDF

4.5 Performance of the fusion algorithm based on GPCA

To simplify the description, some notations, which are listed in Table 2, are used in the following. Meanwhile, the relationships among the notations are further depicted in Fig. 9. The proposed algorithm relies on the property of the HR-SIS, that the spectral response of the two sensors acquiring the MS and PAN image cover the same range. In this case, the correlation coefficient between $M S_{s u m}$ and $P A N_{l}$ is theoretically equal to 1. The property is demonstrated in the following.

Table 2. List of notation

View Table | View all tables in this article

Fig. 9 The relationships among the notation in Table 2

Download Full Size | PDF

Twelve colorful maps with abundant spatial details, as shown in Fig. 10(a), were measured by our HR-SIS. A halogen lamp (MI-150, Edmund) and an LED (IF803, IFIRE) were used as illumination. MS images with spatial size of 170 × 170 pixels were obtained by the spectral branch while the imaging branch captured PAN images with spatial size of 680 × 680 pixels. The correlation coefficients between the $M S_{s u m}$ and $P A N_{l}$ are plotted in Fig. 10(d). All correlation coefficients of the maps are not less than 0.9. The synthetic MS image and downsampled PAN image of the map No. 2, whose correlation coefficient is 0.9, are shown in Figs. 10(b) and 10(c) respectively. The differences between the two images are mainly caused by the defocusing error in the spectral branch.

Fig. 10 (a) Colorful maps with abundant spatial details. Among them, image No. 1-3 were homemade, and image No. 5-12 were obtained from the online image database [30]. Image No. 4 is an Xrite color checker. These pictures, except image No. 4, were printed by color printers (Epson Stylus Photo 1400 for No. 2, 6 and 8; Xerox Workcentre 7346 for No. 1, 3, 5 and 7; Canon iP2780 for No. 8-12). All pictures were taken by Canon 550D color digital camera and white balance was corrected in Photoshop. (b) Synthetic MS image and (c) downsampled PAN image of the map No. 13. (d) Correlation coefficients between the downsampled PAN images and synthetic MS images.

Download Full Size | PDF

To evaluate the performance of the proposed fusion algorithm based on GPCA, several state-of-the-art fusion algorithms, including PCA [31], adaptive Gram–Schmidt (GSA) [32] and modulation transfer functions & generalized Laplacian pyramid (MTF-GLP) [33], are compared with GPCA. Considering the absence of the reference MS image with high spatial and spectral resolution, the assessment of the fusion results follows the Wald’s protocol [34] which states that the fusion results must hold three properties, i.e.,

(i) $M S_{d}^{*}$ should be as close as possible to $M S_{o}$ ;
(ii) $M S_{o}^{*}$ should be as close as possible to $M S_{o}$ ;
(iii) The spectral set of $M S_{o}^{*}$ should be as close as possible to that of $M S_{o}$

To characterize the similarity between two MS images several metrics have been raised in the past decades [23]. In this paper, the CC, SAM, and ERGAS were utilized. These metrics are defined below. Assume that $X^{r e f}$ and $\hat{X}$ denote the reference MS image and fused MS image, respectively.

1) CC: The cross correlation (CC) defined below indicates the spatial distortion of the fused MS image,

CC (\hat{X}, X^{r e f}) = \frac{1}{N} \sum_{i = 1}^{N} CCS ({\hat{X}}_{i}, X_{i}^{r e f})

where N is the number of spectral bands.

X_{i}^{r e f}

and

{\hat{X}}_{i}

are the i-th band image of the reference and fused MS image, respectively. CCS is the cross correlation for between single-band images. Higher value of the CC indicates better performance.

2) SAM: The spectral angle mapper (SAM) characterizes the spectral distortion between MS images and can be defined as,

SAM (\hat{X}, X^{r e f}) = \frac{1}{m \times n} \sum_{i = 1}^{m} \sum_{j = 1}^{n} arc \cos (\frac{< {\hat{x}}_{i, j}, x_{i, j}^{r e f} >}{{‖ {\hat{x}}_{i, j} ‖}_{2} \cdot {‖ x_{i, j}^{r e f} ‖}_{2}})

where m and n are the number of rows and columns in a single-band image.

{\hat{x}}_{i, j}

and

x_{i, j}^{r e f}

are spectrum of

\hat{X}

and

X_{}^{r e f}

at the spatial location of (i, j). Lower value of the SAM indicates better performance.

3) ERGAS: The erreur relative globale adimensionnelle de synthèse (ERGAS) is a global fusion quality indicator, which is defined as,

ERGAS (\hat{X}, X^{r e f}) = 100 \times \frac{1}{r} \sqrt{\frac{1}{N} \sum_{i = 1}^{N} {(\frac{N \times RMSE ({\hat{X}}_{i}, X_{i}^{r e f})}{\sum_{j = 1}^{N} X_{i}^{r e f}})}^{2}}

where r is the spatial resolution ratio between the PAN and MS image. REMS is the root mean squared error, which is defined as,

RMSE ({\hat{X}}_{i}, X_{i}^{r e f}) = \frac{{‖ {\hat{X}}_{i} - X_{i}^{r e f} ‖}_{F}^{2}}{m \times n}

Lower value of the ERGAS indicates better performance.

The CC and SAM between $M S_{o}^{*}$ and $M S_{o}$ were calculated to characterize the spatial and spectral distortions of $M S_{o}^{*}$ , respectively. Meanwhile, the ERGAS between $M S_{d}^{*}$ and $M S_{o}$ was used to indicate the global deviation of . $M S_{d}^{*}$ The evaluating results of different fusion algorithms on the 12 maps in Fig. 10(a) are shown in Fig. 11. The proposed algorithm based on GPCA performs competitively with other state-of-the-art algorithms. Additionally, the fused MS images by the proposed algorithm were compared with the upsampled MS images without fusion. The CC and SAM between $M S_{o}$ and $M S_{o}^{*}$ fused by the proposed algorithm are plotted as red lines, while the CC and SAM between $M S_{o}$ and $M S_{u}^{*}$ are plotted as blue lines in Figs. 12(a) and 12(b). Note that the improvements in the SAM by the fusion are not obvious when the target images don’t have large color variation such as images No. 5-12. Because interpolations introduce less deviations when the data changes more gently.

Fig. 11 Evaluating results of different fusion algorithms. (a) ERGAS between $M S_{d}^{*}$ and $M S_{o}$ . (b) CC and (c) SAM between $M S_{o}^{*}$ and $M S_{o}$ . Note that the ideal values of the ERGAS, CC and SAM are 0, 1 and 0, respectively.

Download Full Size | PDF

Fig. 12 Comparison between fused MS images and upsampled MS images

Download Full Size | PDF

Meanwhile, the local accuracy between $M S_{o}$ and the $M S_{o}^{*}$ fused by the proposed fusion algorithm was further evaluated. The NRMSEs between the $M S_{o}^{*}$ and $M S_{o}$ at the pixels along the red dotted line, as shown in Fig. 13(a), are plotted in Fig. 13(b). The cross section profile of $M S_{s u m}$ is also plotted in Fig. 13(b). It can be seen that, the NRMSE is higher near the edges of the color-blocks than that in the interior areas. This is mainly caused by the blur introduced during the up-sample of the unfused MS image.

Fig. 13 (a) RGB picture of the color checker obtained by a commercial color camera. (b) The red line indicates the NRMSE between $M S_{o}^{*}$ and $M S_{o}$ at the pixels along the red dotted line, while the blue line is the cross section profile of $M S_{s u m}$ .

Download Full Size | PDF

The computing time of the reconstruction and fusion algorithm was also evaluated. The whole post-process procedure was implemented 30 times in Matlab R2012b with an Intel Core i5-4200H CPU. There are slight differences in computing time between each repetition. It is mainly caused by the fact that the condition of the computer is changing slightly over time. The average and standard deviation of the computing times are shown in Fig. 14. The computing time will severely limit the acquisition speed of the HR-SIS. However, it can be further reduced by utilizing a graphics processing unit (GPU).

Fig. 14 Computing time of the reconstruction and fusion algorithm.

Download Full Size | PDF

5. Conclusions

In this paper, the high resolution snapshot imaging spectrometer (HR-SIS) was proposed and experimentally demonstrated. Meanwhile a matching fusion algorithm based on GPCA was developed. A high spectral resolution MS image with low spatial resolution and a high spatial resolution PAN image were acquired by the spectral branch and imaging branch of the HR-SIS, respectively. The fusion procedure based on GPCA was carried out to produce a high spatial-spectral resolution MS image. The HR-SIS can acquire an MS image with spatial size of 680 × 680 pixels and the spectral resolution of 250 cm⁻¹ in a snapshot. The spectral accuracy, spatial resolution and insensitivity to metamerism of the HR-SIS were experimentally demonstrated. The NRMSE of the spectral accuracy can achieve around 10% under proper illumination. The spectral resolution of the HR-SIS is better than 10 nm at 632.8nm and the spatial resolution of the system is 3.56 lines/mm. The HR-SIS is compact and precise in spatial-spectral domain. Meanwhile, in contrast to other snapshot imaging spectrometer based on information fusion [18–20], the HR-SIS is insensitive to metamerism. Because the snapshot imaging spectrometers described in [18–20] all utilize RGB cameras while two monochromatic cameras are used in the HR-SIS. The proposed fusion algorithm was evaluated by observing twelve colorful maps. The property of the HR-SIS, on which the proposed fusion algorithm relies, was highlighted by an experiment. The proposed GPCA algorithm was compared with other state-of-the-art fusion algorithms. Under the Wald’s protocol, the GPCA algorithm is competitively with others on the ERGAS, CC and SAM. Meanwhile, the fusion algorithm also can be implemented in many other snapshot imaging spectrometers, such as techniques described in [7,15]. The HR-SIS can be applied in biomedical imaging, microscopy, endoscopy and air surveillance in concert with an unmanned aerial vehicle (UAV).

Funding

National High Technology Research and Development Program of China (2015AA042401).

References and links

1. D. T. Dicker, J. M. Lerner, and W. S. El-Deiry, “Hyperspectral image analysis of live cells in various cell cycle stages,” Cell Cycle 6(20), 2563–2570 (2007). [CrossRef] [PubMed]

2. R. M. Levenson and J. R. Mansfield, “Multispectral imaging in biology and medicine: slices of life,” Cytometry A 69(8), 748–758 (2006). [CrossRef] [PubMed]

3. D. J. Mulla, “Twenty five years of remote sensing in precision agriculture: key advances and remaining knowledge gaps,” Biosystems Eng. 114(4), 358–371 (2013). [CrossRef]

4. F. D. van der Meer, H. M. A. van der Werff, F. J. A. van Ruitenbeek, C. A. Hecker, W. H. Bakker, M. F. Noomen, M. van der Meijde, E. J. M. Carranza, J. B. Smeth, and T. Woldai, “Multi- and hyperspectral geologic remote sensing: A review,” Int. J. Appl. Earth Obs. 14(1), 112–128 (2012). [CrossRef]

5. G. Yang, “Bioimage informatics for understanding spatiotemporal dynamics of cellular processes,” Wiley Interdiscip. Rev. Syst. Biol. Med. 5(3), 367–380 (2013). [CrossRef] [PubMed]

6. N. Bedard, R. A. Schwarz, A. Hu, V. Bhattar, J. Howe, M. D. Williams, A. M. Gillenwater, R. Richards-Kortum, and T. S. Tkaczyk, “Multimodal snapshot spectral imaging for oral cancer diagnostics: a pilot study,” Biomed. Opt. Express 4(6), 938–949 (2013). [CrossRef] [PubMed]

7. L. Gao, R. T. Kester, N. Hagen, and T. S. Tkaczyk, “Snapshot Image Mapping Spectrometer (IMS) with high sampling density for hyperspectral microscopy,” Opt. Express 18(14), 14330–14344 (2010). [CrossRef] [PubMed]

8. M. W. Kudenov and E. L. Dereniak, “Compact real-time birefringent imaging spectrometer,” Opt. Express 20(16), 17973–17986 (2012). [CrossRef] [PubMed]

9. A. Hirai, T. Inoue, K. Itoh, and Y. Ichioka, “Application of multiple-image fourier transform spectral imaging to measurement of fast phenomena,” Opt. Rev. 1(2), 205–207 (1994). [CrossRef]

10. B. Ford, M. Descour, and R. Lynch, “Large-image-format computed tomography imaging spectrometer for fluorescence microscopy,” Opt. Express 9(9), 444–453 (2001). [CrossRef] [PubMed]

11. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. 47(10), B44–B51 (2008). [CrossRef] [PubMed]

12. A. Gorman, D. W. Fletcher-Holmes, and A. R. Harvey, “Generalization of the Lyot filter and its application to snapshot spectral imaging,” Opt. Express 18(6), 5602–5608 (2010). [CrossRef] [PubMed]

13. N. Hagen and M. W. Kudenov, “Review of snapshot spectral imaging technologies,” Opt. Eng. 52(9), 090901 (2013). [CrossRef]

14. L. Gao and L. V. Wang, “A review of snapshot multidimensional optical imaging: measuring photon tags in parallel,” Phys. Rep. 616, 1–37 (2016). [CrossRef] [PubMed]

15. A. Hegyi and J. Martini, “Hyperspectral imaging with a liquid crystal polarization interferometer,” Opt. Express 23(22), 28742–28754 (2015). [CrossRef] [PubMed]

16. L. Wang, Z. Xiong, D. Gao, G. Shi, and F. Wu, “Dual-camera design for coded aperture snapshot spectral imaging,” Appl. Opt. 54(4), 848–858 (2015). [CrossRef] [PubMed]

17. A. Toet, L. J. van Ruyven, and J. M. Valeton, “Merging thermal and visual images by a contrast pyramid,” Opt. Eng. 28(7), 789–792 (1989). [CrossRef]

18. C. Ma, X. Cao, X. Tong, Q. Dai, and S. Lin, “Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System,” Int. J. Comput. Vis. 110(2), 141–155 (2014). [CrossRef]

19. C. Ma, X. Cao, R. Wu, and Q. Dai, “Content-adaptive high-resolution hyperspectral video acquisition with a hybrid camera system,” Opt. Lett. 39(4), 937–940 (2014). [CrossRef] [PubMed]

20. Y. Murakami, K. Nakazaki, and M. Yamaguchi, “Hybrid-resolution spectral video system using low-resolution spectral sensor,” Opt. Express 22(17), 20311–20325 (2014). [CrossRef] [PubMed]

21. M. C. Abrams, S. P. Davis and J. W. Brault, Fourier Transform Spectrometry (Academic, 2001), Chap. 8.

22. H. Ghassemian, “A review of remote sensing image fusion methods,” Inf. Fusion 32, 75–89 (2016). [CrossRef]

23. L. Loncan, S. Fabre, L. B. Almeida, J. M. Bioucas-Dias, L. Wenzhi, X. Briottet, G. A. Licciardi, J. Chanussot, M. Simo, N. Dobigeon, J. Y. Tourneret, M. A. Veganzones, W. Qi, G. Vivone, and N. Yokoya, “Hyperspectral pansharpening: a review,” IEEE Geosci. Remote S. 3(3), 27–46 (2015). [CrossRef]

24. G. Vivone, L. Alparone, J. Chanussot, M. D. Mura, A. Garzelli, G. A. Licciardi, R. Restaino, and L. Wald, “A Critical Comparison Among Pansharpening Algorithms,” IEEE Trans. Geosci. Remote 53(5), 2565–2586 (2015). [CrossRef]

25. V. K. Shettigara, “A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set,” Photogramm. Eng. Remote Sensing 58(5), 561–567 (1992).

26. C. Thomas, T. Ranchin, L. Wald, and J. Chanussot, “Synthesis of multispectral images to high spatial resolution: a critical review of fusion methods based on remote sensing physics,” IEEE T. Geosci. Remote 46(5), 1301–1312 (2008). [CrossRef]

27. H. F. Kaiser, “The varimax criterion for analytic rotation in factor analysis,” Psychometrika 23(3), 187–200 (1958). [CrossRef]

28. X. Otazu, M. G. Audicana, O. Fors, and J. Nunez, “Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods,” IEEE Trans. Geosci. Remote 43(10), 2376–2385 (2005). [CrossRef]

29. S. Bennett and J. Lasenby, “ChESS – Quick and robust detection of chess-board features,” Comput. Vis. Image Un. 118, 197–210 (2014).

30. Signal and Image Processing Institute, University of Southern California, “The USC-SIPI Image Database,” http://sipi.usc.edu/database/

31. S. K. Mitra and L. Lucchese, “Using saddle points for subpixel feature derection in camera calibattion targets,” in Proceedings of IEEE Asia-Pacific Conference on Circuits and Systems (IEEE, 2002), pp. 191–195.

32. B. Aiazzi, S. Baronti, and M. Selva, “Improving component substitution pansharpening through multivariate regression of MS +Pan Data,” IEEE Trans. Geosci. Remote 45(10), 3230–3239 (2007). [CrossRef]

33. B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva, “MTF-tailored multiscale fusion of high-resolution MS and pan imagery,” Photogramm. Eng. Remote Sensing 72(5), 591–596 (2006). [CrossRef]

34. L. Wald, T. Ranchin, and M. Mangolini, “Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images,” Photogramm. Eng. Remote Sensing 63(6), 691–699 (1997).

Group Number
Element	-2	-1	0	1	2	3	4	5	6	7	8	9
1	0.250	0.500	1.00	2.00	4.00	8.00	16.0	32.0	64.0	128.0	256.0	512.0
2	0.281	0.561	1.12	2.24	4.49	8.98	17.96	35.9	71.8	143.7	287.4	574.7
3	0.315	0.630	1.26	2.52	5.04	10.08	20.16	40.3	80.6	161.3	322.5	645.1
4	0.354	0.707	1.41	2.83	5.66	11.31	22.63	45.3	90.5	181.0	362.0	724.1
5	0.397	0.794	1.59	3.17	6.35	12.70	25.40	50.8	101.6	203.2	406.4	812.7
6	0.445	0.891	1.78	3.56	7.13	14.25	28.51	57.0	114.0	228.1	456.1	912.3

Symbol	Description
*MS₀*	Original MS image
*MS_sum*	Sum of different bands of *MS₀*
*PAN₀*	Original PAN image
**$M S_{h}^{}$***	Fusion of *MS₀* and *PAN₀*
**$M S_{d}^{}$***	Downsampled version of $M S_{h}^{}$ at the scale of MS₀*
*MS_l*	Downsampled MS image by the spatial resolution ratio between *MS₀* and *PAN₀*
*PAN_l*	Downsampled PAN image at the scale of *MS₀*
**$M S_{o}^{}$***	Fusion of *MS_l* and *PAN_l*
**$M S_{u}^{}$***	Upsampled version of *MS_l* at the scale of *MS₀*

Group Number
Element	-2	-1	0	1	2	3	4	5	6	7	8	9
1	0.250	0.500	1.00	2.00	4.00	8.00	16.0	32.0	64.0	128.0	256.0	512.0
2	0.281	0.561	1.12	2.24	4.49	8.98	17.96	35.9	71.8	143.7	287.4	574.7
3	0.315	0.630	1.26	2.52	5.04	10.08	20.16	40.3	80.6	161.3	322.5	645.1
4	0.354	0.707	1.41	2.83	5.66	11.31	22.63	45.3	90.5	181.0	362.0	724.1
5	0.397	0.794	1.59	3.17	6.35	12.70	25.40	50.8	101.6	203.2	406.4	812.7
6	0.445	0.891	1.78	3.56	7.13	14.25	28.51	57.0	114.0	228.1	456.1	912.3

Symbol	Description
*MS₀*	Original MS image
*MS_sum*	Sum of different bands of *MS₀*
*PAN₀*	Original PAN image
**$M S_{h}^{}$***	Fusion of *MS₀* and *PAN₀*
**$M S_{d}^{}$***	Downsampled version of $M S_{h}^{}$ at the scale of MS₀*
*MS_l*	Downsampled MS image by the spatial resolution ratio between *MS₀* and *PAN₀*
*PAN_l*	Downsampled PAN image at the scale of *MS₀*
**$M S_{o}^{}$***	Fusion of *MS_l* and *PAN_l*
**$M S_{u}^{}$***	Upsampled version of *MS_l* at the scale of *MS₀*

High resolution snapshot imaging spectrometer using a fusion algorithm based on grouping principal component analysis

Abstract

1. Introduction

2. Principle of HR-SIS

3. Fusion algorithm based on GPCA

4. Results and Discussion

4.1 Spectral accuracy

4.2 Spectral resolution

4.3 Spatial resolution

4.4 Insensitivity to metamerism

4.5 Performance of the fusion algorithm based on GPCA

5. Conclusions

Funding

References and links

Cited By

Figures (14)

Tables (2)

Equations (17)

Optics Express