1. Introduction

Multispectral imaging techniques are utilized as an effective tool to measure and analyze targets noninvasively. In addition, the utilization of multispectral images has been expected in the field of color image reproduction with high fidelity [1–5]. However, multispectral image acquisition requires bulky imaging systems or time-consuming processes that include data scanning along either the wavelength or spatial axis. As a result, the applications are limited to several typical fields, such as remote sensing and microscopic imaging. Recently, snapshot spectral imaging techniques have been developed, which directly acquire a spectral image in a single exposure (e.g., see [6]). However, thus far, the image quality is insufficient compared to that obtained by a conventional scanning-type imaging system. To utilize multispectral imaging techniques in wider applications, compact and one-shot imaging instruments that realize high accuracy are strongly anticipated.

Image sensors based on color filter arrays (CFAs) are widely used to capture color images, and interpolation techniques for each type of CFA are vigorously investigated. Recently, multispectral image acquisition techniques using the multispectral filter array (MSFA) were proposed as an extension of the CFA-based method [7–9]. However, such an approach is inappropriate when the number of spectral bands is large (e.g., more than ten bands), because the spatial sampling becomes too sparse to reconstruct spectral images by interpolation-based methods.

On the other hand, a hybrid-resolution spectral imaging scheme has been proposed as a new type of spectral imaging technique [10–15]. In this scheme, spectral images are estimated from two types of data with different spatial and spectral resolutions: for instance, high-resolution RGB images and corresponding low-resolution spectral data. Although the data obtained by an MSFA-type image sensor can be regarded as a kind of low-resolution spectral data, a hybrid-resolution approach has not yet been applied.

This paper proposes a new scheme for hybrid-resolution multispectral imaging based on high-resolution RGB images and the data obtained by an MSFA-type image sensor with narrow-band color filters. Because the previously proposed methods presuppose low-resolution spectral data measured at the same position for all bands, this paper presents a new reconstruction method that can be applied to the data of an MSFA.

Subsection 2.1 first introduces the data acquisition model of the proposed hybrid-resolution multispectral imaging, and Subsection 2.2 then presents a low-dimensional model of multispectral images. On the basis of the low-dimensional model, Subsection 2.3 proposes the reconstruction method for multispectral images. Section 3 shows the simulation results, and Section 4 gives the conclusions.

2. Method

This paper proposes a hybrid-resolution multispectral imaging, which produces a K-band multispectral image from a set of RGB images and the data of an MSFA with narrow-band filters of K colors (Fig. 1 ). These data can be obtained in one shot, if a camera is configured with four image sensors (three sensors for RGB and one for an MSFA) accompanied with a four-channel separation prism, as shown in Fig. 2 . The data acquisition model and reconstruction method for multispectral images are presented below.

 

Fig. 1 Conceptual diagram of hybrid-resolution multispectral imaging using multispectral filter array (MSFA).

Download Full Size | PDF

 

Fig. 2 An example of a configuration for a one-shot camera for hybrid-resolution multispectral imaging.

Download Full Size | PDF

2.1 Data acquisition model

Let us first introduce the data acquisition model, which is assumed in the proposed method. For simplicity, the original spectral image is treated below as discrete signals.

Let r_i be an L-dimensional column vector representing the spectral reflectance function of the original spectral reflectance image at pixel i, where L is the number of spectral samples and $1\le i\le N_x\times N_y=N$, with N the number of pixels. The original spectral reflectance image is represented by an $N\times L$ matrix,

First, the ideal K-band multispectral image is defined. The K-band multispectral data corresponding to r_i is represented by a K-dimensional column vector,

If we define ${\tilde{f}}_k$as the kth N-dimensional column vector of F, another representation of F is

Let us now introduce the representation of the data of an MSFA. The data obtained at the pixels assigned to kth band color filters is

Next is a model for RGB image capturing. Color imaging devices can be modeled as linear systems if the nonlinearity of the system is adequately corrected. Hence, the three-band RGB image is represented by

From {c₁,…,c_K} and G, we want to reconstruct the ideal K-band image F.

2.2 Localized low-dimensional model for multispectral images

Before presenting the reconstruction method, let us introduce a localized low-dimensional model for multispectral images to serve as the basis of the proposed reconstruction method.

A widely used approach is the representation of spectral reflectance functions by a linear combination of a relatively small number of basis functions. The required number of basis functions depends on the target data and the accuracy of the representation. If we consider the spectral reflectance functions included by a relatively small region of a spectral image, the variation of those spectral reflectance functions is limited. As a result, it is expected that only a small number of basis functions should be needed to represent the spectral reflectance functions in that region.

Based on this assumption, we derive a model for multispectral images in relation to RGB images. The model is a kind of low-dimensional approximation of multispectral images, which works within localized regions. Now we select a region, indexed by j, in which any spectrum can be approximately represented by a linear combination of three L-dimensional spectral basis vectors $u_q^j$ as

Using the operator []_j, Eqs. (3) and (7) are rewritten as

Substituting Eq. (9) into Eqs. (11) and (12), we have

Except for the cases in which the 3 × 3 matrix UH_RGB is singular,

Substituting Eq. (15) into Eq. (13),

Equation (17) indicates that any image band of a multispectral image can be approximately represented by a linear combination of the corresponding region of the RGB images, if the region is selected to satisfy both Eq. (8) and the nonsingularity condition on UH_RGB.

2.3 Multispectral image reconstruction method

In the proposed method, multispectral images are divided into regions, each of which satisfies Eq. (8) as far as possible. Then, the multispectral images are reconstructed independently for every region by using the MSFA observation. The reconstruction is based on the model of Eq. (17), where the three-dimensional vector a_kj is derived for each region and each band.

By referring to Eq. (6), the MSFA data for this region is represented by

If m_kj is the number of the pixels assigned to the kth color filter included in this region, [c_k]_j is an m_kj-dimensional column vector. By substituting Eq. (17) and setting $D_{kj}={\left[S_{k}G\right]}_j$, we have

Because we have both [c_k]_j and D_kj as measurement data, the weighting coefficients a_kj can be derived by minimizing the error between the sides of Eq. (19). If a simple linear regression analysis is applied, we have

Then, the kth image band of this region is estimated as

By repeating the calculation in Eq. (21) for all bands and all regions, the whole multispectral image is reconstructed.

Noise was ignored in the observation model represented by Eqs. (6) and (7). In actual, there should be random noise on the measurement data in real imaging systems. If noise is nonnegligible, some regularization methods should be introduced into the regression to derive the weighting coefficients. With an appropriate regularization method, the proposed method works properly, which will be presented in another paper. In addition, there may be the error caused by a misalignment among four image sensors. This kind of error should be removed before reconstructing multispectral images because the assumption of Eq. (17) does not hold with this error. The observation of a test target can be used to remove the misalignment.

To end this subsection, let us discuss how to determine the regions described above. The smaller the size of a region, the higher the probability that the region includes only spectra with similar features. As a result, the condition in Eq. (8) is easily satisfied. However, as the size becomes smaller (i.e., as m_kj becomes smaller), the regression to derive Eq. (20) becomes unstable. Therefore, these opposite influences should be considered in deciding the size of the regions. Regardless of the size, there are several ways to divide images into regions. A simple method is that an image be divided horizontally and vertically into small blocks. Because neighboring pixels in multispectral images are more likely to have similar spectral characteristics, simple block division can be effective. Another method is that the pixels with similar RGB values be collected into a single region.

3. Simulation results

Three hyperspectral images described by Foster et al. [16] were used for simulations. Because each band of the original hyperspectral images was captured in a time sequence, there are slight misalignments between image bands, especially in the regions of fluttering leaves. By eliminating the effect of these misalignments in the simulations, parts of the original images were cut out and resized. The resized images are shown in Fig. 3 as color images. The image of Scene3 consists of $256\times 256$ pixels, sampled every 10 nm from 400 nm to 720 nm (33 dimensions). The image of Scene5 consists of $480\times 320$ pixels, sampled every 10 nm from 400 nm to 710 nm (32 dimensions). The image of Scene6 consists of $480\times 320$ pixels, sampled every 10 nm from 400 nm to 720 nm (33 dimensions). For Scene 5, data for 720 nm were originally excluded, because the signal at this wavelength was excessively noisy.

 

Fig. 3 Hyperspectral images used in the simulations (presented as color images).

Download Full Size | PDF

In the simulations, data were assumed to have been captured by four image sensors with the same number of pixels as the hyperspectral images. Then, 16-band images with the same numbers of pixels were reconstructed. Three systems, including the proposed approach, were compared (Fig. 4 ).

 

Fig. 4 Configurations of 16-band imaging systems compared in the simulations.

Download Full Size | PDF

For the proposed hybrid-resolution approach, three image sensors were assigned to RGB, and the remaining one 16-band MSFA-type image sensor (RGB + 16FA) was assigned. The spectral sensitivity of the RGB channels was assumed to be that of typical HDTV video cameras. The spectral sensitivity functions of the 16 narrow-band color filters were those obtained by approximately dividing the wavelength range spanned by the RGB sensitivities into 16 equally. The spectral sensitivities of them are shown in Fig. 5 (left and middle panels). The image was divided horizontally and vertically into 16 × 16-pixel regions, and the reconstruction in Eqs. (20) and (21) was applied to each region. The size of the regions was approximately selected at the smallest possible under the condition that the regression be performed stably.

 

Fig. 5 Spectral sensitivities of R, G, and B color filters (left), 16 narrow-band color filters (middle), and four color filters for RG1G2B (right) used in simulation.

Download Full Size | PDF

For comparison, two different types of imaging system were simulated. One of these was a 4-band imaging system called RG1G2B. The four image sensors were assigned to R, short-pass G, high-pass G, and B channels. The spectral sensitivities of the four color filters are shown in the right panel of Fig. 5. From the 4-band image data, 16-band data were reconstructed pixel by pixel through Wiener estimation with the spectral correlation matrix formulated on the basis of a Markov process [17]. In another system, called 4FA × 4, every image sensor was a four-band MSFA-type image sensor, but with different color filters; i.e., the first MSFA image sensor was assigned to the color filters of bands 1–4, the second was assigned to the color filters of bands 5–8, and so on. Since each image band was sampled every four pixels, each image band was independently reconstructed by linear interpolation in the spatial domain.

For reference, ideal 16-band images were generated using the spectral sensitivities of the 16 narrow-band color filters of the MSFA. These are called the original 16-band images below.

Figure 6 shows the normalized root-mean-square error (NRMSE) for each image band of the 16-band images reconstructed by the three methods. The error of 4FA × 4 is generally higher than others across the bands, while the error of the proposed system is generally lower. The error of RG1G2B is high in several specific image bands. We can see the effectiveness of the proposed method from these results.

 

Fig. 6 Simulation results: NRMSE of 16-band images.

Download Full Size | PDF

To reveal the spatial features of the reconstructed 16-band images, enlarged 70 × 70-pixel regions of Scene3 and Scene6 are shown in Figs. 7 and 8 , respectively. Three-band images selected from a 16-band image were allotted to RGB channels and rendered as color images. The selected band numbers are indicated at the top of each figure. In the images by 4FA × 4 we can see apparent spatial degradation, including blur and color artifacts. Conversely, in the images by RGB + 16FA (proposed approach) and RG1G2B there is no such spatial degradation. A closer look at the images by RG1G2B shows a slight color change in the images of (11,6,1) and (15,10,5) in Fig. 7, which indicates the existence of error in the corresponding spectral components. As for RGB + 16FA, a distinct error in the form of a spurious magenta line arises in the image of (12,7,2) in Fig. 8. We detect such errors only in three regions, including this one in the image of Scene6, which is expected to be improved by introducing the regularization techniques into the deriving of a_jk. Except for this kind of error, there is no perceived error in the reconstruction of images by the proposed RGB + 16FA.

 

Fig. 7 Enlarged 70 × 70-pixel region of 16-band image of Scene3. For this presentation, three-band images were allotted to RGB channels. The corresponding band numbers are indicated at the top.

Download Full Size | PDF

 

Fig. 8 Enlarged 70 × 70-pixel region of 16-band image of Scene6. For this presentation, three-band images were allotted to RGB channels. The corresponding band numbers are indicated at the top.

Download Full Size | PDF

To see the spectral features contained in the reconstructed 16-band images, principal components of 16-band images were evaluated. The principal components of the original 16-band image, V = (v₁,…,v₁₆), can be derived as the eigenvectors of FF^T. These are sixteen 16-dimensional vectors ordered by the magnitudes of the corresponding eigenvalues. In a similar way, the principal components of the 16-band images reconstructed by RGB + 16FA, RG1G2B, and 4FA × 4 were derived; i.e., V_{RGB + 16FA}, V_RG1G2B, and V_{4FA × 4}, respectively. Figure 9 shows V^TV_{RGB + 16FA}, V^TV_RG1G2B, and V^TV_4FAx4 in matrix form, where the range of 0–1 is shown in 8-bit grayscale. Because of the orthonormality of principal components, the matrix becomes an identity matrix when the principal components derived from the reconstructed 16-band image are equivalent to V.

 

Fig. 9 Correspondence between principal components derived from original and reconstructed 16-band images. Ideally, matrix becomes an identity matrix.

Download Full Size | PDF

In the case of RG1G2B, we can derive only four principal components, as the information on principal components with orders higher than fourth is completely lost. As for 4FA × 4, we cannot see any correspondence between two sets of principal components at orders higher than third. Conversely, for RGB + 16FA the power is concentrated along the diagonal, which indicates that the spectral features are well reconstructed.

4. Conclusions

In this paper, we proposed a new scheme for hybrid-resolution multispectral imaging using the data of an MSFA-type image sensor as low-resolution spectral data. By reconstructing a multispectral image as a linear combination of RGB images, the degradation in spatial resolution can be suppressed despite the low sampling ratio of each image band. In addition, since the weighting coefficients for the linear combination are derived for each small region, it was confirmed that spectral features of more than three dimensions can be reconstructed. For further improvement, the following topics will be investigated in the future: a selection method for the region in the reconstruction, a more appropriate or more advanced method of regression to derive the weighting coefficients, including regularization techniques, and verification of the localized low-dimensional model for multispectral images.