Spectra estimation from raw camera responses based on adaptive local-weighted linear regression

Jinxing Liang; Kaida Xiao; Michael R. Pointer; Xiaoxia Wan; Changjun Li

doi:10.1364/OE.27.005165

1. Introduction

Digital cameras can be used to provide spectral data for many applications and thus the development of algorithms to calculate these spectral data from image RGB data is of prime importance. Different digital camera-based spectral imaging systems have been developed for practical applications, such as a camera with bandpass filters [1–3], a camera utilizing multiple illuminants [1,4,5], and a camera with three-channel responses (but a single RGB image) under a specific illuminant [6–13]. The latter system, which can be used to estimate the spectral reflectance from a single RGB image, has received considerably more interest because of the low cost of the devices with high resolution and their convenience in practical applications. The inherent problem of image registration which exists in optical bandpass filter-based spectral imaging systems [1–3,14] can be overcome by just one exposure using a conventional digital RGB camera. Accurate spectral and colorimetric estimation are both critical for practical applications and, since the digital RGB values are readily available from an image, an algorithm that accurately estimates the spectral reflectance from these RGB camera responses can be very useful.

Methods to derive spectral data from camera responses can be divided into two types: model-based and training-based [15]. Since it is complex and expensive to characterize the camera sensitivity functions for the model-based method, the more easily implemented training-based method is both more convenient and more practical. Many training-based spectral estimation methods have been proposed in recent years [5–13,16]. Connah [6], Heikkinen [5], and Shen [8] proposed the use of a nonlinear regression method based on a polynomial model for spectral estimation, with consideration being given to the potential over-fitting problem in the polynomial-based regression model. Xiao et al. [10] illustrated that it is effective to combine the polynomial model with the eigenvector space of principal component analysis (PCA) for skin reflectance estimation from camera responses. These methods all use global training-based techniques.

For the training-based spectral estimation method, the selection and processing of the training samples is crucial to the performance in terms of estimation accuracy. In general: 1) the performance is better if the training and testing samples are as similar as possible in that they come from the same volume of color space [10,15]; 2) the local training-based approach is perhaps more practical for spectral estimation since too many training samples may increase the calculation time but without an accompanying increase in the estimation accuracy [17,18]; and 3) the performance is usually improved if the training samples are assigned appropriate weights, according to their distance from the test sample, when calculating the transformation matrix [12,13,16].

Li [9] and Cao [19] proposed estimation methods based on local linear approximation. In their studies however, the estimation accuracy is limited by the hypothesis of mapping the linearity of tristimulus color space to spectral reflectance space. Similar to the Xiao method [10], Zhang et al. [11] proposed a spectral estimation method based on CIE XYZ tristimulus values under multiple illuminants by selecting local training samples. Heikkinen et al. [7] also proposed a regularized kernel-based learning framework in local training form for spectral estimation. While these methods are local training-based methods, the weighting of the training samples was not considered.

Shen and Xin [16] proposed a spectral estimation method from scanner responses by weighting the pre-selected local training samples. And in our previous study [12], we proposed a local-weighted nonlinear regression method for spectral estimation from camera responses. The weighted nonlinear regression model has also been recently tested by Amiri and Fairchild [13] in a global weighting form. These methods can yield reasonable spectral estimation accuracy, but they are not adaptive enough in selecting of the local training samples for each of the testing samples in predicting their spectral reflectance.

The estimation of spectral reflectance from camera responses is actually to deal with a linear inverse problem [15]. However, most of the existing methods are based on the nonlinear regression model. Although spectral estimation based on raw linear camera responses has been proposed in some methods [11,13], spectral estimation from raw camera responses based on the linear regression model has not been well studied [20]. In addition, for the current weight-based spectral estimation methods, the weighting matrix is constructed based on the distance between the testing and the training samples in the camera RGB color space. The camera RGB color space is device-dependent and non-uniform [21]. The distance calculated in the camera RGB color space does not represent the true perceptual color difference. It makes more sense to calculate the distance in the device-independent and perceptually uniform color space. Furthermore, different existing methods yield different performance because they are based on different concepts. It is however, difficult to simultaneously minimize spectral and colorimetric errors as illustrated by Amiri and Fairchild [13]. Sometimes, the accuracy of the estimated spectral reflectance is satisfactory while the colorimetric accuracy is not considered good enough, and sometimes the situation is the opposite. Thus, an estimation method that can simultaneously obtain the best spectral and the best colorimetric accuracy is required.

With the above in mind, this study intends to develop a more accurate spectral estimation method from raw camera responses. The novelty of this research for spectral estimation is to use Gaussian weighted linear regression model for spectral estimation, making sure that both the input data and the estimation algorithm are in line with the linear imaging model. At the same time, the adaptive selection of the optimal local training samples is integrated into the estimation method by using Gaussian weighting. The weighting matrix is calculated in CIELAB uniform color space. The superior performance of this proposed method is evaluated and compared with existing methods by using a standard color chart and a set of textile samples, with a digital RGB camera. Overall performance is evaluated in terms of both spectral and colorimetric accuracy by using ten times ten-fold cross-validation [22]. Statistical analysis is performed by using one-way ANOVA and Tukey’s post hoc tests (α = 0.05). The comparison of computational times between the different methods is also reported.

2. Imaging model

Based on the color imaging principle of digital still RGB cameras [23–25], the raw responses are first acquired through a linear combination of the camera system sensitivity functions, the spectral power distribution of the light source, the surface spectral reflectance of the object and the system noise. Assuming a linear opto-electronic transfer function in the camera, the imaging model for raw responses can be formulated by Eq. (1)

d_{i} = \int_{λ_{\min}}^{λ_{\max}} l (λ) s_{i} (λ) r (λ) d λ + n_{i} .

where the camera responses d_i are related to the channel i (i = 1, 2, 3) of a pixel; λ is the wavelength, ranging from λ_min to λ_max in the visible wavelength range; l(λ) is the spectral power distribution of the light source; s_i(λ) is the spectral sensitivity of the ith channel of the system; r(λ) is the spectral reflectance of a pixel; n_i is the system noise of the digital camera.

After the raw image is acquired, it is usually post-processed to obtain a more colorful output image and, in general, the linearity of the original response will be modified during this post-processing. Figure 1 shows the linear difference between the raw responses extracted from a .NEF format image (Nikon Electronic Format - a raw image format used solely on Nikon cameras) and the post-processed responses extracted from a corresponding .tiff format image. The grayscale responses of the Gretag Macbeth ColorChecker classic chart that were captured by a Nikon D7000 digital camera were used to illustrate these linear differences. The response values are plotted versus the luminance factor in Fig. 1. It is apparent that the linearity of the camera responses is significantly reduced by the in-camera post-processing, where the R-square values of linear fitting for each channel are reduced from 0.9994, 0.9986, 0.9976 to 0.8268, 0.8177, 0.8172, for the R-, G- and B-channel respectively.

Fig. 1 Linearity comparison between raw and post-processed camera responses as a function luminance factor: (a) R-channel, (b) G-channel, and (c) B-channel.

Download Full Size | PDF

It is clear from Fig. 1 that after the raw image is processed, the responses of each channel are no longer linear but better represented by a nonlinear law which is more complex than a simple power law. In addition, the post-processing methods of different camera manufacturers are usually different and are difficult to accurately simulate or characterize [11,23–25], thus the post-processing step is hard to accurately model. Therefore, the better choice for spectral estimation from camera response is to use the linear raw responses and hence the linear imaging model formulated by Eq. (1) forms the basis for the proposed method in this study. Ignoring the system noise, Eq. (1) can be written in matrix notation as expressed by Eq. (2)

d = M r .

where d is the response vector; M is the overall spectral sensitivity matrix including the product of the matrix form of l(λ), and s_i(λ); r denotes the spectral reflectance vector. Note that the system noise is ignored on the assumption that it is small in comparison with the signal: it is recognized that this is an assumption that is difficult to prove because the actual noise is itself difficult to measure. For spectral reconstruction, the goal is to reconstruct the high-dimensional reflectance

\tilde{r}

from the low-dimensional camera responses vector d indicated by Eq. (3)

\tilde{r} = Q d .

where Q denotes the transformation matrix. The accuracy of reflectance reconstruction is determined by the method of solving for the transformation matrix Q [15,19,26]. The widely used pseudo-inverse (PI) algorithm is used in the method proposed in this study [19].

3. Proposed method

3.1 Raw responses extraction

Raw camera responses without post-processing play a very important role in many research fields including photography, image processing, machine vision, etc [27–29]. and several methods for obtaining the raw data from images have been developed [24], including RawDigger which is bespoke software available via the internet. A detailed comparison test of several methods showed that they all output the same Bayer image. In our study, we used RawDigger to extract the coordinates of samples and used the ‘DNG (Digital Negative) approach’ that is described by Sumner [24] to read and process a Bayer image in MATLAB.

In order to obtain a three-channel raw image from the Bayer image, a demosaicing step is necessary to interpolate between the pixels [30,31]. The influence of different demosaicing methods on the accuracy of spectral estimation was tested by using the spectral estimation methods described by Shen et al. [8]. We found that the demosaicing step does not affect the accuracy of the spectral estimation. Therefore, the demosaicing method built into MATLAB R2016a [30] was selected for the Bayer image demosaicing process. It should be noted that the demosaicing test in this study was based on uniform color samples. If the estimated spectral image is to be used for high-fidelity color reproduction, a demosaicing method that can effectively avoid Moiré patterns should be selected.

3.2 Proposed method

Figure 2 presents the framework of the proposed method. At first, the training and testing samples are both captured under the same imaging conditions, and the raw responses of the samples are extracted from the captured images as described in section 3.1. The spectral reflectance of each of the training samples is measured with a spectrophotometer.

Fig. 2 Framework of the proposed spectral reflectance estimation method from raw camera responses based on Gaussian weighted linear regression model.

Download Full Size | PDF

Secondly, the training samples are used to derive a matrix to convert from device-dependent camera RGB color space to CIE XYZ color space based on the least squares method. With the established conversion matrix, both the raw camera responses of the training and testing samples are converted from from device-dependent camera RGB color space to CIE XYZ color space. Then the CIELAB coordinates of both the training and testing samples can be calculated from the corresponding CIE XYZ tristimulus values [32].

Thirdly, based on the general assumptions for the training-based spectral estimation method, an optimization strategy is proposed by use of a local-weighted method. The Gaussian weighting function that is widely used in machine learning [33] is selected for this study as it can help to adaptively determine the optimal local training samples during the weighting process. The weighting matrix is calculated as follows. Firstly, the Euclidean distance between each of the testing samples and the training samples is calculated in CIELAB color space, as shown in Eq. (4)

s_{j} = \sqrt{{(L_{test}^{*} - L_{train, j}^{*})}^{2} + {(a_{test}^{*} - a_{train, j}^{*})}^{2} + {(b_{test}^{*} - b_{train, j}^{*})}^{2}} (j = 1, 2, \dots, N) .

where

L_{test}^{*}

,

a_{test}^{*}

, and

b_{test}^{*}

denotes the colorimetric values of a testing sample in CIELAB color space [32];

L_{train, j}^{*}

,

a_{train, j}^{*}

, and

b_{train, j}^{*}

denotes the colorimetric values of the jth training sample; N denotes the number of training samples; s_j denotes the Euclidean distance (the CIELAB color difference) between the jth training sample and the testing sample. Secondly, the weighting coefficient is calculated for each of the training samples, as shown by Eq. (5)

w_{j} = \exp (\frac{s_{j}}{- 2 k^{2}}) (j = 1, 2, \dots, N) .

where w_j is the weighting coefficient of the jth training sample; k is the parameter used to determine the effective training samples and their weighting coefficient [33]. For each testing sample, the corresponding weighting coefficients for the training samples will be normalized between 0 and 1 based on the maximum value. If a suitable value of k is set in Eq. (5), the optimal local training samples will be adaptively determined when calculating the weights of the training samples. Thirdly, the weighting matrix W is established according to Eq. (6)

W = {[\begin{array}{l} w_{1} 0 \dots 0 \\ 0 w_{2} 0 ⋮ \\ ⋮ 0 ⋱ 0 \\ 0 \dots 0 w_{N} \end{array}]}_{N \times N} .

Fourth, after the weighting matrix has been constructed, the raw responses are expanded based on the linear model as indicated by Eq. (7)

d_{\exp} = [1 r g b]' .

where d_exp is the 4 × 1 vector of the expanded raw responses; r, g and b denote the raw responses for R-, G- and B-channel of a pixel. With the expanded raw responses, the transformation matrix Q is calculated by Eq. (8)

Q = R_{train} W {(D_{train,exp} W)}^{+} .

where W is the N × N weighting matrix for the training samples; R_train denotes the 31 × N spectral reflectance matrix of the training samples; D_train,exp denotes the expanded 4 × N raw responses matrix of the training samples; and the superscript ‘ + ’ indicates the pseudo-inverse matrix operator. Therefore, the spectral reflectance of the testing sample is estimated by using the 31 × 4 transformation matrix Q, as shown in Eq. (9)

{\tilde{r}}_{test} = Q d_{test,exp} .

where

{\tilde{r}}_{test}

denotes the estimated spectral reflectance of the testing sample; d_test,exp denotes the expanded raw responses vector of the testing sample.

4. Experiment

The experiment was carried out using a Verivide DigiEye imaging system software and a VeriVide light cabinet, in a dark room (with all the lights in the laboratory turned off during the experiment). The inner walls of the light cabinet were painted neutral and provided a approximately diffuse illumination. The D65 simulator tubes were fitted with translucent diffuse reflectors and provided continuing illumination for approximately uniform over the chart area.

The Xrite ColorChecker semi-gloss chart (CCSG, 140 color patches that gave an approximately uniform sampling of CIELAB color space), and a set of textile samples (Textile, 172 textile samples that were approximately uniformly distributed in the color space) were used for the experiment. The color distribution of the samples of the CCSG chart and the textile samples are plotted in an a*-b* plane of CIELAB color space in Fig. 3. The spectral reflectances of the CCSG chart and textile samples were measured using a Konica Minolta CM-2600d and a CM-3600A spectrophotometer respectively, and a Nikon D7000 digital RGB camera controlled by the DigiEye imaging system software was used to capture the images. The imaging plane of the digital camera was set to be approximately parallel to the sample placement plane. The exposure time and aperture size were carefully set to acquire the maximum unclipped digital signals for the white patches, with a minimum ISO value of 100. The average digital values of a fixed area (50 × 50 pixels for CCSG and 120 × 120 pixels for Textile, approximating to the spectrophotometer measurement aperture area) from the centre of each sample patch were used.

Fig. 3 Color distribution of the samples in an a*-b* plane of CIELAB color space: (a) color distribution of the CCSG chart, and (b) color distribution of the textile samples.

Download Full Size | PDF

To test the model more robustly and more fairly, a ten times ten-fold cross-validation approach [22] was used to evaluate the proposed method. For both color chart and textile samples, the full sample set is randomly partitioned into approximately ten equal sized subsets. Of the ten subsets, a single subset is retained as the validation data for testing the model, and the remaining nine subsets are used as training data for each fold test. The average CIELAB color difference (ΔE_ab) under several commonly used CIE illuminants for color evaluation including D65, D50 and A, were used as suitable colorimetric evaluation metrics, and the color matching functions (CMF) of the CIE1931 standard observer were used for the colorimetric calculations. The root-mean-square error (RMSE) and goodness-of-fit coefficient (GFC) were used as spectral evaluation metrics. The calculation of the RMSE and the GFC are defined by Eq. (10) and Eq. (11) respectively

RMSE = \sqrt{\frac{1}{n} {(\tilde{r} - r)}^{T} (\tilde{r} - r)},

GFC = \frac{{\tilde{r}}^{T} r}{‖ {\tilde{r}}^{T} \tilde{r} ‖ \cdot ‖ r^{T} r ‖} .

where

\tilde{r}

denotes the reconstructed spectral reflectance; r denotes the measured reference spectral reflectance; subscript ‘T’ denotes the matrix transpose; n denotes the wavelength sampling number of the spectral reflectance in the visible spectrum: for a wavelength range from 400 nm to 700 nm at 10 nm intervals, n = 31.

5. Results and analysis

In this section, the proposed method is implemented and compared with the currently existing methods in 5.1. Meanwhile, the factors that will influence the estimation accuracy of the proposed method is also investigated and discussed in 5.2 and 5.3.

5.1 Methods implementation and comparison

The proposed adaptive local-weighted linear regression method (ALWLR) and the currently existing methods were first implemented and compared under the defined experimental environment by using the raw camera responses. The compared existing methods included the ordinary least squares (OLS) method [6], the regularized least squares (RLS) method [5,6], the partial least squares (PLS) method [8], the principal component analysis (PCA) method [10], the locally linear approximation (SR-LLA) method [9], the optimized adaptive estimation (o-AE) method [16], the regularized kernel (Kernel) method [7], the global weighted nonlinear regression method proposed by Amiri and Fairchild (the Amiri method) [13], the local nonlinear regression method proposed by Zhang and Wang et al. (the Zhang method) [11], and the local-weighted nonlinear regression method proposed by Liang and Wan (the Liang method) [12].

As noted in section 3.2, the value of k in Eq. (5) is crucial for the weighting process. By adjusting the value of k, optimal local training samples (a different number of effective training samples for each of the testing samples) will be automatically determined as part of the weighting step. Therefore, the influence of the value of k on the estimation accuracy of the proposed method was first explored.

Figure 4 shows the relationship between the mean spectral reflectance RMSE, the mean CIELAB color difference of the CCSG chart and textile samples and the value of k within the usual range of values. It is found that for both the CCSG chart and the textile samples, the evaluation metrics initially decreased with an increase in k, and then they increase after they reach an optimal value of 0.18 and 0.22 respectively. Thus, we consider these values to be optimum for the CCSG charts and the textile samples respectively. Furthermore, it is clear from Fig. 4 that the evaluation metrics of both RMSE and CIELAB color difference show exactly the same trend with the changes in the value of k. Therefore, the proposed method in this study can simultaneously minimize the spectral and colorimetric errors.

Fig. 4 (a) the relationship between spectral estimation accuracy and k for the CCSG chart, and (b) the relationship between spectral estimation accuracy and k for the textile samples.

Download Full Size | PDF

With the selected optimal values of k for CCSG chart and textile samples, the proposed method was compared with the existing methods as listed above. All the existing methods were implemented in their optimal conditions, while six principal components were used for the PCA based method. The mean spectral estimation results of these methods using ten times ten-fold cross-validation are calculated and summarized in Table 1. The RMSE and GFC were used for spectral accuracy evaluation, and the CIELAB color difference under three illuminations was used for colorimetric accuracy evaluation. The first five methods in Table 1 are global training-based methods and the later five methods are local training-based methods, and the last one is the proposed local training-based method.

Table 1. Estimation accuracy of the methods for CCSG and textile samples in terms of RMSE, GFC and CIELAB color difference using ten times ten-fold cross validation

View Table | View all tables in this article

As can be seen from Table 1, the proposed method has the smallest value in RMSE and CIELAB color difference and the largest value in GFC compared to the existing methods, which clearly indicates that it outperforms in both spectral and colorimetric estimation. For the CCSG chart, the mean RMSE and CIELAB color difference for the proposed method are approximately 2.38 and 2.21, whereas the best performance for all existing methods is 2.76 and 2.46 (Liang). The proposed method reduces the predictive error by approximately 0.4 for both RMSE and CIELAB color difference. The maximum error for the proposed method (6.62 and 6.85 for RMSE and CIELAB color difference respectively) is much smaller than Liang’s method (10.35 and 9.08 for RMSE and CIELAB color difference respectively). For the textile chart, the mean RMSE and CIELAB color difference for the proposed method is approximately 2.68 and 1.85 respectively. The best performance for all existing methods is 3.01 (Liang) and 2.21 (o-AE). The proposed method enhances the accuracy by approximately 0.3 for both RMSE and CIELAB color difference. And the maximum error for the proposed method (7.53 and 5.21 for RMSE and CIELAB color difference respectively) is still much smaller than most of the existing methods.

To illustrate both overall predictive accuracy and their error distribution of all methods, Fig. 5 is plotted using boxplots to represent all data tested in the ten times ten-fold validation. Figures 5(a) and 5(b) represent distributions of RMSE and CIELAB color difference for the CCSG chart, while Figs. 5(c) and 5(d) represent results of RMSE and CIELAB color difference for the textile set. The top of the blue rectangle indicates the upper quartile, a horizontal red line near the middle of the rectangle indicates the median, and the bottom of the blue rectangle indicates the lower quartile. The topmost outlier or a vertical line extends from the top of the rectangle to indicate the maximum value, and the bottom end outlier or the vertical line extends from the bottom of the rectangle to indicate the minimum value. The outliers (blue and red circles in Fig. 5) that are above or below the box body indicate that their values are greater than the upper quartile plus 1.5 times the interquartile range or less than lower quartile minus 1.5 times the interquartile range.

Fig. 5 Boxplot distributions of spectral error and colorimetric error of the proposed method and the existing methods: (a) boxplot distribution of the RMSE of CCSG chart, (b) boxplot distribution of the CIELAB color difference of CCSG chart, (c) boxplot distribution of the RMSE of textile samples, and (d) boxplot distribution of the CIELAB color difference of textile samples.

Download Full Size | PDF

As illustrated in Fig. 5, for both the CCSG and the textile samples, the median error of the proposed method is smaller than the existing methods, and the boxplot distribution of estimation results of the proposed method is more compact than the existing methods. In addition, the maximum error of the proposed method is also smaller than that for most of the existing methods. And furthermore, there are more outliers with bigger error values in most of the existing methods for CIELAB color difference as indicated in Figs. 5(b) and 5(d). The boxplot of the overall error distribution for the ten times ten-fold validation in Fig. 5 shows that there are more small error samples estimated by the proposed method, which is more intuitive to prove the superiority of the proposed method. Moreover, the one-way ANOVA and Tukey’s post-hoc tests were performed to test the difference between the mean values of the error distributions in Fig. 5 with an acceptable confidence level of α = 0.05. The statistical analysis of all methods showed that the proposed method was significantly different from the mean values of the analyzed metrics of the compared methods in Table 1.

For a more comprehensive comparison of the proposed method with the existing methods, the computational times of the proposed method and existing methods for the CCSG chart and the textile samples with the ten times ten-fold cross-validation are recorded under the same calculation conditions (ASUS X550V laptop, 4G memory, Intel(R) Core(TM) i5-3230M CPU, 2.60GHz, 64-bit system, MATLAB 2016a software) and are summarized in Table 2. The computational efficiency is in seconds. It can be seen from Table 2 that except for the PCA, Kernel and Zhang methods, all the methods show very similar computational efficiency, and the computational efficiency of the proposed method is even slightly better than some of the existing methods.

Table 2. The comparison of computational times (in seconds) between the different methods for CCSG chart and textile samples using ten times ten-fold cross-validation

View Table | View all tables in this article

5.2 The influence of regression model on the proposed method

In the introduction section, we have illustrated that the estimation of spectral reflectance from camera responses is actually to deal with a linear inverse problem [15]. However, a common problem with existing methods is that most carry out the spectral estimation based on nonlinear regression models, even if the raw linear camera responses are used in [11,13]. Therefore, the influence of regression model on the spectral estimation accuracy of the proposed method was explored.

Except for the linear regression models that are based on first-order polynomial expansion in the proposed method, the non-linear regression models that are based on second-, third-, and fourth-order (with 10, 20 and 35 items respectively in their standard forms) polynomial expansion were also tested. It should be known that when the non-linear regression models were used in the proposed method, and with the changing of the tested parameter k, the metrics of RMSE and CIELAB color difference do not always show the same changing trend. The optimal parameter value of k was selected based on the minimum of RMSE for this case. Therefore, 0.25, 0.35 and 0.37 were selected for second-, third-, and fourth-order polynomial regression model of CCSG chart, and 0.31, 0.37 and 0.53 for textile samples, respectively. Spectral estimation results of the proposed method solved by first-, second-, third-, and fourth-order polynomial regression model are summarized in Table 3, where the ‘ALWLR-1st’, ‘ALWLR-2nd’, ‘ALWLR-3rd’, and ‘ALWLR-4th’ represent the proposed method solved by first-, second-, third-, and fourth-order polynomial regression model, respectively.

Table 3. Spectral estimation results of the proposed method solved by first-, second-, third-, and fourth-order polynomial regression model

View Table | View all tables in this article

For a more intuitive comparison of the results of this regression models, the mean RMSE and mean CIELAB color difference of each regression model are plotted in Fig. 6.

Fig. 6 (a) Mean RMSE of the proposed method solved by first-, second-, third-, and fourth-order polynomial regression model, and (b) Mean CIELAB color difference of the proposed method solved by first-, second-, third-, and fourth-order polynomial regression model.

Download Full Size | PDF

As can be seen from Table 3 and Fig. 6, the linear regression model gave the best performance in general for the evaluation metrics overall. From the linear regression model to nonlinear regression models, as the degree of polynomial increases, the overall spectral estimation accuracy becomes worse for the evaluation metrics. Moreover, the maximum error for both RMSE and CIELAB color difference are also increases generally with the degree of polynomial increases. Results of one-way ANOVA and Tukey’s post-hoc test (with an acceptable confidence level of α = 0.05) showed a general statistically significant difference between the linear and non-linear regression models of the result in Table 3. Further more, the calculation efficiency actually decreases as the degree of polynomial increases, and the overfitting problem should also be considered for the higher degree polynomial models. Therefore, the linear regression model is more practical for spectral estimation from the raw camera responses.

5.3 The influence of color space on the proposed method

Another common problem that has been raised in the introduction section concerns the color space for distance calculation in constructing the weighting matrix. For all the existing methods, the distance is calculated in the device-dependent and perceptually non-uniform camera RGB color space, rather than in a device-independent and perceptually uniform color space. As the estimated spectral data are for both spectral and color applications, it should be more meaningful to calculate the weighting matrix in a device-independent and perceptually uniform color space, such as the CIELAB color space used in the proposed method. To test whether the color space for weighting matrix calculation will affect the spectral estimation accuracy as expected, the influence of the device-dependent camera RGB color space and CIELAB color space on the spectral estimation accuracy was tested.

The optimal values of k for CCSG color chart and textiles samples were also found in the device-dependent camera RGB color space as shown in Fig. 7. The optimal values of k for the CCSG color chart and the textiles samples are 0.12 and 0.09, respectively. The spectral estimation results of the proposed method with the weighting matrix calculated in CIELAB color space and device-dependent camera RGB color space are summarized in Table 4. ‘ALWLR-LAB’ and ‘ALWLR-RGB’ represent the proposed method with the weighting matrix calculated in CIELAB color space and device-dependent camera RGB color space, respectively. It is clear from Table 4 that ALWLR-LAB performs than ALWLR-RGB as expected. Therefore, it makes sense to calculate the distance in the device-independent and uniform color space for weighting matrix construction in the proposed method. The one-way ANOVA and Tukey’s post-hoc tests were also performed to test the difference between the mean values of the experiment results in Table 4 with an acceptable confidence level of α = 0.05. The statistical analysis showed that there was a general significant difference between the experiment results for the weighting matrix calculated in CIELAB and that in device-dependent camera RGB color space.

Fig. 7 (a) the relationship between spectral estimation accuracy and k for the CCSG chart in device-dependent camera RGB color space, and (b) the relationship between spectral estimation accuracy and k for the textile samples in device-dependent camera RGB color space.

Download Full Size | PDF

Table 4. Spectral estimation results of the proposed method with weighting matrix calculated in CIELAB color space and device-dependent camera RGB color space

View Table | View all tables in this article

6. Discussion

The purpose of this study was to investigate a new method to improve the accuracy of the estimation of spectral data from the raw camera responses in terms of spectral and colorimetric accuracy. Experiments illustrated that the proposed method improves the estimation accuracy in comparison with the existing methods. The factors that have contributed to the improvement on estimation accuracy (1), and the factor that influences the parameter selection of the proposed method (2) and some concerns of the proposed method for practical application (3) are discussed below, respectively.

(1) The outperformance of the proposed method is mostly due to simultaneously integration of the linear regression model (see Section 5.2) based on linear raw camera responses, the superior Gaussian weighting approach and the device-independent and perceptually uniform color space for weighting matrix calculation (see Section 5.3).
With Gaussian weighting, the selection of local training samples and the weighting of training samples are merged as one step. The local training samples are adaptively selected for each of the testing samples in the weighting process. This is different from the current local-weighted methods [12,16] with two separate weighting steps, where the local training samples are first selected and then the weighting coefficient is calculated. The introduce of Gaussian weighing function may be one of the reasons to ensure that the spectral and colorimetric error always exhibits the same trend with a change in the value of the parameter value k (cf. Figure 4 and Fig. 7). Thus, the spectral error and the colorimetric error can always be simultaneously minimized with the Gaussian weighing approach in the proposed method.
(2) The main factor that influences the selection of the optimal value of k in the proposed method is the difference of the data set, or more specifically, the color or spectral distribution characteristics of the data set in color or spectral space. It is easy to see from Fig. 3 that there is a significant distribution difference between the CCSG chart and the textile samples in the a*-b* plane of CIELAB color space. And the only factor that has led to the difference between the optimal value selection of the proposed method for the CCSG chart and textile samples (cf. Figure 4 and Fig. 7), is the difference of their color/spectral distribution characteristics. However, there are currently no metrics that can accurately describe the color or spectral distribution characteristics of a data set in color or spectral space, although principal component analysis (PCA) could possibly be used. The development of such metrics could form the aim of future research.
(3) In practical applications, a color chart or a sample database in different fields, such as printing, textile, cosmetics and so on, is easily acquired. Thus the optimal value of parameter k in the proposed method is easy to confirm for a specific application by using the multiple k-fold cross-validation training approaches. Figure 8 shows the stability of the estimation accuracy of the proposed method for CCSG chart and textile samples with ten times ten-fold cross-validation. The optimal values of k are 0.18 and 0.22 for CCSG chart and textile samples respectively as confirmed in Fig. 4. As is clear from Fig. 8, for a fixed value of k, the estimation accuracy shows a very small deviation between the results of the ten times ten-fold cross-validation, which means good stability is obtained between the ten times ten-fold cross-validation. Based on this stability analysis of the proposed method, we conclude that if the appropriate value of k is confirmed in a specific application, it will be appropriate for that specific application.

Fig. 8 (a) the stability of the proposed method for CCSG chart with ten times ten-fold cross-validation, and (b) the stability of the proposed method for textile samples with ten times ten-fold cross-validation.

Download Full Size | PDF

7. Conclusion

In conclusion, this study proposes a superior spectral estimation method to calculate spectral data from raw camera responses by use of the Gaussian weighted linear regression model. The performance of this method was comprehensively compared to the existing methods. Results showed that our method exhibited the best accuracy in terms of both the spectral and colorimetric estimation. The factors that contributed to the proposed method are discussed in detail. This method can be applied for spectral reflectance measurement in many fields including printing, textile, food, medical, cultural heritage, and so on.

Funding

National Natural Science Foundation of China (NSFC) (61575090, 61775169); China Scholarship Council (CSC) (201706270143); National Basic Research Program of China (Program 973) (2012CB725302).

References

1. F. H. Imai and R. S. Berns, “Spectral estimation using trichromatic digital cameras, ”in Proceedings of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives (Society of Multispectral Imaging of Japan, 1999), pp. 42–49.

2. Y. Murakami, T. Obi, M. Yamaguchi, and N. Ohyama, “Nonlinear estimation of spectral reflectance based on Gaussian mixture distribution for color image reproduction,” Appl. Opt. 41(23), 4840–4847 (2002). [CrossRef] [PubMed]

3. J. Liang, X. Wan, Q. Liu, C. Li, and J. Li, “Research on filter selection method for broadband spectral imaging system based on ancient murals,” Color Res. Appl. 41(6), 585–595 (2016). [CrossRef]

4. R. Shrestha and J. Y. Hardeberg, “Spectrogenic imaging: a novel approach to multispectral imaging in an uncontrolled environment,” Opt. Express 22(8), 9123–9133 (2014). [CrossRef] [PubMed]

5. V. Heikkinen, T. Jetsu, J. Parkkinen, M. Hauta-Kasari, T. Jaaskelainen, and S. D. Lee, “Regularized learning framework in the estimation of reflectance spectra from camera responses,” J. Opt. Soc. Am. A 24(9), 2673–2683 (2007). [CrossRef] [PubMed]

6. D. Connah and J. Y. Hardeberg, “Spectral recovery using polynomial models,” Proc. SPIE 5667, 65–75 (2005). [CrossRef]

7. V. Heikkinen, R. Lenz, T. Jetsu, J. Parkkinen, M. Hauta-Kasari, and T. Jääskeläinen, “Evaluation and unification of some methods for estimating reflectance spectra from RGB images,” J. Opt. Soc. Am. A 25(10), 2444–2458 (2008). [CrossRef] [PubMed]

8. H. Shen, H. Wan, and Z. Zhang, “Estimating reflectance from multispectral camera responses based on partial least-squares regression,” J. Electron. Imaging 19(2), 020501 (2010). [CrossRef]

9. H. Li, Z. Wu, L. Zhang, and J. Parkkinen, “SR-LLA: A novel spectral reconstruction method based on locally linear approximation,” inProceedings of IEEE Conference on Image Processing (IEEE, 2013), pp. 2029–2033. [CrossRef]

10. K. Xiao, Y. Zhu, C. Li, D. Connah, J. M. Yates, and S. Wuerger, “Improved method for skin reflectance reconstruction from camera images,” Opt. Express 24(13), 14934–14950 (2016). [CrossRef] [PubMed]

11. X. Zhang, Q. Wang, J. Li, X. Zhou, Y. Yang, and H. Xu, “Estimating spectral reflectance from camera responses based on CIE XYZ tristimulus values under multi-illuminants,” Color Res. Appl. 42(1), 68–77 (2017). [CrossRef]

12. J. Liang and X. Wan, “Optimized method for spectral reflectance reconstruction from camera responses,” Opt. Express 25(23), 28273–28287 (2017). [CrossRef]

13. M. M. Amiri and M. D. Fairchild, “A strategy toward spectral and colorimetric color reproduction using ordinary digital cameras,” Color Res. Appl. (first published 15 April 2018, in press).

14. J. Brauers, N. Schulte, and T. Aach, “Multispectral filter-wheel cameras: geometric distortion model and compensation algorithms,” IEEE Trans. Image Process. 17(12), 2368–2380 (2008). [CrossRef] [PubMed]

15. A. Ribes and F. Schmitt, “Linear inverse problems in imaging,” IEEE Signal Process. Mag. 25(4), 84–99 (2008). [CrossRef]

16. H. L. Shen and J. H. Xin, “Spectral characterization of a color scanner based on optimized adaptive estimation,” J. Opt. Soc. Am. A 23(7), 1566–1569 (2006). [CrossRef] [PubMed]

17. V. Cheung and S. Westland, “Methods for optimal color selection,” J. Imaging Sci. Technol. 50(5), 481–488 (2006). [CrossRef]

18. T. Eckhard, E. M. Valero, J. Hernández-Andrés, and M. Schnitzlein, “Adaptive global training set selection for spectral estimation of printed inks using reflectance modeling,” Appl. Opt. 53(4), 709–719 (2014). [CrossRef] [PubMed]

19. B. Cao, N. Liao, and H. Cheng, “Spectral reflectance reconstruction from RGB images based on weighting smaller color difference group,” Color Res. Appl. 42(3), 327–332 (2017). [CrossRef]

20. V. Babaei, S. H. Amirshahi, and F. Agahian, “Using weighted pseudo-inverse method for reconstruction of reflectance spectra and analyzing the dataset in terms of normality,” Color Res. Appl. 36(4), 295–305 (2011). [CrossRef]

21. G. Paschos, “Perceptually uniform color spaces for color texture analysis: an empirical evaluation,” IEEE Trans. Image Process. 10(6), 932–937 (2001). [CrossRef]

22. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection,” inProceedings of the Fourteenth International Joint Conference on Artificial Intelligence, C. S. Mellish, ed. (Academic,1995), pp. 1137–1143.

23. R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew, “Color image processing pipeline,” IEEE Signal Process. Mag. 22(1), 34–43 (2005). [CrossRef]

24. R. Sumner, “Processing raw images in matlab,” http://www.rcsumner.net/raw_guide/RAWguide.pdf

25. J. Nakamura, Image sensors and signal processing for digital still cameras (CRC press, 2017).

26. M. M. Amiri and S. H. Amirshahi, “A hybrid of weighted regression and linear models for extraction of reflectance spectra from CIEXYZ tristimulus values,” Opt. Rev. 21(6), 816–825 (2014). [CrossRef]

27. J. E. Garcia, A. G. Dyer, A. D. Greentree, G. Spring, and P. A. Wilksch, “Linearisation of RGB camera responses for quantitative image analysis of visible and UV photography: a comparison of two techniques,” PLoS One 8(11), e79534 (2013). [CrossRef] [PubMed]

28. C. Aguerrebere, J. Delon, Y. Gousseau, and P. Musé, “Study of the digital camera acquisition process and statistical modeling of the sensor raw data,”(2013). <hal-00733538v4>

29. C. Steger, M. Ulrich, and C. Wiedemann, Machine Vision Algorithms and Applications (Wiley, 2008).

30. H. Malvar, L. He, and R. Cutler, “High-quality linear interpolation for demosaicing of Bayer-patterned color images,” Proc. IEEE 3, 485–488 (2004).

31. L. Swirski, “CFA Interpolation Detection,”https://pdfs.semanticscholar.org/2ff3/8c5a1aab7e8ed1ff9e1c11c4a26690b56e21.pdf

32. CIE, CIE 15:24 Colorimetry (Vienna, 2004).

33. P. Harrington, Machine Learning in Action (Manning Publications, 2012), Chap. 8.

Method	Results	CCSG			Textile
Method	Results	RMSE%	GFC%	ΔE_ab	RMSE%	GFC%	ΔE_ab
OLS	Mean	3.03	99.52	2.64	3.26	98.22	2.56
OLS	Maximum	6.87	99.99	6.09	8.48	99.91	9.55
RLS	Mean	3.04	99.53	2.65	3.26	98.22	2.56
RLS	Maximum	6.86	99.99	6.10	8.53	99.91	9.51
PLS	Mean	2.98	99.53	2.64	3.21	98.23	2.48
PLS	Maximum	6.67	99.99	6.39	8.15	99.91	8.31
PCA	Mean	3.10	99.48	2.78	3.47	98.00	3.17
PCA	Maximum	6.74	99.99	6.51	9.03	99.87	8.34
Amiri	Mean	3.17	99.37	2.74	3.20	98.21	2.54
Amiri	Maximum	7.77	99.99	6.60	7.50	99.91	9.36
SR-LLA	Mean	4.33	98.63	5.01	3.39	97.68	2.31
SR-LLA	Maximum	12.13	100.00	33.94	10.38	99.96	6.75
Kernel	Mean	2.89	99.45	2.54	3.09	98.31	2.49
Kernel	Maximum	6.77	100.00	5.64	7.20	99.94	7.35
Zhang	Mean	3.27	99.45	2.86	3.05	98.64	2.37
Zhang	Maximum	8.60	100.00	7.48	8.51	99.93	10.28
o-AE	Mean	3.18	99.44	2.74	3.53	98.39	2.21
o-AE	Maximum	7.02	99.99	7.47	8.64	99.89	5.43
Liang	Mean	2.76	99.51	2.46	3.01	98.54	2.22
Liang	Maximum	10.35	100.00	9.08	10.19	99.97	10.17
ALWLR	Mean	2.38	99.71	2.21	2.68	98.73	1.85
ALWLR	Maximum	6.62	100.00	6.85	7.53	99.97	5.21

Method	CCSG	Textile
OLS	5.20	5.39
RLS	5.33	5.35
PLS	5.74	5.89
PCA	8.21	8.27
Amiri	5.73	5.96
SR-LLA	5.23	5.31
Kernel	9.82	15.43
Zhang	7.60	7.90
o-AE	5.59	5.65
Liang	5.67	5.82
ALWLR	5.52	5.70

Method	Results	CCSG			Textile
Method	Results	RMSE%	GFC%	ΔE_ab	RMSE%	GFC%	ΔE_ab
ALWLR-1st	Mean	2.38	99.71	2.21	2.68	98.73	1.85
ALWLR-1st	Maximum	6.62	100.00	6.85	7.53	99.97	5.21
ALWLR-2nd	Mean	2.58	99.69	2.51	2.68	98.75	1.91
ALWLR-2nd	Maximum	7.17	100.00	7.01	6.98	99.96	5.72
ALWLR-3rd	Mean	2.84	99.64	2.76	2.83	98.75	2.13
ALWLR-3rd	Maximum	9.03	100.00	8.58	7.97	99.96	8.60
ALWLR-4th	Mean	3.55	99.48	4.76	3.28	98.61	2.28
ALWLR-4th	Maximum	15.08	100.00	31.22	12.12	99.95	9.51

Method	Results	CCSG			Textile
Method	Results	RMSE%	GFC%	ΔE_ab	RMSE%	GFC%	ΔE_ab
ALWLR-LAB	Mean	2.38	99.71	2.21	2.68	98.73	1.85
ALWLR-LAB	Maximum	6.62	100.00	6.85	7.53	99.97	5.21
ALWLR-RGB	Mean	2.54	99.45	2.21	2.79	98.60	2.05
ALWLR-RGB	Maximum	6.92	100.00	5.87	7.42	99.97	5.83

Method	Results	CCSG			Textile
Method	Results	RMSE%	GFC%	ΔE_ab	RMSE%	GFC%	ΔE_ab
OLS	Mean	3.03	99.52	2.64	3.26	98.22	2.56
OLS	Maximum	6.87	99.99	6.09	8.48	99.91	9.55
RLS	Mean	3.04	99.53	2.65	3.26	98.22	2.56
RLS	Maximum	6.86	99.99	6.10	8.53	99.91	9.51
PLS	Mean	2.98	99.53	2.64	3.21	98.23	2.48
PLS	Maximum	6.67	99.99	6.39	8.15	99.91	8.31
PCA	Mean	3.10	99.48	2.78	3.47	98.00	3.17
PCA	Maximum	6.74	99.99	6.51	9.03	99.87	8.34
Amiri	Mean	3.17	99.37	2.74	3.20	98.21	2.54
Amiri	Maximum	7.77	99.99	6.60	7.50	99.91	9.36
SR-LLA	Mean	4.33	98.63	5.01	3.39	97.68	2.31
SR-LLA	Maximum	12.13	100.00	33.94	10.38	99.96	6.75
Kernel	Mean	2.89	99.45	2.54	3.09	98.31	2.49
Kernel	Maximum	6.77	100.00	5.64	7.20	99.94	7.35
Zhang	Mean	3.27	99.45	2.86	3.05	98.64	2.37
Zhang	Maximum	8.60	100.00	7.48	8.51	99.93	10.28
o-AE	Mean	3.18	99.44	2.74	3.53	98.39	2.21
o-AE	Maximum	7.02	99.99	7.47	8.64	99.89	5.43
Liang	Mean	2.76	99.51	2.46	3.01	98.54	2.22
Liang	Maximum	10.35	100.00	9.08	10.19	99.97	10.17
ALWLR	Mean	2.38	99.71	2.21	2.68	98.73	1.85
ALWLR	Maximum	6.62	100.00	6.85	7.53	99.97	5.21

Spectra estimation from raw camera responses based on adaptive local-weighted linear regression

Abstract

1. Introduction

2. Imaging model

3. Proposed method

3.1 Raw responses extraction

3.2 Proposed method

4. Experiment

5. Results and analysis

5.1 Methods implementation and comparison

5.2 The influence of regression model on the proposed method

5.3 The influence of color space on the proposed method

6. Discussion

7. Conclusion

Funding

References

Cited By

Figures (8)

Tables (4)

Equations (11)

Optics Express