Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group
Journal Home About Issues in Progress Current Issue All Issues Feature Issues

Dataset artificial augmentation with a small number of training samples for reflectance estimation

Jingjing Zhang, Zewei Wang, and Yuke He

Open Access Open Access

Abstract

The accuracy of the spectral reflectance estimation approaches highly depends on the amount, coverage, and representation of valid samples in the training dataset. We present a dataset artificial augmentation approach with a small number of actual training samples by light source spectra tuning. Then, the reflectance estimation process is carried out with our augmented color samples for commonly used datasets (IES, Munsell, Macbeth, Leeds). Finally, the impact of the augmented color sample number is investigated using different augmented color sample numbers. The results show that our proposed approach can artificially augment the color samples from CCSG 140 color samples to 13791 color samples and even more. The reflectance estimation performances with augmented color samples are much higher than with the benchmark CCSG datasets for all tested datasets (IES, Munsell, Macbeth, Leeds, as well as a real-scene hyperspectral reflectance database). It indicates that the proposed dataset augmentation approach is practical for improving the reflectance estimation performances.

© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement

1. Introduction

Spectral reflectance is independent of ambient light and is hence named the ‘fingerprint’ information of an object [1]. The spectral reflectance information is therefore useful for illuminating engineering applications, such as color tuning and prediction [2,3,4], visual enhancement [5], agricultural field [6,7], artwork protection illumination [8,9], and computer vision applications, such as material/ object recognition and characterization [10,11,6], for image enhancement [12,5], for color constancy [13], and geometry (shape) estimation from shading [14]. Moreover, it is also valuable for realistic material reproduction under a variety of illumination conditions in computer graphics [15], color restoration [16], color prediction [2], and relighting [17].

Although surface hyperspectral reflectances can be simultaneously obtained for an entire scene by a hyperspectral camera, the hardware facility has low practicability since it is costly, bulky, and complicated to operate [18]. That is one of the reasons that reflectance information remains limited in commercial computer vision and signal processing.

Hyperspectral reflectance estimation from color images obtained using regular digital cameras is a hot research topic because of its simplicity, low cost, and high speed [19]. Several techniques have been adopted. The simplest methods use a single camera, resulting in only three signals (RGB) from which to estimate the spectral reflectance [20], using techniques such as principle component analysis (PCA) [21], regression approaches [22,23], Wiener estimation [24], and colorimetric methods [25,26]. Nevertheless, compared with hyperspectral cameras, the estimation accuracy can hardly satisfy the requirements for many practical applications. In recent years, much research has been devoted to improving estimation accuracy by developing fusion algorithms such as hybrid polynomial regression and PCA framework [27], Wiener estimation using spectrally localized and weighted training samples (WE-SL&W) method [28], and back propagation neural network (BPNN) with an improved Sparrow Search Algorithm [29]. In addition, convolutional neural networks [19,30–32] are built to improving estimation accuracy. Later, light source spectra and camera responsivities optimization/selection method are proposed based on neural network training [32–35].

As is well known, the accuracy of the above reflectance estimation approaches, especially the kinds of training-based reflectance estimation approaches [19,30–35], highly depends on the amount, coverage, and representativeness of valid samples in the training dataset. Generally, a large amount of samples improves the reflectance estimation accuracy. In addition, the high accuracy performance of the training-based methods relies on established assumptions of training samples. Test samples should be as similar as possible [36], indicating that the training samples should represent the test samples. Therefore, a series of spectral reflectance sets are proposed to represent the colors in our daily life, for instance, the IES TM-30 4880 spectral reflectance set [37], the Leeds 100000 reflectance database [38], the full set of 1269 Munsell samples, and the 24 Macbeth ColorChecker samples.

To build the training dataset consisting of ground truth reflectances and corresponding RGB values of the training color samples, firstly, the ground truth reflectances of the training samples should be measured one by one. RGB values should be captured by a camera under a known light source. Consequently, building the reflectance estimation training dataset with many color samples can be time-consuming since the color samples should be replaced one by one. More importantly, unlike Munsell cards or NCS cards, the IES TM-30 spectral reflectance set [37], which was proved robust among the Munsell, IES, and Leeds datasets for reflectance estimation [35], is a series of artificial spectral curves. It is hence difficult for us to measure the ‘virtual’ IES TM-30 samples in an actual system.

In this study, instead of time-consuming replacing color samples one by one and unrealistically measuring ‘virtual’ color samples, we propose a novel dataset artificial augmentation approach with a small number of actual training color samples. Firstly, by tuning the light source spectra which can be easily realized, various camera response (RGB) values of the small number of color samples are practically captured. Then, supposing the light source spectra are not changed, in order to obtain the above various RGB values, the reflectances of the color sample could be considered artificially ‘changed’. Hence new color samples can be virtually fabricated, and training datasets are augmented by theoretical calculations from the camera imaging model. Next, the augmented datasets’ reflectances and the RGB values are used to train the mapping relationship between the object reflectances and RGB values captured by the camera under the supposed unchanged light source spectra. Finally, the trained mapping relationship between the object reflectances and RGB values are used to estimate reflectances of the unknown color samples under the unchanged light source spectra. This work is organized as follows. First, the proposed approach is descirbed is section II. Then the performances of the approach analyzed in section III.

2. Dataset artificial augmentation

2.1 Camera response model

Without considering the imaging noise, the camera response is a linear combination of the spectral power distribution of the light source, the camera responsivity, and the surface spectral reflectance of the object, which can be expressed by

$${d_i} = \mathop \smallint \nolimits_{{\lambda _1}}^{{\lambda _2}} {c_i}(\lambda )s(\lambda )r(\lambda )d\lambda $$
where di represents the response of the ith channel of the camera (for a color camera, i = r(red), g(green), b(blue)), λ is the wavelength, ranging from 400 nm to 700 nm in the camera-sensitive wavelength range. ci(λ) is the sensitivity function of the ith channel of the camera, s(λ) is the spectral power distribution of the light source, and r(λ) is the surface spectral reflectance of the illuminated object. The matrix form of Eq. (1) can be expressed as
$$\mathbf{d} = \mathbf{CSr}$$

2.2 Dataset augmentation approach

As shown in Fig. 1, the camera response values of the color samples can be obtained (benchmark) when the light source spectra are fixed. By tuning the light source spectra, different camera response (RGB) values of the color sample are captured. Under the circumstance, if we suppose the light source spectra are not changed, the reflectances of the color sample can be artificial “changed”, which can be inferred from Eq. (2). Therefore, new artificial color samples can be fabricated (augmented) by tuning the light source spectra, assuming that the light source spectra are fixed during the light source spectra tuning process.

 figure: Fig. 1.

Fig. 1. Dataset augmentation schematic diagram.

Download Full Size | PDF

A dataset augmentation method is proposed based on the above consideration. The process mainly includes four steps, and the flowchart is shown in Fig. 2. Firstly, new raw camera responses are obtained by tuning the light source spectra. Secondly, to satisfy the authenticity principle, the obtained raw camera responses are converted to CIE XYZ color space using a matrix based on the least squares method. The new raw camera responses whose LAB values are outside a coverage boundary of the practical color datasets collection are discarded. Thirdly, the new color samples are augmented from the new raw camera responses by estimating reflectances from the new raw camera responses using the adaptive local-weighted linear regression method [39]. Finally, the augmented color samples whose XYZ values under the tuned light source spectra have a huge deviation from the benchmark color sample are discarded to satisfy the validity principle. The artificially augmented and the benchmark color samples are combined into the final training dataset. The process is described in detail as follows.

 figure: Fig. 2.

Fig. 2. The flowchart of the proposed artificial dataset augmentation process.

Download Full Size | PDF

As illustrated in Fig. 1, note that only the RGB values of the benchmark color samples are required to be captured under the tuning light source spectra. The artificially augmented color samples are theoretically calculated and purely virtual; hence we do not need to fabricate the augmented color samples for real tests. The performance of the augmented color samples should be evaluated by spectral reflectance estimation using the augmented color samples, which are described in Section 2.3.

2.2.1 Artificially generating new raw camera responses

A common spectrally tunable light source consists of three color channels, and its spectral model is

$$S(\lambda )= {k_r}{S_r}(\lambda )+ {k_g}{S_g}(\lambda )+ {k_b}{S_b}(\lambda )$$
where Sr(λ), Sg(λ), and Sb(λ) respectively represent the maximum spectral radiation of the three channels of the light source at the rated output (also represent the reference light source spectra). kr, kg, and kb represent the output proportional coefficient of the spectral radiation, respectively, which can be tuned by the three color RGB values of the projector. When k is 1, the light source obtains the maximum output radiation, and the light source at this time is the reference light source, denoted as Sref (λ), which can be written as
$$\mathbf{S} = {S_{ref}}(\lambda ){[{{k_r},{k_g},{k_b}} ]^T} = [{{\mathbf{S}_\mathbf{r}},{\mathbf{S}_\mathbf{g}},{\mathbf{S}_\mathbf{b}}} ]{[{{k_r},{k_g},{k_b}} ]^T}$$
where Sr, Sg, and Sb respectively represent the vector of Sr(λ), Sg(λ), Sb(λ).

From Eq. (2) and Eq. (4), the camera response vector of the color sample under the reference illumination (R = G = B = 255) can be given by

$$\left[ {\begin{array}{c} {{\boldsymbol{d}_{\boldsymbol{r},\boldsymbol{ref}}}}\\ {{\boldsymbol{d}_{\boldsymbol{g},\boldsymbol{ref}}}}\\ {{\boldsymbol{d}_{\boldsymbol{b},\boldsymbol{ref}}}} \end{array}} \right] = \left( {{{\left[ {\begin{array}{ccc} {{\mathbf{C}_\mathbf{r}}}&{{\mathbf{C}_\mathbf{g}}}&{{\mathbf{C}_\mathbf{b}}} \end{array}} \right]}^{\mathbf \top }}{\mathbf{S}_{\mathbf{ref}}}} \right)\odot \mathbf{r}$$
where dr,Ref., dg,ref, db,ref represent the response vector of the color sample under the reference light source, respectively. Cr, Cg, and Cb are the vectors of the three color channels of the camera responsivities, respectively. Sref represents the vector of Sref (λ). Consequently, the augmented camera response vector (dr, dg, db) of the color sample under the illumination of the spectrally tunable light source can be given by
$$\left[ {\begin{array}{c} {{\mathbf{d}_\mathbf{r}}}\\ {{\mathbf{d}_\mathbf{g}}}\\ {{\mathbf{d}_\mathbf{b}}} \end{array}} \right] = \left[ {\begin{array}{c} {{\boldsymbol{d}_{\boldsymbol{r},\boldsymbol{ref}}}}\\ {{\boldsymbol{d}_{\boldsymbol{g},\boldsymbol{ref}}}}\\ {{\boldsymbol{d}_{\boldsymbol{b},\boldsymbol{ref}}}} \end{array}} \right]\left[ {\begin{array}{c} {{\boldsymbol{k}_{\boldsymbol{r}}}}\\ {{\boldsymbol{k}_{\boldsymbol{g}}}}\\ {{\boldsymbol{k}_{\boldsymbol{b}}}} \end{array}} \right]$$

Here, a projector with three color channels is used as a spectrally tunable light source. For a three-channel projector, the tuning range of each color channel is from 1 to 255. Therefore, using one color sample, the proposed method could theoretically augment as many as 2553 new color samples by automatically tuning the projector's input RGB values and the camera's image capture process. However, 2553 (16,581,375) is tremendous for a reflectance estimation training set. Hence it is impractical to measure 2553 groups of RGB values. More importantly, due to the information redundancy, the augmented dataset consisting of new color samples should be selected based on three principles in our consideration, authenticity, validity, and representation. Authenticity indicates that the newly augmented color samples should be within the color space of practical color samples [40] (dataset collection of CCSG, Munsell, IES TM30, Macbeth, and Leeds100000). Validity indicates that the error requirement should be met. Representation suggests that the newly augmented color samples should be distributed uniformly in the color space.

2.2.2 Discard camera responses whose color space outer the boundary of the datasets

Training samples are used to derive a matrix from converting from device-dependent camera RGB color space to CIE XYZ color space based on the least squares method. With the established conversion matrix, the training and testing samples’ raw camera responses are converted from device-dependent camera RGB color space to CIE XYZ color space [41].

Camera responses whose whose color space is outer the boundary of the dataset collection of CCSG, Munsell, IES TM30, Macbeth, and Leeds100000 are discarded.

2.2.3 Artificially augment new samples

New color samples are augmented based on the converted XYZ values by the adaptive local-weighted linear regression (ALWLR) method [39]. Here, to understand the whole process smoothly, we briefly describe it in the following parts.

The Euclidean distance $\Delta E_{ab}^\ast $ between the augmented color samples and the known color samples in CIE LAB color space can be calculated as [42]

$$\Delta E_{ab}^\ast{=} \sqrt {{{({{L^\ast } - L_i^\ast } )}^2} + {{({{a^\ast } - a_i^\ast } )}^2} + {{({{b^\ast } - b_i^\ast } )}^2}^2} \; \; \; \; ({i = 1\textrm{, }2\textrm{, } \ldots \textrm{, }n} )$$
where L*, a*, b* are the coordinates of the extended response value in the known sample set space in the CIELAB color space, $L_i^\ast $, $a_i^\ast $, and $b_i^\ast $ are the coordinates of the known dataset in the CIELAB color space, n is the number of available color samples.

Next, the available color samples with first K minimum Euclidean distance values are chosen to generate new color samples artificially. The weighting coefficient wj of the available color-selected samples is given by

$${w_j} = \frac{1}{{\Delta E_{ab,j}^\ast{+} \beta }}\; \; ({j = 1\textrm{, }2\textrm{, } \ldots \textrm{, }K} )$$
where subscript j is the jth sample of the optimal local sample, $\Delta E_{ab,j}^\ast $ is the Euclidean distance between the jth local optimal sample and the augmented color samples in the CIELAB color space, β is increased to avoid division by zero for a very small value, we use β=0.001. K is 10. The weighting coefficient vector W is defined as
$$\mathbf{W} = [{{w_1}\textrm{, }{w_2}\textrm{, }{w_3}\textrm{, } \ldots \textrm{, }{w_K}} ]$$

After determining the weighting coefficient vector, W is expanded based on the dimensions of the selected K available sample reflectance to make the two consistent. The result is recorded as Wr, and finally, the reflectance corresponding to the K samples is multiplied by Wr and summed to obtain the composite reflectance of the extended sample,

$${r_{exp}} = {r_c}\odot {\boldsymbol{W}_{\boldsymbol{r}}}$$

In the above formula, rc is the 61×K (wavelength range 400-700 nm, an interval of 5 nm) reflectance composed of K available samples, and rexp is the composite reflectance of the augmented sample that can be obtained by summing rexp in columns target reflectance vector.

2.2.4 Discarding the augmented color samples with a big deviation from the benchmark color samples

Since the new augmented color samples are derived from benchmark color samples (CCSG), most characteristics of the new augmented color samples should be inherited from the benchmark (similar to image data augmentation by image crop, flipping, and rotation et al., which is widely applied in computer vision [43]). Therefore, we use a constraint to limit the differences in color characteristics between the new augmented color samples and the benchmark under the spectral tuning light source. The augmented color samples whose XYZ values considerably differ from the benchmark color samples (CCSG) are discarded. The equation is expressed by

$$\sqrt {(( {X^\textrm{2}}\textrm{ - }X_{\textrm{benchmar}k}^2\textrm{) + }( {Y^\textrm{2}}\textrm{ - }Y_{\textrm{benchmark}}^2\textrm{) + }( {Z^\textrm{2}}\textrm{ - }Z_{\textrm{benchmark}}^2\textrm{))/3}} \times 100\%\le 10\%$$

After decoration, the rest color samples are the final augmented dataset.

2.3 Spectral reflectance estimation with augmented color samples

To verify the performance of our proposed approach, a commercially available dual imaging system [35] is used, composed of a 3-channel light source (e.g., a data projector) and a 3-channel color camera. The spectral power distribution (SPD) of the RGB projector (M420X, NEC corp., Japan) and the camera responsivities are plotted in Fig. 3. Similar to [44], the SPD was measured by a spectrometer QE65 Pro (Ocean Optics Inc., USA) that is equipped with a direct-view telescope TEL301 (Bentham Instruments Inc., UK).

 figure: Fig. 3.

Fig. 3. (a) Spectral power distribution of the projector, (b) Camera responsivities

Download Full Size | PDF

For the sake of simplicity and economy, a common-used CCSG (ColorChecker Semigloss Chart) with 140 color samples is used as the original samples to augment new color samples artificially.

For a comprehensive assessment, the spectral reflectance of a surface is estimated from the nine signals using a neural network approach [35], as well as the ALWLR approach [39] for a convenient mapping between light source spectra, spectral reflectances, and camera-specific RGB values rather than other methods [18]. Both methods need to train the relationship between the spectral reflectance and a 9-element vector consisting of the camera output RGB values “captured” under each of the three light sources (projector) spectra. The training samples are separate CCSG color samples (benchmark) or a combination of the CCSG color samples and the artificially augmented color samples. After training, four different known datasets are used to test the reflectance estimation accuracy and robustness: IES TM30 (4880 samples), Munsell (1269 samples), Macbeth (24 samples), and the Leeds (100,000 samples).

Similar to [35], a 3-layer radial basis neural network is adopted. The input layer is a 9-element vector consisting of the camera output RGB values ‘captured’ under each of the 3 light source (projector) spectra. The hidden layer has neurons with a Gaussian radial basis function. The output layer has neurons with a linear function, which adds the weighted output of the hidden layer. The latter takes as input the weighted Euclidean distance between the input vectors. Initially the hidden layer has no nods. The nods of the hidden layer are 200.

The spectral reflectance estimation accuracy is evaluated using four metrics: root mean square error (RMSE) and the goodness of fit coefficient (GFC) of the estimated spectral reflectance and the ground truth for spectral accuracy assessment [45,46]; and the mean CIE 1976 u'v’ chromaticity differences (Δu'v’) and luminance difference ratios (ΔY/Y) between the estimated and ground-truth chromaticity and luminance values obtained under some reference illuminant for color accuracy evaluation [24,47]. The four parameters are given by:

$$\left\{ {\begin{array}{c} {RMSE = \sqrt {\frac{1}{N}\mathop \sum \nolimits_{i = 1}^N {{({{R_i} - {{\hat{R}}_i}} )}^2}} }\\ {GFC = \frac{1}{N}\mathop \sum \nolimits_{i = 1}^N |{{R_i}{{\hat{R}}_i}} |/\left( {\sqrt {\mathop \sum \nolimits_{i = 1}^N {R_i}^2} \sqrt {\mathop \sum \nolimits_{i = 1}^N {{\hat{R}}_i}^2} } \right)}\\ {\varDelta u^{\prime}v^{\prime} = \sqrt {{{({u^{\prime} - \widehat {u^{\prime}}} )}^2} + {{({v^{\prime} - \widehat {v^{\prime}}} )}^2}} }\\ {\varDelta Y/Y = |{Y - \hat{Y}} |/Y} \end{array}} \right.$$
where Ri and ${\hat{R}_i}$ refer to the ground truth and the estimated value of the reflectance, respectively; u’ and v’ are the ground truth in the CIE1976 chromaticity diagram, $\hat{u}^{\prime}$ and $\hat{v}^{\prime}$ are the corresponding estimated values; Δu'v’ is the u'v’ deviation in the CIE1976 chromaticity diagram; Y and $\hat{Y}$ respectively designate the ground truth and the estimated value calculated using the CIE 1931 color matching functions (cfr. 1924V-lambda curve); and ΔY designates the deviations between Y and $\hat{Y}$. In this work, the equal-energy-white (CIE illuminant E) has been adopted as the reference illuminant. N is the number of samples of spectral reflectance in the visible spectrum with spectral wavelengths ranging from 400 nm to 700 nm, and we take an interval of 5 nanometers; therefore, N = 61.

For the three-channel projector, the tuning range of each color channel is from 1 to 255. In order to augment the color samples diversely, the three color channels of the projector are tuned with a specific interval. The input RGB values are normalized from 1 to 255 to 0 to 1, making it easy to analyze and compare the impact of the intervals on the final artificial augmentation results. Firstly, the three color channel intervals are equal and increase from 0.2 to 0.5 with a step of 0.1. Then, to compare the influence caused by different color channels, one color channel interval increases from 0.1 to 0.4 with a step of 0.1 while keeping the other two channel intervals constant at 0.3.

2.4 Impact on reflectance estimation performance of the augmented color sample number

Augmented color sample number is highly related to the spectral tuning time and the reflectance estimation performance. The impact of the augmented color sample number needs to be investigated using different augmented color sample numbers.

2.5 Applicability verification for actual scenario application

Spectral reflectance estimation is useful for illuminating engineering and computer vision applications. Most of the applications are real scenarios with multiple colors. Therefore, to verify the applicability of our proposed color sample augmentation approach, the performance is tested with a real-scene hyperspectral reflectance database (CAVE datasets [48]) and the earliest dated example of fifteenth-century European printing (St Christopher Woodcut [49]).

3. Results and discussion

3.1 Dataset augmentation

Table 1 lists the number of spectral tuning times, the augmented sample number, and the corresponding augmentation efficient ratio with different color channel intervals. It is clear that new color samples could be augmented by our proposed method.

Tables Icon

Table 1. Color channel interval and corresponding augmented sample number

View Table | View all tables in this article

Figure 4 shows the augmented sample number as a function of the spectral sampling interval. It can be found from Fig. 4(a) that with the RGB interval decreasing from 0.5 to 0.2, the augmented color sample number increases from 229 to 4408, which is far more than the original sample number of 140. It indicates our proposed dataset approach is effective in color sample augmentation. It indicates our proposed dataset approach is effective in color sample augmentation. Meanwhile, the number of spectral tuning times increases from 1120 to 17500, corresponding to the augmentation efficient ratio (augmented sample number/number of spectral tuning times) from 20.4% to 25.2%. It can be seen from Fig. 4(b) that the spectral sampling channel has little impact on the final augmented sample number.

 figure: Fig. 4.

Fig. 4. Augmented sample number as a function of spectral sampling interval (a. three channels with equal interval simultaneously, b. single color channel respectively)

Download Full Size | PDF

Figure S1 (see Supplement 1) shows a visual illustration of the augmented color samples and benchmark CCSG samples in CIE LAB color space. It can be seen that the coordinates of the augmented color samples are uniformly distributed in the LAB color space, similar to the CCSG color samples. In addition, the augmented color sample number increases with the decreasing sampling interval.

3.2 Spectral reflectance estimation with augmented color samples

3.2.1 Neural network approach

The results of the four evaluation measures for training the benchmark CCSG dataset, with newly augmented 829 color samples, 2126 color samples, 4408 color samples, and 6194 color samples, are shown in Fig. 5. Note that the benchmark CCSG dataset is always included in the training dataset. It can be seen from Fig. 5(a) that the RMSE values with augmented color samples are much lower than with the benchmark CCSG datasets for all tested datasets (IES, Munsell, Macbeth, Leeds). Other three reflectance estimation measures, including GFC, Δu'v’, and ΔY/Y outperform those with benchmark color samples. It indicates that the proposed dataset augmentation approach can improve the robustness of the trained neural network and is thus practical for improving the reflectance estimation performances. Note that The RMSE values with 2126 augmented color samples are lower than the RMSE values with 829 color samples.

 figure: Fig. 5.

Fig. 5. The results of spectral reflectance estimation using the augmented training sample set using the neural network reflectance estimation approach.

Download Full Size | PDF

The mean, median, and maximum (minimum in case of GFC) values of RMSE and GFC for four datasets (IES, Munsell, Macbeth, Leeds) are listed in Table 2. Based on previous studies, the acceptable threshold for RMSE for most of the applications is 0.03 [28]. GFC ≥0.99 represents good recovering for colorimetric purposes; GFC ≥ 0.999 indicates quite good recovering; and finally, GFC ≥ 0.9999 signifies almost an exact mathematical recovering [50]. Therefore, the ratio whose RMSE value is lower than 0.03 in the dataset, and whose GFC value is higher than 0.9999, 0.999, and 0.99 are also given in Table 2. It is shown that 72.4%, 99.3%, 100%, and 93.4% of the color samples in the four datasets are lower than 0.03, respectively. It indicates that our proposed method basically satisfies the requirement for RMSE accuracy. In addition, Table 2 also shows that 88.6%, 99.3%, 99.9%, and 95.0% of the color samples in the four datasets are higher than 0.99, respectively. It indicated that our proposed method also satisfies the requirement for GFC accuracy.

Tables Icon

Table 2. Test results of RMSE and GFC for IES, Munsell, Macbeth, IES datasets

View Table | View all tables in this article

In order to evaluate the color accuracy of the proposed approach, the mean, median, and maximum values of Δu'v’, ΔY/Y, and ΔE00 (DE2000 color difference [51]) under three illuminants D65, A, F11 for four datasets (IES, Munsell, Macbeth, Leeds) are listed in Table 3. Based on previous studies, ΔE00 falls in the range from 3.2 to 6.5 indicates the difference between the two colors is observable, but the impression given by both is basically the same. ΔE00 falls in the range from 1.6 to 3.2 indicates the difference between the two colors is basically indistinguishable and can be considered the same color most of the time [52]. It can be seen from Table 3 that the mean ΔE00 values are all lower than 1 unit under three illuminants for four datasets.

Tables Icon

Table 3. Test results of Δu'v’, ΔY/Y, and ΔE00 under D65, A, F11 for IES, Munsell, Macbeth, IES datasets

View Table | View all tables in this article

For statistical analysis, Table S1 (see Supplement 1) lists the ratio of the color samples whose DE2000 color difference is lower than 1.6, 3.2, and 6.5 for the IES, Munsell, Macbeth, and Leeds datasets, respectively. It can be seen that ΔE00 all 100% falls in the range of ≤ 6.5 (except under F11 for Leeds 99%), indicating sufficient color accuracy of the proposed approach.

Figure S3 (see Supplement 1) shows the performance of using the benchmark dataset and our augmented 4408 color samples in terms of the probability density of the DE2000 color difference for IES 4880 dataset. It is clear that our proposed method outperforms using benchmark dataset without dataset augmentation. Hence, our proposed method is proven efficient for improving color accuracy for reflectance estimation.

As a visual illustration of the improved estimation accuracy using augmented versus benchmark datasets, the estimated and ground-truth spectral reflectance curves have been plotted in Fig. S2 for a select number of samples of the Macbeth ColorChecker set (see Supplement 1). It is clear that the visual agreement for using both datasets is quite good but that using the benchmark datasets tends to lead to more minor deviations. On average, the RMSE is 0.01, indicating that optimized system functions can recover sample spectral reflectance with good to reasonable accuracy.

3.2.2 Adaptive local-weighted linear regression approach

The results of the four evaluation measures for training the benchmark CCSG dataset, the augmented 829 color samples, 2126 color samples, 4408 color samples, and 6194 color samples, are shown in Fig. 6. It can be found from Fig. 6(a) that the RMSE values for the IES dataset and Leeds dataset are slightly lower than the original benchmark dataset. However, the RMSE values for the IES and Leeds datasets are higher than the original benchmark dataset. It can be found from Fig. 6(b) that the GFC values for the IES dataset and Leeds dataset are higher than the original benchmark dataset. However, the GFC values for the IES and Leeds datasets are lower than the original benchmark dataset.

 figure: Fig. 6.

Fig. 6. The results of spectral reflectance estimation using the augmented training sample set using the ALWLR approach.

Download Full Size | PDF

It can be concluded from Fig. 5 and Fig. 6 that without color sample augmentation, the ALWLR approach presents higher performance than the neural network approach. However, after augmentation, the RMSE values with the neural network approach are much lower than that with the ALWLR approach. In conclusion, it is effective to improve the reflectance estimation accuracy using our proposed color sample augmentation approach with neural network training. The results of the following part are carried out with the neural network approach.

3.3 Impact on reflectance estimation performance of the augmented color sample number

The results of the four evaluation measures with a function of augmented color sample number are shown in Fig. S4 (see Supplement 1). It can be found from Fig. S4(a) that the RMSE values reach the lowest with an augmented color sample number of 4408 for testing IES dataset and Macbeth dataset. The RMSE values come at the minimum with an augmented color sample number of 2128 for testing the Munsell datasets. The RMSE values come at the minimum with an augmented color sample number of 6503 for testing the Leeds datasets. Therefore, the increment of augmented sample number does not always lead to better reflectance estimation performance since information redundancy during the augmentation process. Thus, augmenting color samples as much as possible in practical applications is unnecessary. Augmenting color samples to an appropriate number in practical applications is time-economic and efficient. In this study, the reflectance estimation performances are high for all four datasets using augmented 4408 color samples. As listed in Table 1, the corresponding spectral tuning time is 17500. Since there are 140 color samples on the CCSG color chart, 140 groups of RGB values can be acquired by one-time imaging. By calculation, 62 minutes is enough for augmenting 4408 color samples if single spectra tuning and imaging need 30 seconds (including enough light source stabilization time and enough imaging time).

3.4 Applicability verification for actual scenario application

Four hyperspectral reflectance data in the CAVE dataset and the hyperspectral reflectance data of European printing St Christopher Woodcut are used as test datasets. The original images and the RMSE data maps are shown in Fig. S5 (see Supplement 1). It can be seen from Fig. S5 that the RMSE values using augmented color samples are much lower than the benchmark CCSG dataset, indicating the proposed color sample augmentation approach are also applicable and practical for real scenarios.

4. Conclusion

Building the reflectance estimation training dataset with many color samples can be time-consuming since the color samples should be replaced one by one. More importantly, some datasets (e.g., the IES TM-30 spectral reflectance set) are ‘virtual’ and hard to be trained in the actual reflectance estimation process. This work presents a novel dataset artificial augmentation approach with a small number of actual training samples by light source spectral tuning, which can be easily realized.

The results show that our proposed approach can artificially augment the color samples from CCSG 140 color samples to 13971 color samples and even more. The reflectance estimation performances with augmented color samples are much higher than with the benchmark CCSG datasets for all tested datasets (IES, Munsell, Macbeth, Leeds). It indicates that the proposed dataset augmentation approach is practical for improving the reflectance estimation performances. Note that the increment of augmented sample number does not always lead to better reflectance estimation performance since information redundancy during the augmentation process. Augmenting color samples to an appropriate number in practical applications is time-economic and efficient. The reflectance estimation performances are high for all four datasets using augmented 4408 color samples. By calculation, only 62 minutes is enough to augment those 4408 color samples.

Finally, the performances are tested with real-scene hyperspectral reflectance databases, indicating that the proposed color sample augmentation approach is also applicable and practical for real scenarios.

This work focuses on the proof of the principle proposed augmentation approach for color samples artificially by spectral tuning using a CCSG color chart. Since spectral tuning and RGB values acquisition can be realized automatically by programming, the augmentation approach can be realized automatically without changing color samples, which is convenient for users. A camera with a wide dynamic range could help accurately capture the RGB values under different tuning light source spectra. This work could be quite powerful and can be applied in computer vision applications and illuminating engineering applications.

The used CCSG color chart (human-made surfaces) can be replaced with any other objects in the proposed method. Since the reflectances of the natural objects were different from that of color samples and human-made samples [52], benchmark samples consisting of both human-made samples and real samples could be helpful to improve the accuracy of the proposed method, which needs further study.

Funding

Fundamental Research Funds for the Central Universities (CUGL180404).

Acknowledgments

We thank the financial support from the Fundamental Research Funds for the Central Universities.

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but can be obtained from the authors upon reasonable request.

Supplemental document

See Supplement 1 for supporting content.

References

1. K. J. Dana, “Capturing computational appearance: more than meets the eye,” IEEE Signal Process. Mag. 33(5), 70–80 (2016). [CrossRef]  

2. M. Tejada-Casado, R. Ghinea, and M. M. Perez, “Reflectance and color prediction of dental material monolithic samples with varying thickness,” Dent. Mater. 38(4), 622–631 (2022). [CrossRef]  

3. J. J. Zhang, R. Hu, and B. Xie, “Energy-saving light source spectrum optimization by considering object's reflectance,” IEEE Photonics J. 9(2), 1 (2017). [CrossRef]  

4. X. Wang, Z. Wang, Y. Meuret, K. Smet, and J. Zhang, “Point-by-point visual enhancement with spatially and spectrally tunable laser illumination,” Opt. Express 30(25), 45327 (2022). [CrossRef]  

5. A. S. Parihar, K. Singh, and H. Rohilla, “Fusion-based simultaneous estimation of reflectance and illumination for low-light image enhancement,” IET Image Process. 15(7), 1410–1423 (2021). [CrossRef]  

6. A. Amziane, O. Losson, and B. Mathon, “Reflectance estimation from multispectral linescan acquisitions under varying illumination application to outdoor weed identification,” Sensors 21(11), 3601 (2021). [CrossRef]  

7. H. A. Khan, Y. Nakamura, and R. T. Furbank, “Effect of leaf temperature on the estimation of photosynthetic and other traits of wheat leaves from hyperspectral reflectance,” J. Exp. Bot. 72(4), 1271–1281 (2021). [CrossRef]  

8. D. Durmus, D. Abdalla, and A. Duis, “Spectral optimization to minimize light absorbed by artwork,” Leukos 16(1), 45–54 (2020). [CrossRef]  

9. D. Vázquez, A. A. Fernández-Balbuena, H. Canabal, C. Muro, D. Durmus, W. Davis, and S. Mayorga, “Energy optimization of a light projection system for buildings that virtually restores artworks,” Digital Appl. Archaeol. Cultural Heritage 16, e00128 (2020). [CrossRef]  

10. H. Q. Zhao, W. Yunli, and S. Liu, “Spectral reflectance characterization and fiber type discrimination for common natural textile materials using a portable spectroradiometer,” J. Archaeol. Sci. 111, 105026 (2019). [CrossRef]  

11. G. J. Tu, H. Karstoft, and L. J. Pedersen, “Illumination and reflectance estimation with its application in foreground detection,” Sensors 15(9), 21407–21426 (2015). [CrossRef]  

12. X. Y. Fu, Y. H. Liao, and D. L. Zeng, “A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation,” IEEE Trans. Image Process. 24(12), 4965–4977 (2015). [CrossRef]  

13. E. L. Dixon and A. G. Shapiro, “Spatial filtering, color constancy, and the color-changing dress,” J. Vision 17(3), 1 (2017). [CrossRef]  

14. T. T. Ngo, H. Nagahara, and K. Nishino, “Reflectance and shape estimation with a light field camera under natural illumination,” Int. J. Comput. Vis. 127(11-12), 1707–1722 (2019). [CrossRef]  

15. J. Filip, R. Vavra, and M. Havlicek, “Predicting visual perception of material structure in virtual environments,” Comput. Graph. Forum 36(1), 89–100 (2017). [CrossRef]  

16. K. Zolotukhina and I. Soltys, “The reflectance spectra of the model printing inks,” Proc. SPIE 12126, 269–274 (2021). [CrossRef]  

17. X. X. Xing, W. M. Dong, and X. P. Zhang, “Spectrally-based single image relighting,” Entertainment for Education. Digital Techniques and Systems 6249, 509–517 (2010). [CrossRef]  

18. R. M. H. Nguyen, D. K. Prasad, and M. S. Brown, “Training-based spectral reconstruction from a single RGB image,” in Processings of IEEE conference on European Conference on Computer Vision (2014), pp. 186–201.

19. R. Deeb, J. Van de Weijer, and D. Muselet, “Deep spectral reflectance and illuminant estimation from self-interreflections,” J. Opt. Soc. Am. A 36(1), 105–114 (2019). [CrossRef]  

20. L. T. Maloney and B. A. Wandell, “Color constancy - a method for recovering surface spectral reflectance,” J. Opt. Soc. Am. A 3(1), 29–33 (1986). [CrossRef]  

21. R. Shrestha, A. Mansouri, and J. Y. Hardeberg, “Multispectral imaging using a stereo camera: concept, design and assessment,” EURASIP J. Adv. Signal Process. 2011(1), 57 (2011). [CrossRef]  

22. V. Heikkinen, C. Camara, and T. Hirvonen, “Spectral imaging using consumer-level devices and kernel-based regression,” J. Opt. Soc. Am. A 33(6), 1095–1110 (2016). [CrossRef]  

23. V. Heikkinen, “Spectral reflectance estimation using Gaussian processes and combination kernels,” IEEE Trans. Image Process. 27(7), 3358–3373 (2018). [CrossRef]  

24. P. Urban, M. R. Rosen, and R. S. Berns, “Spectral image reconstruction using an edge preserving spatio-spectral Wiener estimation,” J. Opt. Soc. Am. A 26(8), 1865–1875 (2009). [CrossRef]  

25. X. Zhang, Q. Wang, and J. Li, “Estimating spectral reflectance from camera responses based on CIE XYZ tristimulus values under multi-illuminants,” Color Res. Appl. 42(1), 68–77 (2017). [CrossRef]  

26. A. M. Nahavandi and M. A. Tehran, “A new manufacturable filter design approach for spectral reflectance estimation,” Color Res. Appl. 42(3), 316–326 (2017). [CrossRef]  

27. S. Li, K. Xiao, and P. Li, “Spectra reconstruction for human facial color from RGB images via clusters in 3D uniform CIELAB and its subordinate color space,” Sensors 23(2), 810 (2023). [CrossRef]  

28. M. Safdar and P. Emmel, “Toward non-metameric reflectance recovery by emulating the spectral neighborhood using corresponding color information,” J. Opt. Soc. Am. A 39(6), 1066–1075 (2022). [CrossRef]  

29. L. Zhang, C. Wwang, M. Fang, and W. Xu, “Spectral reflectance reconstruction based on BP Neural Network and the improved sparrow search algorithm,” IEICE Trans. Fundamentals E105.A(8), 1175–1179 (2022). [CrossRef]  

30. S. Georgoulis, K. Rematas, and T. Ritschel, “Reflectance and natural illumination from single-material specular objects using deep learning,” IEEE Trans. Pattern Anal. Mach. Intell. 40(8), 1932–1947 (2018). [CrossRef]  

31. L. X. Wang, X. X. Wan, and G. S. Xiao, “Sequential adaptive estimation for spectral reflectance based on camera responses,” Opt. Express 28(18), 25830–25842 (2020). [CrossRef]  

32. W. Y. Zhang, H. Y. Song, and X. He, “Deeply learned broadband encoding stochastic hyperspectral imaging,” Light: Sci. Appl. 10(6), 969–975 (2021). [CrossRef]  

33. Y. Fu, T. Zhang, and Y. Q. Zheng, “Joint camera spectral response selection and hyperspectral image recovery,” IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 256–272 (2022). [CrossRef]  

34. T. Zhang, Z. Y. Liang, and Y. Fu, “Joint spatial-spectral pattern optimization and hyperspectral image reconstruction,” IEEE J. Sel. Top. Signal Process. 16(4), 636–648 (2022). [CrossRef]  

35. J. J. Zhang, Y. Meuret, and X. G. Wang, “Improved and robust spectral reflectance estimation,” Leukos 17(4), 359–379 (2021). [CrossRef]  

36. J. X. Liang and X. X. Wan, “Optimized method for spectral reflectance reconstruction from camera responses,” Opt. Express 25(23), 28273–28287 (2017). [CrossRef]  

37. IES, TM-30-20: IES method for evaluating light source color rendition, Illuminating engineering Society of north America (2020).

38. K. A. G. Smet, J. Schanda, and L. Whitehead, “CRI2012: A proposal for updating the CIE colour rendering index,” Light: Res. Technol. 45(6), 689–709 (2013). [CrossRef]  

39. J. X. Liang, K. D. Xiao, and M. R. Pointer, “Spectra estimation from raw camera responses based on adaptive local-weighted linear regression,” Opt. Express 27(4), 5165–5180 (2019). [CrossRef]  

40. A. David, P. T. Fini, and K. W. Houser, “Development of the IES method for evaluating the color rendition of light sources,” Opt. Express 23(12), 15888–15906 (2015). [CrossRef]  

41. Commission internationale de l’éclairage, colour fidelity index for accurate scientific use. CIE 224, (2017).

42. “Joint ISO/CIE Standard: CIE Colorimetry – Part 4: 1976 L*a*b* Colour Space,” ISO 11664-4:2008(E)/CIE S 014-4 E:2007(E).

43. C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” J. Big Data 6(1), 60 (2019). [CrossRef]  

44. J. J. Zhang, K. A. G. Smet, and Y. Meuret, “Tuning color and saving energy with spatially variable laser illumination,” Opt. Express 27(19), 27136–27150 (2019). [CrossRef]  

45. B. Cao, N. F. Liao, and H. B. Cheng, “Spectral reflectance reconstruction from RGB images based on weighting smaller color difference group,” Color Res Appl. 42(3), 327–332 (2017). [CrossRef]  

46. Commission internationale de l’éclairage, Colormetry. Vienna (Austria). Technical Report, CIE 15 No (2005).

47. F. Yasuma, T. Mitsunaga, and D. Iso, “Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum,” IEEE Trans. Image Process. 19(9), 2241–2253 (2010). [CrossRef]  

48. D. H. Foster, K. Amano, and S. M. C. Nascimento, “Frequency of metamerism in natural scenes,” J. Opt. Soc. Am. A 23(10), 2359–2372 (2006). [CrossRef]  

49. J. L. Nieves, F. Pérez-Ocón, J. Hernández-Andrés, and J. Romero, “Spectral-reflectance function recovery for improved colour-constancy experiments,” Displays 23(5), 213–222 (2002). [CrossRef]  

50. “Joint ISO/CIE Standard: Colorimetry-Part 6: CIEDE2000,” ISO 11664–6:2008(E)/CIE S 014-6:2007(E).

51. BenQ, “The secret to how movies move you: the importance of color accuracy (2021 Update),” (2021/05/13), https://www.benq.com/en-hk/knowledge-center/knowledge/the-importance-of-color-accuracy.html.

52. D. Durmus, “Differences in spectral reflectance functions of natural and artificial surfaces,” in Optica Advanced Photonics Congress 2022, Technical Digest Series (Optica Publishing Group, 2022), paper JW3A.24.

Supplementary Material (1)

NameDescription
Supplement 1       Supplemental 1 for additional figures and table

Data availability

Data underlying the results presented in this paper are not publicly available at this time but can be obtained from the authors upon reasonable request.

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (6)
Fig. 1.
Fig. 1. Dataset augmentation schematic diagram.

View in Article | Download Full Size | PDF

Fig. 2.
Fig. 2. The flowchart of the proposed artificial dataset augmentation process.

View in Article | Download Full Size | PDF

Fig. 3.
Fig. 3. (a) Spectral power distribution of the projector, (b) Camera responsivities

View in Article | Download Full Size | PDF

Fig. 4.
Fig. 4. Augmented sample number as a function of spectral sampling interval (a. three channels with equal interval simultaneously, b. single color channel respectively)

View in Article | Download Full Size | PDF

Fig. 5.
Fig. 5. The results of spectral reflectance estimation using the augmented training sample set using the neural network reflectance estimation approach.

View in Article | Download Full Size | PDF

Fig. 6.
Fig. 6. The results of spectral reflectance estimation using the augmented training sample set using the ALWLR approach.

View in Article | Download Full Size | PDF

Tables (3)

Tables Icon

Table 1. Color channel interval and corresponding augmented sample number

View Table | View all tables in this article
Tables Icon

Table 2. Test results of RMSE and GFC for IES, Munsell, Macbeth, IES datasets

View Table | View all tables in this article
Tables Icon

Table 3. Test results of Δu'v’, ΔY/Y, and ΔE00 under D65, A, F11 for IES, Munsell, Macbeth, IES datasets

View Table | View all tables in this article

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

d i = ∫ λ 1 λ 2 ⁡ c i ( λ ) s ( λ ) r ( λ ) d λ
d = C S r
S ( λ ) = k r S r ( λ ) + k g S g ( λ ) + k b S b ( λ )
S = S r e f ( λ ) [ k r , k g , k b ] T = [ S r , S g , S b ] [ k r , k g , k b ] T
[ d r , r e f d g , r e f d b , r e f ] = ( [ C r C g C b ] ⊤ S r e f ) ⊙ r
[ d r d g d b ] = [ d r , r e f d g , r e f d b , r e f ] [ k r k g k b ]
Δ E a b ∗ = ( L ∗ − L i ∗ ) 2 + ( a ∗ − a i ∗ ) 2 + ( b ∗ − b i ∗ ) 2 2 ( i = 1 ,  2 ,  … ,  n )
w j = 1 Δ E a b , j ∗ + β ( j = 1 ,  2 ,  … ,  K )
W = [ w 1 ,  w 2 ,  w 3 ,  … ,  w K ]
r e x p = r c ⊙ W r
( ( X 2  -  X benchmar k 2 ) +  ( Y 2  -  Y benchmark 2 ) +  ( Z 2  -  Z benchmark 2 ))/3 × 100 % ≤ 10 %
{ R M S E = 1 N ∑ i = 1 N ⁡ ( R i − R ^ i ) 2 G F C = 1 N ∑ i = 1 N ⁡ | R i R ^ i | / ( ∑ i = 1 N ⁡ R i 2 ∑ i = 1 N ⁡ R ^ i 2 ) Δ u ′ v ′ = ( u ′ − u ′ ^ ) 2 + ( v ′ − v ′ ^ ) 2 Δ Y / Y = | Y − Y ^ | / Y
Select as filters
Select Topics Cancel
Top
       
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.
Privacy | Terms of Use