Open Access
Add to BibSonomy
Abstract
A method for spectral reflectance factor reconstruction based on wideband multi-illuminant imaging was proposed, using a programmable LED lighting system and modified Bare Bones Particle Swarm Optimization algorithms. From a set of 16 LEDs with different spectral power distributions, nine light sources with correlated color temperatures in the range of 1924 K - 15746 K, most of them daylight simulators, were generated. Samples from three color charts (X-Rite ColorChecker Digital SG, SCOCIE ScoColor paint chart, and SCOCIE ScoColor textile chart), were captured by a color industrial camera under the nine light sources, and used in sequence as training and/or testing colors. The spectral reconstruction models achieved under multi-illuminant imaging were trained and tested using the canonical Bare Bones Particle Swarm Optimization and its proposed modifications, along with six additional and commonly used algorithms. The impacts of different illuminants, illuminant combinations, algorithms, and training colors on reconstruction accuracy were studied comprehensively. The results indicated that training colors covering larger regions of color space give more accurate reconstructions of spectral reflectance factors, and combinations of two illuminants with a large difference of correlated color temperature achieve more than twice the accuracy of that under a single illuminant. Specifically, the average reconstruction error by the method proposed in this paper for patches from two color charts under A + D90 light sources was 0.94 and 1.08 CIEDE2000 color difference units. The results of the experiment also confirmed that some reconstruction algorithms are unsuitable for predicting spectral reflectance factors from multi-illuminant images due to the complexity of optimization problems and insufficient accuracy. The proposed reconstruction method has many advantages, such as being simple in operation, with no requirement of prior knowledge, and easy to implement in non-contact color measurement and color reproduction devices.
© 2024 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
Corrections
25 January 2024: A typographical correction was made to the author listing.
1. Introduction
Spectral reflectance is generally considered to be the main optical attribute of an opaque object. Traditionally, spectrophotometers have been widely used to measure the spectral reflectance factors of opaque objects. However, these instruments cannot be used to measure non-contactable objects such as corrosive or high-temperature materials, curved surfaces, 3D objects, etc. Digital photography with high resolution cameras can accurately capture the color and appearance of a target object under a stable lighting condition, and can be used to measure the colors of objects [1,2]. Digital cameras have the advantage of being easy-to-use non-contact instruments, useful for distinguishing small color changes in objects on a pixel basis. Spectral reflectance reconstruction provides an effective way for color measurement, and avoids the metamerism phenomenon in camera characterization based on CIE X Y Z tristimulus values [3]. Consequently, over the past two decades, many researchers have focused on studying the reconstruction of spectral reflectance factors from digital camera responses. The usual method for reconstructing the spectral reflectance factor of a given object is to collect camera response data using a black-and-white camera with tunable narrowband filters, or a color camera under a single light source, and then apply specific algorithms, such as pseudo-inverse (PI) [4 –7], Wiener estimation (WE) [4,6,8 –11], finite-dimensional modeling [12 –16], back propagation neural network (BPNN) [17], regularized least-squares (RLS) [18], and so forth.
In the past decade, a variety of spectral reflectance factor reconstruction methods based on wideband imaging have been proposed. In 2010, Kandi [17] proposed BPNN for estimateing the spectral reflectance of printed samples from digital camera responses under two light sources. The networks were configured with six input neurons, 20 hidden neuron layers, and 31 output neurons providing spectral reflectance values. The mean accuracy of the spectral reflectance reconstructions achieved was approximately 2 CIELAB color difference units. In 1999, Hardeberg [5] divided the spectral sensitivity of a camera in the visible spectrum into 2-3 regions, and used 3 groups of LEDs as active lighting sources to cover these regions, obtaining an average spectral reconstruction accuracy of 2.18 CIELAB color difference units. In 2017, Zhang et al. [7] proposed a spectral reflectance estimation method based on CIE tristimulus values X Y Z under multiple virtual light sources, a polynomial model, local training samples, and the pseudo-inverse method, obtaining mean accuracies of 1.05 and 2.11 CIEDE2000 color difference units under the illuminants and light sources tested for the different color charts, respectively. In the same year, Liang et al. [19] proposed a spectral reflectance reconstruction method based on nonlinear extended camera response and a pseudo-inverse algorithm, which improved accuracy by smoothing spectral reflectance curves, and achieved a mean accuracy of 1.0 CIEDE2000 color difference units. Also in 2017, Cao et al. [20] proposed a method for reconstructing spectral reflectance by weighting samples in selected training groups. This method, it was found, could effectively reduce the impact of noise on the reconstruction of spectral reflectance, but required a large number of training samples and a long training time for each color tested. In 2018, Chu et al. [16] proposed a multi-spectral imaging color measurement system with an active LED lighting source and a black-and-white high-speed camera, using the output response of the camera to reconstruct the spectral reflectance factors of objects with an average accuracy of 2.3 CIELAB color difference units for the 24 samples from the X-Rite ColorChecker. In 2021, Li et al. [10] proposed reconstructing spectral reflectance from raw RGB camera data using a weighted polynomial and the Wiener estimation method. This method predicted CIE X Y Z tristimulus values under multiple light sources using the weighted least squares method, extended by a third-order polynomial model, and the best performance was achieved using six light sources. In 2022, Wang et al. [11] proposed establishing a basic equation from raw camera data, including the covariance function of noise and reflectance. System noise was estimated from training data, and the reflectance factors were reconstructed through the use of multiple light sources and a modified Wiener estimation method (named DWE). For two commercial cameras, the best accuracies of this system for Munsell colors were 1.59 and 1.40 CIELAB color difference units over the combination of three light sources: D65, U30, and HZ.
It is worth mentioning that for printed samples the average value of visual threshold reported by Huang et al. [21] was 1.1 CIELAB color difference units. In the automotive industry, color differences below 1.0 CIELAB units are important [22], and so improving the accuracy of the reconstructed spectral reflectance factors is an urgent need.
Multispectral imaging (MSI) is a method for capturing images of an object by considering multiple narrowband wavelength intervals in the visible and invisible parts of the spectrum. MSI systems usually have the disadvantage of time-consuming operation, high manufacturing costs, and low signal-to-noise ratio due to the narrow passband of filters. On the other hand, multi-illuminant imaging (MII) is an easy method for capturing images of an object using multiple wideband light sources in place of filters. In the current paper, a spectral reflectance factor estimation method based on MII and modified Particle Swarm Optimization (PSO) algorithm was proposed. Specifically, a system with 16 programmable LEDs and a color industrial camera were used to produce nine light sources with different correlated color temperatures, and to capture object images under such light sources. The spectral reflectance factors of objects were then reconstructed from camera responses using the proposed PSO and classic algorithms.
2. Spectral reconstruction based on particle swarm optimization algorithm
The relationship between camera response and object spectral reflectance factor is usually expressed by Eq. (1):
To solve Eq. (2) is to find the (N × C) conversion matrix${\; }{\boldsymbol G}$, so that:
where $\hat{{\boldsymbol R}}$ is a (N × M) matrix denoting the estimated spectral reflectance factors, and V is the (C × M) matrix of camera responses. The key to achieving the desired spectral reflectance factor reconstruction relies on the correct computation of transformation matrix${\; }{\boldsymbol G}$. Many algorithms have been proposed for solving Eq. (3), such as pseudo-inverse (PI), Wiener estimation (WE), back propagation neural network (BPNN), etc. The pseudo-inverse algorithm seeks a matrix ${\boldsymbol G}$ while disregarding the additive noise of the system and may be affected by higher requirements of data acquisition in the early stage. The classic Wiener estimation method requires a priori knowledge of second-order statistics with respect to the original spectral reflectance and system noise, usually estimated by the covariance or autocorrelation matrices of reflectance and noise in the training data. If system noise during training and testing is different, the accuracy of reconstruction by this method may be affected. To solve the nonlinear problem of camera responses, scholars have also proposed several methods using polynomial extensions. For example, the 20-term polynomial model proposed by Wang et al. [23] can meet general requirements (named Nonlin), although the number of polynomial terms required is too large and over-fitting may occur. Amiri et al. [24] proposed a weighted pseudo-inverse method with a 17-term polynomial (named wt-nonlin), where each test sample must be trained separately, implying a considerable amount of time for restraining noise and selecting from a large number of training samples. Unfortunately, these classic reconstruction algorithms may obtain a transformation matrix ${\boldsymbol G}\; $that corresponds to a local optimal solution when the optimized problem is complicated. Therefore, in the current paper a global optimization algorithm based on Particle Swarm Optimization (PSO) is proposed for the reconstruction of spectral reflectance factors.PSO [25] is a bionic algorithm which simulates bird flock preying behavior, and is one of the most used swarm intelligence algorithms. Each particle in the swarm represents one potential optimal solution of the problem, e.g., the transformation matrix ${\boldsymbol G}$ in Eq. (3). The global optimal solution of the problem can be found through the individual particles’ own optimal solutions as well as information on the interaction between particles in the population. Among the advantages of the swarm intelligence algorithm are its notable robustness, as well as the fact that individuals interacting in the group are distributed, with no direct control center, and so the failure of a few of those individuals may not affect the solution of the problem. The swarm intelligence algorithm also possesses the characteristics of strong self-organization, and a simple structure that is easy to implement and expand. It was therefore applied to the field of multi-illuminant spectral reconstruction for the current paper.
The PSO algorithm was first proposed by J. Kennedy and R. Eberhart in 1995 [25], and, after years of improvement, several modified PSO algorithms have been derived and widely used in many fields. These include the fully informed PSO (FIPSO) [26], the bare bones PSO (BBPSO) [27], and the quantum-behaved PSO (QPSO) [28].
The original PSO algorithm contains the following important parameters [25,29]: (1) the individual learning factor of particles, also known as individual acceleration factor ${c_1}$; (2) the particle social learning factor, also known as social acceleration factor ${c_2}$; (3) inertia weight of velocity designed as w; (4) after k iterations, the velocity of the i th particle is $v_i^k$; (5) after k iterations, the position of the i th particle is $x_i^k$; (6) fitness value $f(x )$ at position x; (7) after k iterations, $\; pBest_i^k$ is the best position that the i th particle passes by; (8) after k iterations, $gBes{t^k}$ is the best position that all particles pass through; (9) population size or particle swarm. While it is easy to fall into local optimal solutions for small population sizes, larger population sizes may imply longer computation times while improving convergence and leading to global optimal solutions. When the particle swarm size achieves a certain value, further increases no longer have a significant effect. Generally, sizes in the range 20∼50 are sufficient.
After k iterations, the updated velocity and position of the i th particle can be defined as:
In PSO, the reconstructed reflectance factor $\hat{{\boldsymbol R}}(j )$ for the j th wavelength band can be expressed as:
where${\; }j \in [{1,31} ]\; $for a spectral range from 400 nm to 700 nm at 10 nm intervals$,{\; }{\boldsymbol G}(j ) \; $is the general best position $gBes{t^k}\; $after k iterations (also the j th row of matrix ${\boldsymbol G}$ in Eq. (3)), and V are camera responses. We used the 400-700 nm wavelength range because this is the range of the X-Rite Ci64UV spectrophotometer employed for reference measurements of spectral reflectance samples of our samples (section 3.2).Fitness value $f(x )$ at position x is defined as the root mean square error (RMSE) between the reconstructed reflectance factor $\hat{{\boldsymbol R}}(j )$ and the true reflectance factor ${\boldsymbol R}(j )$ for the total number of M training samples:
The basic PSO algorithm has many parameters. Improved PSO algorithms, such as the previously mentioned FIPSO, BBPSO or QPSO algorithms, reduce the number of parameters, increasing convergence speed and improving optimization ability.
Kennedy [27] proposed using a Gaussian distribution to control particle evolution, as shown in Eq. (8), which is known as the Bare Bones Particle Swarm Optimization (BBPSO) algorithm. BBPSO is a more concise algorithm than PSO, eliminating the velocity attribute of particles and proposing evolution following a random Gaussian distribution. The simple collaborative probability search method introduced in the BBPSO can improve both the search efficiency and the accuracy of the algorithm, and avoids the complex parameter settings of PSO. It is therefore widely used.
In order to improve the local exploration ability of the BBPSO algorithm and make it easy to go beyond local optimal solutions, it is proposed here that either a Gaussian or a Cauchy distribution be used for a diversity of particles (hereafter named GCBBPSO algorithm), as shown in Eq. (9):
In order to further improve the global exploration ability of this GCBBPSO algorithm, a Cauchy perturbation was added to update the optimal position of all individuals, as shown in Eq. (11):
Using this Cauchy perturbation, the particles are explored again around the optimal position of the swarm. If a better solution is found, the optimal position of the swarm is updated; if no better solution is found, the original optimal position of the particles is maintained. It is helpful for the particles to transcend potential local optimal solutions by the introduction of this Cauchy perturbation. The pseudo-code for the reconstruction of spectral reflectance factors using the proposed GCBBPSO algorithm is as follows:
In this pseudo-code, maxIteration represents the maximum number of iterations, depending on the desired reconstruction accuracy, and was 20,000 in the current study.
The proposed GCBBPSO can be further extended as GCBBPSO’ based on nonlinear extended camera responses V in order to address the nonlinear relationship between camera responses and reflectance factors.
3. Experimental method
Our workflow for spectral reflectance factor reconstruction based on MII is shown in Fig. 1, where the flowcharts marked in blue dotted, and green broken boxes are the training, and testing parts, respectively.
3.1 Fitting multiplex LED light sources
Traditional light booths, also known as color cabinets, provide six different fixed light sources at most, with different correlated color temperatures. Users manually select from these light sources for specific purposes without any kind of computer control. With the continuous development and improvement of LED lighting technology, it is an inevitable trend to use LEDs as light sources for future color measuring instruments [30,31]. In addition to the characteristics of high luminous efficiency, environmental friendliness, long life, small size, anti-vibration, etc., LEDs also have advantages of fast response speed and easy computer control.
In order to study the impact of light sources on the reconstruction of spectral reflectance factors, 16 LEDs with different spectral power distributions (SPDs) were controlled by a computer to generate the nine light sources employed in the experiment. These light sources were constructed by curve fitting techniques, in an attempt to approach the SPDs of several CIE illuminants with correlated color temperatures (CCTs) in a relatively wide range (1924 - 15746 K) [32]. The intensity of each LED could be programmed to control the amount of current passing through it, and different light sources were constructed.
The relative SPDs of the 16 LEDs used in this study are shown in Fig. 2(a), while the relative SPDs of the nine selected light sources used in the experiment are shown in Fig. 2(b). Table 1 shows information regarding these nine light sources including name, illuminance, CCT, and general color rendering index (Ra). We selected the three standard illuminants currently adopted by the CIE (A, D50 and D65) plus five additional CIE daylight illuminants with CCTs in the range of 7473 -15746 K, plus one horizont light (HZ) which is close to candle light or light at sunset or sunrise. Program-controlled multiplex LED systems can generate light sources with specific SPDs, as well as sources with high values of color rendering indices (except for D160). To simplify the description, the names of the light sources used in this study (Table 1, except for HZ) were the same than those of the corresponding CIE illuminants.
Fig. 2. Spectral power distribution of (a) 16 selected LEDs, (b) the nine light sources used in the experiment; A and D90 are plotted with broken and dotted lines for special attention.
Table 1. Information on the nine light sources used in the experiment.
3.2 Acquisition of experimental data
Images were obtained using a DigiColor light booth produced by Zhejiang SCOCIE Instrument, China, as shown in Fig. 3. LED clusters, including the 16 abovementioned LEDs (Fig. 2(a)), were evenly distributed on both bottom sides of the booth, with four LED clusters on each side. The light emitted at each side was scattered by a diffuse plate, reflected by an aspherical mirror, and diffused through the inner walls and arch dome of the light booth to uniformly illuminate the object placed on the floor of the booth. This particular light booth provided a closed environment which allowed uniform controlled lighting of the object to be measured. An industrial color camera was mounted perpendicularly at the top, in the center of the dome, with a vertical distance of 80 cm to the floor. Specifically, a Daheng color camera model ME2P-1230-23U3C was used, with a lens of 16 mm focal length, providing a resolution of 4096 × 3000 pixels and 12 bits depth. This camera is suitable for industrial inspection, medical and scientific research, education, and security applications. The camera controlling software was generated from a software development kit provided by Daheng, including camera setting, image acquisition, and image processing, and was integrated with the LED controlling software. The camera settings used in the experiment were fixed in aperture, shutter speed, and white balance (D65) to avoid overexposure of patches in the three color charts employed (Fig. 4) under the nine light sources considered. The images captured were used without further image processing after demosaicing.
Fig. 3. Scheme of the DigiColor light booth (Zhejiang SCOCIE Instrument, China) used in the experiment.
Fig. 4. Sets of samples in the three color charts ((a) SG140, (b) SP240, and (c) ST240) used in the experiment.
The three color charts shown in Fig. 4 were used in the experiment, a ColorChecker Digital chart (SG140) from X-Rite, a ScoColor V2.0 chart (SP240), and a ScoColor V3.0 chart (ST240), both of the latter from Zhejiang SCOCIE Instrument, China. Table 2 shows information on these three color charts, including full and abbreviated names, the number of color patches and gamut volume in the CIELAB color space L*, a*, b* [32], assuming the illuminant D65 and the CIE 1931 standard colorimetric observer. It can be seen that the three charts are made from two different materials and have different gamut volumes. For each sample in these three charts, the spectral reflectance factor in the range of 400-700 nm with intervals of 10 nm was measured, using a portable spectrophotometer Ci64UV from X-Rite, with medium aperture and specular component included. Manufacturer’s instructions were followed for these measurements, after proper calibration of the instrument using supplied white and black standards. In order to reduce measurement uncertainties caused by random noise, each color patch in the three color charts was measured three times, and the average value was taken as the target data of the experiment.
Table 2. Information on the three color charts used in the experiment.
In our experiment, the above three color charts were in turn placed on the floor of the light box, and the computer software lit up the LED clusters according to the recipe of the nine light sources. At the same time, the software drove the camera to collect the image from the color chart under each light source. Overall, 27 images (3 color charts × 9 light sources) were collected, and the average RGB values of each color patch in each color chart were extracted using self-developed image processing software. Considering that the color charts had 140 or 240 samples, a total of 140 × 9 + 240 × 2 × 9 = 5,580 sets of RGB data were obtained, and were used as the original experimental data for the current study.
4. Results and discussion
4.1 Impact of training samples on reconstruction accuracy using the GCBBPSO algorithm
To explore the effect of training samples on the accuracy of reconstruction of spectral reflectance factors, the samples from each one of the three color charts employed (SG140, SP240, and ST240) were selected in turn as training samples, and the samples in the two remaining color charts were treated as test samples. The GCBBPSO proposed algorithm was used to conduct the current study, based on each of the nine light sources. Reconstruction accuracies in terms of CIEDE2000 color difference $\Delta {\textrm{E}_{00}}$ units [33,34] are shown in Table 3. One CIEDE2000 unit is roughly equivalent to 0.65 CIELAB units [35].
Table 3. Training (3 color charts) and testing accuracy ($\Delta {{\boldsymbol E}_{00}} \; $units) under nine light sources.
Table 3 shows that reconstruction accuracy for test samples under different light sources was basically similar (mean in range 2.11 - 2.62 CIEDE2000 units), especially under daylight illuminants and ‘Test 3’. This means that if the reconstruction models are trained using ST240, which had the largest gamut volume, very similar prediction accuracies will be given for all tested samples. The main requirement for selecting a given set of training samples is not only a large number of color patches, but also a large color gamut to ensure that the reconstruction model can achieve the highest reconstruction accuracy for all tested samples. Therefore, in the following studies, the samples in the ST240 color chart will be used as training samples, SG140 and SP240 chart as test samples.
4.2 Impact of different algorithms on reconstruction accuracy
In order to reveal the effect of different algorithms on the reconstruction accuracy of spectral reflectance factors, the performance of several algorithms was compared, including the proposed GCBBPSO algorithm and its nonlinear extension version GCBBPSO’, the original PSO, BBPSO, and six other commonly used algorithms: WE, PI, DWE, RLS [18], Nonlin [23], and wt-nonlin [24]. Here, a 17-term polynomial model for the nonlinear extension of camera responses was used in the GCBBPSO’, Nonlin and wt-nonlin algorithms. RLS is the regularized least squares algorithm, usually used for solving linear problems. Test results for the SG140 and SP240 color charts under the nine light sources are shown in Tables 4 and 5, respectively. The last columns in Tables 4 and 5 show the average values of the corresponding algorithms under all nine light sources, while the last rows show the average values of all algorithms under each illuminant (numbers in bold indicate lowest average values).
Table 4. Reconstruction accuracy for SG140 ($\Delta {{\boldsymbol E}_{00}}$ units), using different algorithms under nine light sources.
Table 5. Reconstruction accuracy for SP240 ($\Delta {{\boldsymbol E}_{00}}$ units), using different algorithms under nine light sources.
As shown in Tables 4 and 5, the differences in reconstruction accuracy between the ten tested algorithms were not very large under a single illuminant. For example, the maximum differences between different algorithms were less than 0.5 $\Delta {E_{00}}$ units for the two groups of tests except for that of the HZ light source. If we compare the performance of the ten algorithms, it can be seen that the three algorithms based on particle swarm optimization (PSO, BBPSO, and GCBBPSO), and the two classic algorithms (WE and PI) provided almost the same results for the samples in the SG140 (Table 4) and SP240 (Table 5) color charts. This suggests that the transformation matrix achieved by the WE and PI algorithms is perhaps the global optimal solution under any single light source, and the three versions of the PSO algorithm arrived at just such a solution for the current relatively simple optimization problem. On the other hand, potential noise processing mechanisms in our algorithms seem not to be relevant, at least for single illuminants. From Tables 4 and 5, it can be seen that the three algorithms with nonlinear extended camera responses (GCBBPSO’, Nonlin and wt-nonlin) performed best, implying the existence of a relevant nonlinear relationship between spectral reflectance factors and camera responses. From Tables 4 and 5, average accuracy for all illuminants and algorithms tested was 2.52 and 2.27 CIEDE2000 units (equivalent to 3.9 and 3.5 CIELAB units [35]), respectively.
4.3 Impact of CCT of different illuminants on reconstruction accuracy
From Tables 4 and 5, regardless of the selected algorithm, reconstruction accuracy increases with the correlated color temperature of the light source from 1924 K to 10052 K (i.e., HZ to D100 sources). The highest reconstruction accuracy was achieved for a CCT around 10000 K (D100 source), and then there was a decreasing trend for sources with CCTs in the range of 10052 K to 15746 K (i.e., D100 to D160 sources). From current experimental data, it can be concluded that, using only one light source, the best reconstruction accuracy of spectral reflectance factors was achieved for a daylight source with a CCT of around 10000 K. Specifically, from Tables 4 and 5, average accuracy for all algorithms tested under D100 was 1.34 and 1.30 CIEDE2000 units (equivalent to about 2.0 CIELAB units [35]) for SG140 (Table 4) and SP240 (Table 5), respectively. In any case, this level of accuracy still exceeds the requirements (i.e., 1.0 CIELAB unit) of some industries [21,22].
4.4 Impact of combination of several illuminants on the reconstruction accuracy
In order to further improve the reconstruction accuracy of spectral reflectance factors, we examined the operation of the algorithms employed with data obtained under more than one light source.
First, the effect on reconstruction accuracy of 36 two-sources combinations from the nine light sources was studied. Reconstruction accuracy in terms of${\; }\Delta {E_{00}}\; $for the test samples in the SG140 and SP240 color charts is shown in Tables 6 and 7, respectively. Under all two-source combinations tested, the last columns in Tables 6 and 7 show the average accuracy for the eight algorithms (excluding PSO and BBPSO which showed results divergent to those from remaining algortihms), while the last rows show the average accuracy for each algorithm.
Table 6. Reconstruction accuracy for SG140$\; (\Delta {{\boldsymbol E}_{00}}\;$units) under different two-source combinations. Mean values in the last column excluded divergent results from the PSO and BBPSO algorithms.
Table 7. Reconstruction accuracy for SP240 ($\Delta {{\boldsymbol E}_{00}}$ units) under different two-source combinations. Mean values in the last column excluded divergent results from the PSO and BBPSO algorithms.
As shown in Tables 6 and 7, adding a second light source can improve reconstruction accuracy. For example, the average accuracy of the eight algorithms (except PSO and BBPSO) under most two-source combinations (except HZ + A) is below 1.5 $\Delta {E_{00}}\; $units, roughly equivalent to 2.3 CIELAB units [35]. It can also be noted that the reconstruction accuracy of different two light sources combinations is slightly different. The last rows of Tables 6 and 7 show that the original PSO and BBPSO failed to solve the current problem, while the remaining eight algorithms performed similarly in such a way that average accuracy ranges from 1.07 to 1.32 $\Delta {E_{00}}\; $units under all two-source combinations. It can also be seen from Tables 6 and 7 that the highest accuracy was found for the combination of A and D90, where the averages for the eight algorithms were 0.94 and 1.08 $\Delta {E_{00}}\; $units (around 1.45 and 1.66 CIELAB units [35]) for SG140 and SP240, respectively.
We further investigated the optimal number of combinations of light sources for achieving the best reconstructions of spectral reflectance factors, and the average results found for the different algorithms are shown in Tables 8 and 9, where the values in bold indicate the best source combinations for each algorithm. From Tables 8 and 9, it can be noted that the PSO and BBSPO did not work well under combinations of more than two sources due to the low diversity of particles and the complexity of the optimization problem. However, the proposed GCBBPSO and GCBBPSO’, based on BBPSO, worked sufficiently well in all cases tested. It is confirmed that, compared with the original PSO and BBPSO algorithms, the proposed GCBBPSO and GCBBPSO’ algorithms are effective for finding optimal solutions to complex problems like this.
Table 8. Average reconstruction accuracy ($\Delta {{\boldsymbol E}_{00}}$ units) for SG140 under different numbers of combinations of our nine light sources. Bold numbers indicate best accuracy for each method.
Table 9. Average reconstruction accuracy ($\Delta {{\boldsymbol E}_{00}}$ units) for SP240 under different numbers of combinations of our nine light sources. Bold numbers indicate best accuracy for each method.
From Tables 8 and 9, it is noted that most algorithms achieved better spectral reconstruction accuracy using two or three sources. However, a further increase in the number of light sources has little impact on reconstruction accuracy and increases both storage resources and computation times. In addition, some algorithms, such as PI, DWE, RLS, Nonlin, PSO and BBPSO, may predict spectral reflectance factors with relatively low accuracy. For example, for the SG140 chart (Table 8) the best reconstruction accuracy for the WE algorithm was reached using three sources, while for the PI, Nonlin, wt-nonlin, DWE, RLS and GCBBPSO this was achieved using only two. Similarly, for the SP240 chart (Table 9), most algorithms reached their best accuracy using two or three sources. Based on the above analyses and discussion, it is recommended that the A and D90 light sources be used for optimal reconstruction of spectral reflectance factors in multi-illuminant imaging.
From Tables 8 and 9, it can also be observed that the WE and GCBBPSO’ algorithms performed similarly, and in fact outperformed the other algorithms in most cases. However, compared with iterative algorithms based on PSO, WE has the advantage of simplicity and speed. It is worth mentioning that there are many different versions of the Wiener estimation algorithm, e.g., the DWE and WE used in the current study. However, WE [8] performed slightly better than DWE in the current experiment. DWE and WE used different methods for estimating second-order statistics for the original spectral reflectance and system noise, based on different assumptions of camera channel independence: DWE employed covariance matrices [10], while WE utilized autocorrelation matrices [8] of reflectance and noise in the training data. For single light sources (Tables 4 and 5) or pairs of light sources (Tables 6 and 7), the performance of DWE and WE is similar. However, the more light sources there are in a combination, the larger the difference between DWE and WE because the noise of different channels should be relatively independent for different light sources, and DWE may overestimate the noise by using a covariance matrix.
In comparing the performance of the Nonlin, wt-nonlin, and GCBBPSO’ algorithms, with a nonlinear extension of camera response, the GCBBPSO’ outperformed the other two algorithms.
As an example of the findings, Fig. 5(a) shows the measured and predicted spectral reflectance factors (under A + D90) from different algorithms (without considering BBPSO) for the color patch with the smallest color difference (Row 2, Column G in the SG140 chart, with a mean value of 0.35 $\Delta {E_{00}}$ units). Similarly, Fig. 5(b) shows analogous results for the color patch with the largest color difference (Row 4, Column G in the SG140 chart, with a mean of 2.24 $\Delta {E_{00}}$ units).
Fig. 5. Measured and reconstructed spectral reflectance factors of two color patches with (a) minimum and (b) maximum average CIEDE2000 color differences under a combination of A and D90 for each of the algorithms tested.
5. Conclusions
A method of spectral reflectance factor reconstruction based on wideband multi-illuminant imaging and modified particle swarm optimization (GCBBPSO, and GCBBPSO’) algorithms is proposed. In the experiment, nine light sources with correlated color temperatures in the range 1924 K - 15746 K were employed by a program-controlled multiplex LED lighting system. Images of samples in three training and/or testing color charts were captured by a color camera under nine different light sources, and the spectral reconstruction models were trained and/or tested using ten different algorithms, including the proposed GCBBPSO and GCBBPSO’ algorithms under a total of 511 different light source combinations, with from one to nine light sources in each combination. The results show that the proposed GCBBPSO and GCBBPSO’ algorithms can solve complicated optimization problems such as the one under study, and predict spectral reflectance factors from multi-illuminant images with reasonable accuracy. It is suggested that the GCBBPSO’ or WE algorithm be used for predicting spectral reflectance factors from multi-illuminant images. The best average accuracy achieved in the current experiment was 0.94 and 1.08 CIEDE2000 color difference units, using a combination of A and D90 light sources for the two color charts tested, respectively.
Funding
Ministerio de Ciencia e Innovación and Agencia Estatal de Investigación (PID2022-138031NB-I00/SRA/10.13039/501100011033); National Natural Science Foundation of China (61671329, 61775170).
Acknowledgment
Donald J. Murphy McVeigh for English revision.
Disclosures
The authors declare no conflicts of interest.
Data Availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. H. Shen and J. H. Xin, “Spectral characterization of a color scanner by adaptive estimation,” J. Opt. Soc. Am. A 21(7), 1125–1130 (2004). [CrossRef]
2. D. Li, G. Cui, Z. Zhao, et al., “Color measurement system based on convergent binocular stereo vision,” Proc. SPIE 12462, 124621 (2023). [CrossRef]
3. M. M. Amiri, S. Garcia-Nieto, S. Morillas, et al., “Spectral reflectance reconstruction using fuzzy logic system training: Color Science Application,” Sensors 20(17), 4726 (2020). [CrossRef]
4. W. K. Pratt and F. Davarian, “Fast computational techniques for pseudoinverse and Wiener image restoration,” IEEE Trans. Comput. C-26(6), 571–580 (1977). [CrossRef]
5. J. Y. Hardeberg, “Acquisition and reproduction of color images: colorimetric and multispectral approaches,” Ph.D. dissertation (Ecole Nationale Superieure des Telecommunications, 1999).
6. H. Shen, J. H. Xin, and S. Shao, “Improved reflectance reconstruction for multispectral imaging by combining different techniques,” Opt. Express 15(9), 5531–5536 (2007). [CrossRef]
7. X. Zhang, Q. Wang, J. Li, et al., “Estimating spectral reflectance from camera responses based on CIE XYZ tristimulus values under multi-illuminants,” Color Res. Appl. 42(1), 68–77 (2017). [CrossRef]
8. H. Shen, P. Cai, S. Shao, et al., “Reflectance reconstruction for multispectral imaging by adaptive Wiener estimation,” Opt. Express 15(23), 15545–15554 (2007). [CrossRef]
9. H. Haneishi, T. Hasegawa, A. Hosoi, et al., “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39(35), 6621–6632 (2000). [CrossRef]
10. F. Li and C. Li, “Spectral reflectance reconstruction based on camera raw RGB using weighted third-order polynomial and Wiener estimation,” Spectrosc. Spect. Anal. 41(10), 3281–3285 (2021). [CrossRef]
11. H. Wang and C. Li, “Spectral reconstruction algorithm based on multi-light source and Wiener estimation,” Laser Optoelectron. Prog. 59, 1933001 (2022). [CrossRef]
12. V. Cheung, S. Westland, C. Li, et al., “Characterization of trichromatic color cameras by using a new multispectral imaging technique,” J. Opt. Soc. Am. A 22(7), 1231–1240 (2005). [CrossRef]
13. M. Shi and G. Healey, “Using reflectance models for color scanner calibration,” J. Opt. Soc. Am. A 19(4), 645–656 (2002). [CrossRef]
14. F. H. Imai and R. S. Berns, “Spectral estimation using trichromaitc digital cameras,” in Proceedings of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives (Society of Multispectral Imaging of Japan, Chiba, Japan, 1999).
15. M. Soriano, W. Oblefias, and C. Saloma, “Fluorescence spectrum estimation using multiple color images and minimum negativity constraint,” Opt. Express 10(25), 1458–1464 (2002). [CrossRef]
16. J. Chu, G. Cui, Y. Liu, et al., “A method for measuring surface color based on spectral tunable LED light source and multispectral imaging technology,” Acta Opt. Sin. 38(8), 0833001 (2018). [CrossRef]
17. S. G. Kandi, “Estimating Spectral and Colorimetric Data of Printed Samples from Digital Camera Responses under two Illuminants by Neural Network,” J. Printing Sci. Technol. 47, 392–400 (2010). [CrossRef]
18. H. Shen, J. H. Wan, and Z. C. Zhang, “Estimating reflectance from multispectral camera responses based on partial least-squares regression,” J. Electron. Imaging 19(2), 020501 (2010). [CrossRef]
19. J. Liang and X. Wan, “Optimized method for spectral reflectance reconstruction from camera responses,” Opt. Express 25(23), 28273–28287 (2017). [CrossRef]
20. B. Cao, N. Liao, and H. Cheng, “Spectral reflectance reconstruction from RGB images based on weighting smaller color difference group,” Color Res. Appl. 42(3), 327–332 (2017). [CrossRef]
21. M. Huang, H. Liu, G. Cui, et al., “Evaluation of threshold color differences using printed samples,” J. Opt. Soc. Am. A 29(6), 883–891 (2012). [CrossRef]
22. M. Melgosa, J. Martínez-García, L. Gómez-Robledo, et al., “Measuring color differences in automotive samples with lightness flop: A test of the AUDI2000 color-difference formula,” Opt. Express 22(3), 3458–3467 (2014). [CrossRef]
23. Y. Wang and H. Xu, “Colorimetric characterization for scanner based on polynomial regression models,” Acta Opt. Sin. 27, 1135–1138 (2007).
24. M. M. Amiri and M. D. Fairchild, “A strategy toward spectral and colorimetric color reproduction using ordinary digital cameras,” Color Res. Appl. 43(5), 675–684 (2018). [CrossRef]
25. J. Kennedy and R. Eberhart, “Particle swarm optimization,” Proceedings of ICNN'95 - International Conference on Neural Networks, Perth, WA, Australia, (1995), pp. 1942–1948 vol.4, doi: 10.1109/ICNN.1995.488968.
26. R. Mendes, J. Kennedy, and J. Neves, “The fully informed particle swarm: Simpler, maybe better,” IEEE Trans. Evol. Computat. 8(3), 204–210 (2004). [CrossRef]
27. J. Kennedy, “Bare bones particle swarms,” Proceedings of the IEEE Swarm Intelligence Symposium80–87, (2003).
28. J. Sun, W. Fang, X. Wu, et al., “Quantum-Behaved Particle Swarm Optimization: Analysis of Individual Particle Behavior and Parameter Selection,” Evol. Comput. 20(3), 349–393 (2012). [CrossRef]
29. Y. Shi and R. Eberhart, “A modified particle swarm optimizer,” IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360), 69–73, (1998).
30. J. Liang, K. Xiao, and X. Hu, “Investigation of light source effects on digital camera-based spectral estimation,” Opt. Express 29(26), 43899–43916 (2021). [CrossRef]
31. J. Liang, K. Xiao, M. R. Pointer, et al., “Spectra estimation from raw camera responses based on adaptive local-weighted linear regression,” Opt. Express 27(4), 5165–5180 (2019). [CrossRef]
32. CIE 015:2018. “Colorimetry, 4th Edition” CIE Central Bureau, Vienna.
33. M. R. Luo, G. Cui, and B. Rigg, “The development of the CIE 2000 colour-difference formula: CIEDE2000,” Color Res. Appl. 26(5), 340–350 (2001). [CrossRef]
34. ISO/CIE 11664-6:2022. Colorimetry — Part 6: CIEDE2000 colour-difference formula. CIE Central Bureau, Vienna.
35. M. Melgosa, G. Cui, C. Oleari, et al., “Revisiting the weighting function for lightness in the CIEDE2000 colour-difference formula,” Color. Technol. 133(4), 273–282 (2017). [CrossRef]
