Open Access
Add to BibSonomy
Abstract
Recently, the miniature spectrometer based on the optical filter array has received much attention due to its versatility. Among many open challenges, designing efficient and stable algorithms to recover the input spectrum from the raw measurements is the key to success. Of many existing spectrum reconstruction algorithms, regularization-based algorithms have emerged as practical approaches to the spectrum reconstruction problem, but the reconstruction is still challenging due to ill-posedness of the problem. To alleviate this issue, we propose a novel reconstruction method based on a solver-informed neural network (NN). This approach consists of two components: (1) an existing spectrum reconstruction solver to extract the spectral feature from the raw measurements (2) a multilayer perceptron to build a map from the input feature to the spectrum. We investigate the reconstruction performance of the proposed method on a synthetic dataset and a real dataset collected by the colloidal quantum dot (CQD) spectrometer. The results demonstrate the reconstruction accuracy and robustness of the solver-informed NN. In conclusion, the proposed reconstruction method shows excellent potential for spectral recovery of filter-based miniature spectrometers.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
In recent years, miniature spectrometers featuring light-weight, small volume, and smooth operation have attracted significant attention [1,2]. It can be widely used in many areas, such as biology, chemistry [3], and medical applications [4]. Particularly, miniature spectrometers have great potential to be integrated and achieve on-chip spectral sensors [1–3]. Among all existing miniature spectrometers, those based on absorptive filters [5–8] are one of the promising candidates, which use absorptive filters as spectral sampling channels and recover spectra from channel measurements via reconstruction methods.
Among many miniature spectrometers, the colloidal quantum dot (CQD) spectrometer [9] is one of the most promising miniature spectrometers for practical applications. CQDs, as one kind of semiconductor nanocrystals, possess desirable optical and processing properties. For instance, changing the CQD’s particle size and composition can adjust its absorption spectrum continuously [9,10]. This property makes it easier to produce hundreds of different filters in one CQD spectral device. Additionally, CQDs have great processing versatility due to the solution-phase synthesis process [11]. For example, CQDs can be deposited from solution via spraying, printing [12], etc. This property makes it easy to adapt to the desired film structure and shape compatible with other devices.
Like most other low-cost filters adopted in the miniature spectrometer systems, the CQD filters are also with broad passbands, and many of them have over-lapping operational bands. As a result, these filters are usually highly correlated, which could cause many issues to recover high-quality spectra under noisy environments [13,14]. Mathematically, the spectrum reconstruction problem for the filter-based miniature spectrometer can be formulated as a linear inverse problem [13,15–17]. Since the ordinary least squares often fail to deliver an accurate spectral estimate due to the ill-posed nature of the problem, regularization techniques utilizing the prior knowledge of the underlying problems have been explored to improve the spectrum reconstruction quality. For example, non-negative least squares [15] have proven to be effective in spectrum reconstruction by enforcing the non-negative nature of the spectrum. Besides, other natures of the spectrum have also been explored for spectrum reconstruction, such as the sparsity of the underlying spectrum, see [14,16,17] for examples. Despite their success, they could still lead to noticeable distortions due to both the ill-posedness of the underlying problem and measurement errors [13,16].
Recent advances in neural networks (NNs) [18] have enjoyed great success in the imaging-related fields, such as image classification [19,20], image segmentation [21], and hyperspectral imaging [22–24], which have opened up new opportunities for designing accurate and robust algorithms for spectrum reconstruction. The application of convolutional neural networks (CNNs) to the compressive sensing spectrometers [25] is of particular interest to us. In [25], a CNN is trained to predict spectra from the feature extracted by applying a direct inversion operator. Later on, a residual CNNs (ResCNNs) is proposed to deliver better reconstruction performance than CNNs and sparse recovery [26]. These research findings demonstrate the potential of the NN to facilitate spectrum reconstruction.
Motivated by the above research findings, we propose a novel reconstruction method by leveraging existing spectrum reconstruction solvers and the conventional NN. More specifically, we first employ an existing spectrum reconstruction solver to extract spectral features from the raw measurements of CQD filters and then we use the extracted features to train the NN to learn a map from the features to the ground-truth spectrum. We refer to this network as solver- informed NN. The key difference between our approach and the previous studies [22,25–27] is that we use an existing spectrum reconstruction solver to extract the features instead of using the direct inversion operator. This is supposed to extract more relevant features to enhance the NN’s predictive capability, compared with applying the direct inversion operator. We demonstrated the feasibility and the performance of the proposed method on a synthetic dataset and a real experimental dataset collected by our CQD spectrometer.
The rest of the paper is organized as follows: Section 2 briefly reviews the optical structure of CQD spectrometer and conventional regularized reconstruction methods. We next discuss the proposed NN-based spectrum reconstruction methods in Section 3. We then illustrate the effectiveness and performance of the proposed methods via several experiments in Section 4 and Section 5. We conclude the paper in Section 6.
2. Background
2.1 Colloidal quantum dot spectrometer system
To setup the stage for the later discussion, we shall first introduce the experiment setup and the spectrum reconstruction problem. Our CQD spectrometer system is illustrated in Fig. 1, which consists of a CQD filter array and a charge-coupled device (CCD). The CQD filter array is placed above the CCD chip and each CQD filter corresponds to a set of pixels in CCD. We let $x(\lambda )$ denote the intensity of incident light at wavelength $\lambda$ and $T_i(\lambda )$ denote the transmission function of the $i$th CQD filter. And the spectral sensitivity of CCD pixels is specified with $\eta (\lambda )$, which is assumed to be identical for all pixels in CCD. Each CQD filter and its corresponding CCD pixels can be regarded as one spectral sampling channel with transmission function $R_i(\lambda )=T_i(\lambda )\eta (\lambda ),i=1,2,\ldots ,N$, where $N$ is the number of CQD filters.
The measurement of the sampling channel in the CQD spectrometer system is determined by the incident light $x(\lambda )$ and the channel’s transmission function $R_i(\lambda ),i=1,2,\ldots ,N$. This process can be formulated as the following mathematical problem [13,16]:
where $I$ is a $N \times 1$ dimensional channel measurement vector, $x$ is a $S \times 1$ dimensional incident light spectrum ($S$ is the number of spectral data points) that has to be reconstructed and $\xi$ is a measurement error vector with dimension of $N \times 1$. $R$ is a $N \times S$ dimensional sensing matrix, each row in which denotes a spectral transmission function of one channel. The form of $R$ is given as follows:2.2 Regularized spectrum reconstruction methods
In general, the reconstruction problem is susceptible to measurement errors. In order to produce better-reconstructed spectra, regularization techniques are widely used in the literature of spectrum recovery [13,14,16,17,28]. In this section, we shall briefly discuss several popular choices, including regularized non-negative least squares (Reg NNLS) and sparse recovery based solver, which serve as the baseline solutions later on.
2.2.1 Regularized non-negative least squares
As we mentioned earlier, solving the reconstruction problem (1) directly by the ordinary least squares usually leads to severe distortions due to the existence of measurement errors [13]. Alternatively, modified least squares algorithms are applied to the spectrum reconstruction problem, such as Reg NNLS [13,28], which has been found particularly useful for spectrum reconstruction. In the framework of Reg NNLS, the spectrum reconstruction problem is formulated as the following constrained optimization:
where $\alpha$ is the regularization parameter for controlling the strength of regularization term $\Vert Lx \Vert _2^2$; $L$ is the regularization matrix that often associates with the prior knowledge of the underlying problem. For instance, the matrix $L$ can take the first-order or second-order derivative form to preserve the smoothness of $x$. Besides, we can also enforce $x\geq 0$ to preserve the non-negative nature of the spectrum, which is important for our application.2.2.2 Sparse recovery based solver
Another popular class of spectrum reconstruction algorithms is the sparse recovery based solver, which has been found effective in improving the reconstruction accuracy and resolution given the same number of filters [14,16,17]. In this line of research, the sparsity assumption of the spectrum is exploited, which refers to that the spectrum is sparse directly in the spectral frequency domain or is sparse on a specific basis, i.e., $x=\Phi c$, where $\Phi$ is a sparsifying basis that could transform the non-direct sparse spectrum $x$ into a sparse signal $c$. The sparse approximation can be formulated as the following $\ell_0$-norm minimization problem:
where $\alpha$ is the regularization parameter balancing between the residual error and the sparsity of solution. We set $c\geq 0$, because $c$ is the non-negative weight of sparsifying basis [16]. However, the $\ell_0$-norm minimization problem is a NP-hard problem [29], and it usually can be approximated by the following $\ell_1$-norm minimization problem:In this framework, identifying the sparsifying basis is crucial. Many different sparsifying basis, such as Gaussian kernels [16,17] and dictionary learning [14], are proposed under different contexts.
3. NN-based spectrum reconstruction method
Thanks to the use of regularizers based on smoothness and sparsity, the reconstruction quality of the conventional reconstruction methods have been significantly improved. Nevertheless, these methods still often result in severe distortions in the reconstructed spectrum due to the ill-posedness of the problems under very noisy environments [13,16]. To overcome these limitations, we introduce novel reconstruction methods based on the NN framework to improve the spectrum reconstruction performance of the CQD spectrometer system. Next, we shall describe two approaches: the plain NN based on raw measurements (served as one of the baseline solutions) and our proposed network based on the spectral data extracted from an existing spectrum reconstruction solver. As we mentioned in the introduction, this idea is largely inspired by the recent success of the NN for inverse problems in imaging [27,30] and spectrum reconstruction [25,26].
3.1 Plain NN architecture
A natural idea to utilize NN to reconstruct the spectrum is to train a multilayer perceptron [31] that takes the pairs of the raw measurements and the true spectrum as its input and output. Unlike the regularized reconstruction methods in which an explicit mathematical formulation of the underlying model is needed, the NN is instead used as a black box to learn a map between the raw measurements and the true spectrum directly. More specifically, we employed a multilayer perceptron with two hidden layers illustrated in Fig. 2. Each hidden layer has $32$ hidden units. The sigmoid function is chosen as the activation function to preserve the non-negative nature of the spectrum. As for the loss function, we use the mean square error (MSE):
where $M$ is the number of spectra in training set, $\hat x_k$ and $x_k$ represent the $k$-th predicted spectrum vector and true spectrum vector, respectively. $S$ is the length of the spectrum vector, which is also the number of neurons in the output layer.3.2 Solver-informed NN
It is worth noting that the plain NN approach is purely data-driven (raw measurements) without respecting the model’s prior knowledge too much. To incorporate/embed more relevant model information of the corresponding spectrometer system, we propose a solver-informed NN to leverage the relevant information of the existing reconstruction solver and improve the reconstruction performance the NN. The idea of the solver-informed NN is inspired by the idea of combining a direct inversion operator with a NN in the previous studies [22,25–27], which is found very useful in CT image reconstruction [27] owing to embedding more model information of the imaging system. Unlike the previous studies using a direct inversion operator to extract the relevant features, we propose a solver-informed NN, whose structure is illustrated in Fig. 3. It consists of two steps: (1) first apply an existing spectrum reconstruction solver, such as the Reg NNLS solver, to the raw measurements to extract relevant features (2) train a NN to map the extracted features to the true spectrum. We shall discuss more details in the rest of this section.
- • Solver-based feature extraction. Compared with the raw measurements, the spectral features extracted from the spectrum reconstruction model could not only filter out noises and irrelevant information to a certain level, but also contain more informative features to facilitate NN to learn the reconstructed spectrum. Besides, the NN does not need to learn the transformation between the raw measurement and the spectral feature, which is necessary for the plain NN. This transformation is provided directly from the existing spectrum reconstruction solver. As a result, this requires less number of hidden layers to achieve reasonably accurate predictive performance.
In this paper, we focus our work on the spectrum reconstruction of the CQD spectrometer. The measurements of the CQD spectrometer contains perturbations from different sources, such as dark noise, round error and inhomogeneity of incident light. In this case, we utilize the regularized solvers, which could handle perturbations partially and extract more useful spectral feature information. In general, the regularized solver can be formulated as the following problem:
where $\alpha$ is a regularization parameter. $G(x)$ denotes a regularization function of spectral signal $x$, such as smoothness, sparsity or other types of prior knowledge. We mainly investigate the following two popular classes of regularized solvers we discussed in Section 2:- - Reg NNLS solver: where $L$ takes the form of first-order regularization:
- - Sparse recovery based solver:$$\begin{aligned}\hat{c} &= \min _c \quad \Vert I-R\Phi c\Vert_2^2 +\alpha \Vert c\Vert_1, \quad s.t. \quad c\geq 0, \\ \hat x &= \Phi \hat{c}, \end{aligned}$$where $\Phi$ is the sparsifying basis that transforms spectra into sparse signals. Herein, we use the Gaussian kernels as the sparsifying basis due to its two advantages [16]: (1) Gaussian kernels can preserve the smoothness of the spectrum. (2) Gaussian kernel can be generated only by two parameters, namely central location and width.
- • The NN framework design. Since we take advantage of the existing reconstruction solver, we expect that a shallow NN would be enough to approximate the true underlying spectra reasonably well. In the following example, we employed a multilayer perceptron with two hidden layers, whose structure is described in Section 3.1. Each hidden layer has $32$ nodes. The sigmoid function is used as the activation function for both hidden layers and the output layer. The MSE function is used as the loss function.
Fig. 2. The architecture of the plain NN: taking pairs of the raw measurements and the true spectrum as input and output
Fig. 3. The flowchart of the solver-informed NN: (1) applying a specific spectrum reconstruction solver to extract spectral features from the raw measurements ($2$) using the spectral features as an input of the NN to learn a map from the features to the true spectrum.
Remark. We remark that the solver to extract spectral features can be flexible based on the availability of the reconstruction solvers for the underlying problems.
4. Experiments
4.1 Experiment setup
To demonstrate the feasibility of our proposed reconstruction method, one CQD spectrometer with $108$ CQD filters has been used in the following example. Before performing the spectral measurement, the sensing matrix of the CQD spectrometer was calibrated using the monochrometer. Considering the resolution of the monochrometer as well as the distribution of feature wavelengths of the current CQD filters, the optimal operational spectral range of our CQD spectrometer is from $400$ nm to $750$ nm with a space of $5$ nm. Fig. 4(a) (is the visualization of the sensing matrix in the CQD spectrometer system and Fig. 4(b) shows some spectral transmission functions in the CQD spectrometer system.
Fig. 4. (a) Visualization of the sensing matrix in the CQD spectrometer system, each row represents a spectral transmission function of one channel. (b) Some spectral transmission functions in the CQD spectrometer system.
4.2 Data preparation and training process
We investigate the performance of the proposed approach on two datasets. One is synthetic by using the glossy Munsell color spectral dataset [32], while the other one comes from the real experiment.
- • The synthetic dataset. The synthetic dataset consists of $1600$ spectra, which are generated by using the glossy Munsell color spectral dataset and the spectrum of a white LED light source. The glossy Munsell color spectral dataset comprises $1600$ reflection coefficients of glossy Munsell color chips. Conventionally, the spectrum measured by one spectrometer is the modulation of the spectrum of the incident light source [14]. Hence, we multiply the Munsell color dataset by the spectrum of a white LED light source and take them as the target spectra needed to be reconstructed. It is worth noting that we preprocess these spectra to make their wavelength range and sampling space the same as the operational band of our CQD spectrometer, i.e., $400$-$750$ nm with a space of $5$ nm. Meanwhile, each spectrum ’s height is normalized between $0$ and $1$.
We corrupt the measurement in (1) with the Gaussian white noise [17,26]. We use the signal-to-noise ratio (SNB$10\log {\frac {\Vert Rx\Vert _2^2}{\Vert I- Rx \Vert _2^2}}$) in dB to represent the noise level. The Matlab function $\tt{awgn}$ [33] is used to add the noise and the noise level of $30$ dB, $35$ dB and $40$ dB are considered in our experiment.
- • The real experimental dataset. The experimental dataset is a real dataset collected by the current CQD spectrometer. It consists of $704$ spectra, which comes from a white LED light source filtered by different color plastics and their combinations. Additionally, to reduce the interference of out-of-band light signals [34], we adopted a visible light pass filter in the process of collecting experimental data. Ground-truth spectra were measured by the commercial spectrometer (ocean optics USB2000+) [35]. Since these spectrometers usually have high resolutions (< $1$ nm), we down-sampled the true spectra to match the operational spectral resolution of our CQD spectrometer.
Figure 5 illustrates some spectra randomly chosen from two datasets. The number of spectra in training, validation and test set in both datasets is shown in Table 1. The validation set is used for determining the structure of NN framework and other hyperparameters. To facilitate the training process of NN, we normalize the raw data of input to the range $[0,1]$.
Fig. 5. Some randomly chosen spectra from the (a) synthetic dataset and (b) real experimental dataset. Different colored lines represent different spectra.
Table 1. The size of the training, validation, and test set for two datasets
Tensorflow 2.0 [36] was used to train the network. We used Adam optimizer [37] with a learning rate of $0.0002$, a batch size of $32$, and the max epoch of $50000$. When the loss value on the validation set does not decrease within $1000$ epochs, we stop training and use the model weights from the epoch with the smallest loss value on validation set. The Glorot uniform (also called Xavier uniform) initializer was used [38]. All computations were conducted on a desktop with intel core $i7-6700$ HQ CPU.
4.3 Setup for conventional reconstruction methods
To evaluate the performance of the proposed method, we compared it with conventional reconstruction methods introduced in Section 2.2. More specifically, for Reg NNLS solver (8), we take the first-order regularization. The regularization parameter $\alpha$ is adaptively selected via L-curve method [39]. The MATLAB function $\tt{lsqnonneg}$ was used to solve this regularized non-negative least squares problem.
Remark 1. To get the input for the Reg NNLS solver-informed NN, we set $\alpha =10, 5, 1$ for the synthetic dataset ($30$ dB, $35$ dB, $40$ dB) and $\alpha =0.5$ for the experimental dataset. These values are estimated based on our empirical experiments. We set the value for $\alpha$ in this way for two reason: (1) selecting the regularization $\alpha$ for each spectrum based on the L-curve method can be time-consuming (2) the current solver-informed NN does not require to feed the best solution of Reg NNLS solver into the NN to get a reasonably accurate prediction. The solutions from the existing solver only serve as the coarse representation of the true spectra to provide the relevant feature for spectrum reconstruction.
For sparse recovery (10), we choose the sparsifying basis $\Phi =(\phi _1,\phi _2,\ldots ,\phi _p)$ composed of Gaussian kernels with different full width at half maximum (FWHM, nm) and center positions. Each $\phi _i(\lambda ),i=1,2,\ldots ,p$, has the center wavelength position $\lambda _i$ and FWHM $w_i$ defined as follows:
Remark 2. Similar with the Reg NNLS solver-informed NN, it is not necessary to feed very accurate solutions from the sparse recovery based reconstruction solver into the NN when performing the sparse recovery based solver-informed NN. In this case, we set $\alpha =1, 0.5, 0.1$ for the synthetic dataset (30 dB, 35 dB, 40 dB) and $\alpha = 5$ for the experimental dataset, respectively.
5. Results and discussion
To qualify the reconstruction quality, we use the relative reconstruction error $\epsilon$ defined as follows:
where $\hat {x}$ and $x$ present the reconstructed spectrum and ground-truth spectrum, respectively.5.1 Synthetic dataset
5.1.1 Reconstruction performance of the plain NN framework
To evaluate the performance of the plain NN framework, we first conduct the spectrum reconstruction on the synthetic test set (with different noise levels) by using Reg NNLS, sparse recovery based solver, and the plain NN framework . As shown in Table 2, for all three noise levels, the plain NN outperforms the Reg NNLS and the sparse recovery based solver in terms of the average reconstruction error and the standard deviation, which shows good reconstruction accuracy and robustness of the plain NN to various spectra.
Table 2. The average reconstruction error and its standard deviation over the synthetic test set (with different noise levels) based on different reconstruction methods
To further confirm the reconstruction performance, Fig. 6 shows the reconstruction results of three spectra in the synthetic test set with the noise level of SNR = 40 dB. We can see that it is challenging for both Reg NNLS and sparse recovery based solver to capture all peaks in the spectra. Meanwhile, for both conventional reconstruction methods, the phase error in reconstructed spectra is more obvious. In contrast, the reconstructed spectra by the plain NN are more consistent with the ground-truth spectra. Particularly, multiple peaks in the spectra can be better captured with fewer phase errors.
Fig. 6. Reconstruction results of three spectra in the synthetic test set with the noise level of SNR = 40 dB based on different reconstruction methods: (a) Reg NNLS, (b) sparse recovery based solver and (c) the plain NN framework.
5.1.2 Reconstruction performance of the solver-informed NN
Although the plain NN outperforms the Reg NNLS and the sparse recovery based solver in terms of reconstruction error, there are still some mismatches in the spectra, shown in Fig. 6(c). Next, we shall investigate if the solver-informed NN can help improve the reconstruction performance. Besides the plain NN, we also consider a more sophisticated neural network – CNN [25] for spectrum reconstruction for comparative evaluation. In this case, we trained a moderate depth (five hidden layers including three convolutional layers and two fully connected layers) convolutional network based on the raw measurement data by the Adam optimizer with a batch size of 32 and the max epoch of 50000. And we used the same early stopping strategy as that of the plain NN and solver-informed NNs used.
Table 3 shows the average reconstruction error and one standard deviation on the synthetic test set (with different noise levels) for the plain NN, the Reg NNLS solver-informed NN, the sparse recovery based solver-informed NN, and the CNN. When small noise level presents (SNR $=40$ dB), both solver-informed NNs outperform the plain NN with roughly $14\%$ smaller in average reconstruction error and $24\%$ smaller in standard deviations. When a larger noise level presents in the data set, both solver-informed NNs still perform better than the plain NN in terms of reconstruction errors and standard deviations. This clearly demonstrates that the combination of the conventional spectrum reconstruction solver and neural work can effectively improve the reconstruction performance of the solver-informed NN. In addition, it can be seen that when the SNR=40dB, the reconstruction performance of CNN is similar to that of the plain NN, but when the SNR decreases, the CNN tends to have better robustness to various spectra than the plain NN. But for all SNRs, the solver-informed NN still shows slightly better performance.
Table 3. The average reconstruction error and its standard deviation over the synthetic test set (with different noise levels) based on different NN frameworks
To further confirm the reconstruction performance, we show reconstruction results of three spectra in the synthetic test set with a noise level of SNR = 40dB for the plain NN, the Reg NNLS solver-informed NN, and the sparse recovery based solver-informed NN in Fig. 7. The solver-informed NN can capture the fine details of the spectra almost perfectly. This again demonstrates that the feature extracted from the existing reconstruction solver are embedded with more prior knowledge of the underlying physical system and can be advantageous in improving the predictive capability and robustness of the NN.
Fig. 7. Reconstruction results of three spectra in the synthetic test set with the noise level of SNR = 40 dB based on different NN frameworks: (a) the plain NN, (b) the Reg NNLS solver-informed NN and (c) the sparse recovery based solver-informed NN.
Remark 1. We remark that as we mentioned in Section 4.3, it is not necessary to feed a very accurate reconstruction result (from the existing solvers) to the pretrained NN. Therefore, we don’t have to go through the time-consuming fine selection process for the regularization parameters for each spectrum during the online stage (reconstruction). In this case, the reconstruction process for the solver-informed NN is much faster than the conventional regularized algorithms as the latter typically requires more time to select regularization parameters for each spectrum.
Remark 2. We acknowledge that compared with the plain NN, one drawback of the solver-informed NN is that an additional computational cost is required to run the existing spectrum reconstruction solver for each spectrum to generate the corresponding input for the following pretrained network during the online stage. Nevertheless, as we mentioned in Section 4.3, it is not necessary to feed a very accurate reconstruction results (from the existing solvers) to the pretrained NN. Therefore, we don’t have to go through the time-consuming selection process for the regularization parameters for each espectrum during the online stage. If this additional cost is affordable during the online stage, the solver-informed NN can help achieve better reconstruction performance.
5.2 Experimental dataset
5.2.1 Reconstruction performance of the plain NN framework
To further investigate the performance of the proposed approach for practical applications, we next examine a real experimental dataset. Table 4 shows the average reconstruction error and one standard deviation over the experimental test set for the Reg NNLS, the sparse recovery based solver, and the plain NN framework. We can see that the plain NN outperforms the Reg NNLS and the sparse recovery based solver with roughly $80\%$ smaller in the average reconstruction error and $50\%$ smaller in the standard deviation. It clearly demonstrates that the plain NN framework can effectively deal with the noise in real measurements and deliver more accurate reconstructed spectra with a more stable performance.
Table 4. The average reconstruction error and its standard deviation over the experimental test set based on different reconstruction methods
To further confirm that, we compare three examples of spectra in the experimental test set reconstructed by different methods shown in Fig. 8. For both conventional reconstruction methods, there is a considerable mismatch between the reconstructed spectra and the ground truth. For example, it tends to over/under-estimate the peaks of the spectra, particularly for spectra with multiple peaks. In contrast, the plain NN approach shows good agreements with ground-truth spectra in general.
Fig. 8. Reconstruction results of three examples of spectra in experimental test set test set based on different reconstruction methods: (a) Reg NNLS, (b) sparse recovery based solver and (c) the plain NN framework.
5.2.2 Reconstruction performance of the solver-informed NN
To demonstrate the performance of the solver-informed NN, we list the average reconstruction error and one standard deviation over the experimental test set for the plain NN, both solver-informed NNs, and the CNN in Table 5. It is clear that when compared with the plain NN the average reconstruction error can be further reduced by leveraging the merits of the existing reconstruction solver and the neural network. Qualitatively, the average reconstruction error of both Reg NNLS and sparse recovery based solver-informed NNs is further reduced by $40\%$, compared with the plain NN. In addition, we can see that the CNN has a better reconstruction performance than the plain NN on the experimental test set. And the reconstruction accuracy of the CNN is comparable to that of both solver-inform NNs. Yet, the solver-informed NNs have slightly better reconstruction robustness to various spectra.
Table 5. The average reconstruction error and its standard deviation over the experimental test set based on different NN frameworks
This can be further demonstrated in Fig. 9, where three examples of spectra reconstructed by different methods are compared. It can be seen that the reconstructed spectrum by both solver-informed NNs shows better agreement with the true spectrum comparing with the plain NN, particularly around the peaks of the spectrum .
Fig. 9. Reconstruction results of three spectra in the experimental test set based on different NN frameworks: (a) the plain NN, (b) the Reg NNLS solver-informed NN and (c) the sparse recovery based solver-informed NN.
6. Conclusion
In this paper, we introduce a novel reconstruction framework (solver-informed NN) for the CQD spectrometer. Unlike most of the existing reconstruction methods, the proposed NN would first perform the feature extraction from an existing spectrum reconstruction solver and then train a NN to map from the extracted features to the underlying spectra. Compared with the conventional regularized spectrum reconstruction solvers and the plain NN approach, we have demonstrated that by utilizing more relevant feature from the existing spectrum reconstruction solver, the solver-informed NN can significantly improve the reconstruction performance to achieve higher reconstruction accuracy and stability on a synthetic dataset and a real dataset collected by the CQD spectrometer. The results demonstrate that the proposed solver-informed NN holds a great promise to deliver accurate and stable reconstruction performance for filter-based miniature spectrometers, including the CQD spectrometer, under noisy measurements.
Funding
Beijing National Research Center For Information Science And Technology (BNR2019ZS01005); Simons Foundation (Grant No. 504054).
Acknowledgments
J. Zhang and J. Bao are supported by Beijing National Research Center for Information Science and Technology (BNR2019ZS01005), and Beijing Innovation Center for Future Chips, Tsinghua University. X. Zhu is supported by Simons Foundation (Grant No. 504054).
Disclosures
The authors declare no conflicts of interest.
References
1. R. F. Wolffenbuttel, “State-of-the-art in integrated optical microspectrometers,” IEEE Trans. Instrum. Meas. 53(1), 197–202 (2004). [CrossRef]
2. S.-W. Wang, C. Xia, X. Chen, W. Lu, M. Li, H. Wang, W. Zheng, and T. Zhang, “Concept of a high-resolution miniature spectrometer using an integrated filter array,” Opt. Lett. 32(6), 632–634 (2007). [CrossRef]
3. C. P. Bacon, Y. Mattley, and R. DeFrece, “Miniature spectroscopic instrumentation: Applications to biology and chemistry,” Rev. Sci. Instrum. 75(1), 1–16 (2004). [CrossRef]
4. S. Kim, D. Cho, J. Kim, M. Kim, S. Youn, J. E. Jang, M. Je, D. H. Lee, B. Lee, D. L. Farkas, and J. Y. Hwang, “Smartphone-based multispectral imaging: system development and potential for mobile skin diagnosis,” Biomed. Opt. Express 7(12), 5294–5307 (2016). [CrossRef]
5. C. Kim, W.-B. Lee, S. K. Lee, Y. T. Lee, and H.-N. Lee, “Fabrication of 2D thin-film filter-array for compressive sensing spectroscopy,” Opt. Laser Eng. 115, 53–58 (2019). [CrossRef]
6. J. Oliver, W.-B. Lee, and H.-N. Lee, “Filters with random transmittance for improving resolution in filter-array-based spectrometers,” Opt. Express 21(4), 3969–3989 (2013). [CrossRef]
7. E. Huang, Q. Ma, and Z. Liu, “Etalon array reconstructive spectrometry,” Sci. Rep. 7(1), 40693 (2017). [CrossRef]
8. A. Emadi, H. Wu, G. de Graaf, and R. Wolffenbuttel, “Design and implementation of a sub-nm resolution microspectrometer based on a linear-variable optical filter,” Opt. Express 20(1), 489–507 (2012). [CrossRef]
9. J. Bao and M. G. Bawendi, “A colloidal quantum dot spectrometer,” Nature 523(7558), 67–70 (2015). [CrossRef]
10. U. Resch-Genger, M. Grabolle, S. Cavaliere-Jaricot, R. Nitschke, and T. Nann, “Quantum dots versus organic dyes as fluorescent labels,” Nat. Methods 5(9), 763–775 (2008). [CrossRef]
11. F. P. G. de Arquer, A. Armin, P. Meredith, and E. H. Sargent, “Solution-processed semiconductors for next-generation photodetectors,” Nat. Rev. Mater. 2(3), 16100 (2017). [CrossRef]
12. T.-H. Kim, K.-S. Cho, E. K. Lee, S. J. Lee, J. Chae, J. W. Kim, D. H. Kim, J.-Y. Kwon, G. Amaratunga, S. Y. Lee, B. L. Choi, Y. Kuk, J. M. Kim, and K. Kim, “Full-colour quantum dot displays fabricated by transfer printing,” Nat. Photonics 5(3), 176–182 (2011). [CrossRef]
13. U. Kurokawa, B. I. Choi, and C. Chang, “Filter-based miniature spectrometers: Spectrum reconstruction using adaptive regularization,” IEEE Sens. J. 11(7), 1556–1563 (2011). [CrossRef]
14. S. Zhang, Y. Dong, H. Fu, S.-L. Huang, and L. Zhang, “A spectral reconstruction algorithm of miniature spectrometer based on sparse optimization and dictionary learning,” Sensors 18(2), 644 (2018). [CrossRef]
15. C.-C. Chang and H.-N. Lee, “On the estimation of target spectrum for filter-array based spectrometers,” Opt. Express 16(2), 1056–1061 (2008). [CrossRef]
16. C.-C. Chang, N.-T. Lin, U. Kurokawa, and B. I. Choi, “Spectrum reconstruction for filter-array spectrum sensor from sparse template selection,” Opt. Eng. 50(11), 114402 (2011). [CrossRef]
17. J. Oliver, W. Lee, S. Park, and H.-N. Lee, “Improving resolution of miniature spectrometers by exploiting sparse nature of signals,” Opt. Express 20(3), 2613–2625 (2012). [CrossRef]
18. Y. Lecun, Y. Bengio, and G. E. Hinton, “Deep learning,” Nature 521(7553), 436–444 (2015). [CrossRef]
19. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, eds. (Curran Associates, Inc., 2012), pp. 1097–1105.
20. S. Li, X. Zhu, Y. Zhang, Y. Zheng, and J. Bao, “Multi-stage spatial feature integration for multispectral image classification,” in 2019 IEEE 13th International Conference on Anti-counterfeiting, Security, and Identification (ASID), (IEEE, 2019), pp. 131–135.
21. L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2018). [CrossRef]
22. D. Gedalin, Y. Oiknine, and A. Stern, “Deepcubenet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks,” Opt. Express 27(24), 35811–35822 (2019). [CrossRef]
23. S. Li, X. Zhu, Y. Liu, and J. Bao, “Adaptive spatial-spectral feature learning for hyperspectral image classification,” IEEE Access 7, 61534–61547 (2019). [CrossRef]
24. S. Li, X. Zhu, and J. Bao, “Hierarchical multi-scale convolutional neural networks for hyperspectral image classification,” Sensors 19(7), 1714 (2019). [CrossRef]
25. C. Kim, D. Park, and H.-N. Lee, “Convolutional neural networks for the reconstruction of spectra in compressive sensing spectrometers,” in Optical Data Science II, vol. 10937 (International Society for Optics and Photonics, 2019), p. 109370L.
26. C. Kim, D. Park, and H.-N. Lee, “Compressive sensing spectroscopy using a residual convolutional neural network,” Sensors 20(3), 594 (2020). [CrossRef]
27. K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, “Deep convolutional neural network for inverse problems in imaging,” IEEE Trans. on Image Process. 26(9), 4509–4522 (2017). [CrossRef]
28. C. Chang and H. Lin, “Spectrum reconstruction for on-chip spectrum sensor array using a novel blind nonuniformity correction method,” IEEE Sens. J. 12(8), 2586–2592 (2012). [CrossRef]
29. R. G. Baraniuk, “Compressive sensing [lecture notes],” IEEE Signal Process. Mag. 24(4), 118–121 (2007). [CrossRef]
30. D. Lee, J. Yoo, and J. C. Ye, “Deep residual learning for compressed sensing MRI,” in 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), (2017), pp. 15–18.
31. M. Gardner and S. Dorling, “Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences,” Atmos. Environ. 32(14-15), 2627–2636 (1998). [CrossRef]
32. University of Eastern Finlana. Spectral Color Research Group, https://www3.uef.fi/fi/web/spectral/-spectral-database.
33. AWGN, https://ww2.mathworks.cn/help/comm/ref/awgn.html?searchHighlight=awgn&s_tid=doc_srchtitle.
34. T. Yang, X. L. Huang, H. P. Ho, C. Xu, Y. Y. Zhu, M. D. Yi, X. H. Zhou, X. A. Li, and W. Huang, “Compact spectrometer based on a frosted glass,” IEEE Photonics Technol. Lett. 29(2), 217–220 (2017). [CrossRef]
35. Ocean Insight.USB2000+, http://www.oceanoptics.cn/product/1311.
36. TensorFlow Core v2.2.0, https://tensorflow.google.cn/api_docs/python/tf/keras/layers/Dense?hl=zh-CN.
37. D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” International Conference on Learning Representations (2014).
38. X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, vol. 9 of Proceedings of Machine Learning ResearchY. W. Teh and M. Titterington, eds. (PMLR, Chia Laguna Resort, Sardinia, Italy, 2010), pp. 249–256.
39. P. C. Hansen, “Analysis of discrete ill-posed problems by means of the L-curve,” SIAM Rev. 34(4), 561–580 (1992). [CrossRef]
40. S. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, “An interior-point method for large-scale ℓ1-regularized least squares,” IEEE J. Sel. Top. Signal Process. 1(4), 606–617 (2007). [CrossRef]
