1. Introduction

Miniature spectrometers are key instruments that are required in various academic and industrial applications such as bio-medical, chemical, and environmental engineering [1]. These spectrometers, built with integrated-filter arrays, provide superior portability, flexibility, and cost-effectiveness [2]. The spectrometers also provide fine details about the various spectral components of the incident light. These spectral components reveal a wealth of information concerning the composition and the structure of the various objects being observed [3]. The ability of a spectrometer to reveal fine information is usually attributed to its resolution. Thus, in recent years, researchers have focused on improving the resolution of spectrometers.

State-of-the-art filter-array-based spectrometers rely on digital signal processing (DSP) algorithms. These algorithms may run on a general-purpose computing platform of cellular phones and laptop computers, as shown in Fig. 1 , before the spectrum estimate is displayed on the screen. They can also run on a specialized DSP chip integrated into the spectrometer unit in the case of stand-alone spectrometers. These algorithms process the raw spectrum acquired by the spectrometer to reduce noise and distortions and thereby aid in improving the resolution.

 

Fig. 1 Schematic of the proposed filter-array-based spectrometer.

Download Full Size | PDF

In the filter-array-based spectrometers, the factors that limit the resolution are the number of filters in the filter array and the shapes of the transmittance functions of these filters [4]. In practical miniature spectrometers, the number of filters is fixed. Thus, in order to improve the resolution, it is often not a realistic option to increase the number of filters. The second factor that affects the resolution is the transmittance functions of the filters. While the transmittance function (TF) is usually designed to meet certain criteria, in practice, owing to low-cost integrated-array fabrication, the TF of each of the filters is non-ideal. These non-ideal filters highly distort the raw spectrum and thereby necessitate the digital processing of the raw spectrum to determine information concerning the spectral components of the input signal spectrum. DSP algorithms or estimators have been reported in the literature [5–8]. The algorithms in [5, 6] are centered on L₁-norm minimization, which is a modern optimization tool in DSP for solving underdetermined system of linear equations, whereas the algorithms in [7, 8] are based on conventional least squares and are designed with an aim to reconstruct the signal spectrum but not to improve resolution.

In this paper, we assume the use of the L₁-norm-minimization-based DSP algorithm in [5] as the spectral estimator and aim to address the following relevant questions: Is it possible to enhance the resolution of the spectrometers by shaping the TFs in a particular way? Namely, is there a certain shape of the TF that is suitable for use with the L₁-norm-minimization algorithm at the back end? If yes, what is that shape, and how can such a TF be designed? Is there a practical way to implement such a TF? Also of great importance is the discussion of suitable performance metrics to compare different TFs. This is necessary to reflect the use of the L₁-norm-based spectral estimator we aim to use. With the new metric, one should be able to investigate the maximum possible resolution achieved by employing different TFs.

As per the discussion of the questions above, the role of TFs in improving the resolution of the spectrometers under the DSP-based spectral estimator needs to be carefully examined. We observe that the TFs whose auto-covariance functions resemble that of a Dirac delta function acquire independent information about the closely spaced spectral components of the input signal spectrum under the DSP-based spectral estimator. Filters with random-TFs possess such auto-covariance. Based on our study, we show that it is possible to design filters with random-TFs using thin-film technology and improve the resolution up to 7-fold compared to the filters with non-ideal transmittances in [5].

The concept of random-TFs goes back to the 1970s. Different aspects of the random-TFs have been studied in the past [9–11]. In [9], the author has studied the maximum information transmission rate of an optical filter with random transmittance. A laser-based velocimeter is designed in [10] based on random transmittance patterns. In [11], the authors have designed a holographic-based sensor using random transmittances that are generated by using random gratings. As per our knowledge, however, no one has designed or reported filters with random transmittance for improving the resolution of a spectrometer.

This paper is organized as follows: Section 2 presents the system model, and Section 3 discusses in detail the need for designing filters with random transmittances for improving resolution. Section 4 discusses the design of random-transmittance filters using thin-film technology, and Section 5 presents the concept of the resolution of a spectrometer. Experimental results are discussed in Section 6, and Section 7 concludes the paper.

2. System description

We consider a spectrometer that consists of a planar filter array with M filters arranged in a 2D fashion as shown in Fig. 1. Each of these filters acts as a wavelength-selective device. Each filter is specified in terms of a transmittance function (TF), $T_i\left(\lambda \right),i=1,2,\cdots ,M,$ which indicates the fraction of input light that the filter allows at each unit of wavelength. We note that $T_i\left(\lambda \right)$ is a continuous (analog) function of wavelength. Each filter is attached to a CCD element that converts light into electrical energy. The output of each of the CCD elements is sampled to form an $M\times 1$ vector $y$, which is called a raw spectrum.

Each sample of the raw spectrum denotes the projection (inner product) of the input signal spectrum onto a TF. That is, the ith sample of the raw spectrum is obtained as

Modern filter-array-based spectrometers [4–8] process the raw spectrum using a DSP algorithm or an estimator to recover the input signal spectrum. The data model for the raw spectrum, $y$, which is an input to the DSP algorithm can be represented [4] as a system of linear equations:

In this paper, since we are dealing with improving resolution, the value of N is set to be greater than M, so N > M. That is, the number of samples of x is greater than the number of filters. The reason for setting N > M is that resolution in filter-array-based spectrometers is limited by the number of filters M (as will be discussed in detail in Section 3.1). Thus, in order to increase resolution beyond this limit set by the fixed number of filters M, the value of N should be greater than M. By setting N > M, we note that the noise-free system of equations in Eq. (3) is underdetermined. We also note then that in Eq. (3), the spacing between the samples of $x$ is $\Delta {\lambda }_N=\frac{W_{\lambda }}{N}$, while the spacing between the samples of the raw spectrum $y$ is $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$.

Recently, sparse representation of the input signal spectrum,$x$, has drawn considerable attention [4–6]. The aim of sparse representation is to decompose the signal spectrum into its constituent spectral (wavelength) components. For example, the signal spectrum of a low-power mercury lamp can be decomposed into its seven wavelength components, whereas the rest of the wavelength components are zero-valued. In general, any signal spectrum vector $x$ in Eq. (3) can be equivalently represented as a sparse vector $s$ using a linear transform matrix G, i.e., $x=Gs$. The vector $s$ is called a K-sparse vector. In this paper, we refer to s as a sparse spectrum. It contains K non-zero values and $N-K$ zero-valued components, $K\ll N$. The K non-zero values of $s$ indicate the intensities of the spectral (wavelength) components that constitute x. The set of K wavelengths corresponding to those intensities is called the support set. Thus, each element of the support set indicates the wavelengths (in nm) that are present in the signal spectrum. The intensities of the remaining N - K wavelength components of s are zero.

The matrix G is called the kernel matrix, and it is of size $N\times N$. The columns of $G$ are chosen as functions that preserve the natural shape of the signal spectrum. Gaussian kernels are widely used for signal spectrum modeling [8]. The other possible non-Gaussian kernels are secant hyperbolic and Lorentzian [1]. To construct G, we need to know only a single kernel function, the width of which is controlled by a parameter called full-width at half-maximum (FWHM). When the FWHM is small, the width of the kernel function is also small, and vice versa. Once the kernel function is chosen, it is sampled, and the sampled version of the kernel function is considered as the first column of G. The remaining $N-1$ columns of G are just shifted versions of the first column [6]. We observe that the Gaussian kernel has a uniform width that models narrowband light sources. For modeling broadband light sources, one may require kernels with varying widths, as discussed in [6]. In this paper, we consider only narrowband sources. With the sparse modeling of the input signal spectrum, Eq. (3) can be written as

3. Proposed transmittance functions

3.1. Introduction: Transmittance functions

The transmittance function (TF) of an optical filter is a waveform that has a shape that dictates the fraction of input light that the filter transmits at various wavelengths. The conventional approach is to attempt to design TFs with an ideal shape, i.e., a sharp brick wall as shown in Fig. 2(a) . In Fig. 2(a), we show only three filters out of 40 in a filter array. Ideal filters allow light only for desired wavelengths, called the passband, and completely stop the remaining wavelengths, called the stopband. In addition, from Fig. 2(a), we note that the passbands of the ideal TFs do not overlap with each other.

 

Fig. 2 Transmittances of (a) ideal filters (b) non-ideal filters.

Download Full Size | PDF

Ideal filters are desired for two reasons. First, the raw spectrum (Eq. (3)) obtained by employing the ideal TFs is a direct estimate of the signal spectrum, and hence, there is no need for a DSP algorithm. This is because each sample of the raw spectrum is a projection (Sec. 2) of the input signal spectrum with an ideal TF. Since ideal TFs are non-overlapping by design, each sample of the raw spectrum only contains information about the spectral (wavelength) components that correspond to a single passband. Second, any two spectral components of the input signal spectrum that are one passband spacing apart are distinguishable (resolvable) in the raw spectrum. If there are any two spectral components separated by less than a passband width, they cannot be resolved. In order to resolve such spectral components, the passband width of each filter should be decreased. Consequently, this increases the number of TFs (number of filters) in order to cover the entire operating bandwidth. Thus, the resolution of the filter-array-based spectrometers with ideal TFs is limited by the number of filters, M. Since the spacing between the samples of the raw spectrum is $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$, it was believed to be the limit on the resolution of filter-array-based spectrometers. This limit is the same for the more conventional grating-based spectrometer with a CCD array consisting of a total of M pixels. That is, by using the ideal system impulse response ($h\left(\lambda \right)=\delta \left(\lambda \right)$, the Dirac delta), the spacing between the samples of the raw spectrum of a grating-based spectrometer is also $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$.

However, filters with ideal TFs are not easy to realize in practice [2]. Therefore, non-ideal TFs are inevitable in spectrometers. The shapes of the practical non-ideal TFs employed in a typical filter-array spectrometer [7] are shown in Fig. 2(b). These TFs are not purposely designed in such shapes but are instead accidentally obtained in the micro-level processing of implementing the ideal filters. Unlike the ideal TFs, the waveforms of non-ideal TFs are smooth functions of wavelength. In addition, as seen from Fig. 2(b), the passband of each filter’s TF leaks into the passbands of neighboring filters. Hence, non-ideal TFs have considerable overlap with each other. Non-ideal TFs are considered worse for the signal spectrum estimation process in conventional wisdom. The reason is that each sample of the raw spectrum now contains information about the spectral components not only from its own band but also from the neighboring filters’ bands, viz. interferences. The interference from the use of non-ideal TFs causes severe distortion in the raw spectrum. Therefore, for the spectrometers with non-ideal TFs, post-processing of the raw spectrum using DSP algorithms is inevitable. These algorithms are designed with the aim of extracting the constituent spectral components (sparse spectrum) of the input signal spectrum from the raw spectrum y, via Eq. (4).

A recent recovery algorithm, reported in [5], is tailor-made for spectrometers. This algorithm is based on a DSP optimization tool called $L_1$-norm minimization. $L_1$-norm minimization is a recent approach more suitable for solving a noise-corrupted underdetermined system of linear equations such as Eq. (4) than a classical least-squares approach [12–14]. The minimization problem for the recovery of the sparse signal spectrum $s$ in Eq. (4) can be expressed as:

Note that the problem in Eq. (5) is a bit different from the standard L₁-norm minimization approach with the inclusion of the additional non-negative constraint, $s\ge 0$. In order to find the non-negative spectral estimate $\widehat{s}$, the authors in [5] have used a DSP algorithm that was reported in [15]. In [15], the authors have derived an algorithm using the modern interior point method. They have shown that the performance of their algorithm is best when compared to the standard L₁-norm minimization algorithms such as L₁ magic [14] and LASSO [16], which do not use the additional information. In addition, they have shown that their algorithm is robust against the observation noise, stable, and competitive with the existing algorithms. Using the algorithm in [15] for spectrometers, the authors in [5] have shown an improvement in resolution to about a factor of six below the resolution limit. In addition, the authors in [5] have provided a hint that the recovery performance (and hence the resolution) of the algorithm depends on the design of the TF matrix D.

Recently, fast algorithms for L₁-norm minimization such as FISTA [17], FOM [18], and ADMM-based algorithms [19] have gained considerable interest among the imaging community. In imaging systems [20–23], the number of measurements (typically more than a few million) is much more than the number of measurements used in the miniaturized spectrometers. Consequently, these imaging systems should be capable of handling large amounts of data efficiently; for spectrometers, however, the accuracy is more important than the efficiency. It is well known that the interior-point-method-based algorithms provide good accuracy. Thus, in this paper, we use the same algorithm that was used in [5]. Readers who may be interested in implementing fast algorithms for spectrometers can refer to the approaches in [17–19].

In this paper, our central goal is to design the TF matrix D in order to enhance the resolution. That is, we use the spectral estimation algorithm in [5] and show how to design the TF matrix D (which in turn determines the filter transmittances) that helps the algorithm to improve resolution. We show that by using our proposed TF matrix, it is possible to enhance resolution 7-fold compared to what was achieved in [5].

3.2. Motivations for new TF design

In this paper, we aim to present a new approach to further enhance (as compared to what was achieved in [5]) the resolution of the spectrometers by designing TFs. The motivations behind designing new TFs are based on the following observations:

Thus, it is crucial to design TFs that 1) capture holistic and independent information about the constituent spectral components and 2) aid in recovering those spectral components.

3.3. Design of new TFs

We observe from the previous section that the amount of additional information acquired in each sample of the raw spectrum is solely proportional to the shape of the TFs. By shaping the TFs through a proper design procedure, it is possible to extract much information about the input signal spectrum. Therefore, the TF matrix D that senses the maximum amount of information about the signal spectrum appears to be the best choice for the spectrometer case, and hence, it is our design goal. We note here that a similar design notion arises in the emerging signal processing research area called compressive sensing (CS) where a sensing matrix [12, 13] takes the role of the TF matrix in acquiring a signal.

In CS, the raw signal samples acquired using the matrix D is modeled as $y=DGs+w=As+w$, which is the same as the raw spectrum model in Eq. (4). In CS, the matrix D is referred to as the sensing matrix; the matrix G, the sparsifying basis; the vector s, a K-sparse signal; and As is called the compressed signal. In CS, the sensing matrix D is chosen such that it captures enough information for the unique identification of the signal s with as few samples of As (the compressed signal) as possible. In order to quantify the ability of the sensing matrix to acquire enough information about the signal in a minimum number of raw samples, the mutual coherence metric, $\mu$, is used [12, 14]. The mutual coherence of the matrix A = DG is defined as follows:

It is well known in CS that sensing matrices D, the entries of which are drawn from i.i.d. samples of a random variable, exhibit low coherence. Such matrices are called random sensing matrices. These matrices are capable of capturing enough information about the signal s to perform reconstruction from a small number of samples of y. Therefore, random sensing matrices are widely employed in CS-based applications.

It appears that the goal of designing a sensing matrix in CS resembles that of TF matrix design. That is, both the TF and the sensing matrix should be designed to capture sufficient information to permit the faithful recovery of the signal. Since both of these goals are met by random matrices [12], in this paper, we consider a random matrix as the TF matrix rather than using the non-ideal TF matrix as in [5–8]. Since each row of the TF matrix denotes a transmittance function of a filter, a row of the random TF matrix is termed a random transmittance function. That is, a filter with a transmittance that exhibits random fluctuations as opposed to possessing a pre-designed transmittance shape is termed a random transmittance function. Thus, our goal of improving resolution reduces to the design of a set of M filters with random TFs. A set of TFs $\left\{T_i\left(\lambda \right)\right\}$ is referred to as random when

1) The auto-covariance function (ACF) of each TF in the set is close to the Dirac-delta function. That is,

2) when the cross-covariance function (CCF) between each pair of TFs in the set is very small. That is, ${\epsilon }_{i,j}={\displaystyle \int \left[T_i\left(\lambda \right)-m_i\right]\left[T_j\left(\lambda \right)-m_j\right]}d\lambda ,i\ne j$, where ${\epsilon }_{i,j}$ is a small number close to zero. We discuss in detail the design of filters with such random transmittance in Section 4 using thin-film technology.

3.4. Significance of the proposed approach

The ideas in this paper mirror some of the prior work in CS with regard to the use of random matrices for signal acquisition and recovery. We briefly compare our approach to other well-known CS imaging approaches such as the single-pixel camera (SPC) [20], the P2C2 video system [21], the hyperspectral design (HSD) [22], and the CASSI system [23].

First, we note that the SPC is a well-known CS-based imaging system [20]. In SPC, in order to compressively sample a scene by optical random linear projection, a digital micromirror device (DMD) was used together with a biconvex lens. The DMD is programmed with random 0/1 Walsh patterns [20]. With the availability of such optical devices, random projections of a scene were made almost effortlessly without going through a pixel-by-pixel scanning, and as a result, it was possible to realize the concept of SPC in practice. The P2C2 video system [21] extends the SPC architecture to video acquisition, and hence it also employs the random 0/1 Walsh patterns. We observe that in both the systems the measurement matrices are 0/1 Walsh matrices.

HSD [22] and CASSI [23] are spectral imaging systems. These systems collect a scene as a set of spatial images. Each image represents a range of an electromagnetic spectrum also known as a spectral band. These spatial images are arranged along various spectral bands to form a three-dimensional structure called a hyperspectral data cube. In HSD, the data cube is modeled as a linear combination of the spectral signatures of endmembers (that is, objects in a scene such as a road, grass, trees, or a roof), where each spectral signature can be obtained from a publicly accessible database such as HYDICE Urban hyperspectral data as used in [22]. Then, the HSD exploits the spatial and spectral redundancies of the data cube by using CS. In particular, HSD adopts the concept of SPC for acquisition of the data cube, and hence they use the DMD with 0/1 Walsh patterns. In contrast to HSD, the CASSI system models the data cube as a linear combination of two-dimensional wavelets and then acquires the data cube using the coded aperture masks. Thus, the measurement matrices in the CASSI system are also random binary 0/1 code matrices. We observe that both the HSD and CASSI systems compressively acquire the spectral data by means of spatial domain random measurement matrices, each element of which is either 0 or 1.

In contrast to the above imaging systems, we note that in our proposed spectrometer, we acquire the spectral data or signal spectrum using random spectral filters (not spatial). That is, as discussed in Section 2, the sensing of the signal spectrum is performed using random analog optical TFs, which are non-negative continuous functions of wavelength. We observe that the set of optical TFs play a sensing role as significant as that of the DMD in single-pixel cameras or the coded aperture masks. In addition, the TFs have values that can be any real number between 0 and 1, in contrast to the 0/1 patterns in the imaging applications. Table 1 summarizes the measurement matrices and their implementation in various imaging applications.

Table 1. Measurement Matrices and Their Implementation in Various Imaging Applications

View Table | View all tables in this article

Second, in optical applications the sensing of signals and their subsequent recovery is accomplished with a sensing matrix stored beforehand. For example, in [20, p. 85], the DMD array stores Walsh, Hadamard, or noiselet patterns. The best matrices or the matrices of interest were designed in the digital domain. They were then used to dither the micromirrors back and forth in the DMD. We call this the digital design first approach here, as depicted in Fig. 3 . In spectrometers, the actual mechanism of sensing is done using the analog TFs, as per Eq. (1), and then the signal spectrum recovery is done using its digital model. We note here that the digital TFs can be losslessly obtained from the TFs of analog thin-film filters (discussed in the next section). Thus, once the analog TFs are designed, they can be digitally represented in the TF matrix in Eq. (4) for spectrum recovery. We call this the analog design first approach in this paper (see Fig. 3), in contrast to the digital design first approach. The comparison of the analog design first approach with the digital design first approach may unearth a noteworthy characteristic of the spectrometer designs. Let us elaborate a little further.

 

Fig. 3 Summary of analog and digital design first approaches.

Download Full Size | PDF

For spectrometers, it is desirable to design a set of random analog TFs with the ACF of each TF shaped as close as possible to the Dirac delta function. The narrower the width of the ACF is, the better the set is at resolving the fine details of the input spectrum. Thus, the resolution of a spectrometer is closely related to the width of the ACF of the TF. In the analog design first approach, due to the optical nature of the thin-film filters, there is an inevitable practical limit on how narrow the width of an ACF can be made.

We note that there is no such limit for the digital design. That is, in the digital case, the width of the ACF can be made as arbitrarily narrow as one wishes. For example, this can be achieved by dividing the operating wavelength range by the number of samples, say N, and hence obtaining any resolution we wish to have, and then by digitally generating a TF of length N by drawing independent samples from a probability distribution such as the 0/1 Walsh matrix with N = 65,536 as in [20] or as in P2C2 video systems [21]. Since the samples of the TF are independently and identically generated from the given distribution, the ACF of the TF is always as close as possible to the Dirac delta function. This means that one can increase the resolution just by increasing N. This may work for image recovery applications, subject to the creation of a DMD array with so many more elements, but not for spectrometers. For spectrometers, the inherent width of the ACF of the analog TFs must be preserved in its digital TFs. If we first design a random (white) digital TF with an ACF very close to the Dirac delta, the digital TF may not be realizable as the TF of an analog filter. Thus, the resolution in the digital domain is a useless concept unless a way of constructing the practical analog counterpart of the digital TF is given.

Our main contribution is twofold. The first part shows a specific way to design a set of optical analog filters with TFs that are white and uncorrelated with each other. The second part illustrates that significant improvement in resolution can be achieved with such a filter design.

3.5. Characterizing random TFs

Usually, the degree of correlation of a function over an interval is characterized in terms of the auto-covariance function (ACF). A function can be correlated with a time lagged (or advanced) version of itself. Thus, the ACF is a function of the time difference (time lag). The ACF reveals the correlation among the samples of the function. Similarly, a random TF can be characterized in terms of their ACF, which is now a function of wavelength difference. The shape of the ACF reveals the degree of similarity (correlation) between the light intensities at two different wavelengths of the random TF. If the shape of the ACF is wide with slowly decaying tails, then the TF is said to be highly correlated. On the other hand, if the TF has low correlation at different wavelengths, then the shape of the ACF is narrow with rapidly decaying tails, resembling a Dirac delta function. The shape of the Dirac delta function is ideal for an ACF. The width of the ACF measures the degree of correlation. That is, an ACF with narrow width exhibits low correlation, and vice versa. A Dirac-delta-like ACF is required in applications such as determining the modulation transfer function of an optical system [24] or detecting displacements or phase gradients in holography [11]. This evidence shows that filters with a Dirac-delta-like ACF are crucial in optical systems and can be designed in practice.

Filters having ACFs that resemble that of a Dirac delta are preferred for L₁-norm-minimization-based spectrometers. The reason is that a Dirac-delta-like ACF has narrow width, and hence two spectral components that are separated by more than this width are sensed differently by the TF. This independent information about the spectral components aids the L₁-norm-based algorithms to resolve closely spaced spectral components. Thus, the resolution of a spectrometer has a close connection to the width of the ACF. By using random TFs, we can reduce the width of the ACF and hence the spacing between two spectral components that are independently sensed. On the other hand, non-ideal TFs exhibit high correlation, and hence the widths of the ACFs are wider than those of the random TFs. Hence, information gathered about the two nearby spectral components is not too independent. This reduces the ability of the spectrometer to distinguish the two spectral components. Thus, our goal in this paper is to design the transmittances of filters with Dirac-delta-like ACFs. We call them random TFs, while the TFs in [5], correlated TFs, in the subsequent discussions.

4. Design of the proposed random transmittance

In the previous section, we introduced random TFs with the goal of improving the resolution of a spectrometer system. Towards this end, we ask ourselves the following question: is it possible, in practice, to design and implement random TFs in the analog domain? If yes, how should one be designed? We have found that filters with random transmittance can be designed and implemented by using thin-film optical filters [25, 26] in the way explained in this section.

A thin-film optical filter consists of multiple layers of high- and low-refractive index materials (dielectrics) deposited on a substrate [26]. Each layer has a thickness usually of one quarter-wavelength. Thin-film filters work based on the interference of light transmitted or reflected at the boundaries between the layers. This interference is wavelength-dependent; that is, depending on the number of layers, the index of refraction, and the thicknesses of the layers, the transmission (or reflection) of light through the filter changes with wavelength, and hence the overall transmittance changes. By controlling the thickness of each layer and the number of layers, optical engineers design various filter transmittances such as band-pass filters. In this section, we show how varying the thickness of each layer generates random transmittances using the design methodology in [25].

4.1 Thin-film-based random transmittance filter design

In this section, we first briefly discuss the generic design of thin-film filters to generate band-pass-like transmittances. We then motivate and show how to design random transmittances using the same thin-film structure by varying only the thicknesses of the layers. Let us consider a thin-film filter with m dielectric layers between the input medium (air) and the output medium (substrate). Let $n_a$and $n_s$denote the refractive indices of the input and output medium, while $n_H$ and $n_L$ refer to the refractive indices of the high-refractive-index and low-refractive-index intermediate dielectric layers, respectively, and $d_k$ is the thickness of layer k. Assuming no dielectric losses, the transmittance of the thin-film filter as a function of wavelength $(\lambda )$ and the angle of the incident light $(\theta )$ is given [25] by

The power parameters in Eq. (8) are calculated using a recursive set of equations shown in Table 2 . We note from Table 2 that the power parameters (and hence the transmittance) depend on the thickness and the refractive index of the different media, as well as on the wavelength and angle of the incident light. Optical engineers use these recursive equations to design the transmittance shapes, such as the band-pass profile, that are required in practice. For example, the typical band-pass transmittances designed using Eq. (8) for various angles of incidence are provided in [25, p. 3765, Fig. (1)]. Also, it is shown in [25, p. 3767, Fig. (3)] that the band-pass transmittances designed using the equations in Table 2 are accurate and are close to the actual transmittances obtained in practice. In Table 2, ${\theta }_k$ denotes the angle that the incident light makes with the normal as it travels from layer k to k + 1, ${\eta }_k$ is the effective complex-valued refractive index seen by the light as it enters layer k. In order to calculate ${\rho }_{TE}$ and ${\rho }_{TM}$, we first start with ${\eta }_m=N_m$, and we apply the equation for ${\eta }_k$ recursively until we arrive at ${\eta }_2$, which when substituted into the equation for $\rho$ in Table 2 yields either ${\rho }_{TE}$ or ${\rho }_{TM}$.

Table 2. Recursive Equations for Calculating Power Parameters

View Table | View all tables in this article

In this paper, we adopt the design equations in Table 2 to generate the random transmittances. We are able to generate the random transmittances based on the following observation: We first observed that thin-film filters can provide band-pass-like transmittance because of the principle of interference of light beams. That is, the incident light, when traveling through the thin-film filter, suffers different phase shifts as a result of propagating through different layers. The phase shifts depend on the refractive indices, the thicknesses of the layers, and the wavelength of the incident light. After transmission through the layers, the different beams that reappear at the filter output combine (add) either constructively or destructively depending on their path lengths. The constructive addition leads to increased transmission, and destructive addition leads to decreased transmission of the incident light. Since these constructive and destructive additions are wavelength-dependent, optical engineers usually design the thicknesses of the layers such that constructive interference occurs only for certain wavelengths and thereby achieve band-pass-like transmittances.

To avoid boundary reflections (and hence to obtain efficient transmission), it is preferred to have a 180° phase shift between the reflected beams at the boundary between the layers. This phase difference directly corresponds to a $\lambda /2$ phase shift of a sinusoidal wave, which can best be accomplished by setting the optical thickness of the layers to $\lambda /4$. Hence, in practice, stacks of quarter-wavelength layers are used as a basic building block [26] for many types of thin-film filters such as band-pass, long-wave-pass. For example, thin-film layers with thickness ${\lambda }_{normal}/4n$ are very common in practice, where ${\lambda }_{normal}$ is the center wavelength of the band-pass filter with normal incidence and n is the refractive index of the layer. Thus, we note that the filter transmittances have a direct relationship with the thicknesses of the layers. Based on these observations, we inferred that it is possible to create variations in the filter transmittances by just changing the thicknesses of the layers.

In this paper, in order to generate random transmittances, we propose to vary the thicknesses of the filters, in contrast with the conventional fixed quarter-wavelength thickness. Specifically, we vary the thicknesses of the layers as ${\lambda }_{normal}/U$, where U is a uniform random variable that has a mean and variance that are chosen such that the average thickness of a layer is on the order of $\mu \text{m}$, similar to the thickness of the quarter-wavelength layer. We note that once we generate the random thicknesses (${\lambda }_{normal}/U$), they are inserted into the design, and hence the thicknesses are no longer random after they are generated. We use the same design equations shown in Table 2. The only parameter that changes in the design is ${\beta }_k$ (in Table 2), which depends on the thickness. The other recursive equations stay the same. We note that we can produce M independent random filters by generating M different sets of random thicknesses.

4.2. Designed random transmittances and their ACF and CCF

Figure 4(a) shows the random TFs of three filters (5, 20, and 34) generated by the thin-film method for the specifications given in Section 6. It is apparent from Fig. 4(a) that it is possible to generate a random transmittance just by varying the thickness of the layers in the thin-film filters. Figure 4(b) shows the ACF of the transmittance of Filter-5. It is evident from Fig. 4(b) that the shape of the ACF of thin-film-based transmittances is Dirac-delta-like with near-zero correlations other than the zeroth lag, which is what we desire in the proposed spectrometers. Figure 4(c) shows the CCF between Filter-5 and Filter-20. We infer from Fig. 4(c) that the CCF values are near zero for all lags, as expected. We obtained similar ACFs and CCFs for the other filters as well. An additional advantage of using the random TFs is that no stringent filter design constraint, like keeping an exact quarter-wavelength thickness, has to be followed, and this opens the way for mass production. In Section 6, we demonstrate the improvement in resolution obtained by using the proposed random transmittances over the non-ideal transmittances in [5–8]. The implementation of the random transmittance filter array is currently underway in our laboratory.

 

Fig. 4 (a) Random transmittances produced by the thin-film method. (b) ACF of Filter-5. (c) CCF between Filter-5 and Filter-20.

Download Full Size | PDF

5. Resolution of filter-array-based spectrometers

Traditionally, the resolution of a spectrometer characterizes its ability to distinguish the peaks of two closely spaced spectral components of the input signal spectrum. The spectrometers that identify the closely spaced spectral components reveal fine details about the input signal spectrum. Thus, the quality of the spectrometers is usually specified in terms of this resolution. This was perhaps good enough for the conventional spectrometers but not appropriate for those that use a DSP algorithm. In this section, therefore, we first introduce a metric useful for defining the resolution that can be achieved by using various TF matrices introduced in Section 3. We then provide a new definition of resolution suitable for DSP-based filter-array spectrometers.

5.1. Performance metric useful for defining resolution

As discussed in Section 3.2, the problem of resolving distinct spectral components is equivalent to the exact recovery of the sparse spectral components of Eq. (4) by DSP algorithms. There are infinitely many different sparse spectra s. Thus, to determine the resolution, it is required to test the recovery of as many sparse spectra s as possible for a given TF matrix. Usually, a Monte Carlo simulation is used to evaluate if, on average, a specified resolution can be achieved by a TF matrix. The drawback of Monte Carlo simulation is that it is time-consuming. Therefore, in this section, we introduce a metric called genie-aided mean square error (g.MSE). It is a closed-form mean square error expression obtained assuming that the estimator knows the support set, i.e., a set of constituent wavelengths of the input signal spectrum. Thus, g.MSE can be used to indicate the minimum MSE obtainable by the $L_1$-norm-based DSP estimator. We will provide a comparison based on the more practical MSE as well and discuss in Section 6 how well the g.MSE predicts the MSE that is achieved in practice.

Consider again the data model $y=DGs+w=As+w$ given by Eq. (4). Given y and A, the aim of the $L_1$-norm-based DSP algorithms is to find an estimate $\stackrel{⌢}{s}$ of s. The DSP algorithm should find: 1) a set of constituent wavelengths (the support set) and 2) the non-zero intensity values corresponding to those constituent wavelengths. Let us suppose that the estimator exactly knows the support set. Then, by removing all the non-support set elements from the data model, the raw spectrum can be written as $y=A_k{s}_k+w$, where $s_k$ is a $K\times 1$ vector that contains the K non-zero intensity values of s, and $A_k$ is the $M\times K$ submatrix, the columns of which are the K corresponding columns of A. Note that this is an over-determined problem now, and finding $\stackrel{⌢}{s}$ essentially reduces to estimating the intensity vector $s_k$ from y.

An estimator that has complete knowledge of the support set is called an oracle estimator [27]. By making use of this prior information, the oracle estimator determines $s_k$ from y using the conventional least-squares approach. The minimum MSE of the oracle estimator is termed a genie-aided MSE (g.MSE). This g.MSE provides a tight lower bound on the accuracy of an estimator that finds $s_k$ from y. In this paper, we use the g.MSE to predict the maximum achievable resolution, because

We now define the g.MSE. Let ${\stackrel{⌢}{s}}_k$be the estimate of $s_k$, the $K\times 1$intensity vector. Let $e_s=(s_k-{\stackrel{⌢}{s}}_k)$ be the error vector. The co-variance of the error vector $e_s$ is then given by C_s = $\mathbb{E}\left[e_s{e}_s^T\right]$ = ${\sigma }^2{\left(A_k{}^T{A}_k\right)}^{-1}$ where $A_k$ denotes an $M\times K$ submatrix of A obtained by taking only those K columns that correspond to K wavelengths in the support set. We note that the expectation operation is taken over the probability distribution of the noise vector. The matrix C_s is of size $K\times K$. Now, the average MSE, $\mathbb{E}\left[{\Vert (s_k-{\stackrel{⌢}{s}}_k)\Vert }_2^2\right]$, is calculated as $\mathbb{E}\left[{\Vert (s_k-{\stackrel{⌢}{s}}_k)\Vert }_2^2\right]=\mathbb{E}\left[e_s^T{e}_s\right]=tr\left(C_s\right)$, where $tr\left(C_s\right)$ is the trace of the matrix $C_s$. Since the input signal spectrum is $x=G_k{s}_k$and $\stackrel{⌢}{x}=G_k{\stackrel{⌢}{s}}_k$, the minimum MSE between x and $\stackrel{⌢}{x}$ with the known support set can be calculated. We define the error vector $e_x=(x-\stackrel{⌢}{x})=G_k(s_k-{\stackrel{⌢}{s}}_k)$. The co-variance of the error vector $e_x$is then given by $C_x=\mathbb{E}\left[e_x{e}_x^T\right]={\sigma }^2{G}_k{\left(A_k{}^T{A}_k\right)}^{-1}{G}_k{}^T$. Now, the average genie-aided MSE is calculated as

We note from Section 2 that a sparse spectrum s contains only K non-zero wavelength components, and the remaining N-K wavelength components are zero. In addition, these K non-zero wavelength components can be present anywhere among the N possible wavelength components. That is, for a given N and K, there are $\left({}_K^N\right)$ possible sets of locations (support sets) where each set contains K wavelengths. Therefore, we can obtain $\left({}_K^N\right)$ g.MSEs. Each g.MSE is for a signal spectrum with K wavelengths. We note that for a fixed K, increasing N increases the number of support sets exponentially. In the next section, we define the resolution of a DSP-based spectrometer based on the g.MSE. In this paper, we always express the g.MSE in decibels (dB) for convenience.

5.2. Resolution: A DSP-based definition

As discussed in Section 3.1, when no DSP is used, the resolution is determined by the number of filters, M, in the filter array. That is, $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$ is the minimum distance between two spectral components that can be distinguished by the spectrometer. The larger the number of filters is, the better the resolution is. However, in miniature spectrometers, the number of filters is fixed, and hence the resolution is limited. Thus, $\Delta {\lambda }_M$ is called the resolution limit of the filter-array-based spectrometers. In order to further improve the resolution (beyond the resolution limit), DSP estimators are used [5]. These estimators exploit the sparse nature of the signal spectrum (Section 2) and employ L₁-norm-based estimators to improve the resolution. In this section, we introduce a new definition to measure the resolution that can be obtained by using the DSP estimators for the filter-array-based spectrometers.

We recall from Section 2 that the spacing between the samples of the input signal spectrum x (or equivalently s) is $\Delta {\lambda }_N=\frac{W_{\lambda }}{N}$ while the spacing between the samples of the raw spectrum is $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$. Since M < N, $\Delta {\lambda }_M>\Delta {\lambda }_N$, and therefore, the raw spectrum is in a low-resolution state ($\Delta {\lambda }_M$), while the signal spectrum x is in a high-resolution ($\Delta {\lambda }_N$) state. The high-resolution state can be realized and determined when a quality estimate of x is obtained for all possible x.

A way to define the resolution of a DSP-based spectrometer therefore is to find the minimal distance between the spectral components that can be accurately detected by the DSP estimator. Since the minimal distance is the distance between any two adjacent spectral components, viz. $\Delta {\lambda }_N=\frac{W_{\lambda }}{N}$, as N increases, the minimal distance decreases, and hence the resolution increases. We may come to the point of diminishing returns when going beyond a certain N. We call this N_max. In addition, we define the minimum distance to be $\Delta {\lambda }_{\min }=\frac{W_{\lambda }}{N_{\max }}$. Thus, to find this “digital resolution,” it is sufficient to find N_max.

We obtain N_max using the g.MSE measure introduced in Section 4.1 as follows: We first fix the parameters N, M, K, G, ${\sigma }^2$, and a TF matrix, which are required to calculate the g.MSE. We then check if the condition g.MSE $\le$ $\delta$ is met for a given percentage of the total $\left({}_K^N\right)$ possible support sets, where $\delta$ is an MSE specified by the user. If the condition is met, it indicates that the DSP estimator is able to detect all the spectral components. We then increase the value of N further and check the condition again. We keep repeating this process (that is, increasing the value of N and checking the condition) until the condition is no longer met. We finally declare the largest value of N that met the condition as N_max. The minimum spacing and hence the maximum achievable resolution of the DSP-based spectrometer is then$\Delta {\lambda }_{\min }=\frac{W_{\lambda }}{N_{\max }}$. In this paper, we increase the value of N starting from M that corresponds to the resolution limit $\Delta {\lambda }_M=\frac{W_{\lambda }}{M}$ of filter-array-based spectrometers. We note that it is not possible to increase N indefinitely with the hope of obtaining a minimum spacing. There is a limit for increasing$N$. This limit depends only on the TF matrix for the fixed parameters M, K, G, and ${\sigma }^2$. Since the g.MSE captures all these parameters, it is enough to define the resolution based on the g.MSE measure.

For a signal spectrum with K spectral components, we define the maximum resolution ($\Delta {\lambda }_{\min }$) of the spectrometer as follows:

The definition of the resolution given by Eq. (10) is general in nature in that it can include prior information. For example, if we know beforehand that the signal spectral components occupy only a certain range of the spectrum, then this definition automatically incorporates this prior information. Since prior information reduces the number of support sets and therefore the change is only in the event $\text{g}\text{.MSE}\le \delta$, the definition of the resolution remains unaffected. Since we use g.MSE, the minimum possible MSE, the corresponding resolution obtained is the maximum possible by employing any kind of DSP-based estimator. In the next section, we evaluate the resolution that can be achieved by the correlated TF matrix in [5] and the proposed random TF matrix.

6. Results and discussion

In this section, we aim to investigate how much gain in terms of resolution improvement can be obtained by using random TFs in comparison with the correlated, non-ideal TFs in [5]. We first investigate the maximum possible resolution by using the definition in Eq. (10). We then use the L₁-based DSP algorithm reported in [5] and see how much of this resolution can be realized in practice. We set the number of filters in the array to M = 40. The non-ideal TFs, for the 40 filters, have been obtained from [7], which are also used in [5]. The random TFs are generated using thin-film filters with m = 4 layers. We used the following refractive indices:$n_a=1$, $n_s=3$, $n_H=4$, and $n_L=1.2$ (Section 4.1). The angle of incidence of the light at the input medium is 45°. The center wavelength ${\lambda }_{normal}=850\text{nm}$. The thicknesses of the four layers are chosen as explained in Section 4.1, with a different thickness for each filter. After we generate the random TFs, we sample them, and arrange them in rows to form a random TF matrix.

We choose a Gaussian kernel with an FWHM of 1 nm to generate the kernel matrix G. The operating wavelength range of the spectrometer is from 300 nm to 700 nm, corresponding to a $W_{\lambda }$ of 400 nm. We vary N in order to find the maximum possible resolution $\Delta {\lambda }_{\min }$ as per Eq. (10). The value of $N$ begins with $M$ (which corresponds to the resolution limit) and is set to increase in steps of 40, i.e., N = 40, 80, 120, etc. For$N=40$, the minimum spacing between two adjacent spectral components is $\Delta {\lambda }_N=\frac{400nm}{40}$ = 10 nm (from Section 3), which is the resolution limit (maximum possible resolution without using a DSP estimator). We define the signal-to-noise ratio (SNR) for the data model in Eq. (4) as$\text{SNR}=\frac{{\Vert DGs\Vert }^2}{M{\sigma }^2}$. We keep the SNR at 30 dB.

We consider a sparse signal spectrum with K = 5 spectral (wavelength) components. The locations of these spectral components within the spectrum are random. Thus, the total number of possible locations (support sets) is $\left({}_5^N\right)$ for a given N. We choose 50,000 support sets at random from this total population. For each of these support sets, we calculate the g.MSE as per Eq. (9). We set $\rho =0.95$. As we vary N, we note the value of the g.MSE below which the g.MSEs of 95% of the support sets lie. Figure 5 shows the plot of the 95th-percentile MSE versus N obtained for the TF matrix in [5] and for the random TF matrix.

 

Fig. 5 g.MSE against resolution (N) for K = 5 and 50,000 support sets with $\rho =0.95$.

Download Full Size | PDF

We now set $\delta =-5\text{dB}$and find N_max. That is, we aim to find the maximum N such that the 95th-percentile MSE is less than$-5\text{dB}$. We observe from Fig. 5 that for the TF matrix in [5], the 95-percentile MSE is less than $-5\text{dB}$ when N_max = 62. Thus, the maximum resolution of the DSP-based spectrometer obtained by using the TF matrix in [5] is $\Delta {\lambda }_{\min }=\frac{W_{\lambda }}{N_{\max }}=\frac{400nm}{62}=\text{6}\text{.5}\text{nm}$. Therefore, the improvement in resolution over the resolution limit is 1.54 (10 nm /6.5 nm). In the case of random TFs, N_max = 405 and the maximum resolution obtained is $\Delta {\lambda }_{\min }=\frac{400nm}{405}=\text{0}\text{.99}\text{nm}$. The improvement in resolution in this case is 10.1 (10 nm/0.99 nm). We observe that the random TFs provide a resolution improvement of approximately a factor of 7 over that of the TFs in [5].

Table 3 summarizes the resolution obtained by using various design approaches and algorithms for the same filter and signal settings as before. From Table 3, we note that the proposed random TFs achieve 7-fold resolution improvement compared to the TFs in [5] and 10-fold resolution improvement when compared to the ideal TFs.

Table 3. Summary of Filter Design Approaches and Corresponding Resolution

View Table | View all tables in this article

We have also considered the non-Gaussian kernels such as Lorentzians and secant hyperbolic (mentioned in Section 2) for the same FWHM. Here, we report that we have obtained a similar resolution improvement to those obtained by using the Gaussian kernel. In addition, we note that comparing the resolution obtained by using ideal TFs with that of the random TFs is equivalent to comparing the resolution of ideal grating-based spectrometers with the random filter-array-based spectrometer. This is because, as discussed in Section 3.1, the resolution limit of both of the spectrometers is the same. Thus, we are in one sense indirectly comparing our proposed approach with the best possible conventional grating-based spectrometers.

We now investigate the practical spectral estimate obtained by using the DSP estimator reported in [5] when both the non-ideal and the random TFs are used in the filter-array spectrometer. For this purpose, we consider a signal spectrum generated by a mercury arc lamp, commonly used in fluorescence microscopy applications. The original signal spectrum of the mercury arc lamp has five (K = 5) prominent spectral components located at 365 nm, 405 nm, 436 nm, 546 nm, and 579 nm [22]. We consider N = 360 and M = 40. Figure 6 shows the original signal spectrum of the mercury arc lamp and an estimate of the signal spectrum obtained by using the correlated TF matrix in [5]. Figure 7 shows the estimated signal spectrum obtained by using the proposed random TF matrix.

 

Fig. 6 Reconstruction of a mercury arc signal spectrum by a non-ideal TF matrix in [5].

Download Full Size | PDF

 

Fig. 7 Reconstruction of a mercury arc signal spectrum by the proposed random TF matrix.

Download Full Size | PDF

From Figs. 6 and 7, we observe the following: First, it is evident that the estimate obtained by using the random TFs is closer to the original signal spectrum than that of the estimate using the non-ideal, correlated TFs. Second, the five spectral components of the original signal spectrum are determined correctly by the random TFs. However, the non-ideal TFs miss all five of the spectral components. Both the non-ideal and random TFs detect additional spectral components that are not present in the original signal spectrum. These additional spectral components appear as small bumps in Figs. 6 and 7.

In order to investigate how much resolution can be obtained using the DSP algorithm, we computed the MSE of the spectral estimates shown in Figs. 6 and 7 and compared them with the theoretical MSE shown in Fig. 5. The MSE of the spectral estimate by the random TFs (in Fig. 7) is about –5.5 dB which is just 1.3 dB away from the theoretical 95th-percentile MSE of –6.8 dB (in Fig. 5) at N = 360. For the non-ideal filters, the calculated MSE of the spectral estimate is 14 dB, which is 1.1 dB away from the MSE (12.9 dB) shown in Fig. 5. Thus, we observe that the resolution improvement deduced from Fig. 5 is actually achievable by using the practical DSP algorithms. The difference in MSE between the theoretical and the practical estimates is because of the detection of additional spectral components by the DSP algorithm. The difference in the practical MSE between the non-ideal TF in [5] and the random TFs using the DSP algorithm is about 18 dB.

In order to show that a resolution of 0.99 nm (see Table 3) is achievable by using random TFs, we consider an original mercury lamp spectrum [28] with prominent and weak spectral lines as follows: 404.656 nm, 407.781 nm, 435.835 nm, 491.604 nm, 546.074 nm, and a pair at 576.959 nm and 579.065 nm. Thus, the sparse spectrum of the mercury lamp contains seven spectral lines or components as shown in Fig. 8(a) . We note that the least separation among these spectral lines is 2.106 nm, which is between the pair 576.959 nm and 579.065 nm. We aim to resolve these components in the sparse domain. We consider the case of N = 400. Figures 8(b) and 8(c) show the estimate of the sparse spectrum using the TFs in [5] and using the proposed thin-film-based random TFs, respectively. It is apparent from Fig. 8(c) that random TFs correctly estimate the seven dominant wavelength components in the sparse spectrum. In addition, random TFs clearly resolve the pair of two closely spaced spectral components at 576.959 nm and 579.065 nm. This evidence supports the fact shown in Table 3 that random TFs are capable of resolving any two spectral components that are more than 0.99 nm apart. In contrast, the estimate obtained by using the TFs in [5] misses all seven of the dominant spectral components as shown in Fig. 8(b). In addition, the estimate contains spectral components that were not originally present in the signal spectrum. Based on these results and observations, we conclude that the filters with random TFs outperform the non-ideal TFs in terms of resolution and reconstruction performance.

 

Fig. 8 (a) Original sparse spectrum of the mercury lamp. (b) Estimated sparse spectrum by using TFs in [5]. (c) Estimated sparse spectrum by thin-film-based random TFs.

Download Full Size | PDF

7. Summary and conclusions

In this paper, we have proposed random-transmittance-based filters to improve the resolution of a spectrometer. We have shown that the optical filters with random TFs are designed to acquire holistic information about the signal spectrum, rather than the localized information that was the target of traditional designs. The holistic information captured by each filter when put together at the spectral estimator provides multiple independent views and helps the DSP algorithm to differentiate the details in the spectral components. We have shown via examples that spectrometers with random TFs provide 7-fold resolution improvement when compared to the use of filters that aim at implementing the ideal TFs. Rather than trying to design filters with the ideal brick wall TF, which has been the design paradigm in the past, our study shows that designing filters with random TFs should be the design paradigm for modern DSP-based spectrometers. This shift in the design paradigm not only brings forth an order-of-magnitude improvement in terms of resolution and MSE but also may relax the difficulty of the filter engineering process, resulting in less stringent design requirements.