Open Access
Add to BibSonomy
Abstract
We designed and built a homemade computed tomography imaging spectrometer (CTIS) of $250 \times 250\;{\rm pixels}$ of spatial resolution and 2 nm spectral resolution. The optical design considers a CTIS optical array coupled to a digital reflex camera. We reconstructed the intensity spectra of a fluorescent source, the diffuse reflectance of a ColorChecker, and samples of Capsicum annuum of three different colors, using the expectation-maximization sequential algorithm, optimized utilizing an array of indices to reduce the reconstruction time. The results obtained with a ColorChecker indicate a high positive correlation of 0.9745 with an average residual difference of 1.31% concerning the spectra obtained with a commercial integrating sphere spectrometer. The feasibility of the proposed CTIS system shows how to detect and evaluate the physiological changes resulting from the decomposition of the green fruit of the Capsicum annuum in a range from 500 to 650 nm.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. INTRODUCTION
Computed tomography imaging spectrometry (CTIS) is a technique for the instant acquisition of a cube of spectral data [1], which, compared to other spectral methods, does not require a temporary scan. CTIS works using a diffraction grating that allows diffraction light in several two-dimensional spectral projections onto a focal plane array (FPA). Using algorithms such as those used in limited angle computed tomography, the set of projections is used to reconstruct a data cube of the original object to determine its irradiance spectrum. Since dimensional spatial data and spectral data are superimposed on each projection, complex processing is required to separate the data [2,3].
Currently, to deal with reconstruction problems, there are several well-known iterative methods such as expectation-maximization (EM), multiplicative algebraic reconstruction technique (MART), mixed-expectation image reconstruction (MERT), algebraic reconstruction technique (ART), some based on convolutional neural networks (CNN) [4], as well as the Vose–Horton method [5] based on heuristic processes. For calibrating CTIS, Descour et al. used a monochromator connected to an optical fiber that they placed in front of the objective lens of the CTIS to vary the wavelengths and record them with their CTIS system. Thus, this procedure allows the calibration to obtain a spectral cube with a spatial–spectral resolution of $11 \times 11\;{\rm pixels}$ and 15 nm [3]. Likewise, Hagen et al. [6] performed the calibration of their system following Descour’s methodology but placing the optical fiber at the focal point of the collimating lens to improve the spectral resolution of the CTIS to $100 \times 100 \;{\rm{pixels}}$. In 2012, Habel et al. [7] used the CTIS itself as a monochromator to calibrate the system using three laser sources with different wavelengths that passed through a pinhole and recorded the spatial coordinates of each diffraction order in the detector of the camera used. In addition, they performed an interpolation process to find the spatial–spectral correspondence required for the subsequent reconstruction of the spectral image ($120 \times 120\;{\rm{pixels}}$ and 5 nm). In 2020, Salazar and Mendez [8] presented a low-cost CTIS design based on the methodology ibídem Habel et al., which allowed the reconstruction of a spectral data cube of $116 \times 100\;{\rm{pixels}}$ and a spectral resolution of 2.07 nm, enabling the detection of wavelengths in a range from 400 to 1052 nm.
In 2021, Douarre et al. [9] used the EM algorithm to reconstruct the spectral cube with a spatial and spectral resolution of $60 \times 60\;{\rm{pixels}}$ and 7.4 nm, respectively, within a range of 400–990 nm. This spectral cube is reconstructed from simulated CTIS images of apple leaves affected by apple scab, using compressed learning to identify the areas affected by the disease on the leaves. The CTIS system has been used as an instrument for the acquisition of instantaneous, complete spectral information (460–740 nm) in combination with a fluorescence microscope, allowing the analysis of physiological responses within living biological specimens [10], which highlights the importance of this spectral image acquisition technique for the study and characterization of organic samples. In the present work, we present the paraxial optical design of a CTIS system that allows selecting a set of commercial lenses according to the optical parameters, which makes possible the construction of a portable CTIS device. Furthermore, it is essential to mention that we perform a spatial and spectral evaluation of our CTIS system, reconstructing the spectral data cube of organic and inorganic samples by using an optimized EM algorithm. Diffuse reflectance spectra of a ColorChecker calibrator and fresh Capsicum annuum samples of three different colors (green, orange, and red) are obtained. Additionally, the spectral response capacity of the proposed CTIS system is evaluated when detecting the physiological changes that derivate from the decomposition process of the green Capsicum annuum.
The paper is structured in five sections that describe the development of the research. Section 2 establishes the main methodological aspects for the design of a CTIS system, considering its optical description, linear operator model, and reconstruction algorithm used. Sections 3 and 4 describe the CTIS assembly process and its calibration, respectively. Finally, Section 5 presents the results obtained by implementing the CTIS system to reconstruct the diffuse reflectance spectra of samples such as ColorChecker and Capsicum. In the case of Capsicum, it is possible to establish a parameter that determines its freshness considering the area under the spectral curve.
2. METHODS
A. Paraxial Optical Description of CTIS
The optical design of the CTIS consists of the following optical elements: an objective lens to focus on an object in an intermediate image plane, where a square aperture is placed to limit the spatial extent of the object and the field of view. Light in the intermediate image plane is then propagated to a collimating lens. The collimated light travels through a two-dimensional diffraction grating and is then focused on the FPA by the re-imaging lens (camera lens) [6,11]. We propose a converging optical system with focal length $f = - 20\; {\rm{mm}}$ that allows analyzing the entire first-order diffraction pattern since there is a low attenuation of the diffracted energy [12]. The paraxial design considers an objective lens ${f_1}$ with a focal length of 50 mm, a collimating lens with a focal length of 100 mm and a diameter $D = 50\; {\rm{mm}}$, a diffraction grating with a period of 5 μm , and finally, a re-imaging lens ${f_3}$ with a focal length of 40 mm. The resolution of the diffraction grating initially constrains the focal length of the re-imaging lens and the detector size to be used, thus considering the general equation of a grating with normal incidence [13]:
where ${d_p}$ is the grating period, $\phi$ is the diffraction angle, ${m_n}$ is the diffraction order, and $\lambda$ is the diffracted wavelength. The minimum size of the detector (${h^\prime}$) is estimated by considering the Lagrange or Smith–Helmholtz theorem [14]. Now, combining this theorem with Eq. (1), we have a general condition limiting the ${f_3}$ value that should be used in the CTIS system:B. CTIS Linear Operator Model
Figure 1 represents the $3 \times 3$ diffraction orders recorded in the FPA, generated by the diffraction grating. The acquired image $g$ formed in the FPA has a spatial resolution of ${M_x} \times {M_y}\;{\rm{pixels}}$ and is a superposition of the responses to the different wavelengths, which can be interpreted as parallel projections of the spectral data cube $\hat f({n_x},{n_y},{n_\lambda})$ in different directions. This cube consists of voxels ${n_{\rm{voxel}}} = {n_x} \times {n_y} \times {n_\lambda}$, where ${n_x}$, ${n_y}$ are defined by the resolution of the diffraction zero order recorded in the image $g$, and ${n_\lambda}$ is the spectral dimension. Here, we will use a visible spectral bandwidth. To obtain a spectral image, the entire data cube must be reconstructed from nine $3 \times 3$ projections of ${n_x} \times {n_y}$ formed by a two-dimensional diffraction grating. Figure 1 shows the nine diffraction orders that are identified with the coordinates $(0,0),(1,0),(1,1),(0,1),(- 1,1),(- 1,0),(- 1, - 1),(0, - 1)$, and $(1, - 1)$ or with the index $i = 1\ldots 9$. The positions of each one of the projected diffraction orders in $g$ are represented by ${r_{\textit{iw}}} = ({c_{{n_x}iw}},{c_{{n_y}iw}})$, wherein $w$ refers to the different wavelengths, and $({c_{{n_x}iw}},{c_{{n_y}iw}})$ are the coordinates of the centroid of each order of diffraction. Likewise, Fig. 1 shows the angle formed by each diffraction order for each one of the wavelengths considered on the image plane $g$.
Fig. 1. Illustration showing the concept of scattered prismatic images and their visualization as projections of the cube of the original object.
So, in contrast to standard scanning instruments, the CTIS’s ability to measure a cube of data in one instant measurement is not straightforward. The relationship between each digitized image and its respective reconstructed intensity spectrum is much more complicated than for scanning instruments, so estimating the data cube of a scene becomes a much more complex process. For CTIS, the traditional approach represents the measurement as an abstract mapping between voxels in the data cube $\hat f({n_{\rm{voxel}}} \times 1)$ and pixels in the diffraction image $g$ recorded in FPA. The image $g$ is rewritten in lexicographical order ($m = {M_x} \times {M_y}$) as a vector $\vec g(m \times 1)$, through a transfer matrix $H(m \times {n_{\rm{voxel}}})$ of the system that maps $\hat f$ into $\vec g$, where the matrix $H$ is measured and obtained during the calibration [7]. CTIS systems can be described by the general linear imaging equation [7,15]:
where $\eta$ is the noise factor. The data cube $\hat f$ can be reconstructed from the digitized image $g$ with its respective spectral projections, by performing the pseudoinverse reconstruction, applying an iterative method that allows estimating the data. The following section details this iterative method.C. Reconstruction Algorithm
To deal with the pseudoinverse reconstruction problem, we use the iterative EM method updating ${\hat f^{(q + 1)}}$ to converge to a solution from the initial assumption ($q = 0$) and ${\hat f^{(0)}} = {H^T}\vec g$, where ${H^T}$ represents the transpose of the matrix $H$. Additionally, assuming $\vec g$ follows the Poisson statistic (a constant light intensity with discrete probability distribution arrives at the detector) so the noise factor $\eta$ of Eq. (3) is not considered, the EM algorithm [16,17] provides a way to find the maximum likelihood solution iteratively:
Once ${H_{\rm{indx}}}$ is obtained, as shown in Fig. 2, the unique elements (non-repeating elements) are determined and stored in a cell array ${R_{\rm{indx}}}$. Simultaneously, it is determined how many times the elements of ${R_{\rm{indx}}}$ are repeated in ${H_{\rm{indx}}}$, creating an index cell ${C_{\rm{indx}}}$ that indicates the positions of the repeated elements. These considerations make it feasible in our algorithm to write the term ${[{H{{\hat f}^{(q)}}}]_m}$ in Eq. (4) as $\vec g({R_{\rm{indx}}}) = \sum \vec f({C_{\rm{indx}}})$, thus obtaining an optimized algorithm that allows us to exponentially reduce the computational time to reconstruct the spectral data cube $\hat f$. The optimized algorithm is implemented in MATLAB version R2019b on a laptop with Intel Core i5-9300H CPU 2.40 GHz and 16 Gb of random access memory (RAM).
3. CTIS SYSTEM ASSEMBLY
The optical components’ readjustment keeps the distances among lenses and the focal ratio concerning the original design.
Optical components parameters consider commercial lenses for the final CTIS design based on Zemax optimization. A Tamron brand objective lens, model 21HC with focal length $f = {{50}}\;{\rm{mm}}$ and a full image circle of 2/3 in., is used as the objective lens. The collimating lens is an Edmund Optics VIS-EXT plano-convex chromatic doublet with $f = {{100}}\;{\rm{mm}}$ and $D = 50\; {\rm{mm}}$. Lastly, as a re-imaging lens, a Nikon AF-S Nikkor G-series lens is considered, with $f = {{40}}\;{\rm{mm}}$ and $f/{2.8}$, as shown in Fig. 3.
Fig. 3. Final schematic of the CTIS system assembled and re-designed with commercial lenses, suitable for the initial focal lengths while maintaining the parameters of the paraxial design.
The 3D schematic CTIS mechanical parts are made using Solidworks design software. The pieces are composed of an AF-S Nikkor 40 mm $f/{{2}.\rm{8G}}$ macro lens attachment for Nikon D5600 brand digital single-lens reflex (DSLR) camera [Fig. 4(a)], system body [Fig. 4(b)], collimator lens holder [Fig. 4(c)], C-mount [Fig. 4(d)], C-mount attachment to system body [Fig. 4(e)], C-mount retaining nut [Fig. 4(f)], and square diaphragm holder [Fig. 4(g)]. Figure 3 shows the final overall schematic of the constructed CTIS; it displays a Tamron 21HC lens with a back focal length of 22.10 mm; the edge of the achromatic doublet is 12.87 mm. The initial optical design has a distance of 137.20 mm between the objective and the collimating lens, maintaining a tolerance of 2.23 mm. It can be adjusted using a coupling and a retaining nut designed for the C-mount of the objective lens to ensure the alignment of the optical system to obtain a collimated light beam.
After the CTIS parts design, we proceeded to machining and assembling, using bronze to build the pieces shown in Figs. 4(c) and 4(d). The parts shown in Fig. 4, when assembled, make up the CTIS mechanical system. Other components employ aluminum in their building. A He–Ne laser of $\lambda = 633\;{\rm nm}$ is used to verify the optomechanical system’s collimation and the intensity distribution that reaches the image camera lens. Figure 5 shows the whole optomechanical system.
The last two essential elements to implement the CTIS are the field aperture and the diffraction grating, as shown in Fig. 3. With the field aperture, a $5 \times 5 \;{\rm{mm}}$ square aperture is fabricated with an Objet500 Connex3 printer with 200 µm precision, using a high resolution opaque black VeroBlackPlus photopolymer. This aperture limits the field angle of the objective lens to 2.86° so that an image size of 1 mm is obtained. Additionally, to avoid introducing noise into the system from the amount of light passing through the 3D printing material, the aperture is darkened using 3M Temflex Vinyl 1700-Black.
To the Nikon AF-S Nikkor 40 mm, a $f/{{2}.\rm{8G}}$ lens was attached to a two-dimensional diffraction grating of 0.200 line/µm through a 52 mm filter adapter ring. As shown in Fig. 5, the system coupled with a Nikon D5600 camera, which has a $23.5 \times 15.6 {\rm{mm}}$ complementary metal-oxide-semiconductor (CMOS) sensor with 24.2 million effective pixels ($6016 \times 4016\;{\rm pixels}$), is used to detect the $g$ image, with its respective spectral projections, produced by the two-dimensional diffraction grating of the CTIS.
The vertical size of the image on the CMOS sensor is 1.00 mm for the zero order of diffraction, 2.96 mm for the first order, and 10.88 mm for the second order. The vertical half of the CMOS sensor used is 7.8 mm. These paraxial calculations guarantee the photographic recording of the two-dimensional first-order diffraction pattern.
We measure the experimental modulation transfer function (MTF), which in general is a graph of the measured intensity (gray levels) as a function of the spatial frequency in pairs of lines per millimeter (lp/mm). A direct way to determine the MTF is to measure the intensity image of an object with known spatial frequency [18,19] using an Edmund USAF 1951 resolution test pattern. The pattern is photographed with the CTIS system, and the contrast is determined. Gray level contrasts are analyzed at each spatial frequency of the test object, obtaining an average profile for each selected spatial frequency region to determine the maximum and minimum intensity values for each frequency.
The values obtained from the contrasts are presented in Fig. 6, where the meridional (red line) and sagittal (blue line) metatarsophalangeals (MTP) are shown. At 4 lp/mm, there is a contrast of 94%, which decays to 38.82% and 58.97%, meridional and sagittal, respectively, at a spatial frequency of 101 lp/mm, which represents a difference of 61.18% and 41.03% to the ideal diffraction-limited MTF (black line). We observe that at a frequency of 64 lp/mm, the best spatial resolution is achieved for the CTIS system with contrasts ${\gt}{{80}}\%$, with a difference of approximately 8% concerning the diffraction-limited MFT. In addition, the spatial resolution is estimated by analogy with the Rayleigh criterion [20], for which a resolution criterion of 9% (horizontal black dotted line) is chosen. Therefore, the implemented CTIS system has a high resolving power that guarantees the spatial quality of the images obtained.
The CMOS sensor’s spectral response per RGB channel evaluates using a halogen point source, and it evidences a low sensitivity in the 400–440 nm and 650–700 nm ranges. The next step to implement the CTIS system is the spatial, spectral calibration of the camera sensor, and this involves interpolating the pixels with the wavelengths recorded by the sensor. The following section will detail all the crucial aspects of this calibration.
4. SPATIO-SPECTRAL CALIBRATION
To calibrate the CTIS, we used a visible near-infrared (Vis-NIR) HL-2000 halogen source coupled to a DMC1-02 monochromator (200–800 nm) from Optometrics Corporation. A 1000 µm pinhole is placed at the output of the monochromator, trying to represent a point source. It is captured by the CTIS located at a distance of 63 cm, because the minimum distance to the object is 44 cm for the Tamron 21HC lens. The CTIS system is controlled remotely by digiCamControl software, a free and open-source application for Windows [21], which can control DSLR cameras using a universal serial bus (USB) or wireless fidelity (WiFi) connection, as shown in Fig. 7.
Table 1. Average Angles of Each Diffraction Order About the ${+}x$ Axis at FPAa
It is important to note that the signals of the 25 monochromator wavelengths selected in a range from 417 to 659 nm are visible to the camera detector in the CTIS. One pinhole image is recorded with the camera for each wavelength, using an exposure time of 1/50 s, speed ISO-100, and $f/2.8$. These images (${I_{\rm{Raw}}}$) are saved and decoded in a Nikon proprietary Raw format (.nef), employing the DCRaw image processor, developed by Coffin [22], following three steps. Step 1: DCRaw is executed in MATLAB to transform ${I_{\rm{Raw}}}$ into a $I$ image in the tagged image file format (tiff). Step 2: The decoded $I$ image separated into its red–green–green–blue (RGGB) color channels (${I_{\rm R}}$, ${I_{\rm G1}}$, ${I_{\rm G2}}$, ${I_{\rm B}}$) from the Bayer mosaic [23] are reduced to half the size of the CMOS sensor ($2008 \times 3008\;{\rm pixels}$) in each RGGB channel. Step 3: Finally, the two green channels (${I_{\rm G1}}$, ${I_{\rm G2}}$) are averaged to obtain a single green channel intensity image produced by the Bayer mosaic and thus construct a color image ${I_{\rm{RGB}}}$ without chromatic interpolation.
Each RGB channel of the recalculated ${I_{\rm{RGB}}}$ images is cropped to an area of interest of $1659 \times 1659\;{\rm pixels}$, including the nine diffraction orders. Then, adaptive thresholding is applied [15] to set a threshold value based on the local characteristics (different gray levels or non-uniform illumination) of each of the 25 registered images. Such threshold value allows us to segment the images to determine the coordinates of the centroids $({c_{{n_x}iw}},{c_{{n_y}iw}})$ of the nine peaks of the diffraction mosaic produced by the pinhole, as depicted in Fig. 1. From the centroids $({c_{{n_x}iw}},{c_{{n_y}iw}})$, one can estimate by linear interpolation, employing least squares, the positions of each wavelength (${\lambda _{\textit{iw}}}$) of the diffraction mosaic occupied in the camera’s CMOS detector. To achieve interpolation, first, distance ${r_{\textit{iw}}}$ concerning the zeroth order of each diffraction order in each of the 25 images must be calculated according to the following equation:
Subscript $i$ represents each diffraction order, (${c_{{n_x}1w}},{c_{{n_y}1w}}$) are the coordinates of the zeroth-order centroid ($i = 1$), and subscript $w$ indicates each of the 25 selected monochromator wavelengths. A new space, whose coordinates are $({r_{\textit{iw}}},{\lambda _{\textit{iw}}})$ starting from calculated values of ${r_{\textit{iw}}}$ of the 25 images and the 25 monochromator wavelengths, obtains the twenty-five computed points. Then, linear interpolation is performed to each diffraction order, getting the line, Eq. (6), that best approximates the 25 points:
where ${m_i}$ represents the slope, and ${b_i}$ represents the ordinate in a ${\lambda _{\textit{iw}}}$ versus ${r_{\textit{iw}}}$ plot. The values of ${m_i}$ and ${b_i}$ vary for each diffraction order, as shown in Table 1. Also, with the centroids, the average ${\theta _i}$ angles of each diffraction order are determined throughTo determine coordinates $({c_{{n_x}iw}},{c_{{n_y}iw}})$ of an unknown wavelength, one enters a value of ${\lambda _{\rm{new}}}$ into Eq. (6) by considering the values of ${m_i}$ and ${b_i}$ for each diffraction order. This procedure determines new ${r_{\textit{iw}}}$ values by substituting them into Eqs. (8) and (9), which are detailed below:
The new coordinates (${c_{{n_x}iw}},{c_{{n_y}iw}}$) in the FPA are determined through ${\theta _i}$ values taken from Table 1. By performing a residual analysis between the positions obtained with the monochromator and those estimated employing the fitting equation, Eq. (6), we have for $\lambda = 417\;{\rm nm}$ a maximum difference of 2.23 pixels, representing a residual of ${\pm}1.5\;{\rm nm}$ between wavelengths. Overall, without considering $\lambda = 417$, an average difference $| {\Delta {r_{\textit{iw}}}} |$ of 0.5 pixels is obtained, representing a variation of ${\pm}{0.41}\;{\rm{nm}}$.
In this sense, the interpolation of 151 wavelengths performs, from 400 to 700 nm, representing a $\Delta \lambda = 2\;{\rm{nm}}$. The interpolated centroids squares of $250 \times 250\;{\rm{pixels}}$ are constructed, thus generating a binary image representing the projections of the data cube on the FPA. This binary image will allow building the calibration matrix $H$, where each column contains nine entries in the position of the diffracted pixels at a wavelength (active pixels). The rest of the elements of that column are zeros. Therefore, we have a matrix $H$ with $2,\!752,\!281 \times 9,\!437,\!500$ elements, considering only the positions of the active pixels by columns, which is rewritten as ${H_{\rm{indx}}}$ with $9 \times 9,437,500$ elements.
The optimized EM algorithm reconstructs a spectral cube with dimensions of ${{250}} \times {{250}}\;{\rm{pixels}}$ and 151 wavelengths. The resulting resolution is better than that of some CTIS devices reported by Descour et al. [3], Hagen et al. [6], Habel et al. [7], and Salazar and Mendez [8], in which the spatial resolution of their data cubes is lower than the one reported in the present work. The total time taken by the EM algorithm to reconstruct the spectral data cube for each RGB channel depends on the number of iterations $q$, since the general behavior of this algorithm is to apply the most significant corrections in the few initial iterations. In contrast, high numbers of iterations tend to be unrealistic solutions [24]. Therefore, if we employ $q = 20$, the total reconstruction time is 75 s. However, it is sufficient with $q = 10$, because the convergence RMS of EM algorithm is reduced to 11%, obtaining a realistic solution, and the computation time is reduced to 37.49 s.
Our CTIS is validated by reconstructing the intensity spectrum of a commercial fluorescent source of 14 W and color temperature of 5000 K, being confined in a rectangular box covered with aluminum foil, letting out a portion of light through a pinhole of 1000 µm, which represents a point fluorescent source that is recorded by the proposed CTIS, producing a diffraction mosaic. The spectral data cube is obtained for each RGB color channel ${\hat f_S}{({n_x},{n_y},{n_\lambda})_{\rm R,G,B}}$ by using the optimized EM algorithm with the matrix ${H_{\rm{indx}}}$ determined in the calibration process. Thus, the spectrum of the point fluorescent source with the 151 wavelengths is reconstructed. Any of the channels could be used alone to rebuild a spectrum. However, the spectral response of each channel covers only a part of the visible spectrum, so we must combine the channels and thus obtain the data cube of the total emission spectral response of the source through the following expression:
In Fig. 8, the spectrum obtained from the CTIS reconstruction shows a maximum shift in the wavelength reconstruction of 1.5 nm for the first two prominent peaks (433.3 and 541.7 nm) and 0.5 nm for the third peak (607.7 nm) of the spectrum of a fluorescent source for the commercial spectrometer, which represents a relative percentage error of 0.35%, 0.27%, and 0.08%, respectively. These results show the sensitivity and efficiency of the proposed CTIS for obtaining emission spectra in a range of 430 nm–640 nm.
Now, to verify the efficiency in the reconstruction of solid objects, a homogeneous illumination system (halogen source) is coupled to the CTIS, obtaining the reflected intensity spectra, which expresses the diffuse reflectance spectra. For this purpose, the objects are illuminated with two 60 W halogen bulbs placed behind diffuser screens. Each source is positioned at ${\pm}{45^ \circ}$ from the optical axis, producing a uniform illumination on the objects, as evidenced in Fig. 9, which are 63 cm from the objective lens of the CTIS.
5. RESULTS AND DISCUSSION
A. Objects Reconstruction and Spatial Assessment of Images Quality
In Fig. 10, the RGB images reconstructed by wavelengths from 404 to 664 nm are presented. It can be observed how the object’s red color begins to be visualized with intensities starting to 604 nm. Likewise, below 444 nm, there is a low RGB representation. That is, there is no correspondence between the wavelength and its chromatic coordinates in the color space. One way to assess the spatial quality of an image is by using metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), normalized RMS deviation (${\sigma _n}$), or correlation coefficient (${\rho _C}$), and the use of their corresponding generated parameters allow us to compare how similar the reconstructed image is to the original image [25–28].
Since the original images are not available for each wavelength, the quality of the image is evaluated by comparing each RGB channel reconstructed by wavelength with the original RGB channels obtained by the Bayer mosaic of the CMOS sensor to record the information in the camera used. Thus, using the metrics mentioned earlier, the R channel of the original image (zero-order image) is compared with the R channel of the 151 reconstructed images associated with each wavelength. Corresponding metrics are also used in the G and B channels of the original image, with each of their respective reconstructed channels of the 151 wavelengths (400 to 700 nm).
Figure 11 shows an example of how the spatial quality of the reconstructed images is evaluated considering the spectral width of the Bayer mosaic [29]. Figure 12 shows the behavior by channels (color lines) and average (black line) of the quality of the reconstructed RGB images, using the PSNR, SSIM, ${\sigma _n}$, and ${\rho _C}$ metrics.
Fig. 12. Evaluation of the quality of the reconstructed RGB image of an object (LEGO) by wavelength.
It is important to note that when the PSNR value is high, it represents a better signal-to-noise ratio to the original image. In the case of digital image processing, it is reported in the literature that the PSNR value is usually acceptable when it is $\ge {{20}}\;{\rm{dB}}$. The highest PSNR value represents the best [30], so the reconstruction quality is guaranteed. In Fig. 12, a PSNR value is evidenced in a range from 23 to 42 dB for the three reconstructed channels, in the regions centered at 450 nm, 550 nm, and 600 nm, respectively. Such PSNR values indicate good quality of reconstructed images when comparing the RGB channels individually; in this sense, an average PSNR interval is obtained by combining the three channels from 20 to 27 dB. Likewise, it is evident in Fig. 12 that the SSIM presents a behavior proportional to that obtained when using the PSNR metric, finding SSIM values in an interval of 0.1–0.9. Evaluating 500 separate channels in an interval from 450 to 650 nm, we found SSIM values of 0.6–0.9, which indicates a low degradation of structural similarity between the original image and the reconstructed ones since the range of SSIM values reported in the literature are given between ${-}{{1}}$ and ${+}{{1}}$, where the value of one corresponds to two identical images [26,31].
Additionally, in Fig. 12, we have the metric ${\sigma _n} \to 0$ in the intervals of 574–650 nm (R channel), 472–574 nm (G channel), and 450–500 nm (B channel), indicating high sensitivity of reconstruction for each RGB channel in those intervals. Thus, we found an inverse behavior of the PSNR to ${\sigma _n}$; when the PSNR increases, ${\sigma _n}$ decreases. Finally, in Fig. 12, the ${\rho _C}$ metric remained between 0.8 and 0.97, which indicates that spatial reconstruction by individual channels is obtained in more than 80%. The values of the metrics that showed less similarity are below 450 nm and above 650 nm, which is due to the low spectral sensitivity of the CMOS sensor used to record the CTIS images.
B. Reconstruction of Diffuse Reflectance Spectra
To determine objects’ diffuse reflectance spectra, we use a dark plate with reflectivity less than 2% and a white plate reference reflecting 99% of the light, coated with barium sulfate (${\rm{BaS}}{{\rm{O}}_4}$) with Reactivos Química MEYER. Reference plates are located 63 cm from the objective lens and illuminated with a uniform source. The reflected light attenuates when it reaches the CTIS; to record the image, we use an exposure time of $1/5$s, ISO-100 speed, and $f/2.8$.
From the recorded images, spectral data cubes of the white ${\hat f_{\cal B}}{({n_x},{n_y},{n_\lambda})_{\rm R,G,B}}$ and dark ${\hat f_{\cal O}}{({n_x},{n_y},{n_\lambda})_{\rm R,G,B}}$ reference plates are reconstructed, using the optimized EM algorithm for each color channel. Thus, the total reflectivity cube is determined, by the use of the Eq. (11) [7], for white ${\hat M_{\cal B}}({n_x},{n_y},{n_\lambda})$ and dark ${\hat M_{\cal O}}({n_x},{n_y},{n_\lambda})$ plates:
C. ColorChecker
A ColorChecker Classic mini x-rite PANTONE calibrator is used to validate the diffuse spectral response reconstruction of our CTIS, consisting of $4 \times 6$ color regions With all 24 colors [32].
The ColorChecker is placed at a distance ${\gt}63\;{\rm cm}$ so that its image occupies the entire area of order zero in the diffraction mosaic. After employing the optimized EM algorithm, as shown in Fig. 13, it is observed that in the highlighted blue box only the first wavelengths of the spectral cube are within the sensitivity range of our CTIS, and the white box presents illumination at all wavelengths of the spectral cube.
Fig. 13. Spectral data cube of $250 \times 250\;{\rm pixels}$ and 151 wavelengths from the ColorChecker.
Selecting some wavelengths from the spectral data cube (see Fig. 14), it is observed that in a range from 440 to 650 nm there is no saturation of the intensity of the spectral response, which facilitates data analysis. We achieved an RGB representation associated with a data cube for each wavelength, as shown in Fig. 14. Unlike other works reported by Habel et al. [7], who reconstructed and showed grayscale images, Huang et al. [4] assigned false-color associated with the intensity distribution for each reconstructed channel.
Figure 15 shows the diffuse reflectance spectrum of a matrix of $4 \times 6$ regions of the ColorChecker obtained using Eq. (12) (scattered colored points) in a range from 450 to 650 nm; compared with a spectrum obtained with a commercial spectrometer (continuous colored lines), Fig. 15 shows some similarities. We used residual analysis and Pearson’s linear correlation [33] ${\rho _c}$ to assess the similarities between the CTIS and OceanOptics USB4000 spectra. The highest correlation is found for the red region (${\rho _c} = 0.9970$), the lowest is for white (${\rho _c} = 0.8175$), and, for the 24 ColorChecker areas, there is an average correlation ${\rho _c} = 0.9745$.
Fig. 15. CTIS comparison reconstruction with OceanOptics USB4000 spectrometer for 24 ColorChecker regions.
In general, all correlations present a strong positive linear relationship (${\rho _c} \gt 0.8$) [33], which indicates a high correspondence of morphologies between the spectral curves, giving an average residual intensity of 1.31%. These results show that it is possible to reproduce the spectra of solid, flat, and homogeneous inorganic samples such as the ColorChecker. However, testing with natural objects is more complex considering the structure; to continue the validation, we use organic specimens that present certain irregularities on their surfaces, such as the Capsicum annuum.
D. Capsicum annuum
Capsicum annum cultivated worldwide is commonly known as a sweet pepper or bell pepper; it is native to South America [34–36]. The world production of peppers for the the year of 2018 was 716,000 tons, with Mexico producing 9000 tons, which positioned it in ninth place among the 11 primary exporting countries of peppers [37].
To determine the spectra of three fruits of different colors: green, orange, and red, CTIS is used with the experimental setup shown in Fig. 9, where the fruits are placed at a distance of 63 cm from the CTIS objective lens. Using the configuration of Fig. 9, a photographic record is made, obtaining ${I_{\rm{Raw}}}$ from the diffraction mosaic of each of the fruits of different colors, as can be seen in Figs. 16(a)–16(c).
Fig. 16. Raw image of Capsicum: (a) green, (b) orange, and (c) red with our CTIS. Spectral data cube of the Capsicum: (d) green, (e) orange, and (f) red. Diffuse reflectance spectra of the Capsicum: (g) green, (h) orange, and (i) red.
Using the optimized EM algorithm, the spectral data cubes are constructed, as shown in Figs. 16(d)–16(f), for each of the colors of the Capsicum samples. Furthermore, the wavelengths for each photographed piece are highlighted on the lateral data cubes.
We now take a neighborhood of $5 \times 5$ voxels of the least saturated regions in the spectral data cubes and the diffuse reflectance spectra obtained using Eq. (12). In Figs. 16(g)–16(i), the colored lines represent the reconstructed spectra voxel by voxel; the black line is the average spectrum of the 25 voxels. The cyan line represents the diffuse reflectance spectrum using a commercial OceanOptics USB4000 spectrometer with an integrating sphere from 500 to 650 nm. Now, averaging the 25 voxels, the noise in voxel reconstruction is filtered, making the average spectrum fit the one obtained with the integrating sphere with a correlation coefficient of ${\rho _c} = 0.9889$, ${\rho _c} = 0.9988$, and ${\rho _c} = 0.9996$ for green, orange, and red peppers, respectively, indicating a significant similarity of the morphology between the CTIS and commercial spectra.
Finally, the ability of the CTIS to record changes in the samples over the days is verified. The green pepper is preserved at room temperature (26.7°C) and relative humidity of 34%. The pepper is photographed with our CTIS during senescence on four different days. The obtained results are compared with the commercial spectrometer, as evidenced in Fig. 17. As days elapse, the pepper darkens due to the tissue decomposition process. Therefore, CTIS can record physiological changes in organic samples such as the Capsicum. Now, calculating the area under the curve (${\cal A}$) in the established spectral range, it is possible to determine an amount associated with the freshness of the fruit (freshness index) in percentage terms (${{\cal I}_\%}$). Figure 17 shows that for the fourth day, the freshness of the fruit reduced to 21.18%; decreasing respiratory rate and low production of ethylene characterizes a non-climacteric fruit [38], such as Capsicum. Our system can determine the freshness index following the methodology proposed by Althaus & Blanke [39].
6. CONCLUSIONS
We have presented a detailed description of the design, construction, and calibration of a CTIS, as well as the performance of an optimized EM algorithm necessary to reconstruct the spectral data cubes that contain diffuse reflectance spectra of inorganic and organic samples with a spatial–spectral resolution of $250 \times 250\;{\rm pixels}$ and 2 nm, from the ColorChecker and different pigmentations of the Capsicum annuum fruit. The EM algorithm is optimized using an array of indices, exponentially reducing the reconstruction time combined with the CTIS calibration, generating a better approximation of the real cube of spectral data per RGB channel. The CTIS reconstruction of color diffuse reflectance spectra regions of the ColorChecker indicated a strong positive correlation for the spectra obtained with a commercial spectrometer, evidencing the potential of our CTIS to improve the estimation of diffuse reflectance spectra. For the case of Capsicum annuum samples, we showed that our CTIS has the performance to detect physiological changes resulting from senescence in a spectral range from 500 to 650 nm, introducing a parameter to assess the freshness index. Finally, one aspect to highlight in the CTIS implementation is that the spectra depend not only on the robustness of the reconstruction algorithm but also on the quality of the optical components and their alignment, calibration, and lighting. Furthermore, we guarantee that the region of interest to be analyzed stays well focused and illuminated when making the photographic record of the diffraction mosaic.
Acknowledgment
We appreciate the support of CONACYT and Centro de Enseñanza Técnica Industrial CETI-Mexico in developing this research topic and Ing. Javier Arriaga P. for machining of the parts used in the CTIS system.
Disclosures
The authors declare no conflicts of interest.
Data Availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
REFERENCES
1. T. Okamoto, A. Takahashi, and I. Yamaguchi, “Simultaneous acquisition of spectral and spatial intensity distribution,” Appl. Spectrosc. 47, 1198–1202 (1993). [CrossRef]
2. M. R. Descour, E. L. Dereniak, and A. C. Dubey, “Mine detection using instantaneous spectral imaging,” Proc. SPIE 2496, 286–304 (1995). [CrossRef]
3. M. R. Descour, C. E. Volin, E. L. Dereniak, K. J. Thome, A. B. Schumacher, D. W. Wilson, and P. D. Maker, “Demonstration of a high-speed nonscanning imaging spectrometer,” Opt. Lett. 22, 1271–1273 (1997). [CrossRef]
4. W.-C. Huang, M. S. Peters, M. J. Ahlebaek, M. T. Frandsen, R. L. Eriksen, and B. Jørgensen, “The application of convolutional neural networks for tomographic reconstruction of hyperspectral images,” Displays 74, 102218 (2021). [CrossRef]
5. M. D. Vose and M. D. Horton, “A heuristic technique for CTIS image reconstruction,” Appl. Opt. 46, 6498–6503 (2007). [CrossRef]
6. N. Hagen, E. L. Dereniak, and D. T. Sass, “Maximizing the resolution of a CTIS instrument,” Proc. SPIE 6302, 168–178 (2006). [CrossRef]
7. R. Habel, M. Kudenov, and M. Wimmer, “Practical spectral photography,” Comput. Graph. Forum 31, 449–458 (2012). [CrossRef]
8. J. Salazar-Vazquez and A. Mendez-Vazquez, “A plug-and-play hyperspectral imaging sensor using low-cost equipment,” HardwareX 7, e00087 (2020). [CrossRef]
9. C. Douarre, C. Crispim-Junior, A. Gelibert, G. Germain, L. Tougne, and D. Rousseau, “CTIS-Net: a neural network architecture for compressed learning based on computed tomography imaging spectrometers,” IEEE Trans. Comput. Imaging 7, 572–583 (2021). [CrossRef]
10. B. K. Ford, M. R. Descour, and R. M. Lynch, “Large-image-format computed tomography imaging spectrometer for fluorescence microscopy,” Opt. Express 9, 444–453 (2001). [CrossRef]
11. C. J. Vandervlugt, “Computed tomographic imaging spectrometry,” Ph.D. thesis (University of Arizona, 2011).
12. L. C. Marquet, “Transmission diffraction grating attenuator for analysis of high power laser beam quality,” Appl. Opt. 10, 960–961 (1971). [CrossRef]
13. F. J. Duarte, “Chapter 11 - Spectrometry and interferometry,” in Tunable Laser Optics, F. J. Duarte, ed. (Academic, 2003), pp. 227–248.
14. R. Kingslake and R. Johnson, Lens Design Fundamentals, 2nd ed. (Elsevier Science, 2009).
15. R. C. Gonzalez, R. E. Woods, and S. L. Eddins, Digital Image Processing using MATLAB, 3rd ed. (Gatesmark, 2020).
16. L. A. Shepp and Y. Vardi, “Maximum likelihood reconstruction for emission tomography,” IEEE Trans. Med. Imaging 1, 113–122 (1982). [CrossRef]
17. N. Hagen, E. L. Dereniak, and D. T. Sass, “Fourier methods of improving reconstruction speed for CTIS imaging spectrometers,” Proc. SPIE 6661, 15–25 (2007). [CrossRef]
18. M. Turkmenoglu, O. Sengul, and L. Yalcıner, “MTF measurements for the imaging system quality analysis,” Gazi Univ. J. Sci. 25, 19–28 (2012).
19. G. Bostan, P. Sterian, T. Necsoiu, A. Bobei, and C. Sarafoleanu, “The slanted-edge method application in testing the optical resolution of a vision system,” J. Optoelectron. Adv. Mater. 21, 22–34 (2019).
20. H. Courrier, C. Kankelborg, B. De Pontieu, and J.-P. Wülser, “An on orbit determination of point spread functions for the interface region imaging spectrograph,” Sol. Phys. 293, 125 (2018). [CrossRef]
21. D. Istvan, digiCamControl (2018).
22. D. Coffin, “Dcraw: Decoding raw digital photos in linux,” (2008), https://www.dechifro.org/dcraw/.
23. J. Nakamura, Image Sensors and Signal Processing for Digital Still Cameras, Optical Science and Engineering (CRC Press, 2017).
24. N. Hagen, “Snapshot imaging spectropolarimetry,” Ph.D. thesis (University of Arizona, 2007).
25. C. Kwan and J. Larkin, “Demosaicing of Bayer and CFA 2.0 patterns for low lighting images,” Electronics 8, 1444 (2019). [CrossRef]
26. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE Trans. Image Process. 13, 600–612 (2004). [CrossRef]
27. B. Sankur, “Statistical evaluation of image quality measures,” J. Electron. Imaging 11, 206–223 (2002). [CrossRef]
28. J. J. Sánchez-Escobar, L. I. Barbosa-Santillán, J. Vargas-Ubera, and F. Aguilar-Valdés, “Inversion method based on stochastic optimization for particle sizing,” Appl. Opt. 55, 5806–5813 (2016). [CrossRef]
29. H. Teranaka, Y. Monno, M. Tanaka, and M. Ok, “Single-sensor RGB and NIR image acquisition: toward optimal performance by taking account of CFA pattern, demosaicking, and color correction,” Electron. Imaging 28, 1–6 (2016). [CrossRef]
30. A. Shahin, Y. Guo, and A. S. Ashour, “Neutrosophic set-based denoising of optical coherence tomography images,” in Neutrosophic Set in Medical Image Analysis (Elsevier, 2019), pp. 123–142.
31. D. R. Bull and F. Zhang, “Measuring and managing picture quality,” in Intelligent Image and Video Compression (Elsevier, 2021), pp. 335–384.
32. C. S. McCamy, H. Marcus, and J. G. Davidson, “Color-rendition chart,” J. Appl. Photogr. Eng. 2, 95–99 (1976).
33. R. A. Fisher, Statistical Methods for Research Workers, 20th ed. (Oliver and Boyd, 1954).
34. J. Sancho and F. Navarro, Pimientos y Pimenton. Estudio Quimico-Fisico (Instituto de Orientacion, Murcia, Facultad de Ciencias de la Universidad de Murcia, 1962).
35. R. M. Monsurat, S. A. Suraj, and O. A. Olumide, “Physicochemical properties and mineral composition of Capsicum annum and Capsicum frutescens oils,” IOSR J. Appl. Chem. 7, 112–116 (2014). [CrossRef]
36. G. E.-S. Batiha, A. Alqahtani, O. A. Ojo, H. M. Shaheen, L. Wasef, M. Elzeiny, M. Ismail, M. Shalaby, T. Murata, A. Zaragoza-Bastida, N. Rivero-Perez, A. Magdy Beshbishy, K. I. Kasozi, P. Jeandet, and H. F. Hetta, “Biological properties, bioactive constituents, and pharmacokinetics of some Capsicum spp. and Capsaicinoids,” Int. J. Mol. Sci. 21, 5179 (2020). [CrossRef]
37. FAOSTAT, “United nations, food and agriculture organization,” Tech. Rep. (FAOSTAT, 2021).
38. V. Paul, R. K. Pandey, and G. C. Srivastava, “The fading distinctions between classical patterns of ripening in climacteric and non-climacteric fruit and the ubiquity of ethylene—an overview,” J. Food Sci. Technol. 49, 1–21 (2011). [CrossRef]
39. B. Althaus and M. Blanke, “Development of a freshness index for fruit quality assessment-using bell pepper as a case study,” Horticulturae 7, 405 (2021). [CrossRef]
