Open Access
Add to BibSonomy
Abstract
Pushbroom hyperspectral imaging (HSI) has been used in many areas from air to land. However, its inherent operational drawback of the bulky slit leads to a limited field of view (FOV) and high energy consumption. Accordingly, a new and versatile HSI system is proposed by employing a smart digital micromirror device (DMD) to replace the mechanical scanning component. Moreover, tunable spatial and spectral resolution is implemented through adjusting the on-chip scanning linewidth and adopting the pixel fusion method, respectively. Meanwhile, three scanning modes including rough scanning, fine scanning, and regional scanning are achieved. These multiple choices increase the system’s flexibility, universality, and intelligence, which is attractive for practically different applications, especially for military and remote sensing fields in need of a large FOV, and medical and food fields in need of tunable resolution for various samples.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
As one of the most frequently employed hyperspectral imaging (HSI) methods [1], pushbroom HSI has outstanding performance for uses in many fields, ranging from remote sensing [2], food quality analysis [3], to medical diagnosis [4]. Different from other HSI methods, it relies on the mechanical slit’s scanning to acquire an object’s three-dimensional (3D) data cube with 2D spatial distributions and 1D spectral feature, thus generating its own advantages. For example, it can achieve both higher spatial resolution and higher spectral resolution [4,5]. Moreover, without extremely complex reconstruction algorithm, it constructs hyperspectral images more simply and precisely [6].
In the existing pushbroom HSI instruments, either the slit or the object has to move in order to induce relative movement between them. For the space-based applications, a spectrometer affixed with a slit needs to be boarded on the moving platforms such as a satellite or an aircraft [6]. And for the ground applications, either the slit or the object can be placed on the moving platforms. For example, A.Siedliska et al [7] and J. H. Cheng et al [8] put different food samples on a conveyor belt or a translation stage, respectively, and then passed them under a fixed slit. On the contrary, H. Akbari et al [9,10] moved the slit along a rail to scan the biological tissue samples. However, the biggest drawback is that the instruments can only obtain one line of the object imagery at a time because of the field of view (FOV) defined by the narrow slit aperture [10,11]. And this drawback not only goes against the rapid imaging of the large-scale scene, but also causes the low signal-to-noise ratio (SNR) and light collection efficiency [12,13]. Consequently, some users prefers to choose other HSI methods despite sacrificing the spatial and spectral resolution. Furthermore, the optimal resolution is always demanded for detection and recognition of different objects [11]. One way is to change the slit width resorting to some mechanical parts, while it reduces the system’s compactness and increases the energy consumption. And the tunable slit is too bulky to be integrated into the miniaturized HSI system [14].
Inspired by the traditional pushbroom method and emerging microelectromechanical systems (MEMS), a new HSI principle based on a smart digital micromirror device (DMD) is presented to get over the above confusions. In fact, the DMD has appeared in the non-scanning spectral imaging systems as binary-coded masks to replace the mechanical ones [1,15,16]. Different from them, we employ the DMD as a scanner to realize rapid scanning on the chip for its fast response. Compared with the bulky slit’s scanning, the energy consumption will be reduced dramatically because of its small size, a low driving voltage and the absence of mechanical moving parts [17]. Simultaneously, we directly capture a 2D object’s image instead of 1D image in one measurement, thus generating a larger FOV than the narrow slit aperture, and also boosting the light collection and SNR. Furthermore, we expand more functions to improve the system’s flexibility, universality and intelligence. Variable spatial resolution and three scanning modes, i.e. rough scanning, fine scanning and regional scanning are achieved by programming control over the DMD’s micromirrors [18]. It will help users to quickly find the target and then analyze the most important region. Besides, taking advantage of the pixel fusion method in the post processing also allows the spectral resolution to be tuned, which is conducive to searching for the most suitable resolution according to the specific application.
2. Principle and design
2.1 Concept and system design
The conceptual layout of the DMD-based spatial and spectral resolution tunable hyperspectral imaging (DSSRT-HSI) system is shown in Fig. 1. It mainly consists of two subsystems, i.e. spatial modulation subsystem and spectral dispersion subsystem. The former includes an objective lens and a DMD. The latter includes two groups of lenses (lens group A and B), a grating and a charge coupled device (CCD). The working principle is summarized as follows. First, an object is focused onto the DMD by an objective lens, and its 2D image is divided into numerous lines by the micromirrors. Then, the DMD, functioning as a light field scanner, linearly reflects the object’s image sequentially to the spectral dispersion subsystem. The reflected light of each chosen 1D image is successively collimated by the lens group A and dispersed by the grating. And the corresponding spectrum is focused on the CCD by the lens group B. After that, the CCD records the object’s full data cube composed of these spectrally dispersed images. The process of the sequential selection implemented by the DMD is similar to a panoramic scan completed by a traditional slit, but there is no mechanical dither during the on-chip scanning. Therefore, the misalignment issue is avoided, which reduces the difficulty in the further image processing.
Regarding the optical design of the DSSRT-HSI system, four design considerations are taken. Firstly, the working waveband covers the visible range from 450 nm to 650 nm. Secondly, as the efficient working area of the DMD and CCD in this research has the dimensions of 10.506 mm × 14.008 mm and 6.6 mm × 8.8 mm, respectively, the magnification of the spectral dispersion subsystem is set to be −1 ×. Thirdly, there should be a right configuration between the spatial modulation subsystem and the spectral dispersion subsystem in order to avoid the light-path interference between the optical components. Finally, the whole optical system should be as small and compact as possible, which is set as the optimization goal.
Based on the above discussion, the optical layout of the DSSRT-HSI system designed by ZEMAX using the ray-tracing method and three different wavelengths’ modulation transfer function (MTF) are illustrated in Fig. 2. Obviously, the MTF approaches the diffraction limit, which indicates a good imaging quality achieved by the designed DSSRT-HSI system. Besides, the important optical performance parameters are described in Table 1.
Table 1. Optical performance of the DSSRT-HSI system
2.2 Tunable spatial resolution
To tune the spatial resolution, we choose different number of adjacent columns on the DMD as a spatial light modulation unit, which operates successively until the entire projected image is scanned through. The DMD used is composed of a matrix of 1024 × 768 pixels, and the size of each pixel is 13.68 μm × 13.68 μm. Supposing that the number of columns contained in a modulation unit is determined as k, the total modulation units equal to n and the scanning linewidth b of the DMD are given by Eqs. (1) and (2), respectively.
The results in Fig. 3 confirm that as the size of the selected modulation unit decreases, the scanning linewidth becomes narrower, enabling the higher lateral spatial resolution in the X direction obtained. The lateral spatial resolution dx is related to the scanning linewidth b and the magnification msm of the spatial modulation subsystem, represented by
According to Eq. (3), it can be further expressed as
Fig. 3 Functional demonstration of the programmable control of spatially linear scanning of DMD. (a)–(c) The 2nd, 10th, and 19th modulation unit’s scanning with k = 50 and n = 21. (d)–(f) The 3rd, 26th, and 40th modulation unit’s scanning with k = 16 and n = 64. (g)–(i) The 15th, 106th, and 215th modulation unit’s scanning with k = 4 and n = 256.
Note that the vertical spatial resolution dy in the Y direction is decided by the pixel size s of the CCD and the magnification m of the whole system, given as
In the DSSRT-HSI system, the size of each CCD pixel is 3.45 µm × 3.45 µm. After calibration, the vertical spatial resolution is 16.4 μm. Therefore, the mentioned tunable spatial resolution in this work just refers to the lateral spatial resolution in the X direction. For actual applications, we ought to select the proper size of the modulation unit to obtain the spatial modulation on request.
2.3 Tunable spectral resolution and spectral image acquisition
Taking the number of the spectral channels c = 7 for example, the process of constructing the spectral images corresponding to various spectral channels and spectral resolution is illustrated in Fig. 4. First, an object is imaged on the DMD, and subdivided into n modulation units as shown in Fig. 4(a). After the DMD scans from left to right, the CCD will capture n spectrally dispersed images corresponding to n modulation units. Taking the leftmost and rightmost modulation units for instance, their images are depicted in Fig. 4(b). Through the spectrum calibration, all the wavelengths’ locations of each spectrally dispersed image on the CCD can be known. Then we divide the working waveband into seven channels with seven central wavelengths from λ1 to λ7 along the dispersive direction. As for the first channel, supposing p1 and pn represent the first and last number of the CCD pixels covered by its dispersive spectrum, respectively, the total pixels p chosen by the first channel can be expressed by
Fig. 4 Schematic procedure to acquire a spectral image of λ1. (a) The image of an object on the DMD. (b) Dispersive spectrum of seven channels collected by the CCD with the inset denoting the pixel fusion. (c) λ1 wavelength’s extraction. (d) The reconstructed spectral image of λ1.
Next, all these p pixels are merged into a super one by the pixel fusion. After the spectrum calibration procedure, the spectral resolution of each CCD pixel is approximately equal to 0.2 nm when k = 2, 4, 6, 8, 16. Hence, the spectral resolution SR of the super pixel is given as
According to Eq. (7), the spectral resolution and the number of spectral channels can be adjusted by selecting different values of p. Obviously, this method sacrifices the spectral resolution, but it greatly reduces the data memory and computational complexity when the high spectral resolution is not so necessary [19]. Conversely, we can also choose one CCD pixel as a single spectral channel, generating the highest spectral resolution of 0.2 nm and one thousand spectral channels. After the channel division, we extract the interested channel’s dispersive spectrum of each modulation unit as illustrated in Fig. 4(c). Finally, the spectral image of the object is constructed by stacking all the dispersive spectra from the interested channels as shown in Fig. 4(d).
3. Results and discussion
3.1 System prototype and functional verification
Based on the above-mentioned optical design, we established a prototype of the DSSRT-HSI system without any mechanical rotation and translation parts, as demonstrated in Fig. 5(a). The DMD (DLP7000) and CCD (Pia2400-17gm) were provided by Texas Instruments and Basler, respectively. Besides, we ensured the precise synchronization between the DMD’s scanning and the CCD’s recording by setting a synchronous signal between them. It is noteworthy that the maximal frame rate of the DSSRT-HSI system depends on the relatively slow CCD. As the CCD’s maximal frame rate is 17 frames per second (fps), thus the system’s shortest acquisition time t is given by Eq. (8). In fact, we should select the proper frame rate to get best exposure time according to the object’s light intensity.
Fig. 5 (a) The prototype of the DSSRT-HSI system. (b) A leaf with diseases and pests. (c) Spectra comparison between the DSSRT-HSI system and Oceanview.
To verify the validity of the proposed method, we measured the same point’s spectra of a tested target as shown in Fig. 5(b) with our DSSRT-HSI system and a commercial spectrometer (FLAME-T-VIS-NIR-ES) called Oceanview, respectively, shown in Fig. 5(c). And the root mean square error (RMSE) and the correlation coefficient r are introduced to evaluate the measuring results as shown in Fig. 6. As we can see from the results, the spectral curves achieved by the DSSRT-HSI system with different scanning linewidth have good correlation with the reference one measured by the commercial spectrometer, proving the validity of our system. Due to the existence of noise generated by the stray light, dark current and so on, there is small deviation of the measurement results. Consequently, some effective denoising techniques should be applied for image processing, which will be the research focus in the near future.
Fig. 6 Relationship using regression linear equation of the normalized intensity measured by the DSSRT-HSI system with different k and Oceanview.
3.2 Spectrum calibration
Spectrum calibration allows locating the arbitrary wavelength on the CCD in the working waveband, and getting the dispersive equation corresponding to each modulation unit of the DMD. The dispersive equation refers to the mathematical relation between the central wavelength and the pixel index of the CCD. During the calibration procedure, a halogen lamp was used as the object directly, and we put twelve filters separately with the known central wavelengths and full width at half maximum (FWHM), respectively, between the lens group B and the CCD. Note that every time a filter was placed, the DMD performed a scan and the CCD recorded the relevant spectrally dispersed images. Then, according to these filters’ spectrum, we calculated the dispersive equations of each modulation unit in different size, e.g. k = 2, 4, 6, 8, 16.
Taking k = 4 as an example, there were 256 modulation units. After placing twelve filters, each modulation unit acquired twelve spectrally dispersed images. Among them, two filters’ spectrally dispersed images of the 127th modulation unit were picked out as shown in Fig. 7(a). The stripy spectrum looks very straight, which indicates that good imaging quality can be received. In order to find the peak positions of twelve central wavelengths on the CCD, the spectral intensity of the middle position indicated by the white line in Fig. 7(a) was extracted from twelve filters’ spectrally dispersed images and plotted with Gaussians in Fig. 7(b). Thereby, the dispersive equation of the central wavelength and pixel index of the CCD was obtained as shown in Fig. 7(c).
Fig. 7 The 127th modulation unit’s fitting results of k = 4. (a) The spectrally dispersed images of the 7th and 11th filter. (b) Twelve filters’ spectral curves of the middle field by Gaussians fitting with the central wavelength of 453.6, 471.5, 500.6, 510.6, 534.3, 548.5, 564.8, 571.4, 610.3, 623.5, 638.4 and 651.3 nm and with the FWHM of 24.44, 16.15, 16.61, 17.36, 20.66, 24.30, 17.54, 15.72, 15.37, 35.05, 15.62 and 30.52 nm. (c) The dispersive curve fitted with an approximate linear function.
The actual spectral resolution of the CCD pixel SRpixelac can be approximately calculated according to Eq. (9).
where Δλf is the FWHM of the filter, and Npixelac represents the number of the pixels on the CCD covered by the FWHM’s spectrum of each filter practically.3.3 Experimental demonstration of imaging application
The DSSRT-HSI system can realize intelligent scanning to avoid the cost of time and data amount to a degree, since it allows for the flexible choices of spatial resolution based on the importance of different regions for a given mission.
Still taking the leaf in Fig. 5(b) for conceptual description, rough scanning was first implemented with the broad scanning linewidth (k = 8), to get a general view of the object. The resulting grayscale spectral images of twenty channels were achieved as shown in Fig. 8. Then, we looked for the region of interest (ROI) such as the leaf’s diseases and pests. Clearly, there were great differences between the two spectral images with the central wavelengths of 530 nm and 610 nm, respectively. Next, to study more spatial characteristics of the ROI, we performed fine scanning with the narrower scanning linewidth (k = 4 and k = 6) to survey the ROI of 538μm × 538 μm indicated by the red box as shown in Fig. 9(b), and acquired the corresponding spectral images. For comparison, we highlighted each ROI’s spectral image by pseudo-color processing based on the difference in gray values. Additionally, three curves of the intensity and the pixel position were extracted from the spectral images of ROI. Note that, the size of the modulation unit k was the only variable, and other factors such as the intensity of the light source, scanning speed of the DMD were kept all the same.
Fig. 9 (a)–(b) The spectral images with SR = 5 nm and the central wavelength of 530 nm and 610 nm, respectively. (c)–(e) The enlarged grayscale and false color spectral images of the ROI with the same central wavelength 610 nm and SR = 5 nm but different modulation unit size k. (f)–(h) The curves of the intensity and the pixel position exacted from the spectral images of the ROI. The green and red lines correspond to the same positions concerned.
As expected, the spectral image with k = 8 in Fig. 9(e) looks blurry and loses some spatial details compared to the ones with k = 6 in Fig. 9(d) and k = 4 in Fig. 9(c), caused by the decrease of spatial resolution. It may inevitably create a negative effect on prospective recognizing and classifying of the disease, since the different gray value is related to the types or degrees of the disease. Conversely, the SNR is increased with the increment of scanning linewidth. This is because the optical signal reflected from the modulation unit with small sizes to the spectral dispersion subsystem is weaker and susceptible to the noises, so it is difficult to extract effective spectral information from the raw spectrally dispersed images. Therefore, there is an unavoidable tradeoff between the SNR and spatial resolution. Besides, there is rapid intensity change existing in three curves as shown in Figs. 9(f)–9(h) around the boundary between the leaf’s abnormal area and the normal one, since the two areas composed of various components reveal a great difference in the absorption of the wavelength of 610 nm. And among them, the curve in Fig. 9(h) apparently gives the least fluctuation variations, and it is possible to exert a bad influence on the refined segmentation around the boundary and the evaluation of disease degrees. Fortunately, the DSSRT-HSI system enlarges the choice scope of the scanning linewidth. Therefore, in the practical applications, we can select the low spatial resolution in the preliminary analysis, and then gradually improve it until different components can be distinguished.
Next, we further study the impact of the various spectral resolutions. The results in Fig. 10 compare four spectral images of the same central wavelength but with different spectral resolution. It can be visually judged that the SNR of the smooth spectral image with the lowest spectral resolution is considerably improved, resulting from the effect of noise reduction generated by the pixel fusion approach. In addition, the characteristics of diseases and pests in different spectral images seem similar, though the higher spectral resolution may contain more details theoretically. This is probably because these characteristics appear in a broad spectral range, and there is no distinct change with different spectral resolution seen by our human eyes. Hence, in this case, the low spectral resolution of 15 nm may be enough to distinguish the diseases and pests from the leaf. However, when faced with some small and inconspicuous spectral differences of similar components, especially at the boundary identification in cancer resection, the system with a high resolution may have the potential to capture them [4]. Hence, we should choose the proper spectral resolution according to practical requirements.
Fig. 10 Spectral images of the same central wavelength 610nm with k = 4 and SR = 1 nm (a), SR = 5 nm (b), SR = 10 nm (c) and SR = 20nm (d), respectively.
The preceding analysis has studied morphology features through images, and we further observe the whole variation trend in term of spectral curves. Two points (point R and point G) of the leaf were measured and the results were processed utilizing the Gaussian fitting method as shown in Fig. 11. On the whole, they change considerably in the reflection intensity in the working waveband. Especially, their peak intensities match two distinct wavelengths, which are highlighted with a green bar and a red bar, respectively. And in case of the wavelengths between 510 nm and 590 nm, the point G in the normal region has the larger reflection intensity, which accords with the fact that the chlorophyll doesn’t absorb the green light. Whereas, for the wavelength more than 600 nm, the reflection intensity of the point R becomes stronger due to the disease or pests. Furthermore, by using the DSSRT-HSI system and the commercial spectrometer, respectively, we tested two monochromatic light emitting diodes (LEDs) whose spectrum has a sharp crest. The results in Fig. 12 not only verify the validity of the DSSRT-HSI system, but also indicate its good performance on spectral resolution. Note that the highest spectral resolution of the DSSRT-HSI system can reach as high as 0.2 nm, and thereby it provides the potential of effectively mining the spectral signatures in depth.
Fig. 11 The measured spectral data and fitted curves of the point R and G with k = 6 and SR = 0.2 nm (The regions nearby two peak intensities are highlighted with a green bar and a red bar, respectively).
Fig. 12 The spectrum of a red LED (a) and a green one (b) measured by the DSSRT-HSI system with k = 4 and Oceanview, respectively.
4. Conclusion
The DSSRT-HSI system is presented by introducing a smart MEMS chip, DMD, as an ideal scanner, and the first-generation prototype is established. Compared with the conventional pushbroom HSI, it captures a 2D scene at a time because of its large FOV determined by the fore optics rather than the slit aperture. Moreover, it achieves the microscopic scanning on a chip, completely free of any mechanical moving parts. We demonstrate its capabilities of tunable spatial and spectral resolution with multiple changes by selecting a diseased leaf as a target, and discuss the resulting impact on the spatial and spectral features. The spatial and spectral resolution can be tuned to the minimum value of 65.1 µm and 0.2 nm at an imaging distance of 9 cm, respectively. Meanwhile, we also show the concept of three scanning modes by implementing rough scanning, fine scanning and regional scanning in sequence. These expanded functions provide a variety of options of resolutions and scanning regions for various user's cases, and result in great benefits such as reducing the amount of data and time consumptionm, and increasing the system’s flexibility and universality significantly. In our future plan, some further improvements will be made in the performance and extra functions by developing various prototypes according to the specific requirements. And we also want to capture a dynamic target by taking full advantage of the high-speed scanning of the DMD.
Funding
Joint Fund for the Equipment Pre-research of Space Science and Technology (6141B06240205); National Natural Science Foundation of China (51622509); 111 Project (B13044).
Acknowledgments
We gratefully acknowledge the useful discussions and encouragement from Miss. Wenli Li, Mr. Zhiwan Yan and Mr.Jiancun Zhao.
References
1. X. Lin, G. Wetzstein, Y. Liu, and Q. Dai, “Dual-coded compressive hyperspectral imaging,” Opt. Lett. 39(7), 2044–2047 (2014). [CrossRef] [PubMed]
2. R. G. Sellar and G. D. Boreman, “Classification of imaging spectrometers for remote sensing applications,” Opt. Eng. 44(1), 013602 (2005). [CrossRef]
3. K. Kamruzzaman, Y. Makino, and S. Oshita, “Rapid and non-destructive detection of chicken adulteration in minced beef using visible near-infrared hyperspectral imaging and machine learning,” J. Food Eng. 170(7), 8–15 (2016). [CrossRef]
4. G. Lu and B. Fei, “Medical hyperspectral imaging: a review,” J. Biomed. Opt. 19(1), 10901 (2014). [CrossRef] [PubMed]
5. Y. Garini, I. T. Young, and G. McNamara, “Spectral imaging: principles and applications,” Cytometry A 69(8), 735–747 (2006). [CrossRef] [PubMed]
6. N. Hagen and M. Kudenov, “Review of snapshot spectral imaging technologies,” Opt. Eng. 52(9), 090901 (2013). [CrossRef]
7. A. Siedliska, P. Baranowski, M. Zubik, W. Mazurek, and B. Sosnowska, “Detection of fungal infections in strawberry fruit by VNIR/SWIR hyperspectral imaging,” Postharvest Biol. Technol. 139, 115–126 (2018). [CrossRef]
8. J. H. Cheng, D. W. Sun, and H. Pu, “Combining the genetic algorithm and successive projection algorithm for the selection of feature wavelengths to evaluate exudative characteristics in frozen-thawed fish muscle,” Food Chem. 197(Pt A), 855–863 (2016). [CrossRef] [PubMed]
9. H. Akbari, Y. Kosugi, K. Kojima, and N. Tanaka, “Detection and analysis of the intestinal ischemia using visible and invisible hyperspectral imaging,” IEEE Trans. Biomed. Eng. 57(8), 2011–2017 (2010). [CrossRef] [PubMed]
10. H. Akbari, K. Uto, Y. Kosugi, K. Kojima, and N. Tanaka, “Cancer detection using infrared hyperspectral imaging,” Cancer Sci. 102(4), 852–857 (2011). [CrossRef] [PubMed]
11. J. Reimers, A. Bauer, K. P. Thompson, and J. P. Rolland, “Freeform spectrometer enabling increased compactness,” Light Sci. Appl. 6(7), e17026 (2017). [CrossRef] [PubMed]
12. A. Jullien, R. Pascal, U. Bortolozzo, N. Forget, and S. Residori, “High-resolution hyperspectral imaging with cascaded liquid crystal cells,” Optica 4(4), 400–405 (2017). [CrossRef]
13. R. G. Sellar and G. D. Boreman, “Comparison of relative signal-to-noise ratios of different classes of imaging spectrometer,” Appl. Opt. 44(9), 1614–1624 (2005). [CrossRef] [PubMed]
14. S. Schuhladen, K. Banerjee, M. Stürmer, P. Müller, U. Wallrabe, and H. Zappe, “Variable optofluidic slit aperture,” Light Sci. Appl. 5(1), e16005 (2016). [CrossRef] [PubMed]
15. Y. Wu, I. O. Mirza, G. R. Arce, and D. W. Prather, “Development of a digital-micromirror-device-based multishot snapshot spectral imaging system,” Opt. Lett. 36(14), 2692–2694 (2011). [CrossRef] [PubMed]
16. M. Dunlop-Gray, P. K. Poon, D. Golish, E. Vera, and M. E. Gehm, “Experimental demonstration of an adaptive architecture for direct spectral imaging classification,” Opt. Express 24(16), 18307–18321 (2016). [CrossRef] [PubMed]
17. T. Yoon, C. S. Kim, K. Kim, and J. R. Choi, “Emerging applications of digital micromirror devices in biophotonic fields,” Opt. Laser Technol. 104, 17–25 (2018). [CrossRef]
18. S. A. Goorden, J. Bertolotti, and A. P. Mosk, “Superpixel-based spatial amplitude and phase modulation using a digital micromirror device,” Opt. Express 22(15), 17999–18009 (2014). [CrossRef] [PubMed]
19. W. Jahr, B. Schmid, C. Schmied, F. O. Fahrbach, and J. Huisken, “Hyperspectral light sheet microscopy,” Nat. Commun. 6(1), 7990 (2015). [CrossRef] [PubMed]
