1. INTRODUCTION

The demand for high-resolution, information-rich image data has been amplified by the widespread industry adoption of machine learning algorithms [1–6]. Meta-optics-enabled spectral and polarimetric imaging exhibits potential in meeting these rising data demands of learning algorithms within the visible spectrum [7–15]. Yet, the integration of meta-optics with infrared thermal imaging remains a relatively unexplored domain. Recent strides in heat-assisted detection and ranging (HADAR) have demonstrated the potential of thermal imaging [16] for machine perception tasks utilizing the infrared spectrum and the atmospheric transparency window. By combining spectral-resolved thermal imaging with artificial intelligence, HADAR offers a platform for machine perception through pitch darkness like broad daylight [16]. Here, our goal is to show that integrating meta-optics with infrared thermal imaging emerges as a crucial enabling factor for such next-generation machine vision algorithms. Our work offers a pathway for realizing compact spectro-polarimetric thermal imagers beyond the conventional technologies that use push-broom, filter-wheel, or interferometer modules. This new class of thermal imagers enhances the ability of machine learning algorithms to capture information such as temperature, material composition, and surface morphology.

Within the broad thermal infrared spectrum, the long-wave infrared (LWIR) region stands out as especially advantageous for heat-assisted detection and ranging, as most objects at room temperature radiate thermal energy at these wavelengths [Fig. 1(a)]. Moreover, the LWIR atmospheric transmission window facilitates the thermal radiation signal to propagate long distances [17]. However, the blurry nature of thermal images poses a significant challenge for machine vision algorithms. For example, perception tasks such as semantic segmentation and ranging rely on texture and contrast for learning features and show poor performance with panchromatic thermal imagery [18]. HADAR overcomes the blurry nature of thermal images by decomposing the LWIR thermal radiation into its spectral channels [16]. Unfortunately, the commonly used mosaic filter technique for spectral decomposition [Fig. 1(b)] [19–24] falls short for LWIR thermal imaging due to the restricted pixel count in LWIR focal plane arrays. Current HADAR spectral imagers rely on cumbersome filter wheels [Fig. 1(c)] with restricted spectral resolution, or fragile and bulky interferometers [Fig. 1(d)] with a confined field of view (FOV) [16]. Such constraints considerably hinder the widespread applicability and adoption of HADAR.

 

Fig. 1. Long-wave infrared (LWIR) spectro-polarimetric thermal imaging. (a) Room-temperature blackbody radiation (shown in red) and atmospheric transmission spectrum (shown as a shaded area). The LWIR spectral region is crucial for thermal imaging due to its peaked room-temperature thermal radiation spectrum and the atmospheric transparency window. (b)–(d) Conventional methods for spectral imaging, such as using (b) a mosaic sensor, (c) a filter wheel, or (d) interferometry, either pose limitations or are infeasible for LWIR thermal imaging. (e) In this study, we propose a new approach for spectro-polarimetric thermal imaging, achieved by combining large-area spinning metasurfaces and compressive sensing reconstruction algorithms.

Download Full Size | PDF

To address these limitations, in this work, we harness the capabilities of large-area meta-optics in the infrared domain to create a platform for spectro-polarimetric thermal imaging. We achieve spectral decomposition by designing the dispersion and polarization through stacked metasurfaces, while the spectral reconstruction is accomplished using compressive sensing and dictionary learning algorithms. Our designs employ 2D structures with large feature sizes (${\gt}1\;{\unicode{x00B5}{\rm m}}$), which can be made using photo-lithography techniques, facilitating large-area fabrication and scalable manufacturing. Despite its compact form, our Spinning MetaCam delivers exceptional spectral resolution across diverse materials and can effectively unveil subtle polarimetric features, enabling applications such as material classification. By leveraging meta-optics and advanced computational spectroscopy methods, we demonstrate three times higher accuracy for machine vision tasks than conventional panchromatic thermal imaging. In the future, the angular speed of the spinning stage can be increased to enable real-time video leading to widespread adoption of HADAR technology similar to RADAR, LIDAR, and SONAR.

2. DESIGN OF THE SPINNING METASURFACE STACK

The architecture of our spinning-metasurface based spectro-polarimetric imaging system is depicted in Fig. 1(e). It comprises a broadband linear polarizer, three anisotropic and dispersive metasurfaces, and an LWIR imaging sensor. The polarizer is utilized to polarize the incoming thermal radiation signals, and the metasurfaces are utilized to realize spectral filtering. We design the metasurfaces with high anisotropy to produce distinct spectral responses for orthogonal polarizations. Additionally, the metasurfaces’ dispersion rotates different wavelengths of radiation to varying polarization orientations. By using the metasurfaces in tandem and axially spinning the polarizer and metasurfaces to different angles, we obtain tunable transmission spectra that sample the incident thermal radiation in its spectral and polarimetric channels. We then reconstruct unknown spectra of imaging targets using compressive sensing and dictionary learning algorithms. Dictionary learning generates a set of basis functions that represent the unknown spectra in a sparse format [25]. Compressive sensing enables accurate reconstruction of the sparse spectra from a limited number of measurements [26]. Combining these two techniques enables accurate and stable spectral reconstruction in the presence of noise and measurement errors [27]. The four-dimensional spectro-polarimetric data generated by our system offers a wealth of physical information about an imaging target, making it a valuable tool for physics-driven machine vision [5,28], facilitating various tasks such as object detection and semantic segmentation [29,30].

 

Fig. 2. Design and characterization of the spinning metasurfaces. (a)–(c) Schematics of the three different metasurface devices. (d) Optical microscope image of a fabricated metasurface. Inset, an optical image of a 25.4 mm-diameter device used for the imaging experiments, highlighting the large-area uniformity. (e) Measured polarized transmission spectra (${t_{\textit{ip}}}$ and ${t_{\textit{is}}}$) of the three metasurfaces (M1–M3), displaying strong anisotropy and distinctive dispersion. (f) Generated tunable transmission spectra by our spinning metasurface stack. The upper boundary of colored areas represents the transmission spectra corresponding to specific spinning angle combinations of the three metasurfaces. The colored areas highlight the differences between the generated transmission spectra. The low correlation between the generated transmission spectra is crucial for improving spectral reconstruction performance. Inset, an optical image of the integrated spinning-metasurface module based on motorized rotatory mounts. The overall size of the module is smaller than ${10}\;{\rm cm} \times {10}\;{\rm cm} \times {10}\;{\rm cm}$, making it a promising platform for next-generation high-contrast LWIR thermal imaging. (g) Normalized spatial transmittance of the module at two representative spinning angle combinations $\Theta = { (0^ \circ}{,0^ \circ}{,0^ \circ})$ shown at the top and $\Theta = { (90^ \circ}{,90^ \circ}{,90^ \circ})$ illustrated at the bottom. The red circles correspond to a transmittance value of 0.5. Note that the spatial transmittance remains relatively consistent across all different $\Theta$ combinations, considering that the field of view is primarily limited by the diameters of the metasurface devices. The estimated field of view of the imaging system across the LWIR range is approximately 20.5°.

Download Full Size | PDF

To quantitatively describe the mechanism of the spinning-angle-controlled transmission spectra, we represent the spectro-polarimetric response of the metasurfaces using Jones matrices. Assuming that the transmission axis of the input linear polarizer is at 0 deg relative to the $x$ axis, the Jones matrix ${J_i}$ of a metasurface $i$ with the principal axis ($p$) at a spinning angle ${\theta _i}$ relative to the $x$ axis can be expressed as

The Jones matrix of the three-metasurface assembly is given by

Thus, the total transmission spectrum of the three spinning metasurfaces strongly depends on the spinning-angle combinations $\Theta = ({{\theta _1},\,{\theta _2},\,{\theta _3}})$ when the constituted metasurfaces are strongly anisotropic and dispersive, i.e., ${t_{\textit{ip}}}(\lambda) \ne {t_{\textit{is}}}(\lambda)$ (see Supplement 1 for the detailed analysis). We note that large differences between the spectral responses of the three metasurfaces (M1, M2, M3) are also introduced to minimize the correlations between the generated spectra, which can significantly improve the spectral reconstruction performance [31]. We note that the metasurfaces do not necessitate rotation at fixed angular frequencies during the imaging process. As depicted in Eq. (2), it is the static angular positioning of the metasurfaces that is pivotal for selectively filtering the incident thermal radiation in its spectral content. We emphasize that our design generates a large set of distinct transmission spectra with only three metasurfaces, while the total number of spectra in the traditional mosaic array is limited to the number of metasurfaces/filters used [7,8,19–23].

 

Fig. 3. Schematic of the spectral reconstruction process. (a) The measured raw signal can be expressed by (b) the pre-calibrated spectral response function of the imaging system multiplied by (c) the spectrum of an imaging target. (c), (d) For the spectral reconstruction, the unknown spectrum ${P_\lambda}$ is projected onto a sparse representation basis ${D_{{k\lambda}}}$ using dictionary learning. (e) This sparse representation ${\phi _k}$ is then used for compressive sensing based reconstruction. The use of compressive sensing and dictionary learning in the reconstruction process significantly improves the reconstruction accuracy, making the spinning-metasurface-based spectro-polarimetric imaging more robust against noise and measurement errors.

Download Full Size | PDF

Accordingly, we design the metasurfaces and experimentally achieve three key characteristics for optimized spectro-polarimetric imaging performance: (1) strong anisotropy and dispersion for efficient wavelength decomposition; (2) high transmission and low self-emission for high signal to noise ratio (SNR); (3) small angular dependence for a large FOV. The unit cells of the three designed metasurfaces are shown in Figs. 2(a)–2(c). Strong dispersive anisotropy of the transmission spectra can be observed in Fig. 2(e). Additionally, we emphasize that large-area devices are generally required for imaging applications to ensure sufficient numerical aperture. All the metasurfaces designed here have feature sizes larger than 1 µm. Large-area devices (25.4 mm in diameter) with high structural quality and uniformity [Fig. 2(d)] can be rapidly fabricated by standard photo-lithography techniques, enabling scalable manufacturing for practical applications. This is in strong contrast to recent works on miniaturized spectrometers [32–40], where the device footprint is on the micrometer scale and thus not suitable for imaging applications.

The tunable transmission spectra produced by our spinning metasurfaces are shown in Fig. 2(f). The distinct spectra are a result of the tuned spinning-angle combination $\Theta$. We integrate the three fabricated metasurfaces tandemly via compact rotatory mounts to independently control the rotation of each metasurface [inset of Fig. 2(f)]. We also optimize the spinning-angle combinations of the three spinning metasurfaces using genetic algorithms to generate largely uncorrelated transmission spectra for optimal spectral reconstruction performance (see Supplement 1 for details). Additionally, we note that increasing the number of metasurfaces can further improve the spectral resolution but simultaneously reduces the SNR as the peak transmissions of the LWIR devices are limited to around 0.6 (see Supplement 1, Fig. S5 for details). However, our method has the potential to scale up into the hyperspectral regime by adding more high-transmission LWIR metasurfaces.

We also evaluate the FOV of our imaging module by integrating it with an LWIR thermal camera and capturing images of a large-area uniform blackbody. To determine the spatial transmission efficiency, we normalize the signal counts of each pixel by the counts at the center of the images. We also define the angular range with transmittance above 0.5 as the effective FOV of a system. As seen in Fig. 2(g), our spinning metasurface module has an FOV of around 20.5 deg, which is difficult to achieve with interferometer-based spectral imagers.

3. ALGORITHM DESIGN FOR SPECTRAL RECONSTRUCTION

To extract the unknown spectro-polarimetric properties of various imaging targets, we use a combination of dictionary learning and compressive sensing algorithms in the reconstruction process. The tunable transmission spectra produced by our spinning metasurfaces [shown in Fig. 2(f)] are not narrowband, which means that the collected raw signals at different spinning-angle combinations $\Theta$ do not directly reflect the spectral radiance at different wavelengths. Instead, the collected signal $I(\Theta)$ at each pixel can be described as an integral of the spectral response function $R(\Theta ,\lambda)$ multiplied by the ground truth spectrum $P(\lambda)$ that we wish to obtain, i.e., $I(\Theta) = \int_{{\lambda _{{\min}}}}^{{\lambda _{{\max}}}} R(\Theta ,\lambda)P(\lambda){\rm d}\lambda$. To solve for this equation, we discretize the spectral range of interest and express it in a tensor form as shown in Eq. (3),

We emphasize that directly solving Eq. (3) does not produce accurate spectral reconstructions [33,41]. In theory, we can use measured signals ${I_\Theta}$ and the pre-calibrated response function ${R_{\theta \lambda}}$ to directly determine unknown spectra ${P_\lambda}$ at each pixel of a scene. However, in practice, two limitations impede the performance of spectral reconstruction: the problem becomes underdetermined when there are many discretized wavelength bands, and measurement noise affects both ${I_\Theta}$ and ${R_{\Theta \lambda}}$, making the direct reconstruction method unstable and the results inaccurate.

 

Fig. 4. Spectro-polarimetric thermal imaging results. (a) An optical image of the “PURDUE” imaging target that is constructed from titanium letters on a glass substrate (${75}\;{\rm mm} \times {50}\;{\rm mm}$). Inset, a zoomed-in optical microscope image of the micro-structures in the letters, which generate distinctive spectral and polarimetric signatures. (b)–(e) Reconstructed spectra of four representative pixels (corresponding to the letter “R”, “U”, “E”, and the glass substrate, respectively) compared with the ground truth spectra measured by a Fourier-transform infrared spectrometer. (f) Reconstructed spectral frames at 6 representative wavelengths. The contrast between different frames demonstrates that the system can effectively reveal the LWIR spectral properties of various materials and structures. (g), (h) Degree-of-linear-polarization and angle-of-linear-polarization frames. Distinctive polarimetric signatures can be observed for each letter in the images. (i) Simulated spectral reconstruction results. The ground truth spectra (solid lines) are Gaussian peaks with 0.6 µm FWHM centered at different wavelengths (8.5 µm, 9.3 µm, 10.2 µm, 11 µm, 11.8 µm, 12.7 µm, and 13.5 µm). The reconstructed spectra (dotted lines) show good agreements with the ground truth.

Download Full Size | PDF

To enhance both the precision and consistency of spectral reconstruction, we leverage the capabilities of compressive sensing and dictionary learning algorithms to solve Eq. (3) (Fig. 3). Our approach begins with dictionary learning, a process we employ to generate a dictionary comprising 32 basis functions, designated as $D$. The spectra utilized for the dictionary learning are derived from the infrared emissivity spectra drawn from the ECOSTRESS Spectral Library [42,43], as well as blackbody thermal radiation spectra across various temperatures. The resultant dictionary can provide sparse representations for any thermal radiation spectrum in the space of spectra we are studying. We point out that the efficiency of this sparse coding is fundamentally linked to the prevalent spectral similarities found within the LWIR thermal radiation spectra. We thus project the unknown spectrum ${P_\lambda}$ as a linear combination of the basis functions in the dictionary [Figs. 3(c) and 3(d)]. We have

Our reconstruction method significantly improves the reconstruction accuracy, making spinning-metasurface-based spectro-polarimetric imaging more robust against noise and measurement errors.

 

Fig. 5. Material classification using the Spinning MetaCam. (a) Simulated spectral reconstruction results for materials commonly found in everyday life. (b) Calculated confusion matrix illustrating material classification without the use of spinning metasurfaces. The classification in this case is solely reliant on the integrated intensity of thermal radiation within the LWIR region. (c) Calculated confusion matrix presenting material classification with spinning metasurfaces incorporated. The inclusion of spectral information substantially augments the performance of material classification, improving the mean average precision from 0.293 to 0.834.

Download Full Size | PDF

4. SPECTRO-POLARIMETRIC IMAGING AND MACHINE VISION

To evaluate the performance of our prototype imaging system, we conduct experiments using a custom-designed “PURDUE” target made of letters constructed from titanium and a glass substrate [Fig. 4(a)]. Each letter has unique micro-grating structures [inset of Fig. 4(a)] that generate distinctive spectro-polarimetric signatures in the thermal radiation signal. The glass substrate also features a characteristic emission peak around 11 µm. Note that we heat the image target to 150°C to generate high signal intensity. The reconstructed spectra of four representative pixels are shown in Figs. 4(b)–4(e). We compare them with the ground truth spectra measured by a Fourier-transform infrared spectrometer, validating the effectiveness of our reconstruction approach. The reconstructed spectral frames [Fig. 4(f)] also exhibit high contrasts between different wavelengths, demonstrating that the system can effectively reveal the LWIR spectral properties of different targets. We note that the relatively low reconstruction accuracy at shorter wavelengths (8–10 µm) results from the low transmission (low SNR) and the high correlation (similarity) between the tuned spectra [Fig. 2(f)].

To characterize the spectral resolution of our system, we conduct a numerical simulation of the spectral reconstruction performance using Gaussian peaks centered at varying wavelengths. This testing method is a widely adopted way to determine the fundamental resolution constraints of reconstruction-based spectroscopy [21,22,24,33,35,41]. The simulation incorporates the tunable transmission spectrum set [as depicted in Fig. 2(f)] produced by the spinning-metasurface module and utilizes the same dictionary learning and compressive sensing algorithms previously discussed. Figure 4(i) reveals that Gaussian peaks with a 0.6 µm full width at half maximum (FWHM) centered above 10 µm can be precisely reconstructed with negligible mean square errors (MSEs). Additional simulations also show that our system is capable of detecting narrower spectral peaks with FWHM as low as 0.1 µm (see Supplement 1, Fig. S4 for details). We again observe that the performance of spectral reconstruction is dependent on wavelength. A higher level of accuracy is achieved at longer wavelengths due to the significant differences between the tuned transmission spectra in this region [Fig. 2(f)]. This implies that we could further enhance the spectral resolving power by refining the design of the metasurfaces and generate a broader array of mutually uncorrelated tunable transmission spectra.

We also obtain the polarimetric information including degree of the linear polarization (DOLP) and the angle of linear polarization (AoLP) using the designed system. For polarimetric imaging, we collectively rotate the spinning metasurfaces and the input polarizer, selecting four different polarizations (0°, 90°, 45°, and ${-}{45^ \circ}$) while maintaining the same spectral transmission. We use the first three Stokes parameters to quantify the polarimetric information associated with each pixel, i.e., ${S_0} = {I_0} + {I_{90}}$, ${S_1} = {I_0} - {I_{90}}$, and ${S_2} = {I_{45}} - {I_{- 45}}$, where ${I_0}$, ${I_{90}}$, ${I_{45}}$, and ${I_{- 45}}$ are the light intensity at polarization angles of 0°, 90°, 45°, and ${-}{45^ \circ}$, respectively. The DoLP and AoLP are then calculated at each wavelength through ${\rm DoLP} = \sqrt {S_1^2 + S_2^2} /{S_0}$ and ${\rm AoLP} = {\rm arctan}({S_2}/{S_1})$. As shown in Figs. 4(g) and 4(h), we can clearly distinguish between different letters based on their polarimetric signatures in the thermal radiation signal. The four-dimensional spatial-spectro-polarimetric data-tesseract provides significantly more insight associated with an object, making it a powerful tool for a wide range of imaging applications.

To prove the effectiveness and reveal the potential of the Spinning MetaCam, we conduct numerical tests demonstrating its applicability in material classification. Figure 5(a) shows that our spinning metasurfaces can accurately reconstruct the LWIR thermal radiation spectra of a diverse range of materials commonly found in everyday life (ground truth obtained from the ECOSTRESS Spectral Library [42,43]). These results also highlight the spectral fidelity and universality of our spectral reconstruction method. Employing these thermal radiation spectra, we execute material classification by identifying the material whose ground truth spectrum exhibits the least mean square error (MSE) when compared to the reconstructed spectrum. We repeat this spectral reconstruction and material detection procedure 100 times for each material and use the outcomes to generate a confusion matrix (see Supplement 1 for details). Figures 5(b) and 5(c) compare the resulting confusion matrices derived with and without the use of spinning metasurfaces, while maintaining a consistent noise level of 5%. Without the spinning metasurfaces, material classification relies solely on the integrated intensity of the thermal radiation over the LWIR region. This approach exhibits a low precision for most materials, being effective only for those materials with substantially disparate thermal radiation intensities, such as metallic paint. In stark contrast, material classification significantly improves in accuracy when the spectra collected by the Spinning MetaCam is utilized. We note that instances of lower accuracy only arise when there is a pronounced similarity in the LWIR spectra of materials, as observed between construction glass and felsic. For a more quantitative analysis, we perform the material classification test at varying noise levels ranging from 1% to 10%, and subsequently calculate the mean average precision (mAP) [44]. Representing the overall accuracy of classification for all classes of materials at all different noise levels, the mAP reaches an impressively high value of 0.834, providing a significant improvement from the low mAP of 0.293 obtained without the spinning metasurfaces.

5. CONCLUSION

Our results provide an innovative approach for spectro-polarimetric thermal imaging by combining meta-optics and computational spectral reconstruction. The low-SWaP (size, weight, and power) system opens the door for physics-driven machine vision. The high-dimensional thermal image data can significantly improve the performance of tasks such as depth estimation, object detection, and semantic segmentation when only radiative heat signal is available. Furthermore, we foresee that spectro-polarimetric thermal imaging can also be a powerful tool for scientific research, allowing for non-destructive characterization in the infrared region to investigate a wide range of novel physical phenomena, such as anisotropic thermal conduction [45] and directional or nonreciprocal radiative heat transfer [46–48]. Overall, our work provides a key development in the rapidly growing field of thermal imaging, offering a pivotal technology for heat-assisted detection and ranging.