1. Introduction

Over the past few decades, computational imaging has made remarkable progress thanks to advancements in optical instruments and the exponential growth of computing power. One ultimate objective of computational imaging is to simultaneously capture the complete plenoptic function [1], a seven-dimensional (7D) function encompassing 3D space (x, y, z), 1D time (t), 1D spectrum (λ), and 2D angle (u, v) [2,3]. However, traditional optical imaging systems typically only measure light intensity over a 2D spatial (x-y) grid, overlooking a significant portion of the photon's information and leading to inefficient data acquisition.

To capture a high-dimensional (high-D) plenoptic function, most systems acquire a 1D or 2D subset of a datacube at a time, necessitating tedious scanning of the remaining dimension(s) to obtain a complete datacube. For instance, hyperspectral imagers, which capture a 3D plenoptic datacube (x, y, λ), often perform scanning either in the spatial domain [4,5] or the spectral domain [6–8], resulting in low light collection efficiency and prolonged acquisition times. Snapshot techniques like image mapping spectrometry (IMS) [9–12], computed tomography imaging spectrometry (CTIS) [13,14], and coded aperture snapshot spectral imaging (CASSI) [15–17] optically map a 3D plenoptic datacube to a 2D plane while maximizing the light throughput [18,19]. Among these techniques, CASSI stands out with its significant advantage in data acquisition efficiency, achieved by leveraging compressed sensing (CS) [20] to minimize the number of necessary measurements. The compression ratio, defined as $\boldsymbol{cr} = \boldsymbol{Nyquist\; sampling\; number}/\boldsymbol{Actual\; sampling\; number}$, plays a crucial role in reducing the data load of high-D measurements. However, the substantial data compression ratio inherent in CASSI can often render the reconstruction problem ill-posed, resulting in compromised image quality when capturing complex scenes. Recent efforts have been made to enhance the reconstruction quality by reducing the compression ratio [21–28]. Two major approaches include (i) capturing multiple snapshots with different coded apertures [21–23] and (ii) utilizing a non-diffracted image as a prior in the reconstruction [24–28]. While these methods improve reconstruction quality, they either require multiple shots or additional optics and cameras, thereby increasing system cost and complexity.

Capturing a high-D plenoptic datacube, such as hyperspectral volumetric imaging (x, y, u, v, $\lambda$), in a single snapshot while preserving high image quality poses a formidable challenge. Existing approaches often rely on Nyquist sampling with a unit compression ratio [29–32] or adopt complex multi-modal systems [29,33,34]. For instance, non-compressed IMS-based hyperspectral volumetric imagers [32] face a significant trade-off among sampling along various plenoptic dimensions due to limited detector array size. Multi-modal systems [29,33,34] integrate multiple cameras with distinct modalities and necessitate intricate calibration procedures. Acquiring a high-D datacube with a substantial compression ratio using a single camera remains demanding. In our previous research [35], we sought to address this challenge by employing compressed sampling of the light field through sparsely spaced angles. This approach allowed us to achieve a relatively high compression ratio, albeit with a trade-off of reduced light throughput due to the use of narrow slits.

To surpass the limitations of the previously mentioned techniques, herein we present coded aperture snapshot hyperspectral light field tomography (CASH-LIFT), a novel approach that employs a highly efficient cascaded compressed sensing scheme to achieve superior imaging results. Light field tomography (LIFT), which acquires en-face projections of perspective images [36,37], serves as the first compressed stage to undersample spatial and angular dimensions using a line spread function. The subsequent CASSI system functions as the second compressed module, further encoding the spectral dimension of the compressed light field image from the previous stage. By incorporating 1D optical blurring along the dispersion direction from the LIFT stage, we effectively distribute the computational load across each stage, thereby mitigating the ill-posed nature of the image recovery problem. This approach results in enhanced spatial resolution, suppressed aliasing artifacts, and improved spectral reconstruction fidelity. Furthermore, by leveraging the compressibility of natural scenes in multiple dimensions, CASH-LIFT enables highly efficient data acquisition to capture the 5D plenoptic function (x, y, u, v, $\lambda$) in a snapshot while maintaining a high light throughput.

2. Method

2.1 Cascaded compressed image formation model

CASH-LIFT introduces a two-stage cascaded compressed process to transform high-dimensional (high-D) plenoptic function into a 2D representation. The approach combines LIFT and CASSI techniques to sequentially perform compressed measurements across angular, spatial, and spectral domains. In the first stage, a dove prism array and cylindrical lens array capture compressed light field information, enabling synthetic refocusing using the resulting sub-pupil disparities. The second stage involves using a coded aperture and a dispersive element, such as a Wedge prism, to encode and disperse the compressed light field [ Fig. 1(a,b)]. Accordingly, we can separate the forward image formation model into two steps. The first step describes the light field compression by the LIFT process, and it can be formulated as:

 

Fig. 1. Optical schematic and image formation model. (a) Comparison of the working principles between CASSI and CASH-LIFT. ${C_{xy}}$, spatial encoding operator; ${S_x}$, spectral shearing operator; I, spatial-spectral integration; ${R_\theta }$, angular rotation operator; ${T_x}$, spatial integration operator. (b) The Dove prism array and cylindrical array are located at the Fourier plane of the main lens. The 2D light field sub-image is downsampled along the invariant axis of the cylindrical lens. A binary mask and a wedge prism modulate the LIFT output image and encode high-dimensional spectral data into 2D space. The wedge dispersed direction is parallel to the non-power axis of the cylindrical lens. (c) The reconstruction pipeline of our imaging method. Leveraging the reduced information content along the spectral dispersion direction, the spectral projection cube can be efficiently reconstructed in the CASSI stage, reducing computation load. The object can then be reconstructed in five dimensions ( $x, y, \theta, \varphi, \lambda$) with higher light throughput and spatial resolution by combining coded aperture imaging, computational tomography, and light field imaging.

Download Full Size | PDF

The subsequent stage involves the spatial-spectral scene encoding performed by CASSI:

Here ${\boldsymbol{C}_{\boldsymbol{xy}}}$ is a spatial encoding operator, which indicates the function of a random binary mask. ${\boldsymbol{S}_{\boldsymbol{x}}}$ is a spectral shearing operator along the dispersion direction of a diffractive element [$x$ axis in Fig. 1(a)]. $\boldsymbol{I}$ is a spatial-spectral integration operator. m is the raw measurement acquired by a monochrome camera, and e denotes the noise. The spatial-spectral encoding mechanism ($\boldsymbol{I}{\boldsymbol{S}_{\boldsymbol{x}}}{\boldsymbol{C}_{\boldsymbol{xy}}}$) can also be characterized by a sensing matrix $\mathbf{\Phi }$ (or transport matrix from the light transport perspective) [see Supplement 1]. In a conventional CASSI system, a compressed measurement of size $({N_x} + {N_\lambda }) \times {N_y}$ is used to reconstruct the complete hyperspectral datacube ${N_x} \times {N_y} \times {N_\lambda }$, resulting in a significant compression ratio ${N_x} \times {N_y} \times {N_\lambda }/[{({N_x} + {N_\lambda }} )\times {N_y}]$ [Fig. 1(a)]. In contrast, in our method, the convolution operation in the LIFT stage reduces the spatial information content along the spectral dispersion direction. As a result, the effective information that requires recovery in the CASSI stage is diminished by a factor of n, leading to a lower compression ratio $c{r_2} = {N_x}/n \times {N_y} \times {N_\lambda }/[{({N_x} + {N_\lambda }} )\times {N_y}]$. The reduction in effective information not only mitigates the computational burden of the CASSI system but also enables improved reconstruction quality.

Therefore, the image formation model of the entire system can be formulated as:

The overall compression ratio equals the multiplication of the compression ratio of each stage:

There are two special cases: (i) $n = 1.$ The system is equivalent to a combination of a conventional light field system and CASSI. The PSF of each sub-image has a circular shape, uniformly convolving the input scene in both x and y directions. The compression ratio of the CASSI stage remains the same as that of the original CASSI system. (ii) $n = {N_x}$. The resultant system has a long LSF and distributes most compression to the LIFT stage. To strike a balance in the computational burden between the stages, it is essential to ensure that both $c{r_1}$ and $c{r_2}$ are greater than one. Consequently, the parameter n must be chosen within an optimal range between ${N_\theta }$ and ${N_x} \times {N_\lambda }/({N_x} + {N_\lambda })$ to ensure optimal performance.

2.2 Image reconstruction

The reconstruction of a hyperspectral cube involves solving two optimization problems sequentially, which can be formulated as:

To extend our method to a 3D scene, we introduce a translation step for each sub-pupil image before passing it to the iterative reconstruction [35–37]. This translation is performed with respect to the spatial coordinates ($u,v$) of each Dove prism in the array. We translate the sub-pupil image a distance of the vector $s^{\ast}$($u,v$), where $s$ is a parameter that depends on the axial location of the synthetic focal plane. Provided that a Dove prism rotates the image counterclockwise by angle $\theta$, and the non-power direction of the cylindrical lens aligns with the sub-pupil coordinate axis $v$, we apply a corresponding shift to each sub-pupil projection [35]:

The quality of the estimated hyperspectral datacube obtained through the optimization algorithm heavily relies on the accurate characterization of sensing matrix $\mathbf{\Phi }$. System calibration plays a critical role in mitigating the disparity between the theoretical system operator $\boldsymbol{I}{\boldsymbol{S}_{\boldsymbol{x}}}{\boldsymbol{C}_{\boldsymbol{xy}}}$ and the experimental system-specific operator $\mathbf{\Phi }$. For our system, the calibration of dispersion shift was obtained by illuminating the mask using a broadband light source with an optical tunable bandpass filter. The spectral channel resolution can be calculated from the pixel shift (dispersion direction) between adjacent channels [see Supplement 1]. In the current setup, the spectral range and average resolution are 150 nm and 7.5 nm, respectively. It is important to note that the spectral range and resolution can be adjusted by employing different dispersion elements or utilizing a different focal length relay system. This allows for tailoring the system's imaging performance for specific applications.

2.3 Optical setup

The system schematic is shown in Fig. S1 in Supplement 1. The beam is collimated by a main lens (AC254-300, f = 300 mm, Thorlabs. Inc), which is located 300 mm from the object plane. A custom 16 Dove prism array (height = 2 mm, spacing between adjacent prism = 2.9 mm) is placed at the Fourier plane of the first lens. The 5 × 1 cylindrical lens array (f = 75 mm, height = 2 mm) covers all Dove prisms with an extended length of 17 mm. The LIFT output image is focused onto a coded mask and encoded by a random binary pattern. The mask was fabricated on a quartz substrate with chrome coating by photolithography (Frontrange-photomask, Inc). The smallest coded feature of the pattern is sampled by approximately 2 × 2 pixels on the camera (15 µm). We use two photographic lenses (50 mm, f/1.8, Yongnuo, Inc.) as a 4f system to relay the coded image to a monochromatic camera (16 bits, 7.4um pixel size, 4864 × 3232, Lt16059H, Lumenera, Inc). To disperse the image, we position a round wedge prism (PS814-A, 10° Beam Deviation, Thorlabs, Inc.) at the Fourier plane of the 4f relay system. For the illumination source, we use a halogen lamp (HL250-AY, AmScope, Inc.) bandlimited by a combination of a 450 nm long pass filter (FELH0450, Thorlabs, Inc.) and a 600 nm short pass filter (FESH0600, Thorlabs, Inc.). The field of view (FOV) of the system is approximately 8 mm × 8 mm, with a systematic magnification of 0.25×.

3. Simulation

To accurately evaluate the imaging performance of our CASH-LIFT system, we maintain consistency between the simulation parameters and the real experimental optical system. The mask size and spectrum used in the simulation were aligned with those employed in the physical setup. As depicted in Fig. 2(a), CASH-LIFT effectively mitigates the spatial blurring effect commonly encountered in CASSI systems, particularly for scenes with lower spectral sparsity. Additionally, it demonstrates the capability to resolve small line pairs with narrow spacing, as small as approximately 44um (3 times the minimal mask feature size). This indicates the system's ability to capture fine details and resolve high-frequency features, thereby enabling high-resolution imaging. Given that the LIFT stage utilizes multiple sub-images to capture light field information, we further conducted a comparison between our CASH-LIFT and CASSI employing a 16 sub-images sampling scheme, known as Light Field CASSI [Fig. 2(a), middle column]. In Light Field CASSI, the PSF of each sub-image has a circular shape and is uniformly convolved in both x and y directions. In contrast, CASH-LIFT efficiently down-samples light field information using a cylindrical lens with a line-shaped PSF, significantly reducing the reconstruction burden in the subsequent CASSI reconstruction and, thereby, improving the image quality. Figure 2(b) shows the reconstructed spectral curves with a mean squared error of the normalized spectral difference measured at 0.04 (compared to 0.06 in CASSI).

 

Fig. 2. Simulation results of USAF target/planar object reconstruction. (a) Comparison of spatial resolution between CASSI, light field CASSI, and CASH-LIFT at a representative spectral channel. (b) Evaluation of spectral reconstruction quality between CASSI and CASH-LIFT. (c) Comparison of reconstruction quality between Hyper-LIFT and CASH-LIFT under different SNR (Signal-to-Noise Ratio), characterized with PSNR (Peak Signal-to-Noise Ratio), and SSIM (Structural Similarity Index Measure) as metrics.

Download Full Size | PDF

In Fig. 2(c), we compare the robustness of Hyper-LIFT and CASH-LIFT by adding Gaussian and Poisson noise to the simulated raw image. We scaled the signal intensity and assessed the reconstruction quality accordingly. In Hyper-LIFT, a slit array is employed for 1-D sampling. However, this approach sacrifices light throughput, resulting in a decrease of less than 1% and consequently leading to poor reconstruction quality, particularly in low signal-to-noise ratio (SNR) scenarios. On the other hand, CASH-LIFT benefits from its higher light throughput design, achieving 50% light efficiency over the random binary mask. This larger photon budget enables CASH-LIFT to achieve enhanced reconstruction quality even in challenging low SNR conditions.

4. Experimental results

4.1 Enhanced resolution demonstration on a USAF target

The effectiveness of CASSI relies on the inherent sparsity observed in the spectral data of natural scenes. However, verifying the necessary conditions for high-fidelity compressed sensing reconstruction in practical coded aperture designs can be challenging, leading to limitations in the quality of the reconstructed spectral data [22]. In our experiments, we compared the performance of the CASSI system with our CASH-LIFT system using a United States Air Force (USAF) resolution target. The CASSI reconstruction suffered from spatial blurring, resulting in excessively smooth recovered images [ Fig. 3(a)]. In contrast, the two-stage compression in the CASH-LIFT system significantly reduces the reconstruction burden in each stage, thereby improving image resolution and mitigating spatial blurring effects. The contrast level, defined as $({{I_{max}} - {I_{min}}} )/({{I_{max}} + {I_{min}}} )$, is measured [Fig. 3(b)]. For line pair groups 1-4 to 1-6, the CASH-LIFT system demonstrates contrast levels of 1, 0.89, and 0.62, while the contrast levels of the CASSI system are 0.55, 0.23, and 0.12.

 

Fig. 3. Enhancement of spatial resolution by cascading LIFT and CASSI. (a) Comparison of reconstruction results between the CASH-LIFT system and CASSI system when imaging a USAF resolution target. The reconstruction result of a representative spectral channel is presented. (b) Pixel intensity profiles of sampling lines i, ii, and iii in (a).

Download Full Size | PDF

4.2 Hyperspectral imaging of planar objects

To further illustrate the resolution enhancement, we compared our method with CASSI on a planar object, using the Structural Similarity Index Measure (SSIM) and Peak-to-Noise Ratio (PSNR) as metrics to assess the reconstruction quality [ Fig. 4(a)]. The broadband light spectrum was captured using both our system and a reference fiber spectrometer (O STS-VIS-L-25-400-SMA, Ocean Optics Inc.). The reconstructed spectral intensity distribution was normalized and compared to the ground truth provided by the spectrometer [Fig. 4(b)].

 

Fig. 4. Validation of reconstruction quality and synthetic refocus. (a,b) Comparison of reconstruction results between the CASH-LIFT system and CASSI system when imaging a sparse object. The reconstruction quality is assessed using SSIM and PSNR metrics. The spatial recovery performance is related to spectral sparsity, as indicated by the red dashed box representing single-wavelength illumination (BW = 1 nm), while the remaining image corresponds to broadband illumination (BW = 150 nm). We observe significant spatial performance improvement while maintaining similar spectral performance compared to CASSI. (c) The resolving capability gains robustness in dealing with samples of different orientations. (d) Reconstruction image of an artificial eye phantom depicting a complex scene with sparse features. (e) Validation of synthetic refocusing. The reconstruction result of a representative spectral channel is presented. The relationship between sub-image positions and the ability of our system to restore sharp focus to a previously defocused object by shearing. This plot demonstrates the dependence between the degree of shearing and the object's depth.

Download Full Size | PDF

As demonstrated in Fig. 4(a), the fidelity of CASSI reconstruction is highly influenced by the sparsity in both the spatial and spectral domains. Under sparse spectrum conditions, such as the scenario with 1 nm bandwidth (indicated by the red dashed box), CASSI can resolve high-spatial-frequency components (sharp features and edges) of the object in the reconstruction. However, the reconstruction image exhibits more artifacts and spatial blurring when dealing with broadband spectrum conditions. This is due to the correspondence between spatial blurring in CASSI and the sparsity in the spectral domain. In other words, CASSI encounters limitations in image quality when reconstructing less spectrally sparse scenes. By effectively distributing the computational burden to both stages, CASH-LIFT mitigates the sparsity requirement in the spectral domain for CASSI reconstruction, leading to improved reconstruction quality. Furthermore, the resolving capability of CASSI is also influenced by the relative orientation of the object with respect to the dispersion direction. In Fig. 4(c), the CASSI reconstruction quality of the vertical and oblique bars is relatively lower compared to the horizontal bar. This discrepancy is attributed to the spatial blurring introduced by spectral shearing, which predominantly affects the dispersion direction (horizontal). In contrast, CASH-LIFT achieved better image quality and improved resolving ability for objects with different orientations.

We also tested our system by imaging a standard eye phantom (wide-field model eye, Rowe Technical Design), which features a realistic eye lens and a vasculature-like pattern painted on the back surface [Fig. 4(d)]. We observed that sparse objects with fine spatial details and complex spectra benefit the most from our method. Moreover, CASH-LIFT captures light field information, enabling 3D reconstruction. We evaluated the synthetic refocusing performance by placing a planar object at different depths relative to the imaging system's object plane. For each depth, we captured an image and reconstructed a series of focal stack images for each spectral channel, as depicted in Fig. 4(e). By shifting each projection accordingly during post-processing, the object comes into focus at its specific depth.

4.3 Dynamic hyperspectral volumetric imaging

CASH-LIFT's snapshot 3D hyperspectral imaging capability enables simultaneous recording of a hyperspectral datacube at different depths. We demonstrated this capability by imaging the letter ‘USAF’ on a resolution target (R3L3S1N, Thorlabs Inc) illuminated with rainbow light [ Fig. 5(a)]. To achieve a broad-spectrum projection, we utilized a linear variable visible bandpass filter (LVF) (88365, Edmund Optics Inc) placed in front of a broadband halogen light source (HL250-AY, AmScope Inc). The LVF has a coating thickness that linearly varies along one dimension, resulting in continuous variation in spectral transmission.

 

Fig. 5. Dynamic 3-D hyperspectral imaging experiments. (a) An illumination system was designed to capture 3-D hyperspectral video. The linear variable filter creates rainbow illumination, while the tunable filter linearly scans the center wavelength. The object was mounted on a motorized stage controlled by a computer, and the tunable filter, motorized stage, and camera were synchronized. (b) The upper row displays representative frames of the letters, while the lower row shows the letters placed at another depth. Through synthetic refocus in post-processing, clear images of dynamic scenes at various depths can be reconstructed.

Download Full Size | PDF

To project the broad spectrum onto the field of view, we used a lens pair with a focal length ratio of 3.3:1 (MAP1030100-A, Thorlabs Inc) to demagnify the linear filter and relay it to an intermediate plane where the letter object was positioned. The tunable filter temporally sweeps the center wavelength through the entire spectrum band with a speed of 10 nm/s. This configuration ensured that at each frame, only one lateral location of the object exhibited a distinct color. Furthermore, the letter object was mounted on a motorized stage (PT1-Z8, Thorlabs Inc) that moved at an approximate speed of 2 mm/s along the depth direction. We reconstructed a sharp image sequence (12 Hz) for each spectral channel and depth using synthetic refocusing in post-processing [Fig. 5(b), Visualization 1]. This approach allowed us to obtain dynamic imaging results capturing the variation of objects or illumination over different depths in the scene.

4. Discussion and conclusion

Cascaded compressed imaging offers a highly efficient approach for capturing high-dimensional light datacubes. In conventional multidimensional imaging, acquiring optical information along an additional dimension can significantly increase the data load, presenting challenges in terms of data transfer, storage, and processing. However, cascaded compression addresses this issue by compressing the optical information along the desired axis of interest and mapping it onto a lower-dimensional spatial axis. This approach enables efficient data acquisition by capturing and representing only the essential information rather than measuring the complete dataset. For instance, in the CASH-LIFT system, instead of capturing duplicated sub-aperture images with disparity information as done in conventional light field systems, LIFT provides an efficient means to capture light field information using cylindrical lenses to reduce the data dimensions. The snapshot capability and high data acquisition efficiency of CASSI distinguish it from traditional scanning-based spectral imagers.

Our two-stage compressed imaging scheme can be extended to measure other plenoptic dimensions by adapting different coded aperture systems as the second stage, incorporating spatial encoding, shearing, and integration operators. The first stage can utilize the LIFT module to obtain the projection that downsamples spatial information along the desired shearing direction, leveraging optical blurring as a prior during reconstruction to reduce the computational burden. While currently combining LIFT and CASSI for (x, y, u, v, λ) imaging, the cascaded compressed framework can be expanded to capture additional dimensions of light, such as the time (t) dimension using a streak camera [41], and the polarization (p) dimension using a birefringent crystal [42,43].

Moreover, the competitive advantage of distributing the computational burden in cascaded compressed imaging extends beyond the CASSI stage (encoding, shearing, and integration stage) and also applies to the reconstruction of the LIFT stage. Similar to sparse-view computed tomography (CT), LIFT reconstruction is prone to artifacts and anisotropic resolution due to the limited number of projection measurements [35]. However, in the case of CASH-LIFT, instead of utilizing a narrow slit to select a single line of the projection image as in Hyper-LIFT, the entire frame is preserved as the first stage measurement. Although this results in a reduced spectral resolution, it broadens the power spectrum of each sub-aperture image captured. Under the same number of projection conditions, the additional sampling information in the Fourier domain contributes to improved reconstruction quality [Supplement 1].

Further improvement is possible by implementing appropriate regularizations and adjusting weighting coefficients to adapt to different scene types. In our method, the use of total variation (TV) regularization has proven effective in preserving image quality during optimization, with first-order TV regularization preserving sharp edges and second-order TV regularization resolving smooth transitions [44]. By synergistically combining these regularizers, we can elevate the image reconstruction process to achieve a level of superior image quality beyond what each regularization method can accomplish individually. Additionally, integrating deep learning techniques holds great promise for enhancing snapshot compressive imaging and efficiently solving ill-posed inverse problems while reducing computation time [45,46]. Embracing these advancements will lead to even better results in various applications.

In conclusion, we have successfully developed a novel cascaded compressed imaging scheme, CASH-LIFT, which efficiently maps high-dimensional datacubes to a 2D sensor through cascading LIFT and CASSI techniques. This innovative approach enables the simultaneous capture of high-dimensional information of the plenoptic function from a single 2D acquisition with a single camera, setting it apart from other hybrid and multi-model-based hyperspectral light field imaging systems. Our CASH-LIFT system demonstrated impressive 5D hyperspectral volumetric dynamic imaging over a substantial spatial volume (∼8mm × 8mm × 12.5 mm) and a spectral range of 150 nm (25 spectral channels) with high reconstruction quality. Notably, the cascaded compressed system design, employing light field sub-image downsampling in the LIFT stage, significantly enhances the reconstruction quality of CASSI, especially in less sparse spectrum cases, while retaining its snapshot advantage. This unique advantage positions the cascaded compressed CASH-LIFT approach as a promising solution to address the challenges in multidimensional optical imaging and have a major impact on the broad applications in both basic and applied sciences, such as snapshot optical coherence tomography and 3D multispectral fluorescence microscopy.