1. Introduction

In an imaging system, each incoming light ray carries abundant information, which is described as the plenoptic function by Adelson and Bergen [1]. This seven-dimensional (7D) plenoptic function, $P\left(x,y,z,\theta ,\phi ,\lambda ,t\right)$, gives the spatial $\left(x,y,z\right)$ and angular $\left(\theta ,\phi \right)$ coordinates, wavelength $\left(\lambda \right)$, and time $\left(t\right)$ of an incoming light ray. Gao and Wang [2] further expanded the function to nine dimensions by including the polarization orientation and ellipticity angles $\left(\psi ,\chi \right)$. Conventional imaging systems which only record the information from two dimensions ($x,y$), miss a mass of messages delivered by the light rays.

Spectral information in the wavelength $(\lambda )$ reveals the chemical and molecular characteristics of the object [3–6]. Spectral imaging has been widely used in many fields, such as biology and biomedicine [3,7], remote sensing [4,8], and food quality control [5,6], due to its three-dimensional (3D) ($x,y,\lambda$) imaging capabilities. On the other hand, the morphological and functional information of the object, carried by the volumetric spatial data$\left(x,y,z\right)$, can be very useful in biomedical imaging [9–12], photography [13], object recognition [14,15], and particle image velocimetry [16,17]. In the past few decades, numerous efforts have been made to push the limits of imaging systems and capture light information in more dimensions. In recent years, spectral-volumetric imaging, which is capable of capturing a four-dimensional (4D) datacube ($x,y,z,\lambda$) of the incoming light rays, has drawn considerable interest with applications in biomedicine [18,19,29], remote sensing [20,21], object recognition [22], and multimedia [23]. Most spectral-volumetric imaging systems rely on scanning [18–25] or multi-shot [26,27] which require prolonged acquisition time that restricts their use in dynamic imaging. Fusing data from multi-sensor [28–31] is a feasible strategy to capture a 4D datacube ($x,y,z,\lambda$) in a single snapshot, but these systems are bulky and suffer from alignment error. Feng et al. have reported on compressed sensing to achieve snapshot 4D spectral-volumetric imaging [32]. However, this technique is computationally expensive and highly relies on the ill-posed assumption that the 4D datacube $\left(x,y,z,\lambda \right)$ is sparse in a given domain.

Here, we demonstrate a snapshot spectral-volumetric imaging (SSVI) system by introducing the paradigm of light-field imaging into Fourier transform imaging spectroscopy. In the SSVI system, we employ a single detector to record the light-field image coupled with the interference from a birefringent polarization interferometer. A convolutional neural network has been developed to decouple the light-field image and the interference. We derive the 3D volumetric datacube $\left(x,y,z\right)$ and 3D spectral datacube $\left(x,y,\lambda \right)$ from the light-field image and the interference, respectively. Combining these two datacubes with the 4D light-field datacube $\left(x,y,\theta ,\phi \right)$ gives the complete, 7D plenoptic function, $P\left(x,y,z,\theta ,\phi ,\lambda ,t\right)$. To our knowledge, this is the first time a complete plenoptic function of light field has been recorded by a single detector. Compared with multi-sensor-based systems [28–31], SSVI is compact, robust, and inexpensive. In addition, SSVI preserves the high-frequency details of the object in both spatial and spectral domains.

2. Methods

2.1 Snapshot spectral-volumetric imaging system

The configuration of the SSVI system is shown in Fig. 1. The incident light from an object is first imaged by an objective lens, forming the first intermediate image at a field stop. This intermediate image is relayed by a relay lens to a virtual intermediate image, indicated as the second intermediate image in Fig. 1(a). After that, an elemental image (EI) array, denoted as the third intermediate image, is formed through a microlens array (MLA). Finally, the third intermediate image is relayed by a relay lens onto a charge-coupled device (CCD).

 

Fig. 1 Snapshot spectral-volumetric imaging system configuration. (a) System sketch. MLA, microlens array; CCD, charge-coupled device; EI, elemental image. (b) Configuration of the birefringent polarization interferometer (BPI). Components between the polarizer I and the Wollaston prism have been omitted for clarity. The red arrow and circle indicate the polarization eigenmodes of the Wollaston prism, while the black ones denote the polarization of light rays. The transmission axes of the two polarizers are both oriented at 45° with respect to the polarization eigenmodes of the Wollaston prism. (c) The rotated BPI and CCD. Other components have been omitted.

Download Full Size | PDF

The SSVI system incorporates a birefringent polarization interferometer (BPI) which contains two polarizers and a Wollaston prism, as shown in Fig. 1(b). The Wollaston prism was custom-made from quartz. Based on the birefringence of the prism, the BPI separates the incident light rays into two paths which are then converged by relay lens II and interfere on the CCD. As depicted in Fig. 1(c), the BPI is rotated about the $z$-axis by a small angle ($\delta$) with respect to the $y$-axis. Thus, the theoretical optical path difference (OPD) on the CCD can be derived as [33,34]

The MLA design choice affects the resolution in the lateral $\left(x,y\right)$, spectral $\left(\lambda \right)$ and angular $\left(\theta ,\phi \right)$ dimensions. In a focused light-field camera, the lateral resolution is determined by the pitch (microlens diameter) and numerical aperture (NA) of the MLA, while the angular resolution is controlled by the number of the microlenses in the MLA [35]. On the other hand, the spectral resolution of a BPI-based imaging spectrometer is related to both the pitch and number of the microlenses in the MLA [33,34]. Here, we employ an MLA which contains 15 × 15 microlenses with a pitch of 1mm × 1mm. The focal length of each microlens is 10.9mm which gives $NA=0.0459$. A bi-telecentric lens with 0.44 × magnification is used to relay the third intermediate image onto the CCD. In the BPI, we employ a customized Wollaston prism with a wedge angle $\alpha =7.6°$ and an $x$ offset $x_0~1.5\text{mm}$, which makes the largest OPD around 28 $\text{μm}$.

2.2 Reconstruction algorithm

Mathematically, we denote the plenoptic function of an incoming light ray from the object as $P(x,y,z,\theta ,\phi ,\lambda ,t)$. This can be reduced to $P(x,y,\theta ,\phi ,\lambda )$ under the assumptions that the function does not vary within a single integration time of the detector and the radiance along a ray is a constant [36]. Without loss of generality, we can consider only the rays in the $yoz$ plane and the function can be further simplified as $P(y,\phi ,\lambda )$.

For a centered lens system, the output ray vector $[y,\phi ]\text{'}$ at the image point can be derived as [37]

$\text{Θ}$, $\text{Φ}$, and $\text{Λ}$ are the detection range of the SSVI system in the two-dimensional propagation-angle domain and the spectral domain, respectively. $d_m$ and $d_n$ are the distances from the optical axis to the center of the microlens, through which the light rays pass, along the $x$- and $y$-axis respectively. If we assume a Lambertian surface, the spectra $\left(\lambda \right)$ of light rays are independent of the propagation angles $\left(\theta ,\phi \right)$ and Eq. (5) can be decomposed as

Figure 2 shows the image processing pipeline, which consists of four steps, namely (I) light-field-interference decouple, (II) depth reconstruction, (III) interferogram extraction, and (IV) spectral datacube reconstruction. As indicated in Eq. (7), the light-field and interference are multiplicatively coupled in the raw image $I\left(x,y\right)$. Therefore, the first step is to decouple the light-field and interference, or in other words, to remove the interference from the raw image. Convolutional neural network (CNN) is a powerful tool which has been successfully used to remove image patterns, e.g. rain streaks [38] and dirt on a window [39]. Moreover, the interference in the raw image captured by SSVI is much simpler than rain streaks or dirt on a window, because it tilts in a specific angle and its frequency is limited by the spectral response of the CCD. Herein, we develop a CNN, as shown in Fig. 2(a), to decouple the light-field and interference. We refer to the proposed decoupling convolutional neural network as DC-CNN. To convert the multiplicative coupling mode into an additive one, we take the logarithm of the raw image, i.e.

 

Fig. 2 Flowchart of image processing pipeline. (a) Framework of the light-field-interferogram decoupling convolutional neural network (DC-CNN). Log, logarithm; Conv, convolution; ReLU, rectified linear unit; BN, batch normalization. (b) Light-field image. (c) An epipolar plane image extracted from the light-field image. (d) Disparity map. (e) Depth map. (f) An epipolar plane image extracted from the raw image. (g) Interferogram of a single pixel. (h) Spectrum of a single pixel. FFT, fast Fourier transform. (i) Spectral datacube.

Download Full Size | PDF

Inspired by the residual learning strategy [40], we set the interference instead of the light-field image of objects as the learning target, because the light-field image is much more complex than the interference of objects. As shown in Fig. 2(a), the DC-CNN architecture consists of two hidden layers and an output layer. The forward propagation of our network can be represented by three operations

To generate the training set, we used the spectral images from the ICVL spectral data set released by Arad and Ben-Shahar [41]. These images were captured by a line scanner camera (Specim PS Kappa DX4 hyperspectral) with a lateral resolution of 1392 × 1300 over 519 spectral bands, and then downsampled to 31 spectral channels from 400nm to 700nm. We further downsampled the spectral images to the same size of a single elemental image captured by the SSVI system and replicated each downsampled image by 5 × 5. Then a synthesized elemental image array with interference, $I$, was derived through Eqs. (1) and (7), while a corresponding elemental image array without interference, $E$, was simply calculated by accumulating the slice images over all spectral channels. We define the loss function of the DC-CNN as

In step II, we consider the MLA as an array of stereo cameras and arrange the elemental image array into a four-dimensional datacube $LF\left(x,y,m,n\right)$, where $m$ and $n$ index the elemental images along the $x$- and $y$-axis, respectively. A 2D $y-n$ slice of the datacube, dubbed an epipolar plane image (EPI), is shown in Fig. 2(c). The tilting angle of the ‘line’ structure, $\beta$, in the EPI corresponds to a certain depth of that point in the object space [43]. The disparity between the top and bottom elemental images can be derived by $D=h\cdot \tan \left(\beta \right)$, where $h$ is the width of the EPI along the $n$-axis. We adopt a disparity estimation algorithm based on scale-depth space transform [44] to reconstruct the disparity map. Finally, the depth map of the object can be derived through a calibration procedure [45].

In step III, we extract interference vectors from the raw image under the guidance of the tilting angles derived in the last step. Figure 2(f) shows an example of the EPIs reconstructed from the raw image. The red dashed line indicates the tilting angle at that position. The vector extracted along the red dashed line from bottom to top is a portion of the corresponding interference vector. Concatenating all the vectors extracted from the EPIs along the $m$-axis gives the complete interference vector.

Mathematically, we denote the extracted interference vector as ${\left[L{F}_{raw}\left(x_i,y_i,m_i,n_i\right)\right]}_{i=1,2,\dots ,P\cdot Q}$, where $L{F}_{raw}\left(x,y,m,n\right)$ is the light-filed datacube reconstructed from the raw image. $P$ and $Q$ are the number of elemental images along the $m$- and $n$-axis, respectively. Here, we consider the central elemental image as a reference. According to the extracting strategy described above, the coordinates, $\left(x_i,y_i,m_i,n_i\right)$, can be calculated by

We can get an OPD vector with equal intervals by setting $\delta ={\text{tan}}^{-1}\left(1/Q\right)$ [33,34]. Figure 2(g) plots an example of interference vector at different OPD values.

In step IV, we derive the spectrum at point $\left(x_c,y_c\right)$ by taking the Fourier transformation of the interference vector ${\left[L{F}_{raw}\left(x_i,y_i,m_i,n_i\right)\right]}_{i=1,2,\dots ,P\cdot Q}$ along the OPD vector ${\left[OPD\left(x_i^{raw},y_i^{raw}\right)\right]}_{i=1,2,\dots ,P\cdot Q}$. Figure 2(h) plots the derived spectrum corresponding to the interference in Fig. 2(g). Finally, following this procedure pixelwise yields a reconstructed spectral datacube $S\left(x,y,\lambda \right)$ [Fig. 2(i)].

3. Experiment

To demonstrate the concept of SSVI, we built a prototype (Fig. 3) based on the description in Section 2. We employed two commercial lenses (Canon EF 50mm f/1.8 and EF 50mm f/1.4) as the objective lens and relay lens I, respectively. A monochromatic CCD (BM-500GE, JAI) with 2058 × 2456 pixels was used as the detector. Before the following experiments, we calibrated the system in both spectral- and depth-dimension by the methods described in Ref [34] and [45], respectively.

 

Fig. 3 Prototype of the snapshot spectral-volumetric imaging system.

Download Full Size | PDF

3.1 Real-time spectral-volumetric video

To highlight SSVI’s single-shot multi-dimensional-imaging capability, we visualized a dynamic scene consisting of a static white board with black letters and a green leaf swinging along the line of sight with a period of $~2\text{s}$ [Fig. 4(a)]. The scene was located at a distance of $~0.85\text{m}$ from the SSVI system and a halogen lamp (MI-150, Edmund) was used for illumination. Using the reconstruction algorithm described in Section 2, we derived a real-time 4D $\left(x,y,z,\lambda \right)$ video with a frame rate of 15Hz (Visualization 1). Figures 4(b)-4(i) show the depth map and reconstructed image of eight example frames from the video. Here, the reconstructed images are derived by accumulating the datacube over all spectral channels. As a result, we are able to visualize the real-time motion of the leaf in 3D $\left(x,y,z\right)$. To demonstrate the refocusing ability of the SSVI system, we reconstructed two videos when the system focuses at 450mm and 850mm (Visualization 2). Figures 4(j)-4(m) illustrate four example frames in each reconstructed video.

 

Fig. 4 Real-time spectral-volumetric video. (a) Illustration of the scene including a static paper with letters and a swinging leaf. (b-i) Samples of consecutive depth and image frames reconstructed at 15 Hz frame-rate with a lateral resolution of 110 × 110 pixels. (j-m) Samples of image frames reconstructed when the system focuses at 450mm (upper row) and 850mm (lower row). Note: the full video can be seen in Visualization 1 and Visualization 2.

Download Full Size | PDF

In this experiment, we attached an L-shaped green paper on the swinging leaf. Figure 5(c) shows a close-up of the leaf captured by a commercial RGB camera. A frame of the reconstructed-image video at $t=1.6\text{s}$ is depicted in Fig. 5(a). Since the color of the green paper is quite close to that of the leaf, the contrast between the L-shaped paper and the leaf is low in Figs. 5(a) and 5(c). Figure 5(b) plots the spectra of point A and B which are located on the green paper and the leaf [Fig. 5(a)], respectively. The spectrum of point B increases dramatically around 700nm due to the absorbing property of chlorophyll. By contrast, the spectrum of point A varies smoothly from 500nm to 800nm. This phenomenon, whereby different spectra appear the same color to RGB cameras and human eyes, is referred to as metamerism. Unlike the monochromatic [Fig. 5(a)] and RGB images [Fig. 5(c)], the spectral slice at 728.6nm [Fig. 5(d)] shows high contrast between the L-shaped paper and the leaf. The image of the leaf can be further enhanced by taking the difference between the spectral slice at 728.6nm [Fig. 5(d)] and 673.9nm [Fig. 5(e)].

 

Fig. 5 Response to metamerism. (a) Reconstructed image at $t=1.6s$. (b) Spectra of points A and B indicated in (a). (c) High-resolution RGB image of the leaf captured by a commercial camera. (d) Spectral slice at 728.6nm. (e) Spectral slice at 673.9nm.

Download Full Size | PDF

3.2 Lateral resolution

We assessed the lateral resolution of SSVI by imaging a 1951 USAF resolution test target at ten steps along the optical axis. The lateral resolution was derived from the reconstructed image at each step. Figure 6(a) shows an example of the reconstructed images when the resolution target was located at 840mm from the imaging system, and Fig. 6(b) plots the experimentally measured resolution at ten steps with blue squares. The red dashed line indicates the diffraction limit which is greater than the experimental resolution of SSVI. We attribute this discrepancy to two reasons. First, the aberrations introduced by the optical lenses, especially by the MLA, blur the image. Second, the depth reconstruction error also degrades the imaging performance.

 

Fig. 6 Lateral resolution of SSVI. (a) Experimentally measured resolution at different working distance. (b) Reconstructed image when the working distance is 840 mm.

Download Full Size | PDF

3.3 Spectral resolution and accuracy

To evaluate the spectral resolution of the SSVI system, we imaged a black board illuminated by three lasers [Melles Griot, 25-LHP-925-230, 632.8nm (red); Oxxius, 532S-100-COL-PP, 532nm (green); Coherent, OBIS 488-60 LS, 488nm (blue)]. The board was located at a distance of $~0.9\text{m}$ from the imaging system. Figure 7(a) shows the reconstructed image where the three laser points are labeled as ‘Red’, ‘Green’, and ‘Blue’, respectively. The spectra at the three laser point areas are plotted in Figs. 7(b)-7(d). The solid line in each figure is the fitted Gaussian curve. The insets depict the spectral slice at 632.6nm, 531.6nm, and 487.1nm, respectively. We consider the spectra of these three lasers as delta functions and use the full width at half maximum (FWHM) of the measured spectra to characterize the spectral resolution of SSVI. As indicated in Figs. 7(b)-7(d), the spectral resolution of the system is 15.4nm, 10.7nm, and 8.4nm, corresponding to 384.6 ${\text{cm}}^{-1}$, 378.1${\text{cm}}^{-1}$, and 352.8${\text{cm}}^{-1}$ in the wavenumber domain, at 632.8nm, 532nm, and 488nm, respectively.

 

Fig. 7 Spectral resolution of SSVI. (a) Reconstructed image. (b-d) Spectra at the red, green, and blue laser point. The insets depict the spectral slice at 632.6nm, 531.6nm, and 487.1nm, respectively.

Download Full Size | PDF

To evaluate the accuracy and precision of the spectral-datacube reconstruction, we visualized a ColorChecker (X-Rite, MSCCPPCC0616) which was located at $~0.78\text{m}$ from the imaging system and illuminated by a halogen lamp (MI-150, Edmund). Figure 8(a) shows a photo of the ColorChecker captured by a commercial camera. The color blocks are labeled as ‘1’ to ‘16’, respectively. We employed a commercial fiber spectrometer (Avantes, AvaSpec-ULS 2048-USB2) with a spectral resolution of 1.15nm to measure the spectrum of each block. The comparison between the spectra derived from the Avantes spectrometer and SSVI in each block are shown in Figs. 8(c)-8(r), respectively. Considering the measurements from the Avantes spectrometer as references, we took 10 × 10 pixels in each block area and calculated the normalized root mean square (RMS) error of the corresponding spectra. The blue line in Fig. 8(b) plots the average and standard deviation of the normalized RMS errors in each block. The average normalized RMS error over all blocks is 6.87%. To compare SSVI with a snapshot-spectral-imaging modality, we transferred the SSVI system to the spectral imager described in [34] by extracting the interference vectors according to a registering procedure [46] instead of the disparity map. A spectral datacube, $S\left(x,y,\lambda \right)$, was then reconstructed by taking the Fourier transformation of the interference vectors. The red line in Fig. 8(b) plots the normalized RMS errors of the spectra derived by the spectral imager, where the average normalized RMS error over all blocks is 6.57%. Compared with the original spectral imager, SSVI well maintains the spectral accuracy when introducing the paradigm of light-field imaging.

 

Fig. 8 Quantitative evaluation of the spectral-datacube reconstruction. (a) Photo of the ColorChecker captured by a commercial camera. (b) Average normalized RMS errors of the reconstructed spectra in color blocks. (c-r) Reconstructed spectra from the SSVI system (Blue dots) and the Avantes spectrometer (Red line) in color blocks.

Download Full Size | PDF

3.4 Depth accuracy

To evaluate the depth accuracy of SSVI, we imaged a scene containing two white boards with black letters [Fig. 9(a)]. The back board is perpendicular to the $z$-axis, and the front board is tilted with respect to the $x$-axis by $45°$. As shown in Fig. 9(a), the yellow shadowed area on the back board is occluded by the front board. The distances from the boards to the imaging system were measured and considered as the ground truth [Fig. 9(b)]. We reconstructed the depth map [Fig. 9(c)] using the algorithm described in Section 2. We then calculated an error map representing the absolute difference between the reconstructed depth and the ground truth [Fig. 9(d)], where the RMS error was found to be 7.7mm. To compare SSVI with a light-field imaging modality, we derived a light-field image without interference by summing the two raw images when the transmission axis of polarizer I was oriented at $45°$ and $135°$ with respect to the $x\text{'}$-axis [Fig. 1(b)]. We employed the same algorithm [44] used in SSVI to reconstruct the depth map. Figure 9(e) shows the error map of the reconstructed depth from the light-field image without interference, where the RMS error was found to be 7.4mm. These results show that the integration of spectral imaging into a light field camera as in SSVI maintains the light field camera’s original depth accuracy.

 

Fig. 9 Depth accuracy of SSVI. (a) Schematic of the experimental setup. (b) Ground-truth depth of the scene. (c) Reconstructed depth. (d) Error map of the reconstructed depth from SSVI. (e) Error map of the reconstructed depth from the light-field image without interference.

Download Full Size | PDF

4. Discussion and conclusion

We have introduced a snapshot spectral-volumetric imaging system which can capture the 4D light-field datacube $\left(x,y,\theta ,\phi \right)$ coupled with the interference of the incoming light rays within a snapshot of a single detector. We also proposed a post-processing algorithm to reconstruct the corresponding 4D spectral-volumetric datacube $\left(x,y,z,\lambda \right)$ from the raw image. From these experiments, we have reconstructed a spectral-volumetric video $\left(x,y,z,\lambda ,t\right)$ of a dynamic scene. Combining the video with the light-field datacubes, we derived the complete plenoptic function, $P\left(x,y,z,\theta ,\phi ,\lambda ,t\right)$, of the incoming light rays. To our knowledge, this is the first single-detector imaging system to record the complete plenoptic function of light rays from an object. Additionally, we compare SSVI with two separate imaging modalities, i.e. the spectral and light-filed imaging modalities. The comparison results indicate that the SSVI system maintains both spectral and depth accuracy when integrating the two separate imaging modalities.

Compared to multi-camera systems [28–31], SSVI is compact, robust and inexpensive. Moreover, camera-array-based systems [29,30] capture the multispectral light-field by placing different broadband color filters in front of a camera array. This directly limits the optical throughput of the system as the inverse of the number of spectral channels. SSVI does not suffer this trade-off, as it relies on the Fellget multiplex advantage of a Fourier transform imaging spectroscopy. Additionally, the lateral, depth, and spectral resolution can be easily adjusted using different Wollaston prisms and MLAs in our system.

That being said, SSVI has a relatively low throughput because 75% of the light is lost in the BPI. One future avenue of development is replacing the first polarizer in BPI with a polarizing beam splitter (PBS) and a half-wave plate (HWP). Setting the fast axis of the HWP at 22.5° with respect to the $x\text{'}$-axis makes the polarization of the light rays passing through the PBS and HWP orient at 45° with respect to the $x\text{'}$-axis. We can use these light rays to reconstruct the 7D plenoptic datacube and employ the light rays reflected by the PBS to obtain a high-resolution monochromatic image [46]. The energy loss in this configuration is reduced to 25%. Moreover, the captured high-resolution monochromatic image can be used to improve the lateral resolution of the 7D plenoptic datacube by image fusion [46].

Currently, the post-processing of SSVI is implemented in Matlab 2015 by an Intel Core i7-8700 CPU. It takes less than ten minutes to reconstruct the complete 7D plenoptic datacube from a raw image. Since most computations in the post-processing algorithm can be processed in parallel, we believe that the computing time can be dramatically reduced by using a graphics processing unit (GPU).

It is worth noting that the reconstruction algorithm of SSVI is based on two assumptions. First, the radiance along a light ray is a constant. Equivalently put, the object scene is free of occlusions. Second, we consider the object as Lambertian. In recent years, several algorithms [47–49] have been developed to deal with the object scene with occlusions and specular surfaces in light field imaging. Adopting these algorithms in the reconstruction process of SSVI is our future work.

The experimental results presented here demonstrate that SSVI can be used to derive the complete plenoptic function, $P\left(x,y,z,\theta ,\phi ,\lambda ,t\right)$, of the light rays from a dynamic scene with information in the lateral, spectral, depth, and time domains. In light of its unprecedented 7D imaging capability, we anticipate that SSVI will facilitate a wide range of applications in biological analysis, robotic vision, and many other fields.