Light field imaging can record two-dimensional (2D) spatial $({x,{}y})$ and 2D angular $({\theta ,\varphi})$ information of incident rays [1]. The captured four-dimensional (4D) information endows light field imaging with attractive advantages, including multiple-perspective viewing, digital refocusing, and depth estimation [2]. To date, light field imaging has been widely implemented in microscopy [3–6], photography [7], and endoscopy [8,9]. In existing configurations, microlens arrays (MLAs) are popularly used to sample $({x,{}y})$ information in a field of view and fill in local voids with $({\theta ,\varphi})$ information [4]. Nonetheless, the induced trade-off poses challenges for MLA-based light field imaging to attaining high spatial resolution and high angular resolution simultaneously [10].

To overcome this problem, many efforts have been taken to capture 4D light fields with a camera’s full pixel count [11–14]. Among these approaches, coded-aperture light field (CALF) imaging has sparked increasing research attention [15–21]. This modality uses single or multiple masks to encode the system’s aperture. Despite retaining a camera’s full pixel count, early CALF imaging systems had various limitations, including low pattern adaptability to scenes [22], long acquisition time, and additional error due to pattern misalignment [16].

To improve the flexibility, efficiency, and accuracy in CALF imaging, liquid-crystal spatial light modulators (LC-SLMs) have been implemented for aperture encoding [16–19]. Without any mechanically moving parts, LC-SLMs eliminate the error from mask misalignment. However, these systems suffer major drawbacks in contrast (due to imperfect polarization selectivity [23]), stability (due to the flicker noise [24]), and speeds (due to LC’s limited responsible time [19,25]). Thus far, CALF imaging of dynamic scenes at video rate is rarely performed.

Digital micromirror devices (DMDs) are a promising candidate to solve these problems [26,27]. As a 2D binary amplitude SLM [28], a DMD consists of up to millions of micromirrors, each of which can be independently tilted to either ${+}{{12}}^\circ$ or ${-}{{12}}^\circ$ from its surface normal to reflect incident light to one of the two directions as an “ON” or “OFF” pixel. This operating principle enables DMDs to produce high-contrast binary images. As a micro-electromechanical device, a DMD can generate binary patterns at up to tens of kilohertz [29]. Leveraging these technical advantages, DMDs have been used in phase-space measurements [30] (i.e., the physical-optics equivalence of light field imaging [31]). By placing a DMD on the Fourier plane to rapidly create and scan sub-apertures, this system recorded light field images of three red light-emitting diodes (LEDs) at different depths. However, acting as a diffraction grating, the DMD induced severe spatial dispersion in the acquired images for broadband light [23]. Consequently, a color filter with a 10-nm bandwidth was inserted to lessen the dispersion problem [30]. Thus, the DMD-based CALF imaging could be demonstrated only with static objects using narrowband light.

To surmount these limitations, we have developed a new dispersion-eliminated (DE) CALF imaging system for broadband light field imaging at video rate, as shown systematically in Fig. 1. Design details are presented in Fig. S1. Broadband light from an object is imaged by a ${{4}}f$ imaging system consisting of lenses L1 and L2. DMD1 (Texas Instruments, Discovery 4100, ${{0}.{{7}}^{{\prime \prime}}}$ XGA), placed on the back focal plane of L1, spatially disperses the incident light so that a spectrally smeared image of the object is formed on the intermediate image plane. This image is relayed to the final image plane by another identical ${{4}}f$ imaging system consisting of lenses L3 and L4 and DMD2 (Texas Instruments, Discovery 1100, ${{0}.{{7}}^{{\prime \prime}}}$ XGA). Because the two ${{4}}f$ imaging systems are symmetrical about the intermediate image plane, DMD1-induced dispersion is compensated for. In this way, a clear image of the object is formed on a high-speed color camera (PCO, 1200 hs) placed at the final image plane. In operation, DMD1 divides the system’s aperture into ${{5}} \times {{5}}$ square sub-apertures, each of which contains ${{50}} \times {{50}}$ micromirrors and has a 50% overlap with adjacent ones. An all-OFF pattern is loaded onto DMD2. The camera is synchronized with DMD1. Overall, the DECALF system acquires ${{1280}} \times {{1024}} \times {{5}} \times {{5}}$ $({x,y,\theta ,\varphi})$ light fields at 20 Hz.

 

Fig. 1. Schematic of the dispersion-eliminated coded-aperture light field (DECALF) system. DMD1–DMD2, digital micromirror devices; L1–L4, lenses (Thorlabs, AC254-100-A); M1–M2, mirrors (Thorlabs, PF10-03-G01).

Download Full Size | PDF

The characterization of the DECALF system was carried out by imaging a negative resolution target illuminated by a white LED (Thorlabs, MNWHL4) with a 400 nm–700-nm spectrum. An all-OFF pattern was loaded onto DMD1. The image captured on the intermediate image plane [Fig. 2(a)] shows severe dispersion induced by DMD1. In contrast, a clear image of the resolution target was captured at the final image plane [Fig. 2(b)], demonstrating that the dispersion is compensated for. The minimum resolvable feature sizes were quantified as 22.10 µm (group 4, element 4) in the horizontal direction [Fig. 2(c)] and 19.69 µm (group 4, element 5) in the vertical direction [Fig. 2(d)], both of which agree well with the theoretical values. The slight difference in the two directions is likely attributed to the unmatched surface curves of the two DMDs [32,33]. In addition, the axial resolution, depending on the camera’s pixel size and the system’s angular resolution [34], was determined to be 1.24 mm. Finally, the imaging volume, relying on the ($x,y$) field of view and the depth of field of perspective images, was quantified to be ${15.36} \times {12.29} \times {97.56}\;{\rm{mm}}^3$.

 

Fig. 2. Characterization of the DECALF system. (a) Image on the intermediate image plane. (b) Image on the final image plane. (c), (d) Averaged horizontal and vertical line profiles of selected elements on the resolution target [marked by red and blue lines in (b)]. Error bar: standard deviation.

Download Full Size | PDF

To demonstrate the DECALF system’s performance, we imaged a static three-dimensional (3D) color scene. As shown in Fig. 3(a), the incident white LED light was filtered by a multi-color filter (Izumar, Multi-color 58 mm). After that, a hollow maple-leaf mask and a “1X” symbol were placed at two depths separated by 64 mm. Perspective images were captured by sub-aperture scanning (see Visualization 1). Figure 3(b) shows four representative perspective images captured by opening the leftmost, rightmost, topmost, and bottommost sub-apertures, respectively. The first two panels illustrate the horizontal shift between the 1X symbol and the maple-leaf mask by opening two different sub-apertures along the horizontal direction. Similarly, the vertical shift is evident in the last two panels, corresponding to the opening of two sub-apertures along the vertical direction. Moreover, all perspective images retain the full pixel count of the deployed color camera. Using these perspective images, we digitally refocused the 3D scene to the front, to the back, and over the entire scene [Fig. 3(c)]. Full scanning of the digital refocus is provided in Visualization 2. The distance between the 1X symbol and the maple-leaf mask was quantified as 64.48 mm, which shows good agreement with the pre-set value.

 

Fig. 3. DECALF imaging of a static 3D color scene. (a) Experimental setup. (b) Four representative perspective images. (c) Digital refocusing results.

Download Full Size | PDF

To demonstrate the DECALF imaging’s ability to visualize dynamic objects, we imaged moving microspheres in water. In the experiment, the white LED illuminated polyethylene microspheres (Cospheric, WPMS-1.00 850–1000 µm) randomly distributed in water in a cuvette (Labshops, SKU:Q109), as shown in Fig. 4(a). The transmitted white light entered the DECALF system. Microspheres’ movement was induced by stirring the water. Figure 4(b) shows three all-focused images at 50 ms, 250 ms, and 400 ms, in which the depths of these microspheres (marked as M1–M5) were determined via digital refocusing. By calculating the centroids of each microsphere, time histories of 3D positions of these microspheres are plotted in Fig. 4(c), and the full evolution is shown in Visualization 3. In this experiment, the occlusion of microspheres was not observed. Nevertheless, it is worth noting that DECALF imaging is poised to mitigate such a problem with its current configuration. The acquired perspective images enable viewing the scene from different angles, which increases the chance to observe occluded microspheres. Using light field occlusion modeling [35], the depths of these microspheres could be estimated by the DECALF system.

 

Fig. 4. 3D tracking of moving microspheres using DECALF imaging. (a) Experiment setup. (b) Representative depth-coded images. (c) 3D positions of five microspheres over time.

Download Full Size | PDF

To highlight the dynamic 3D imaging ability of the DECALF system, we imaged six-day-old zebrafish larvae freely moving in a cuvette (Labshops, SKU:Q109) (Fig. 5, Fig S2, and Visualization 4). Water jetting was used to stimulate zebrafish’s escape behaviors. Three representative all-focused images of a zebrafish at 100 ms, 250 ms, and 600 ms are shown in Fig. 5(a). The time trace of the 3D spatial positions of the head of this zebrafish is shown in Fig. 5(b). Using this trajectory, we calculated its instantaneous moving velocities in the $x$, $y$, and $z$ directions [Fig. 5(c)]. To further analyze the zebrafish’s motion, we tracked the tail bending angle $\alpha$ and the fin orientation angle $\beta$. Changes in these angles, along with the zebrafish’s moving distance, are shown in Fig. 5(d). These results illustrate the correlation between the distance and the instantaneous velocities of the zebrafish. In addition, the results show that the tail bending angle is zero at the beginning and the end of the recording window, indicating that the zebrafish kept its tail straight when staying still. In contrast, once it encountered a threatening stimulus, large tail bending angles were observed, resulting in a change in direction followed by a rapid swim with higher instantaneous velocities. These behaviors are reflected in Fig. 5(c) as a sharp oscillation in its moving trace from 100 ms to 350 ms. Finally, the data reveal asymmetrical orientation angles of the left and right fins, indicating drastic changes in direction during the zebrafish’s escape from the stimulus.

 

Fig. 5. 3D tracking of a six-day-old freely moving zebrafish larva using the DECALF system. (a) Representative all-focused frames at 100 ms, 250 ms, and 600 ms. (b) 3D trace of the zebrafish. (c) Instantaneous moving velocities of the zebrafish in the $x$, $y$, and $z$ directions. (d) Time histories of the moving distance, tail bending angle, and fin orientation angle of the zebrafish.

Download Full Size | PDF

To test the DECALF imaging’s assessment of swimming behavioral differences in different zebrafish models, we applied it to imaging a normal zebrafish and a C9ORF72 loss-of-function (C9-LOF) zebrafish (Fig. 6 and Visualization 5). Recently developed to study the pathogenesis of amyotrophic lateral sclerosis, this C9-LOF zebrafish model replicates aspects of this disease, including motor behavioral defects, muscle atrophy, and motor neuron loss [36]. The representative all-focused frames of normal and C9-LOF six-day old zebrafish larvae at three time points [Fig. 6(a)] show no apparent difference in their shapes. However, when water stream stimulation was applied, we observed different behaviors by tracking the 3D positions of both models [Fig. 6(b)]. The normal zebrafish quickly moved away from the site of the startle. In contrast, the C9-LOF zebrafish showed slow responses and a limited moving ability due to motor deficits [36]. This difference is quantitatively reflected in the instantaneous velocities of the normal and C9-LOF zebrafish in the $x$, $y$, and $z$ directions, as shown in Fig. 6(c). While the curves for the normal zebrafish oscillate sharply in all three directions, those of the C9-LOF zebrafish show small changes, especially in the $x$ direction. Altogether, these findings demonstrate the power of DECALF imaging for the behavioral study of disease-model zebrafish in vivo.

 

Fig. 6. Comparison of escape behaviors of a normal and disease-model (C9-LOF) zebrafish. (a) Representative all-focused frames at 50 ms, 100 ms, and 200 ms. Depths are coded with colors. Backgrounds are subtracted for better display. (b) 3D traces after stimulation. (c) Instantaneous moving velocities in the $x$, $y$, and $z$ directions.

Download Full Size | PDF

In summary, we have developed DMD-based DECALF imaging for high-resolution, color light field acquisition using broadband visible light at video rate. DECALF imaging is applied to studying zebrafish’s motion under stimulation. Circumventing the trade-off between spatial and angular resolutions, DECALF imaging enables ${{5}} \times {{5}}$ $({\theta ,\varphi})$ perspectives at the camera’s full $({x,y})$ pixel count of ${{1280}} \times {{1024}}$. DECALF imaging extends the operation scope of DMD-based CALF imaging to broadband light. Compared with conventional CALF imaging that employs a single DMD with a narrow-bandpass filter [30] or monochromatic illumination [23], the accommodation of the full visible spectrum in the DECALF system enhances light throughput. Its dispersion-compensated design also avoids the reduction of spatial resolution by pixel binning and the decrease in image quality due to laser speckles. Furthermore, the broadband imaging circumvents the potential color-induced complexity in the study of animal behaviors [37]. As a universal imaging scheme, the DECALF system is expected to be integrated into a variety of modalities for both macroscopic and microscopic light field imaging.

Our future work will focus on further improving the DECALF system’s technical specifications and imaging capability. For example, the 500-Hz full frame rate of the camera used in this work is much lower than the 22-kHz refreshing rate of the DMD. Replacing it with a high-speed camera [38,39] could largely increase the frame rate of light field imaging. Moreover, other advanced encoding schemes [40–42] could be employed to considerably enhance the signal-to-noise ratio and accuracy in both the acquired perspective images and the recovered light field images. Finally, by enlarging the angular range covered by the perspective images and by employing super-resolution algorithms in digital refocusing [43], the DECALF system will enable accurate depth sensing in the scenario of partial occlusion, shedding new light on in vivo high-speed 3D position tracking.