1. Introduction

Ultrahigh-speed continuous imaging is a critical technology for high-throughput screening of cell structure and behavior [1], drug discovery [2, 3], rare cell detection for cancer diagnostics [4], and numerous other clinical and basic research applications throughout the life and physical sciences [5, 6]. For example, understanding cellular heterogeneity has become essential for investigating drug resistance in cancer treatment wherein cells of interest often comprise less than 0.2% of the total population [6]. Identification and isolation of subpopulations presents a significant challenge for statistically and biologically meaningful analysis and thus demands techniques capable of both high throughput and high information content. To meet this requirement, imaging flow cytometry combines the high acquisition rate of non-imaging traditional flow cytometry with the high information content of optical microscopy [7]. However, while traditional flow cytometry can analyze samples at flow velocities in the range of 10 m/s, imaging flow cytometers remain limited by the image acquisition step to maximum flow velocities of 0.06 m/s [8]. Photonic systems such as time-stretch microscopy [9–13] are poised to close this gap, permitting analysis at flow velocities up to 10 m/s [11] and thus drastically reducing the time to detect rare events such as circulating tumor cells with an incidence of one in several million [4].

High-speed imagers generally fall into two categories: burst sampling and continuous sampling. Using in situ storage, cutting-edge complementary metal-oxide semiconductor (CMOS) [14] and charge-coupled device (CCD) [15] imaging arrays have achieved impressive burst frame rates of 10s of MHz [16]. However, these architectures offer maximum record lengths limited by pixel-level memory constraints to approximately 100 frames. Microscopic imaging up to a 4.4 THz frame rate for 6 frames has been demonstrated in a technique called sequentially timed all-optical mapping photography (STAMP), using spectrally-carved mode-locked laser pulses spatially separated on an imaging array using a diffraction grating [17]. Burst imaging of macroscale objects at up to 100 GHz frame rates for up to 350 frames using a digital micromirror device (DMD) and streak camera in conjunction with compressed sensing (CS) recovery has also been recently demonstrated in a technique named compressed ultrafast photography (CUP) [18]. The STAMP and CUP burst sampling systems achieve incredible burst pixel rates of 1.66 exapixels/sec and 2.25 petapixels/sec, however these sampling rates can only be sustained for time spans of 1.37 ps and 3.5 ns respectively, followed by dead times of at least 1–10 ms for the required image sensor readout.

While burst sampling systems are useful for observing extremely fast but isolated events in a single-shot, many applications (e.g. high-throughput diagnostics) necessitate continuous sampling, which requires tremendous hardware resources to record the massive stream of high-speed image data. Recently, cutting-edge imaging architectures employing ultrafast laser pulses and fiber-optic-based information processing yielded a performance leap in ultrahigh-speed continuous acquisition [9–13]. Still, such approaches remain fundamentally limited in speed, resolution, and image quality by the measurement rate of electronic digitizers [19]. For example, both traditional CCD arrays and state-of-the-art photonic systems such as serial time-encoded amplified microscopy (STEAM) read out the pixel information serially with a single analog to digital converter (ADC). Thus the number of pixels acquired per second is equal to the sampling rate of the ADC.

Notably, real signals such as most natural images are highly compressible and contain far less information than their full capacity as evidenced by the prevalence of modern data compression technology. Moreover, a recent advance in signal acquisition theory known as compressed sensing indicates that, due to their compressibility, real signals can be acquired with far fewer measurements than conventionally deemed necessary [20–24]. Thus cutting-edge ultrahigh-speed imaging systems are inefficient, collecting far more data than is required to accurately characterize the signals of interest and thus limiting their potential operating rate.

Recently, data compression in the optical domain has become a popular topic of research to improve analog-to-digital conversion efficiency. Several systems have been demonstrated for compressive photonic sampling of sparse radio frequency (RF) signals [25–29]. Beyond permitting signal characterization with a sub-Nyquist number of measurements, compression in the optical domain has also enabled extension of the effective sampling bandwidth beyond the electronic subsystem limitations [26, 27] and temporal integration of the pseudorandom measurements to allow for low ADC sampling rates [27, 29]. In addition to compressive sampling, the anamorphic stretch transform (AST) has been proposed to achieve time-bandwidth compression of pulsed optical waveforms by employing sublinear group delay chirping in conjunction with measurement of the complex electric field [30,31]. Very recently, multiple groups have also shown interest in compressed sensing imaging using ultrafast pulses [32–35], but to our knowledge this paper is the first demonstration of ultrafast structured illumination imaging of microscopic objects moving at high speed.

Here we demonstrate an imaging system that harnesses continuous high-rate photonically-enabled compressed sensing (CHiRP-CS) for image acquisition. In the CHiRP-CS imaging approach, ultrahigh-rate spectral shaping is achieved through dispersive chirp processing of broadband laser pulses to enable ultrafast structured illumination of objects flowing through a one-dimensional (1D) field of view. We investigate two different 1D spatial dispersers for low and high magnification imaging of complex test objects printed on transparencies and 25-μm polystyrene microsphere clusters, respectively, placed on a spinning hard disk platter. Compressive measurements are acquired continuously without averaging at a rate of one digital sample per optical pulse. We demonstrate successful reconstruction of 2D images from the 1D compressive measurements at effective 1.46, 4.19, and 7.32-Gigapixel/sec rates from a 90-MHz sampling rate. We also extend the system with optical pulse interleaving to 9.9, 19.8 and 39.6-Gigapixel/sec rates from a 720-MHz acquisition rate.

2. Compressed sensing theory and application to imaging

Real images and most real-world signals are highly compressible and can be accurately represented by relatively few significant coefficients in an appropriate mathematical basis. Sparse approximation—the process of transforming the signal to this basis and saving the most significant coefficients while ignoring the rest—is the foundation of modern data compression technologies such as the Joint Photographic Experts Group (JPEG) and Moving Picture Experts Group (MPEG) formats [36, 37]. Traditionally a signal is sampled according to the Nyquist theorem to acquire a raw digital representation and then a compression algorithm is applied, eliminating as much of the redundancy in the original data as possible. Hence, most of the acquired data is simply thrown away. Consequently, for most applications in high-speed continuous acquisition, the raw image data bandwidth is far larger than is truly necessary.

Compressed sensing is a recent and influential sampling paradigm that advocates a more efficient signal acquisition process. According to CS theory, a K-sparse signal x^⋆ ∈ ℝⁿ is measured through a set of M measurements of linear projections y_i = 〈a_i,x^⋆〉, i = 1,…,M, in which vectors a_i ∈ ℝ^N form the matrix A of size M × N. To reconstruct x^⋆, ℓ₁-minimization is proposed to solve the following problem

The case above deals with imperfect observations contaminated by noise, i.e., y = Ax^⋆ + w where w is some unknown perturbation bounded by a known amount ‖w‖₂ ≤ σ. If the sensing matrix A obeys the Restricted Isometry Property (RIP) [20] and σ is not too large, then the solution $\widehat{\mathit{x}}$ of Eq. (1) does not depart significantly from the optimal solution x^⋆, so long as the number of measurements M is on the order of K logN [20–24]. Thus the CS framework advocates the collection of significantly fewer measurements than the ambient dimension of the signal (M ≪ N).

A notable CS imaging architecture is the single-pixel camera in which light collected from an object is randomly combined via a digital micro-mirror device (DMD) before it is focused onto a single-pixel photodetector [38]. By tuning each micro-mirror in the pixel array, the system creates pseudorandom 2D patterns that modulate the image before summing the optical power using the single detector, thereby optically performing the inner product, y_i =〈a_i,x^⋆〉. This technique has also been extended to macroscopic [39] and microscopic structured illumination imaging [40]. However, in all of these systems the need to mechanically transition the MEMS-actuated micro-mirrors sets the upper limit of the pattern rate to a few kHz, restricting the total image acquisition time. In contrast, the CHiRP-CS architecture we demonstrate here achieves illumination pattern rates more than 20,000× faster. Thus our approach allows for application of CS to the domain of ultrahigh-speed image acquisition.

3. Experimental system

The principle of operation of the CHiRP-CS imaging system (Fig. 1) is to modulate pseudorandom patterns at an ultrahigh rate onto the optical spectra of broadband mode-locked laser pulses and then utilize these spectral patterns to create structured illumination of an object. Light collected from the object is directed onto a single-pixel high-speed photodetector and the energy of each returned laser pulse is recorded continuously by a synchronized real-time ADC. A CS recovery algorithm then constructs an image of the object from far fewer measurements than would be required for conventional Nyquist sampling.

 

Fig. 1 (a) Broadband laser pulses are dispersed in optical fiber to accomplish spectrum-to-time mapping. Each pulse is modulated with a unique ultrahigh-rate pseudorandom binary pattern and then re-compressed in fiber (Dispersion compensation) to an ultrashort duration before passing through a 1D wavelength-to-space mapping diffraction grating and lens that focuses the spectral pattern onto the object plane, providing structured illumination of the object flow. The output pulse energy traveling back through the spatial disperser to the photodiode and ADC represents an optically-computed inner product between the pseudorandom pattern and the object. The image is reconstructed via a sparsity-driven optimization from sub-Nyquist compressive measurements. (b) Temporal overlap of the pulses at the pattern modulation stage. (c) Detailed system schematic for low magnification results in Subsection 5.1. (d) Pulse interleaver and (e) spatial disperser with microscope for high magnification results in Subsection 5.2.

Download Full Size | PDF

Spectral patterning is accomplished using chirp processing in optical fiber [27]. A passively mode-locked erbium-doped fiber laser (MLL) emitting 300-fs pulses at the native 90-MHz repetition rate (centered at 1555 nm) is used in conjunction with a C-band erbium-doped fiber amplifier (EDFA) to amplify the optical pulse train to 200 mW. Dispersive spectrum-to-time mapping is then performed in a dispersion compensating fiber (DCF) with a total group velocity dispersion (GVD) of −853 ps/nm and dispersion slope of 2.92 ps/nm² at 1550 nm. Spectral broadening to a full width of 33 nm is achieved through the high peak power after the EDFA and the moderate nonlinearity (γ = 7.6 W⁻¹km⁻¹) of the DCF, stretching the 300-fs MLL pulses to greater than 28 ns.

Pattern modulation is achieved with an 11.52-Gbit/s pulse pattern generator (PPG) synchronized to the MLL driving a 20-GHz Mach-Zehnder intensity modulator (MZM). This permits 128 pseudorandom binary features per 11.1-ns pulse repetition period. The PPG can output user-programmable patterns up to 1.3 Mbit in length; in practice a customized string of 1.1 Mbit or 8615 patterns is used. Of these, a few patterns are used as a header to determine the alignment between the samples from the ADC and the predetermined pseudorandom patterns for the reconstruction. The PPG modulates the set of patterns continuously permitting uninterrupted sampling and the 95.7-μs repetition period for the set of patterns does not affect the robustness of the sampling approach.

As depicted in Fig. 1(b), the time-stretched pulses overlap partially during pattern modulation. Thus, neighboring pulses share some temporal features, but these features are mapped to different wavelengths and, thereby, involve different regions of the structured illumination pattern. This preserves mutual incoherence between the pseudorandom patterns while permitting many more features per pulse. Three example PRBS-encoded laser pulse spectra are shown in Fig. 2. In practice, we achieve 325 features per pulse within the spectral bandwidth, which sets the horizontal pixel resolution of the reconstructed images.

 

Fig. 2 Example repeating 128-bit pseudorandom binary patterns observed with an optical spectrum analyzer. In practice, we reconstruct 325 horizontal pixels utilizing the full width of the spectrum. Actual sampling patterns are unique to each pulse and cannot be observed on the averaged spectrum.

Download Full Size | PDF

After spectral patterning, the pulses are time-compressed in standard single-mode fiber (SMF) with complementary GVD of +853 ps/nm and dispersion slope of +2.92 ps/nm²at 1550 nm to the DCF. The spectrally-patterned and compressed laser pulses pass through a 1D spatial disperser to serve as ultrafast structured illumination of an object flow.

Here we demonstrate the CHiRP-CS imaging system at two levels of magnification and therefore we construct two different 1D spatial dispersers. The low magnification disperser is composed of a 600-line/mm ruled diffraction grating and 123-mm effective focal length spherical lens. The high magnification disperser employs the same grating with a 1-m focal length spherical lens to form an intermediate structured illumination image before a 200-mm tube lens and a 50× near-IR microscope objective (Olympus LCPLN50XIR, NA=0.65) designed for long working distance. Large-area high-resolution optics are specifically chosen to allow the spectral resolution of the diffraction grating to exceed the minimum feature size. To test the system under operating conditions safe for biological samples, we fix the optical power at 300 μW at the object plane.

Each feature occupies a spectral bandwidth of 12.5 GHz, which corresponds to a shutter speed of 35.2 ps for a transform-limited Gaussian feature inside the disperser. The decreased modulation depth for the fastest (e.g., 010 or 101) alternating features (Fig. 2) is a product of the single feature bandwidth and the pattern modulation rate. By adjusting the pattern modulation rate, it is possible to achieve >15 dB modulation depth for all features across the full spectral width [27] to approach ideal binary patterns with an envelope corresponding to the spectral shape.

Test objects pass through the focused image of the structured illumination and the scattered light returns through the disperser into an optical fiber and amplified 150-MHz photodetector. Thus, the system behaves as a confocal imager. As in prior work focusing on application to imaging flow cytometry [4], the objects move through the system field of view at a constant velocity and 2D images are reconstructed with a vertical dimension that corresponds to both time and vertical spatial extent.

The detected pulse energy, recorded with a synchronized ADC, represents the vector inner product between the spatial profile of the object and the unique spectral illumination pattern. Therefore, only one digital sample per pulse, acquired at the laser repetition rate, is required for each compressive measurement. To achieve the minimum electronic digitization rate for the greatest system sampling efficiency, an externally-clocked ADC is driven with a 90-MHz sampling clock derived from the MLL monitor port input to a 1.2-GHz photodiode with appropriate RF bandpass filters. The phase of the sampling clock is fixed to align the sampling windows with the peaks of the detected voltage waveform.

The low magnification disperser produces a 2.77-mm × 5.4-μm structured illumination line with 8.5-μm × 5.4-μm features at the object plane. In the high magnification disperser, the tube lens and objective (designed for 180-mm tube length) result in a 55.6× demagnification of the structured illumination patterns to create 1.2-μm × 1.2-μm features across a 390-μm 1D field of view. However, in practice, we add a low-power EDFA before the high magnification disperser to compensate the additional coupling loss into the microscope objective. Lower gain in the EDFA at the edges of the spectrum causes slight narrowing of the field of view to 330 μm with 275 horizontal pixels (28-nm spectral width).

Finally, to investigate even higher acquisition rates in the high-magnification system, we also add three time-interleaving fiber Mach-Zehnder interferometers after the time-stretching fiber, before the PRBS MZM to increase the pulse repetition rate to 720 MHz [Fig. 1(d)]. To accommodate the new pulse repetition rate, we also switch to a 1.2-GHz PD and 720-MHz ADC sampling rate.

4. Reconstruction algorithm

To reconstruct the 2D image frames from the 1D compressive pseudorandom measurements, a naïve approach is to recover one image row at a time independently. Instead, we further develop a novel 2D reconstruction algorithm tailored to this imaging apparatus. As depicted in Fig. 3, we utilize ℓ₁-minimization coupled with a discrete cosine transform (DCT) basis at the local level of blocks of pixels called patches: any selected local patch should be sparse. Out of all candidate images that are consistent with the 1D measurements, the iterative optimization algorithm seeks the most sparse set of overlapped patches.

 

Fig. 3 Patch based image recovery from 1D compressive pseudorandom measurements, which iterates between two steps—global reconstruction of the image estimates and maximizing the sparsity level of all local image patches.

Download Full Size | PDF

Similar to conventional image compression such as JPEG, the reconstruction framework focuses on the local image structures. A popular model to quantify local image information is sparsity in an appropriate domain: given a patch or block of pixels $x\in {ℝ}^{N_b\times M_b}$ extracted at random location from an image, the coefficient $\mathit{\alpha }\in {ℝ}^{N_b\times M_b}$ of x under some sparsifying transform $\tilde{\mathbf{\Psi }}(\cdot )$ defined by

The recovery process estimates the set of sparse coefficients ${\left\{{\mathit{\alpha }}^k\right\}}_{k=1}^p$ of the patch set ${\left\{x^k\right\}}_{k=1}^p$ covering the entire image of interest which is consistent with the 1D observations. Denoting ${\left\{{\overline{\mathit{\alpha }}}^k\right\}}_{k=1}^p$ as the sparse coefficients of the patches ${\left\{{\overline{x}}^k\right\}}_{k=1}^p$ extracted from the original image $\overline{G}\in {\mathrm{R}}^{N\times M}$, the 1D compressive measurement process can be written as

The optimization problem in Eq. (2) can be solved efficiently by an iteratively alternating minimization procedure. At iteration t of the algorithm, a noisy estimate G_t of the original image consistent with the observations is reconstructed based on the information from the previous iteration. The estimates of the coefficients ${\left\{{\mathit{\alpha }}_t^k\right\}}_{k=1}^p$ at this iteration can then be found by thresholding the coefficients of the noisy patches ${\left\{x_t^k\right\}}_{k=1}^p$ extracted from G_t.

Because we acquire compressive 1D pseudorandom line scans with a horizontal resolution set by the pulse spectral width and chirp processing parameters, the recovered vertical dimension N_v can be used as a tuning parameter depending on the complexity of the objects under test. In the reconstruction process, we use an effective M_l samples per line, far fewer than the number of pixels per line N_l where the full dimension N = N_l×N_v. Thus, the compression ratio, line rate, and pixel rate are related to the average number of samples needed to reconstruct each line in the image by

5. Experimental results

5.1. Low magnification

We construct a high-speed test image using laser-printed transparencies fixed to the platter of a dismantled 7200-RPM (rotations per minute) hard drive. The printed test objects are positioned on the outer edge of the spinning platter, measured to be moving at 34.3 m/s. The transparencies offer complex customized test objects with microscale features to measure the system performance at low magnification.

Our reconstructed results in Fig. 4 demonstrate imaging of complex objects moving at high speed from far fewer measurements than required in conventional Nyquist sampling. The first column shows optical microscope images of the static test objects for the purpose of comparison. Each subsequent column shows images of the objects moving at high speed taken with our compressive imager. Each of these images is reconstructed from 8400 consecutive measurements acquired in a single shot in 93.3 μs. Each column shows image reconstruction using a different relative percentage of measurements to recover the full image dimension. Therefore, the 6.15, 2.15, and 1.23% compression ratios in Fig. 4 correspond to imaging rates of 1.46, 4.19, and 7.32 Gigapixel/sec, which vastly exceed the present 90-MHz sampling rate.

 

Fig. 4 A laser-printed transparency with three objects of varying complexity (rows a–c) was fixed to the top platter of a 7200-RPM computer hard drive and imaged by the system at compression ratios of 6.15, 2.15, and 1.23%. The unevenness in the illumination is due to the spectral envelope (Fig. 2), which was left un-compensated in these results.

Download Full Size | PDF

The compression ratio is practically limited by the complexity of the object’s spatial features. For example, simpler objects such as the soccer ball in Fig. 4 in row (c) show very little loss of image quality as the compression ratio decreases whereas more complex objects such as the shield in row (b) become noticeably distorted in the horizontal dimension.

5.2. High magnification

To demonstrate the CHiRP-CS system’s potential for high-speed imaging of micron-scale objects [Fig. 1(e)] with correspondingly reduced signal contrast, we acquire images of a cluster of 25-μm undyed polystyrene microspheres dried onto the surface of the platter; the hard disk motor is now driven by a variable DC brushless motor controller. Figure 5 depicts reconstructions of the cluster moving at 12.4, 26.0, and 42.2 m/s from 7000, 3350, and 2140 measurements respectively at measurement rates from 90-MHz up to the interleaved 720-MHz using 7.27, 3.64, and 1.82% of Nyquist sampling. Note, each row of the figure corresponds to a single acquisition at 720 MHz and downsampled versions at 360, 180, and 90 MHz in order to demonstrate the benefit of the increased optical sampling rate for high-speed flows. At 12.4 m/s, the shape of the microsphere cluster is well-represented at all sampling rates and compression ratios, but with some distortion at 90 MHz. There is also some characteristic horizontal blurring within the cluster at the higher compression ratio. At 42.2 m/s, though the reconstructed image contrast is reduced, the cluster shape shows excellent agreement in the 720 MHz case, but there is significant motion distortion that increases with lower sampling rate. At 1.82% compression, the loss of horizontal resolution at very low compression ratios prevents differentiation of the particles, but the overall size and shape of the cluster are well reconstructed. These results demonstrate image reconstruction of very high-speed microscopic flows at effective 9.9, 19.8 and 39.6 Gigapixel/sec rates from a maximum 720 MHz acquisition rate. To our knowledge these measurements are of the fastest flow rates to date for a diffraction limited microscopic line scan imager [11].

 

Fig. 5 A single cluster of 25-μm undyed polystyrene microspheres imaged by the extended system, depicted in Fig. 1(c–e), moving at 12.4, 26.0, and 42.2 m/s using compression ratios of 7.27, 3.64, and 1.82%. The measurements were acquired at the interleaved pulse rate of 720 MHz and downsampled to the equivalent of 360, 180, and 90 MHz pulse rates for comparison.

Download Full Size | PDF

To reconstruct the microspheres as bright objects on a dark background, we acquire a reference trace on the ADC with no objects inside the field of view and compute a difference signal with objects in the field of view and input this into the reconstruction. The static image included for reference in Fig. 5 was acquired with a separate visible light microscope using dark field illumination. Thus, there is some uncertainty in how the interior regions of the microspheres should ideally appear under the system’s near-IR confocal illumination.

6. Discussion

In addition to data compression, the compressive sampling technique presented here also results in considerable benefits for the signal to noise ratio of the measurements. On average, half of a pulse’s spectral features are given a high ‘1’ intensity level and half will be given a low ‘0’ level. Thus the output pulse energy per sample is proportional to half of the unmodulated pulse energy. On the contrary, for conventional systems the energy per sample is inversely proportional to the total number of pixels. For example, in STEAM, considerable optical amplification (25–30 dB) is required to raise the optical signal above the detection noise floor [4, 9]. While the CHiRP-CS approach demonstrated here is entirely compatible with optical amplification of the output signal, it was not necessary for the results presented here.

The system presented here successfully extends CS imaging to continuous ultrahigh sampling rates. Compressive pseudorandom structured illumination reduces the required sampling bandwidth and information storage capacity by shifting signal processing complexity to the image reconstruction process. Thus, for the proposed high throughput flow cytometry application, online processing can be employed to exclude empty frames, but offline processing will be required to complete the image reconstruction and analysis, similar to commercial imaging flow cytometers [7]. The system offers a benefit nonetheless by increasing the achievable image acquisition speeds and by achieving real-time efficient image compression. More test samples can thus be analyzed by the imaging apparatus in less time with more efficient data storage. Image post-analysis can be completed with inexpensive, readily available, and increasingly powerful computing hardware.

Compressive sampling opens a path to significantly higher speeds by increasing the information content gained per digital sample. For conventional Nyquist-sampling systems, the most efficient mode of operation is to acquire one sample per output image pixel. Typically, each image line is encoded on a single laser pulse and each pulse is sampled a number of times corresponding to the number of pixels per line. In contrast, we operate with a higher pulse repetition rate and each pulse is sampled once corresponding to a single compressive measurement. We demonstrate high-speed imaging using a smaller number of measurements corresponding to only a few percent of the total number of image pixels. In other words, at the same ADC sampling rate, this compressive system can perform 10–100× faster. In addition, because the system relies on structured illumination with straightforward single-pixel output photodetection, it can be readily adapted for imaging of fluorescence. Beyond imaging of flows, by employing a 2D spatial disperser [41], the system can be readily adapted to 3D compressive video measurements [42] of ultrahigh-speed phenomena. Furthermore, this all-optical approach to compressive measurements can increase dramatically the speed and efficiency of multiple optical measurement modalities, for example, real-time spectroscopy [43], swept-source optical coherence tomography [44], and high-speed microwave measurement [26,27,45].