1. Introduction

Fluorescence microscopy serves as a powerful tool for biomedical observation and diagnosis, especially when combined with confocal and two-photon techniques [1]. By probing the proteins with fluorophores, one could monitor the physiological state of the specimens, including live cells, which provides functional information for biomedical imaging. However, fluorescence microscopy generally lacks the structural/morphological information of the specimens, since most thin specimens do not absorb or scatter light significantly, i.e., they are transparent or translucent [2]. Complementarily, quantitative phase imaging is an emerging technique for structural characterization of transparent specimens quantitatively. The combination of fluorescence and phase imaging would provide both the functional and structural features for more accurate and flexible observation and diagnosis in a wide range of fields, from basic science to clinical applications.

Conventional methods for simultaneous fluorescence and phase imaging either use multiple well-aligned cameras or a single camera with spatial multiplexing [3, 4]. The former needs expertise demanding alignment and the latter trades off spatial resolution for multiple capture modes [5]. However, single-pixel imaging offers an alternative solution for alignment-free, full-resolution multimodal imaging [6, 7]. By exploiting sequential structured illuminations and correlated measurements with the sample, one could recover the original two-dimensional (2D) information of the sample computationally and compressively [8]. The point-scanning imaging method, such as confocal laser scanning microscopy [1] and optical trapping/focusing [9, 10] falls into the general form of single-pixel imaging where the illumination is a single point focus and scanning is used to acquire an image. We refer single-pixel imaging here to a specific form, i.e., imaging with wide-field structured illuminations and single-pixel detection, where mechanical scanning is not required.

Here, we report an alignment-free full-resolution simultaneous fluorescence and quantitative phase imaging approach using two single-pixel detectors. Single-pixel imaging has been applied for dual-or multiple-modality tasks thanks to the high-sensitivity, wide-spectrum-range and low-cost property of single-pixel detectors. Recent advances include four-angle illuminations for three-dimensional imaging [11], dual wavelengths for infrared and visible imaging [12, 13], multiple-wavelength illumination/detection for color or multispectral imaging [6,14–18], and spatial-frequency multiplexing for multispectral 3D imaging [7]. However, simultaneous fluorescence and quantitative phase imaging using single-pixel detectors without mechanical scanning has not been reported yet, to the best of our knowledge.

Previous single-pixel phase imaging method generally requires phase-mode spatial light modulation and two-beam interferometry [19–21]. However, phase-mode spatial light modulation is relatively slow with a typical refreshing rate of 60 Hz, which limits the acquisition time to a few minutes. Recently, González et al. used the high-speed binary amplitude modulator, i.e., the digital micromirror device (DMD) for phase modulation [22] and obtained single-pixel digital holography within a few seconds [23]. However, these two-beam interferometry methods suffer from phase stability issues [21], since single-pixel imaging requires thousands of sequential measurements and the corrupted phases would damage the reconstruction. Recently, Stockton et al. combined spatial frequency projections with single-point detection to obtain the phase of a line, which requires rotating disk to generate line focus with varying spatial frequencies and line scan to obtain the phase of a 2D sample [24]. Mechanical implementations like rotation and line scanning though compatible to imaging in spectra where spatial light modulators are not available, are undesirable because they could bring about issues like mechanical vibrations, non-uniformity along the focal line scanning dimension, and increased electrical complexity for synchronization in case of high-speed acquisition. A similar approach using the DMD for complex amplitude modulation [25–27] and single-point detection/projection has been reported to form an optical transducer [28]. But no multimodality of this approach is demonstrated for its passive imaging scheme. Therefore, a multimodal single-pixel imaging method (including phase imaging modality) for stable and direct 2D acquisition without mechanical scanning is highly desirable.

By combining reference-free interferometry with single-pixel detection, we encode the phase and fluorescence of the sample in two detection arms at the same time. We employ active imaging, i.e., complex structured illuminations and add a bucket detector for fluorescence imaging to achieve simultaneous fluorescence and phase imaging. It is worth noting that these two modes are conducted in a parallel manner and share the same structured illuminations, which precludes expertise demanding alignment of multiple cameras. Moreover, this method holds great potential for multispectral fluorescence microscopy with additional single-pixel detectors or a spectrometer [6, 14–17]. We further envision that this cost-efficient multimodal system might be extended for imaging in spectra where array detectors are expensive or even unavailable [12, 29–32], which might find broad applications in biomedical science and material science.

2. Single-pixel phase and fluorescence imaging

Single-pixel imaging or the single-pixel camera [8] encodes the sample’s transmission or reflection information into its correlated intensity measurements with sequential structure illuminations. The measurement can be recorded with a single-pixel detector, and then used to reconstruct the sample information computationally together with the illumination patterns.

In this section, we introduce the proposed single-pixel phase and fluorescence imaging method. First, we briefly summarize the forward model and the reconstruction method of single-pixel imaging. Then, we introduce the single-pixel imaging method using reference-free phase-shifting interferometry and complex amplitude modulation. Finally, we show that the phase-shifting patterns can be used for fluorescence imaging simultaneously.

2.1. Single-pixel imaging

For coherent illumination, a 2D sample with complex field $S(\overrightarrow{r})$ illuminated by a spatially modulated pattern $P_k(\overrightarrow{r})$, where k is the index of the pattern (k = 1, ⋯, M) and $\overrightarrow{r}$ is the spatial coordinate vector, the measured intensity can be expressed as

The reconstruction can be conducted from either over-sampled/full-sampled or under-sampled measurements. For the former setting, Moore-Penrose inverse or pseudoinverse can be applied for reconstruction. For the latter case, one can use compressive sensing methods for reliable recovery based on the statistics that natural scenes are inherently sparse inspecific domains [36, 37]. In this work, we use Walsh-Hadamard transform bases instead of Fourier transform bases as the sensing matrix to avoid introducing three-or four-step spatial shifts of the sinusoidal patterns [33] and slow grayscale patterning. Besides, we adopt under-sampled measurements to obtain only a proportion of the Walsh-Hadamard transform spectrum with high coefficients, and impose the inverse transform for reconstruction [6, 12, 18, 35].

2.2. Single-pixel phase imaging

We use the reference-free method for single-pixel phase imaging by measuring the intensity of the center point at the Fourier plane, which is firstly reported by Stockton et al. [24] for line-scan phase imaging and then extended by Shin et al. [28] as an optical transducer. We adapt the passive imaging scheme in [28] to active imaging mode to obtain the fluorescence information of the sample at the same time. We refer to this method as the single-point detection method for phase imaging following [24, 28].

The forward model is different from conventional single-pixel imaging, since the original total-intensity measurements would wash out the phase information induced both by the sample and the structured illuminations. For single-pixel phase imaging, we measure the single-point intensity at the center of the Fourier plane of the correlated field between the sample and the illuminations [24, 28],which can be expressed as

Here $s_k={\displaystyle {\iint }_{\Omega }{H}_k(\overrightarrow{r})S(\overrightarrow{r}){\mathrm{d}}^2\overrightarrow{r}}$ is the integration of the complex field of the sample coded by the illumination, $r={\displaystyle {\iint }_{\Omega }S(\overrightarrow{r}){\mathrm{d}}^2\overrightarrow{r}}$ is a constant, which is the reference beam for the interferometry. With three-step phase-shifting, that is ϕ = 0, π/2, π, the complex integration can be recovered by

2.3. Single-pixel fluorescence imaging

For fluorescence detection, the total intensity of the fluorescence field is measured. Because fluorescence light is incoherent, the forward model follows Eq. (1),

Fluorescence detection can be conducted parallelly with phase detection by shared Walsh-Hadamard pattern illuminations. For P_k,ϕ = (e^jϕ · H_k + 1)/2 and ϕ = 0 or π, all elements of P_k,ϕ are either 0 or 1, we have

Therefore, the fluorescence integration can be recovered with two phase shifts

Similarly, the fluorescence intensity can be reconstructed via the inverse Walsh-Hadamard transform as Eq. (4) shows. Note that Eq. (7) and Eq. (10) look similar, but are different in principle: Eq. (7) is based on phase-shifting interferometry, while Eq. (10) contains subtraction of two phase shifts because each element of the Walsh-Hadamard matrix is either +1 or −1 and negative intensity cannot be displayed on the binary spatial light modulator.

In summary, both the phase and fluorescence of the sample can be reconstructed simultaneously. Since the fluorescence detection arm only uses two of the three-step phase shifts, the total sequential measurements remain the same as the phase detection arm and fluorescence information can be acquired via total-intensity fluorescence detection.

3. Experimental setup

The scheme of the experimental setup is shown in Fig. 1. We use a single-frequency 488 nm laser (Sapphire 488SF-100mW, Coherent) here. The incident light is expanded and collimated by a spatial filter, consisting of an objective lens, a pinhole, and a collimating lens. The diameter of the output beam is around 20 mm to match the size of the DMD (V-7001, ViALUX GmbH). The pixel resolution of the DMD is 1024 × 768 with a pixel size of 13.68 µm. The DMD has a maximum refreshing rate of 22.7 kHz for binary patterns. We use the total internal reflection (TIR) prism to compensate for the 12^◦ rotation angle of the micromirror and rotate the DMD to 45^◦ to ensure the light reflected by the DMD parallel to the optical axis of the incident light, which is widely used in digital light projection with the DMD. A 4f system is built to enable the DMD with complex amplitude modulation using Lee hologram [22, 26]. The 4f system consists of two lenses (L₁ and L₂) and a diaphragm at the Fourier plane with a diameter of ∼1 mm, which directly blocks all the other orders except for the +1st order, as shown in the inset of Fig. 1. The focal length of L₁ and L₂ is 150 mm. Three patterns used in the experiment are shown in top left of Fig. 1, where 0, π/2, π represents three phase shifts (in terms of light path length, instead of spatial shifts of sinusoidal patterns) of the Walsh-Hadamard pattern, and the inset is the cross part of the ϕ = π/2 pattern containing phase difference between the left part and the right part. Then the desired complex amplitude field interacts with the sample field at the sample plane, which is conjugate with the DMD plane, as indicated by the dashed line in Fig. 1. The tube lens (TTL180-A for Olympus, Thorlabs) has an effective focal length of 180 mm. The objective lens (Plan N 10x/0.25, Olympus) has a magnification of 10x and numerical aperture (NA) of 0.25. For larger field-of-view in the quantum dot experiment, we do not use the tube lens and objective lens, and put the sample directly at the conjugate plane of the DMD in the L₁−L₂ 4f system.

 

Fig. 1 Experimental setup for simultaneous phase and fluorescence imaging with single-pixel detectors. Lee hologram [22, 26] is applied to generate the complex amplitude field at the sample plane, which provides both reference beam and phase shifts for interferometry. SF-CW, single frequency-continuous wave; DMD, digital micromirror device; TIR prism, total internal reflection prism; L₁-L₄, lenses (f₁ = f₂ = 150 mm, f₃ = f₄ = 75 mm); OL, objective lens; DM, dichroic mirror; PMT, photomultiplier tube; SMF, single-mode fiber. The dashed line denotes the intermediate image plane of the DMD, which is conjugate with the sample plane.

Download Full Size | PDF

The interaction field is detected by two arms. The first is the phase imaging arm consisting of a lens L₃, a single-mode fiber (with a core diameter of ∼4 µm), and a Si avalanche photodetector (APD120A2, Thorlabs) to measure the intensity of light exit from the single-mode fiber, which is not shown in Fig. 1. The second is the fluorescence arm consisting of a dichroic mirror (DM, MDF-GFP, Thorlabs), a fluorescence filter (FELH0500, Thorlabs), a lens (L4), and a photomultiplier tube (PMT1001/M, Thorlabs). The signals of the two photodetectors are digitalized by a four-channel 14-bit analog-to-digital converter (PCI8514, ART Technology) parallelly. Finally, the raw data is post-processed to retrieve the phase and fluorescence of the sample. Because the DMD for spatial-resolved encoding in single-pixel imaging and the pixelated detector for direct spatial-resolved capturing in conventional photography are reciprocal [38], the images of different modalities in single-pixel imaging are automatically aligned since they use the same spatial-resolved element [11]. In this way, the recovered phase and fluorescence images are inherently aligned thanks to single-pixel detection.

4. Results

We experimentally demonstrate the phase imaging ability on an etched glass sample. The pixel resolution of the result is 128 × 128, and the total number of projected patterns for each sample is 49, 152. The data acquisition for each sample takes 2.5 seconds, with the refreshing rate of the DMD set to 20 kHz. We use 6 × 6 pixel binning of the DMD to increase illumination intensity at the sample plane, i.e., 768 × 768 of the DMD pixels are used in the experiment. Note that 2 × 2 pixel binning is also feasible (and pixel binning of at least 2 × 2 is usually used in Lee hologram to obtain complex-amplitude modulation with high fidelity [22, 26, 27]), we use large pixel binning considering the modulation efficiency and detection sensitivity, because the Lee hologram method has the efficiency of complex field modulation of ∼5% and the intensities of single-point detection and fluorescence detection are relatively low. Then we show simultaneous phase and fluorescence of the quantum dot sample. The pixel resolution is 256 × 256 and the pixel binning is 3 × 3. The total acquisition time for full-sampling is ∼10 seconds, not including the loading time of the binary patterns and switching time among four pattern groups because of the limited number of pre-load patterns (87, 380 < 256 × 256 × 3). For under-sampling with the sampling ratio of 30%, the acquisition time is 3 seconds.

4.1. Phase imaging

The sample used in the phase imaging experiment is an etched glass including three letters “100” at 400nm etching depth. The phase reconstruction result is shown in Fig. 2, and the physical sample surface is estimated from the phase difference of the sample

 

Fig. 2 Phase imaging result of an etched glass sample. (a) Raw phase reconstruction of the etched glass. (b) Background phase of the system. (c) Estimated etching height from the sample’s phase, i.e., the phase difference between (a) and (b). (d) Phase along the highlighted profile in (c), and performance comparison among the ground-truth phase (black dashed line), the result of diffraction phase microscopy (DPM) [39] (blue dotted line), and ours (green solid line). Scale bar: 500 µm.

Download Full Size | PDF

Here $\phi (\overrightarrow{r})$ denotes the phase difference introduced by the sample, λ is the wavelength of the incident light 488 nm in this experiment, and Δn is the difference of the refractive index between the glass and air 0.4630 in the experiment.

As shown in Figs. 2(a) and 2(b), the smooth part of the raw sample phase matches well with the background phase. The estimated height is about 400 nm for the three characters “100”, which agrees well with the fabrication and the result of diffraction phase microscopy (DPM) [39], as shown in Fig. 2(d). Note that the background phase is fixed and needs to be calibrated only once. All the phases here are unwrapped using Goldstein phase unwrapping method [40]. The artifacts of the subtracted phase are due to the slightly misaligned center-frequency points at the Fourier plane when measuring the raw phase and the background phase. The scale bar in Fig. 2 is 500 µm.

4.2. Simultaneous phase and fluorescence imaging

In this experiment, we use the quantum dot sample to demonstrate our simultaneous phase and fluorescence imaging ability. The quantum dot is 20% solution of methylammonium lead bromine (CH₃NH₃PbBr₃). We drop the solution on a piece of pre-cleaned slide glass to form dots or characters, and the results of simultaneous phase and fluorescence imaging are shown in Fig. 3. The sample includes five letters (“FM” and “QPM”) written with quantum dot solution on the slide glass. The letters “FM” denote fluorescence microscopy and these two letters are written with the quantum dot emitting green fluorescence. The letters “QPM” denote quantitative phase microscopy and do not emit fluorescence. Figure 3 shows that phase and fluorescence imaging offer complementary information. Phase imaging contains the structural information, that is the height of the written letters shown in Fig. 3(b). Fluorescence imaging contains the functional information, that is the peripheral of the “FM” letters are fluorescent shown in Fig. 3(a). Both phase and fluorescence images are full-sampled with Walsh-Hadamard bases. It is worth noting that the positions of the contours of both fluorescence and phase images match well without manual alignment, as shown in Fig. 3(c) attributing to the single-pixel image mechanism: the sample position in the image is merely determined by relative position between the sample and structured illuminations. Here we do not subtract the background phase to avoid background misalignment artifacts between two imaging arms. The scale bar in Fig. 3 is 2 mm.

 

Fig. 3 Simultaneous phase and fluorescence imaging of the quantum dot sample. (a) Fluorescence intensity of the quantum dot sample on the slide glass. (b) Phase of the quantum dot sample. (c) Superimposition of automatically aligned phase and fluorescence images. Scale bar: 2 mm.

Download Full Size | PDF

4.3. Compressive phase and fluorescence imaging

We further show that we can recover the phase and fluorescence compressively from under-sampled measurements. We sample a proportion of the Walsh-Hadamard spectrum with high coefficients (assuming the coefficients of nature images concentrate at low spatial frequencies) as stated in [6, 12, 18] and apply the inverse Walsh-Hadamard transform to reconstruct the phase and fluorescence images. We use the “zigzag” strategy to sample the 2D coefficients of Walsh-Hadamard transform space, in order to get as many high spatial frequencies as possible, as suggested in [35]. The sample is the same as the quantum dot sample used in Sec. 4.2.

The reconstructed phase and fluorescence images, peak signal-to-noise ratio (PSNR) of the undersampled fluorescence and phase images are shown in Figs. 4(a), 4(b) and 4(c), respectively. PSNR of an image I with respect to a reference image R is calculated as $\text{PSNR}=20\cdot {\log }_{10}({\text{MAX}}_{\mathbf{I}}/\sqrt{\text{MSE}})$, where MAX_I is the maximum possible pixel value of the image I, MSE is the mean square error of the image I with respect to the reference R, i.e., $\text{MSE}=\frac{1}{mn}{\displaystyle {\sum }_{j=1}^m{[\mathbf{I}(i,j)-\mathbf{R}(i,j)]}^2}$. We use the full-sampled, that is sampling ratio of 100% as the ground truth for quantitative evaluation, since the real ground truth is nearly inaccessible. It can be seen that sampling ratio of 30% already give good estimation (over 30 dB in terms of PSNR) of the scene, for both fluorescence and phase imaging, as shown in Fig. 4(a). Besides, the quantitative indices support the observation. The reconstructed PSNR of both fluorescence and phase images are over 30 dB (PSNR of over 30 dB is generally acceptable in most scenarios), as shown in Figs. 4(b) and 4(c), respectively.

 

Fig. 4 Results of compressive phase and fluorescence imaging under varying sampling ratios. (a) Fluorescence reconstructions are shown in the top row and phases are in the bottom. The sampling ratio increases from 10% to 70%, and the results from fully sampled measurements are given for comparison. Scale bar: 2 mm. Peak signal-to-noise ratio (PSNR in dB, higher is better) of the reconstructed fluorescence (b) and phase (c) images varying sampling ratios from 10% to 90%. The fully sampled measurements are used as the ground truth for calculating PSNR.

Download Full Size | PDF

Biological observations and diagnosis could benefit from simultaneous phase and fluorescence modalities and compressive imaging, especially for long-term imaging of slow phenomena [2]. Simultaneous phase and fluorescence modalities could provide both structural/morphological and functional information of the sample [3–5], which provides a comprehensive tool for the study of embryo development of model organisms, like Danio rerio. Besides, a single-pixel detector could be very sensitive, so the power of the laser source could be limited to a very low, safe and bio-compatible level. In this way, it would impose little impact on the development of organisms, which is beneficial for long-term imaging. The reduced number of measurements could further reduce the total amount of light energy on the organisms, and contribute to faster acquisition of a single frame.

5. Discussion and conclusion

We report a simultaneous phase and fluorescence imaging approach using single-pixel detectors. By combining reference-free interferometry with single-pixel detection, we encode the phase and fluorescence of the sample in two detection arms at the same time. This is the first attempt to explore these microscopic multi-modalities via single-pixel imaging without mechanical scanning, to the best of our knowledge. This approach is capable of multimodal imaging without expertise demanding alignment of multiple cameras or reduced spatial resolution due to spatial multiplexing. We tested the phase imaging accuracy with an etched glass, and demonstrated the ability of simultaneous phase and fluorescence imaging on a quantum dot sample, where the fluorescence and phase information matches well with each other. Besides, we experimentally show that we can get good estimation with low sampling ratio, which is promising for high-speed imaging and detection tasks [41, 42]. This method can be extended to other wavelengths, where array detectors are expensive or even unavailable, such as infrared imaging [12, 13, 28], Terahertz imaging [29, 30, 43], and X-ray imaging [31, 32].

Two main challenges are remaining for further pursuit. The first is the trade-offs of the acquisition speed, pixel resolution and quality of the reconstructed image, since single-pixel imaging (SPI) is a sequential imaging modality. The limitation lies in the refreshing speed of the spatial light modulator (SLM), denoted as f_SLM. The acquisition speed f_SPI is proportional to the refreshing speed of the SLM, and inversely proportional to the total number of pixels N and the sampling ratio η, that is f_SPI = f_SLM/(η · N). One could increase the speed by either breaking the refreshing limit of the SLM, like spatial sweeping [41], or decreasing the sampling ratio while maintaining acceptable reconstruction image quality, like compressive [6, 18, 20] and adaptive [44–46] sampling strategies. To this end, compressive sensing for complex-valued 2D signals is highly desirable since phase information lies in the complex-amplitude signal and limited work has been devoted to that problem, to the best of our knowledge. Another challenge is the light efficiency issue of this method, especially for wide-field fluorescence microscopy. As discussed in Section 4, the efficiency of complex field modulation using Lee hologram is ∼5% because of the diffraction effect of the DMD and only the +1st order beam in use. Though single-pixel detectors with increased sensitivity, like single-photon counting modules, or increased laser power could compensate for that purpose, it could add to the cost of the system, which might limit its usage in budget-thrift applications. This issue could be tackled by binary complex-amplitude modulation methods with higher efficiency or using fast spatial phase modulator (though slower than the DMD), like the fast liquid crystal on silicon (LCoS) [47] or the ferroelectric LCoS [48].

Moreover, with additional single-pixel detectors or a spectrometer, this system could be extended for multispectral fluorescence microscopy [6, 18]. Furthermore, we envision that this cost-efficient multimodal system might find broad applications in biomedical science and material science.