1. INTRODUCTION

The diffraction limit of light has traditionally imposed a restriction on the resolution attainable by classical microscopy, as studied by Abbe and Rayleigh [1,2]. Contemporary superresolution methods, such as localization [3–5], stimulated emission depletion (STED) [6], as well as structured illumination [7], and super-resolution optical fluctuation imaging (SOFI) [8] surpass the diffraction limit, enabling the recovery of subwavelength features. Image scanning microscopy (ISM) is an additional established imaging technique that achieves a twofold enhancement in resolution [9,10]. In its essence, ISM is based on an array of detectors, where each pixel acts as a small pinhole in a confocal scanning laser microscope. Therefore, ISM provides a lateral resolution twice above the diffraction limit, without compromising the collected signal level. Even though it offers a modest improvement, ISM is compatible with a standard confocal microscope configuration, and hence has already been incorporated into commercial products [11].

In a recent addition to the class of superresolution imaging methods, we have shown that a quantum optical effect, namely, photon antibunching, can be integrated in an ISM architecture to yield quantum image scanning microscopy (Q-ISM) [12]. Taking advantage of a quantum correlation measurement, a known source for resolution enhancement [13–16], Q-ISM offers a potential fourfold increase in resolution over the diffraction limit, with only minor modifications to a standard confocal microscope. However, apart from the resolution gain being moderate, the intrinsically weak level of quantum correlation signal demands long acquisition times that impede the practicality of Q-ISM. Such long exposure times stem from the inherently low probability for a detection of a photon pair from a single emitter.

Since the imaging process of Q-ISM inherently necessitates collection of “standard” ISM data, it simultaneously generates two variants of the imaged object: one is the low-resolution, low-noise ISM image, and the other is the noisier high-resolution Q-ISM image. Consequently, an algorithmic reconstruction that combines both images may lead to enhanced recovery of the object at short integration times.

In this work, we suggest combining the images using sparse recovery methods, and in particular ideas based on multiple measurement vector (MMV) recovery [17–19]. Sparsity-based signal recovery has become a useful tool in many applications, such as radar [20–22], ultrasound [23,24], ghost imaging [25], magnetic resonance imaging [26], and superresolution imaging [27–32]. Here we introduce a joint sparse recovery (JSR) algorithm to fuse the complementary ISM and Q-ISM images, exploiting the fact that the light measured in both images emanates from the same emitters.

We first demonstrate the utility of this approach with simulation results. We then continue to present experimental results of dispersed quantum dots (QDs), in which the restored superresolved optical images are correlated with ground truth images taken with a scanning electron microscope (SEM). Through the application of our JSR algorithm to the measurements, we were able to enhance resolution and provide a solid estimate of the emitter density, performing all this with a substantial reduction in the image acquisition time.

Finally, we show that through utilization of our JSR algorithm, quantum correlation signal from a biological sample, collected in 10 ms per pixel is sufficient for augmenting ISM reconstructions. Compared to the acquisition times of Q-ISM images in [12], the approach proposed here facilitates an order of magnitude shorter exposure times, without compromising the quality of the reconstructions.

2. JSR OF Q-ISM AND ISM

A. Acquisition of Q-ISM and ISM Images

In Q-ISM, a modified confocal microscope architecture is utilized in which the standard detector is replaced with a fast single-photon sensitive pixelated detector. During a measurement, the sample, labeled with single-photon emitters, is raster-scanned through a diffraction-limited focused laser beam. The emitted fluorescent light is then separated from the excitation light by a dichroic mirror and imaged upon a detector array [Fig. 1(a)]. Here, the detector array comprises 14 optical fibers in a bundle formation, the core of each of them being much smaller than the diffraction-limited spot size in the imaging plane. On its other end, the bundle fans out to separate fibers that guide the light to 14 individual single-photon avalanche detectors (SPADs). The SPADs are connected to a time-correlated single-photon counting (TCSPC) card that time-stamps the incoming photons with subnanosecond temporal resolution. Analysis of the collected data enables the reconstruction of a Q-ISM image [12]. Naturally, in conjunction with a Q-ISM image, one can construct its ISM counterpart, as it is derived from the same data, by simply counting the number of photons impinging on each detector. The Q-ISM and ISM images essentially represent the reduced variance and the mean of the photon counts data, respectively.

 

Fig. 1. Generation and synthesis of ISM and Q-ISM images. (a) Optical setup. The standard pinhole in a confocal microscope is replaced by a fiber bundle (FB), which guides the impinging fluorescent light to 14 individual SPADs. Exc, excitation laser; L, lens; DM, dichroic mirror; Obj, objective lens; (b) schematics of resolution enhancement in ISM and Q-ISM; (i) laser excitation profile, $h_{\mathrm{exc}}$; (ii) detection probability distribution, $h_{\det }$, shown for two point-like detectors (det number 2 and 4) with unity magnification; (iii) ISM PSF, $h_{\mathrm{ISM}}$, for each detector is the product of its detection distribution, and the excitation profile; (iv) effective Q-ISM PSF, $h_{\mathrm{Q}-\mathrm{ISM}}$, is the product of the two ISM PSFs of the two detectors. For simplicity, we use 1D illustrations and approximate the PSFs to be Gaussian. (c) ISM and Q-ISM images are integrated to generate a joint SR image (experimental results).

Download Full Size | PDF

Figure 1(b) depicts the resolution improvement mechanisms of both ISM and Q-ISM. The laser excitation point spread function (PSF) and the detection PSFs of point-like detectors are illustrated in panels (i) and (ii) of Fig. 1(b), respectively, (assuming unity magnification). Since a single-photon detection event requires both the absorption and detection of a photon, the effective PSF of a scanned image from a single detector is a product of the laser excitation profile and the detection PSF [Fig. 1(b-iii)]. If we approximate both PSFs by identical Gaussian profiles (neglecting the Stokes shift), the ISM PSF is simply the square of the wide-field PSF. Inescapably, the reduction of the detector size, in favor of resolution improvement, comes at the expense of only a small fraction of the light being collected [10]. However, ISM allows one to overcome this limitation by properly adding up contributions from all detectors in the extended array. The adequate summation of the individual images from the detectors is termed pixel reassignment and addresses the fact that different detectors record laterally shifted versions of the same image. Thus, naïvely, the ISM technique is capable of enhancing the resolution of the native microscope by a factor of $\sqrt{2}$, without suffering the penalty of signal level reduction. Furthermore, by digitally amplifying the high spatial frequency content, a process termed Fourier reweighting, one can achieve an image with up to twofold resolution enhancement [10].

Q-ISM images depend on measuring antibunching of the fluorescent light to improve resolution. In this scheme, pairs of fibers constitute a Hanbury–Brown and Twiss coincidence detection measurement, rather than counting the total number of photons. The fact that a particle cannot emit more than one photon at a time, causes the photons to be antibunched. Antibunching manifests itself as a reduction of simultaneous photon pairs compared to photon pairs separated by times longer than the fluorescence lifetime. Q-ISM uses these missing photon pair events, which involve absorption and emission of two photons, as the imaging contrast. Therefore, the effective PSF of Q-ISM is further narrowed and is veritably proportional to the excitation PSF squared multiplied by the detection PSF squared [Fig. 1(b-iv)], implying a straightforward twofold finer resolution than the diffraction limit. By performing Fourier reweighting, the resolution of the image could be increased by up to fourfold, and up to twofold relative to ISM [12]. Similar to ISM, application of pixel reassignment leads to proper summation of the signal from all detector pairs.

The use of an ultrafast detector, with temporal resolution beyond the fluorescence life time of the quantum emitters, has facilitated rapid measurement of the antibunching signal. Nevertheless, since in far-field microscopy the probability of detecting a single photon per emitter per laser pulse is much less than unity, missing events of two-photon detection are rather rare. As a result, the acquisition time required for a high signal-to-noise ratio (SNR) Q-ISM image is appreciably longer than that for an ISM image, thus posing a substantial limitation on Q-ISM in terms of a severe trade-off between SNR, collection time, and photobleaching.

More specifically, relatively long acquisition times, on the order of tens of milliseconds per scan step, were needed to produce a reasonable Q-ISM image superior to the ISM one [12]. In an endeavor to benefit from the enhanced resolution encapsulated in Q-ISM while circumventing exhaustively long measurements, we consider joint reconstruction of the two via sparse reconstruction (SR) algorithms and in particular, multiple measurements methods.

B. JSR Algorithm

We begin this section with a brief introduction to sparsity-based image reconstruction and continue with an outline of the JSR algorithm used here (described in detail in Supplement 1, Section 1). Generally, a measured image of size $M\times M$ pixels can be mathematically modeled by a discrete linear system $\mathit{y}=\mathit{A}\mathit{x}+\mathit{n}$. The vector ${\mathit{y}}_{M^2\times 1}$ is the measured blurred image, $\mathit{n}$ is an unknown noise or perturbation, and ${\mathit{x}}_{N^2\times 1}$ stands for the unknown “true” image to be estimated. Strictly speaking, $\mathit{x}$ is the signal that underlies the inevitable degradation that accompanies the image acquisition process. Both $\mathit{x}$ and $\mathit{y}$ vectors are constructed by stacking the columns of their respective two-dimensional images. In superresolution type reconstructions, we anticipate retrieving features smaller than the pixel size in the original image; hence, the recovered grid is set to be denser than the grid of the captured image; that is, $N>M$. The matrix ${\mathit{A}}_{M^2\times N^2}$ is a known sensing operator whose columns are subpixel-shifted copies of the PSF of the imaging system. A very prominent approach to solve the inverse problem of estimating the latent signal $\mathit{x}$ is through the use of prior information regarding the sparsity of the signal [33–35]. This means that in some basis representation, the signal has only a small percentage of nonzero elements. Sparse optimization has become a dominant tool in the field of compressed sensing [18,19,36–39], which, in the context of superresolution imaging, is aimed at robustly recovering information contained in spatial frequencies that were cut off by low-pass filtering as a result of diffraction [27]. While the requirement for sparsity of the image is very specific, it is nevertheless an incredibly prevalent property in nature [18,27,40].

One approach to combine sparsity in the recovery process is to formulate an optimization problem that minimizes the data error and a regularization term that incorporates the signal sparsity. A popular approach is to use the $l_1$ norm as a measure of sparsity, leading to the convex optimization problem,

An extension to the problem in Eq. (1) is the MMV problem. In this case, one deals with a set of measurements and unknown signals, ${\mathit{y}}_{(k)}$ and ${\mathit{x}}_{(k)}$, respectively, in which the vectors ${\mathit{x}}_{(k)}$ are jointly sparse, i.e., they have nonzero entries in the same locations [18]. Stacking these vectors into the columns of a matrix defines a row-sparse matrix $\mathit{X}$. In MMV optimization, the single-vector $l_1$ norm in Eq. (1) is replaced by a mixed norm of $\mathit{X}$ [19].

As described earlier, Q-ISM produces two independent variants of the underlying image, with complementary properties, from the same scan. The ISM image features low resolution but a high SNR, whereas the quantum enhanced image has a higher resolution, albeit it is noisier. Our goal is to incorporate the information from both images to obtain a better characterization of the density and locations of the emitters, as heuristically shown in Fig. 1(c). Even though the contrast mechanisms of the two images are not the same, the collected light originates from the same emitters. In other words, both images share in common the support of the signal. Therefore, our strategy is to employ a JSR algorithm to achieve superior reconstruction results, better than a sparse recovery from any of the individual images alone. We formulate the optimization problem in the following form:

3. RESULTS

A. Simulation Results

We begin with simulation results of emulations of shot-noise-limited ISM and Q-ISM images from a known scene of emitters (see Supplement 1, Section 2). Figure 2(a) shows the ground truth for these scans, smoothened with a narrow Gaussian PSF. Two data sets were created, with an order of magnitude difference in integration time, which show a considerable modification of the quality of the Q-ISM images. The first scan produced a reasonable Q-ISM image with SNR $\sim 6$, at the brightest pixels [Fig. 2(c)], while the second scan created a very noisy image with SNR $\sim 2$ at most [Fig. 2(h)]. In this context, we underscore that shot noise in Q-ISM is associated with the signal being the difference between two relatively large numbers (see [12]). Regarding ISM, the photon count level is sufficiently high, and hence the two versions do not exhibit any discernible differences, as can be seen in Figs. 2(b) and 2(g). Here, no postprocessing in terms of Fourier reweighting was applied to the raw image prior to algorithmic reconstruction. Thus, Q-ISM features a $\sqrt{2}$ improvement in resolution over ISM. Thereupon, each data set was processed separately. We implemented a sparse recovery algorithm according to Eq. (1) on each of the simulated images alone: ISM SR [Figs. 2(d) and 2(i)] and Q-ISM SR [Figs. 2(e) and 2(j)]. Finally, the solutions to the JSR problem [Eq. (2)] are shown in Figs. 2(f) and 2(k). The free parameters $\lambda$ and $\eta$ were tuned empirically. For consistency, all recovered images were smoothened with the same narrow Gaussian kernel that was used for the ground truth.

 

Fig. 2. Advantage of joint SR, simulation results; (a) ground truth of a scene of single photon emitters. Left-hand side, (b)–(f) optical images and reconstruction results from a scan featuring a 50 nm step size and 100 ms pixel dwell time; (b) ISM image; (c) Q-ISM image; (d) ISM SR; (e) Q-ISM SR; (f) JSR. Right-hand side, (g)–(k) optical images and reconstruction results from a scan featuring a 50 nm step size and 10 ms pixel dwell time; (g) ISM image; (h) Q-ISM image; (i) ISM SR; (j) Q-ISM SR; (k) JSR. Color bars of (b) and (g) represent detected photon counts. Color bars of (c) and (h) represent the number of missing detected photon pairs. Scale bar, 0.25 μm.

Download Full Size | PDF

Visually comparing the left-hand side panels of Fig. 2, we observe that when the correlation signal is sufficiently high, its exclusive reconstruction clearly outperforms the intensity-based one. Although not perfectly matching the ground truth, especially in terms of localizing the emitters, it is able to reveal finer features that are missing in the pure intensity method. Nonetheless, the joint method closely resembles the correlation-based reconstruction. In that case, the intensity image is essentially redundant and holds no added value.

In the lower SNR scenario, shown in the right-hand side panels of Fig. 2, the correlation signal is rather too low to facilitate an independent robust solution. It results in a degraded reconstruction, principally due to noise being erroneously interpreted as emitters. Strikingly, however, reconstruction according to the joint method eliminates these false positives, while preserving the finer details of the ground truth to a considerable extent. Put in a different way, the joint approach is capable of taking advantage of the information existing in a better resolved yet overwhelmingly noisy image, and enhance the low-resolution image reconstruction. We conducted many such simulation realizations, and consistently obtained similar behavior. Thus, the computational synthesis of the intensity and correlation images paves the way for the implementation of much faster Q-ISM measurements. Finally, we comment that as the SNR of the correlation image further decreases, its relative proportion in the data fidelity terms in Eq. (2) needs to be diminished, and consequently JSR reduces to the intensity reconstruction. Thus, adjustment of the relative weight guarantees that the incorporation of the noisy image could only boost the reconstruction process, and not deteriorate it. Since the resolution attainable by the algorithm is dependent on the noise and details of the scene, such as the unknown local emitter density, an unequivocal estimation of the resolution enhancement is rather convoluted. Nonetheless, Supplement 1, Section 3 illustrates the resolution improvement potential through simulations of the elementary case of two particles.

B. Experimental Results

We continue and demonstrate JSR on experimental Q-ISM and ISM data of QDs sparsely dispersed in a plane, which contains imperfections such as fluctuations and inaccurate knowledge of the PSF. The sample was prepared by drop-casting a diluted solution of QDs onto a marked glass coverslip, and was allowed to dry (see Supplement 1, Section 4 for further experiment details). The optical scans were correlated using a SEM, whose resolution was sufficient to separate individual QDs. Thus, SEM measurements played the role of ground truth measurements. The correlative measurement process is depicted in Supplement 1, Section 5.

Figure 3 compares the optical images, analyzed from a scan of a cluster of QDs, their computational reconstructions, and the matching SEM image. The optical images in Figs. 3(a) and 3(b) were processed from a comparably short scan, with 10 ms pixel dwell time, in which the Q-ISM image featured SNR $\sim 2$. The pixel reassignment procedure was carried out using a calibration measurement of a single fluorescent particle, much smaller than the laser beam PSF [12]. The calibration scan also provided knowledge of the ISM and Q-ISM PSFs required for the reconstruction process. The empirically measured PSFs were fitted with two-dimensional Gaussians, which were later used in the algorithm. With the support of the markings on the glass, we were able to locate and scan the very same scene utilizing the SEM [Fig. 3(c)]. Analogous to the simulation results, three possible reconstructions of the scene, ISM SR, Q-ISM SR, and JSR, are presented in Figs. 3(d)–3(f), respectively. All of the recovered images were smoothened with a narrow Gaussian PSF. For a clearer comparison, a refined SEM image in which the QDs were accentuated, was superimposed over each of the reconstructions [Figs. 3(g)–3(i)]. The refined SEM image was procured by pinpointing the QDs in the raw SEM image and marking them with approximately 20 nm circles, the average size of the particles (QDot 625, Thermo Fisher).

 

Fig. 3. SRs and correlative electron microscopy. (a) and (b) Images processed from a confocal scan (50 nm step size, using 10 ms segment from the pixel dwell time) of a cluster of fluorescent QDs; (a) ISM image; color bar represents number of detected photons; (b) Q-ISM image; color bar represents missing detected photon pairs; (c) SEM image of the same cluster that was optically measured. (d)–(f) Algorithmic reconstructions of the scene. (d) ISM SR; (e) Q-ISM SR; (f) JSR. (g)–(i) Refined correlative SEM image overlaid with each of the reconstructions. (g) ISM SR; (h) Q-ISM SR; (i) JSR. Reconstructions are in green, refined SEM image is in purple, and white represents the spatial overlap between them. Scale bar, 0.25 μm.

Download Full Size | PDF

A visual inspection of Fig. 3 leads to the observation that JSR achieves a more accurate characterization of the scene than any of the single measurements and their recoveries, in excellent agreement with the ground truth. On the one hand, the separation between particles in JSR is more salient than in ISM SR, and in particular, more particles are resolved. On the other hand, compared to Q-ISM SR, JSR delivers again a more precise description of the scene, especially mitigating the emergence of spurious emitters, which evidently hamper the reliability of the former.

An additional example of correlative light and electron microscopy of QDs and the corresponding algorithmic reconstructions is provided in Supplement 1, Section 6.

We further implement JSR on ISM and Q-ISM images of a biological sample of fixed 3T3, cells whose microtubules were labeled with QDs (see Supplement 1, Section 4). Although in this case the optical measurements lack a ground truth reference, we can still estimate the performance of JSR in an alternative way. According to the simulation analysis described earlier, at sufficiently high SNR, Q-ISM reconstruction alone is capable of providing optimal results, and the joint reconstruction is somewhat degenerate. Based on that premise, we evaluate JSR at a low signal level by comparing it to the reconstructions at relatively high SNR. Figure 4 presents two data sets that were derived from the same measurement. Figures 4(a) and 4(b) show the ISM and Q-ISM images analyzed from a scan with 100 ms pixel dwell time, whereas Figs. 4(f)–4(g) depict the optical images analyzed from a 10 ms segment of the signal collected in each scan step. The typical SNRs in the 100 ms exposure Q-ISM image and in the 10 ms exposure image are $\sim 3$ and $\sim 1$, respectively. Subsequently, each set of optical images was processed separately. We solved Eq. (1) for each of the images individually: ISM SR [Figs. 4(c) and 4(h)] and Q-ISM SR [Figs. 4(d) and 4(i)]. Ultimately, we implemented our JSR algorithm to solve Eq. (2), its results showing in Figs. 4(e) and 4(j). Once again, the free parameters $\lambda$ and $\eta$ were adjusted manually, and all recovered images were smoothened with a narrow Gaussian PSF. Examining the high SNR data set at Fig. 4, one observes that the Q-ISM reconstruction provides improved resolution compared to the ISM solution. Conspicuously, by resolving streak patterns into separated emitters or emitter clusters, Q-ISM SR achieves an overall more faithful description of the scene. Moreover, careful inspection of the corresponding JSR in Fig. 4(e) reveals appreciable similarity to the Q-ISM SR, explained by the partial redundancy of the ISM data. In contrast, the reconstruction of the noisy low-exposure Q-ISM image no longer succeeds in sustaining the overall structure of the object. Nevertheless, the corresponding JSR [Fig. 4(j)] features great resemblance to the high SNR restorations, validating its credibility and thus facilitating the employment of rapid Q-ISM scans.

 

Fig. 4. JSR of labeled microtubule cell samples. Left-hand side, (a)–(e), optical images and reconstruction results from a scan (50 nm step size, 100 ms pixel dwell time) of microtubules in a fixed 3T3 cell labeled with fluorescent QDs (QDot 625, Thermo Fisher). (a) ISM image; (b) Q-ISM image; (c) ISM SR; (d) Q-ISM SR; (e) JSR. Right-hand side, (f)–(j), optical images and reconstruction results from the same scan in (a) and (b), with a digitally cropped 10 ms pixel dwell time; (f) ISM image; (g) Q-ISM image; (h) ISM SR; (i) Q-ISM SR; (j) JSR. Color bars of (a) and (f) represent detected photon counts. Color bars of (b) and (g) represent the number of missing detected photon pairs. Scale bar, 0.5 μm.

Download Full Size | PDF

4. CONCLUSIONS AND DISCUSSION

We demonstrated a simple JSR algorithm that merged ISM and Q-ISM images. Synthesis of the images allowed us to take advantage of the somewhat concealed valuable information in Q-ISM acquired in an order of magnitude less time than needed for a moderate SNR image. With the experimental conditions in this work, we were able to reduce the pixel dwell time to 10 ms and still benefit from the advantages encompassed in the photon correlation measurement. We speculate that optimization of several experimental parameters (e.g., a higher laser repetition rate and higher quantum yield emitters with improved photostability) may improve the signal and therefore the recovered images. Under these circumstances, it is only reasonable to anticipate that pixel dwell time could be further decreased to the 1 ms scale.

We have shown that SRs are capable of improving the resolution of the input image, under the frequently fulfilled condition that the object is sparse. In this work, we had only few emitters per diffraction limited spot; hence, the object was sparse in the real-space domain. Notwithstanding, depending on the image, other sparse representations could be used, e.g., discrete wavelet bases [40].

The correlation with electron microscope images provided us with ground truth to test the algorithm and determine the best parameters. However, in order to establish the reliability of blind reconstructions of various types of scenes and experimental conditions, a suitable autonomous choice of the parameters is needed. A possible way to realize the optimization of parameters would be through techniques from the field of machine learning (see, for example, [44]).

At this stage, the algorithm was supplied with only two complementary images and their PSFs. Nevertheless, additional implicit information could be integrated into the algorithm. For instance, knowledge of the intensity and second-order correlation can provide an estimation to the number of emitters in a certain area [45,46], which might be exploited to boost the performance of the algorithm.

We note that joint recovery of complementary images in terms of resolution and SNR, presented in this work, could generally be applied in other situations as well, where complementary information is measured simultaneously, but often part of it is unused. For example, application to SOFI microscopy [8], gated-STED [47], quantum imaging by centroid estimation of biphotons [48], and to the proposed structured illumination quantum correlation microscopy [49], might assist in shortening the required exposure times.