1. Introduction

The Rayleigh resolution criterion is commonly used for estimating the resolution limit of optical imaging instruments such as telescopes and microscopes. It states that two incoherent point sources, when projected to the image plane, have to be at a certain distance d_R apart to be distinguishable [1,2]. This distance is approximately equal to the full width at half maximum (FWHM) of the imaging system point spread function (PSF), and is commonly estimated as d_R ≈0.6λ/NA [2], where λ is the wavelength of light and NA the lens numerical aperture. Although the rationale for the Rayleigh resolution criterion [1] can be easily understood, we should note that it is only an empirical rule for estimating resolution in incoherent imaging systems. Indeed, it has been experimentally demonstrated that resolution three times finer than the Rayleigh resolution limit can be achieved through appropriate computational inversions for incoherent imaging [3], and the mechanisms responsible for such superresolving restorations have also been carefully studied [4,5].

In [4], Donoho et al. argue that the non-negativity and sparsity (i.e., the portion of an object that is zero or nearly zero, described as “near-blackness” in [4]) of the object are critical for achieving superresolution in the presence of noise. In [5], Sementilli et al. show that for noise-free imaging systems, least-squares inversions exploiting the non-negativity of the object can sometimes achieve perfect restorations, i.e., restorations identical to the objects. It also shows the effect of noise on spatial frequency extent, and concludes that resolution enhancement is inversely related to the object size. While object non-negativity is a natural constraint to impose for incoherent imaging, it is not practical to restrain the physical size of the object or object sparsity. For this reason, the Rayleigh resolution limit is generally regarded as a reasonable estimate for incoherent imaging resolution [6], and the efforts to establish superresolution techniques in the past few decades have turned to exploiting the physical properties of fluorophores, mainly fluorescence stimulated emission [7,8] and fluorescing state transitions (i.e., the fluorescent molecules undergoing transitions between fluorescing and non-fluorescing states) [9–12]. The main exception is structured illumination microscopy [13], which employs frequency mixing between periodic illumination patterns and the imaging system modulation transfer function (MTF) to access spatial frequencies higher than the imaging system cutoff frequency 1/d_R. With such an approach, structured illumination microscopy fundamentally limits itself to a two-fold resolution enhancement over the Rayleigh resolution limit. This is because the highest accessible frequency through such a frequency-mixing approach is the sum of the highest possible illumination spatial frequency and the upper bound of the imaging system MTF, both of which are ~1/d_R [13]. Surpassing the two-fold resolution enhancement following this methodology ultimately relies on fluorescing state transitions [14].

Returning to the previous discovery that spatial extent of objects and object sparsity can affect resolution enhancement of computational inversions [5], fluorescence microscopy might provide us with an opportunity to artificially increase sparsity and reduce the effective object size by illuminating only a small region, instead of the entire region within the camera field of view. This would certainly reduce the field of view, so one can then steer the small illumination region to many positions and acquire a series of images to cover the region of interest, and apply the computational inversions to individual images. With a standard fluorescence microscope, the smallest illumination region one can easily create is the focal spot formed by focusing a collimated beam through the microscope objective, a common practice in laser scanning confocal microscopy [15]. The goal of this research work is to experimentally determine whether using such a focused-spot illumination in conjunction with non-negative least-squares (NNLS) inversions achieves resolution enhancement more than two-fold over the Rayleigh resolution limit, and to quantitatively assess the resolution enhancement under the photon budgets of common fluorescent samples.

2. Two-point object simulation experiment

Before we examine whether using a focused illumination spot can help enhance superresolution, we first verify that we can reproduce the superresolving restorations through NNLS inversions as described in previous literature [3–5], and further examine the quantitative aspects of these restorations. Extending the concept of experiments in [3] to a two-dimensional imaging scenario at a variety of signal levels, we design a simulation experiment where we: 1) place two incoherent point sources at a distance d equal to or smaller than the Rayleigh distance d_R, 2) numerically model the acquired images with various photon numbers assuming photon shot noise, and 3) restore the objects from these images through NNLS inversions. Figure 1 summarizes our simulation experiment results. Figure 1(a) shows the object consisting of two incoherent point sources and the imaging system point spread function as detected by an array of pixelated detectors on a camera, and Fig. 1(b) shows acquired images with simulated Poisson noise at progressively increasing photon numbers. Because of the close proximity of the two point sources, it is expected that they will be visually indistinguishable in all of the acquired images.

 

Fig. 1 Resolving two incoherent point sources using NNLS inversions at various point-to-point distances and photon numbers. (a), (b) Simulated image formation where a two-point object convolves with the imaging system PSF to form the acquired images, assuming 1:1 magnification. The ‘*’ sign denotes a convolution operation. Each simulated acquired image in (b) is a representative image out of 403 independent simulations, assuming Poisson noise. (c) The criteria determining whether the two point sources are discernable in a particular restoration: successful discernment is declared only when both the average and maximum values in Region1 and Region2 are greater than those in Region0. The two white dots indicate the positions of the two point sources. (d) NNLS restorations corresponding to acquired images in (b). (e) Discernment rates versus photon numbers at various two-point distances. (f) Estimated number of photons per emitter that are required to achieve a 75% discernment rate for various resolution enhancement, defined as d_R/d. See Appendix A for further details.

Download Full Size | PDF

For the NNLS inversions, we create a dictionary of PSFs that represent the images of individual point sources at various positions; these positions are visualized as the yellow grid line intersections in Figs. 1(c) and 1(d). The grid is chosen so that the exact point sources are in between grid intersections, which is the most difficult case computationally. We define d_Dict as the distance between adjacent dictionary PSFs. The inversion algorithm then finds the brightness combination of the dictionary PSFs that minimizes squared differences when compared to the acquired image of the two-point-source object, under the constraint of non-negative PSF brightness. In order to discuss very general models, no additional regularization terms are used throughout this research work, though we note that adding certain maximum likelihood regularizing terms would likely increase resolution even further. Using the criteria described in Fig. 1(c), we determine whether the restorations, examples of which are shown in Fig. 1(d), resolve the two point sources, and then estimate the discernment rate by computationally repeating this experiment over many trials.

Figure 1(e) shows that the discernment rates are positively associated with photon numbers for a variety of point source distances. Consistent with the predictions of previous literature [4,5], we observe the discernment rates approach 100% as the photon number increases, no matter how close the two point sources are. We can also see that more photons are required to distinguish the point sources as they are placed closer (Fig. 1(f)). Interestingly, we find that surpassing the Rayleigh resolution limit by two-fold, represented here by the case where d = d_R/2 (Fig. 1(e), dark blue curve), can be done at a ~70% rate with ~300 photons per emitter, or greater than 80% with 1000 photons per emitter, both of which are well within the photon number range for commonly used fluorophores in fluorescence microscopy [16]. These observations suggest that the belief of the necessity of stimulated emission or fluorescing state transition for achieving more-than-two-fold resolution enhancement over the Rayleigh resolution limit is not precise, since resolution enhancement depends on the number of photons, and can easily go beyond two-fold improvement with reasonable photon numbers.

In this simulation experiment and subsequent NNLS processes in this work, d_Dict is usually smaller than the effective camera pixel size, d_CP. Therefore, to generate a PSF dictionary, we first calculate a group of PSFs, each of which represents the PSF at a particular sub-pixel position. For instance, in the case of Fig. 1(c) where d_Dict = d_CP/8, the dictionary PSFs have 8² = 64 distinct sub-pixel positions. We calculate the 64 PSFs for the distinct sub-pixel positions, and then use these 64 PSFs as templates to create individual dictionary PSFs based on their shifts relative to their templates. This technique of creating PSF dictionaries is used throughout the presented work.

3. Three-line object simulation experiment comparing widefield and focused spot illuminations

Going beyond superresolving point sources in close proximity, we now explore whether using a focused spot to illuminate an extended object at several positions provides better restorations than illuminating the entire object with uniform illumination. Figure 2 shows the simulation experiment we design for such a test. In this simulation experiment, the object consists of three lines of a 1.5d_R length, spaced d_R/2 away from each other (Fig. 2(a)). To further increase the object complexity, we set the brightness of the middle line to 3/4 of the top and bottom lines. In our simulation, each line consists of 19 incoherent point sources equally spaced along the 1.5d_R length. We step the focused illumination spot through 25 positions, where the centers of the illumination spots are represented by the cyan crosses in Fig. 2(a) overlayed with the object.

 

Fig. 2 Comparison of NNLS restorations from widefield and focused-spot illuminations. (a) An object consisting of three lines and an overlay of the object and 25 illumination spot positions, denoted by the cyan crosses. The brightness of the middle line is 3/4 of the top/bottom line. (b) Simulated two-dimensional intensity profiles of the illumination spots. (c) The object illuminated by the 25 focused spots individually. (d) Simulated acquired images with individual focused-spot illuminations. (e) Individual NNLS restorations of acquired images in (d), with d_Dict = d_R/12. (f) Sum of restorations in (e). (g) The effective widefield image. (h) NNLS restoration of (g). (i) Definition of the line restoration quality (LRQ) of a particular object line: the average of the values (Avg) within the line region divided by their standard deviations (Std). (j) LRQs versus number of photons for spot illumination and widefield illumination. Due to the symmetry of Region1 and Region3, we average their LRQs in the plot, and denote such an average as LRQ_1,3. See Appendix B for further details.

Download Full Size | PDF

We simulate the intensity profiles of these illumination spots (Fig. 2(b)), assuming they are created by focusing a collimated beam through the microscope objective at various incident angles. We derive the acquired 25 images (Fig. 2(d)) by convolving the imaging system PSF with the product of individual illumination intensity profiles and the object (Fig. 2(c)). All the acquired images are simulated assuming Poisson noise. We then apply NNLS inversions to individual acquired images and sum up the 25 restorations to obtain the final restoration (Figs. 2(e) and 2(f)).

Considering the image formation process illustrated from Figs. 2(b) to 2(d) as a linear system, we can obtain an effective widefield image (Fig. 2(g)) with nearly uniform illumination by summing up the acquired images in Fig. 2(d). We derive the widefield restoration (Fig. 2(h)) by applying the same NNLS inversion on the widefield image Fig. 2(g). In Fig. 2(i) we define line restoration quality (LRQ) for quantitative evaluations of the three original object lines, denoted as Region1-3. In Fig. 2(j), we can see the focused-spot illumination approach not only shows higher LRQs compared to the widefield illumination approach, but its LRQs also improve faster as the photon number increases. These results show that with the same numbers of detected photons, using a focused spot for illuminating an extended object significantly improves the restoration quality compared to uniform illumination in conventional widefield microscope configurations. We also note that the LRQs of the middle line (LRQ₂) are lower than those of the top and bottom lines, which is consistent with the finding in [3]. Since the lines are spaced d_R/2 apart, this experiment also shows resolution more than two-fold past the Rayleigh resolution limit.

4. Imaging experiments

In order to experimentally evaluate the performance and feasibility of our proposed superresolution imaging approach, we modified an inverted fluorescence microscope by removing its widefield epi-illuminator and replacing that with a custom-built laser scanning unit. Figure 3 shows results of two imaging experiments we performed using this focused-spot illumination microscope in conjunction with NNLS inversions.

 

Fig. 3 Results of two imaging experiments using focused spot illumination in conjunction with NNLS inversions. (a)-(d) Imaging experiment using a 70nm resolution test sample as the object. (e)-(h) Imaging experiment using fluorescently stained mitochondria as the object. (a) Effective widefield image. (b) Restoration from our method, with d_Dict = d_CP/12. (c) Enlarged restoration and widefield image of selected regions. The enlarged widefield images are interpolated using a MATLAB ‘bicubic’ method. (d) One-dimensional intensity profiles of the six DNA origami-fluorescent dye structures. (e) Effective 0.5NA widefield image. (f) Restoration from our method, obtained by summing up the restorations from individual 0.5NA images using focused-spot illumination, with d_Dict = d_CP/3. (g) 1.4NA Widefield image of the same region. (h) Average strengths of the 2D DFTs in (e)-(g) versus spatial frequency radius. See Appendix H for further details for calculating frequency ring average.

Download Full Size | PDF

In the first imaging experiment, we used a 1.4NA objective to image a resolution test sample that contains DNA origami-fluorescent dye structures (Figs. 3(a)-3(d)). These molecular structures are engineered such that each has a DNA origami in the middle of a pair of fluorescent markers, and the DNA origami separates the two fluorescent markers by ~70nm. Since the dominant emission wavelength of this fluorophore is ~530nm, the Rayleigh resolution limit d_R is ~230nm, meaning that resolving the ~70nm separation between each fluorescent marker pair requires more than three-fold resolution enhancement over the Rayleigh resolution limit.

In the second imaging experiment, we add object complexities into consideration by imaging a fluorescently labeled biological sample (Figs. 3(e)-3(i)). Compared to the usage of engineered DNA origami structures, we have little knowledge of the true object that represents a biological sample, and therefore we established a different approach to evaluate the quality of restoration. We first image the sample with a 0.5NA objective using our method, and then switch to a 1.4NA objective to image the sample again under conventional widefield illumination. Such an experiment design allows us to evaluate the 0.5NA restoration by comparing it with the 1.4NA widefield image. If the 0.5NA restoration resolves structural details comparable to the 1.4NA widefield image, we can conclude that our approach achieves more than two-fold resolution improvement, without having complete knowledge of the true object.

For both imaging experiments, the scanning step sizes are approximately equal to their effective camera pixel sizes (i.e., the physical camera pixel size divided by the magnification of the imaging system). For imaging the 70nm resolution test sample, this size is ~103nm. For imaging the biological sample using a 0.5NA objective, this size is ~325nm.

4.1 Imaging the 70nm resolution test sample

In the first imaging experiment, we stepped the focused illumination spot to cover a square area on the resolution test sample, applied NNLS inversions to individual acquired images, and summed up individual restorations to obtain the final restoration. Figure 3(a) shows the effective widefield image, derived by integrating all the acquired images. In the restoration (Figs. 3(b) and 3(c)), we can see many of the fluorescent marker pairs are clearly resolved, while they are visually indistinguishable in the widefield image (Figs. 3(a) and 3(c)). In Fig. 3(d), we measure the restored intensity profiles for six fluorescent marker pairs, and we can see that our method clearly resolves the expected two-peak profiles. Nevertheless, we note that that the measured peak-to-peak distances have an average of ~60nm and a standard deviation of ~7nm. The discrepancy between the average peak-to-peak distance and the 70nm molecular design of the sample may come from the orientations of the fluorescent markers with respect to the DNA origami, as the size of fluorescent molecules (Alexa Fluor 488) is at a ~10nm scale, and each fluorescent marker has ~13 fluorescent molecules according to the manufacturer. Slight mismatch between the PSF of the physical imaging system and the simulated PSF dictionary may also cause systematic errors that contribute to this discrepancy.

4.2 Low NA-high NA comparison

In the second imaging experiment, we imaged mitochondria in a fixed cell labeled with Red CMXRos, whose emission wavelength is ~600nm. Typical mitochondrial compartments are at sub-µm scales [17], and therefore the effective widefield 0.5NA image (Fig. 3(e)), derived from summing up all the acquired images, hardly resolves such structures. In contrast, the 0.5NA restoration (Fig. 3(f)) shows these compartments clearly, and the resolved sub-µm structures are highly consistent with the 1.4NA widefield image (Fig. 3(g)). We can also compare these results in the frequency domain by applying the two-dimensional discrete Fourier transform (2D DFT). In Fig. 3(h), the 0.5NA restoration indeed shows similar strengths as the 1.4NA widefield image at frequencies higher than the theoretical cutoff frequency of the 0.5NA objective at a ~600nm wavelength. While the 0.5NA restoration and the 1.4NA widefield image show certain differences in a few detailed structures, without complete knowledge of the true object, it is difficult to determine whether this is because the 0.5NA restoration resolves even finer structures, or if those differences are actually computational artifacts. Interestingly, near the bottom-right corner of the 1.4NA widefield image is a triangular shape structure that appears blurry because it is outside of the shallow depth of field of the 1.4NA objective. Since the 0.5NA objective has a much longer depth of field, the 0.5NA restoration is able to resolve this structure clearly.

Appendices C, D and F provide further details for the imaging system, the NNLS algorithm, the test samples, and imaging experiment parameters.

5. Examining signal-resolution relation under low-light conditions

From simulation results presented in Figs. 1 and 2, we have seen that the restoration resolution and quality are highly dependent on detected photon numbers, and we expect that similar relations also exist for complex biological samples. In fact, the resolution enhancement could even be arbitrarily high given an unlimited number of photons [4,5]. However, because of photobleaching [18,19], photon budgets in fluorescence microscopy are generally limited. To ensure the feasibility of our method in fluorescence microscopy, we further examine the resolution enhancement under the circumstances where the photon budgets are very limited. In Appendix E, we estimate that the maximum photon counts in an effective widefield image acquired by our microscope, n_MPC, can be as high as 3.2 × 10⁵ before significant photobleaching occurs.

To experimentally examine the effect of photon numbers, Fig. 4 presents results of a series of imaging experiments where we image the mitochondria sample with distinct excitation laser powers. Figures 4(a)-4(c) show that the effective widefield images become less noisy as photon counts increase, and Fig. 4(d) shows their average strengths in the frequency domain exhibit similar decaying trends. This is consistent with traditional understanding of optical resolution that, for widefield images without processing, increased signal-to-noise ratios (SNRs) do not improve resolution. By comparison, the restorations (Figs. 4(e)-4(g)) using our method, as we expect, progressively resolve more details as the detected photon number increases. We should note that, while the theoretical d_R of a 1.4NA objective at a ~600nm wavelength is ~260nm, the FWHM of the PSF is experimentally measured ~330nm (Fig. 4(d) inset), in agreement with PSF FWHM statistics reported in [20] for a similar objective lens. Such discrepancy between theoretical estimation and experimental measurement may result from minor aberrations in the imaging system. Therefore, we estimate the effective d_R for this imaging experiment series to be 330nm, and the cutoff frequency for widefield imaging is consequently 1/d_R ≈3µm⁻¹ (Fig. 4(d) gray arrow). This estimated cutoff frequency agrees well with the decaying trend of the frequency strength curves in Fig. 4(d).

 

Fig. 4 The resolution enhancement provided by our method shows steady increases with signal-to-noise ratios. (a)-(c) Effective widefield images obtained by summing up the acquired focused-spot illumination images at three excitation laser powers. Objects are fluorescently stained mitochondria in fixed cells. The three laser powers are selected such that the photon counts in the effective widefield images are significantly lower than those can be supported by commonly used fluorophores. (d) Average strengths of widefield image 2D DFTs versus spatial frequency radius. The inset shows an experimentally measured imaging system PSF. (e)-(g) Restorations from our method, obtained by summing up the restorations from individual acquired images that are used to generate effective widefield images (a)-(c) respectively. (h) Correlation coefficients of restoration 2D DFTs versus spatial frequency radius. 1/7 threshold is used to determine the effective cutoff frequencies of the restorations. Appendices G and H provide details for estimating signal-to-noise ratios and frequency ring correlation.

Download Full Size | PDF

To characterize the resolution enhancement without complete knowledge of the true objects, we utilize a frequency analysis-based method known as Fourier ring correlation [21,22] for resolution estimation (Fig. 4(h)). For such analysis, we acquire two images at each focused spot position and create two independent data sets. We then apply NNLS to both data sets, perform the 2D DFT on their restorations, and calculate correlation coefficients at various ring regions in their frequency domains. In particular, we adopt the 1/7 correlation threshold from [22] for determining effective cutoff frequencies. Among the results of three distinct SNRs, Fig. 4(a) exemplifies the case where the signal level is extremely low. With maximum photon counts merely around 300, the restoration, Fig. 4(e), already exhibits a ~4.2µm⁻¹ cutoff frequency in the Fourier ring correlation (Fig. 4(h), cyan curve), indicating a ~1.4-fold resolution enhancement over 1/d_R. In addition, by comparing Figs. 4(a) and 4(e), we observe significant contrast improvement using our method. Some of this effect is due to our post-processing, since the NNLS inversion helps eliminate noises in a similar manner as a high-frequency noise filter [4]. When we increase the excitation laser power by ~10 times, the maximum photon counts increase to ~2,000 (Fig. 4(b)), which is still significantly lower than n_MPC, and the restoration (Fig. 4(f)) shows discernible improvement compared to Fig. 4(e). Fourier ring correlation (Fig. 4(h), blue curve) indicates such a signal level is sufficient to increase the cutoff frequency to ~6µm⁻¹, ~2 times greater than 1/d_R. When we further increase the excitation laser power such that maximum photon counts are ~10⁴ (Fig. 4(c)), approximately n_MPC/10, the restoration resolution enhancement increases to ~2.4-fold (Fig. 4(g) and green curve in Fig. 4(h)). These results verify that the superresolution provided by our imaging method is associated with detected photon numbers, and is able to achieve more than two-fold resolution enhancement over the Rayleigh resolution limit with practical photon budgets. Note that while we present Figs. 4(e)-4(g) with PSF dictionary grid sizes that best visualize resolution enhancement of individual restorations, we calculate Fourier ring correlations for all three photon levels based on their restorations at d_Dict≈17.2nm. Figure 5 shows restorations at the three distinct PSF dictionary grid sizes and signal levels.

 

Fig. 5 Restorations obtained using our method, with various dictionary PSF spacings and signal levels. (a)-(c) Center positions (blue dots) of dictionary PSFs at d_Dict ≈103.2nm, 51.6nm, or 17.2nm, respectively. The green dashed circle indicates the first dark fringe of a diffraction-limited focused illumination spot. (d)-(f) Effective widefield images at three signal levels. (g)-(o) Restorations obtained using our method with 9 distinct combinations of d_Dict and the signal level.

Download Full Size | PDF

6. Discussions

Consistent with the findings in previous theoretical and simulation studies, we experimentally show that NNLS inversions have the capability to achieve signal-dependent superresolution in incoherent imaging. We further demonstrate that using a focused spot for illumination enhances this superresolution capability significantly, and we experimentally demonstrate more-than-2-fold resolution enhancement over the Rayleigh resolution limit. Most importantly, we show such resolution enhancement is achievable with practical photon budgets. By not using stimulated emission or exploiting fluorescing state transition, our method is not subject to the limitations of most existing superresolution techniques. For instance, compared to stimulated emission depletion microscopy [7,8], our method does not throw away fluorescent photons by stimulated emission to increase resolution, but instead utilizes all detected fluorescent photons for NNLS inversions, and therefore can potentially achieve similar resolution enhancement with less photobleaching. Compared to superresolution techniques that require photoswitchable dyes or photoactivable fluorescent proteins for fluorescing state transitions [9,10,14], our method is applicable to any fluorescent proteins and dyes and thus sets no limits to the choice of fluorophores. Compared to those that utilize fluorescing state transitions of common fluorescent dyes to achieve superresolution [11,12], our method still excels in simplicity, because adjusting the fluorescing state transition rate appropriately for image acquisition requires careful sample preconditioning [23].

Another critical difference between our method and those utilizing stochastic fluorescing state transitions [9–12] is the limit of imaging speed. Superresolution techniques that require stochastic fluorescing state transitions typically acquire many hundreds to thousands of images to average out and/or characterize temporal behaviors of the fluorophores. Such practices set fundamental limits to imaging speed, and these limits can range from seconds to hours [9–12], depending on the particular techniques and fluorophores in use. In contrast, because our method relies on only fluorescence excitation and emission, the fundamental limit to imaging speed is the fluorophore excited-state lifetime, typically at nanosecond scales [24]. Therefore, the imaging speed of our method can be increased almost arbitrarily through instrumentation improvements, such as creating multiple illumination spots simultaneously, using faster cameras, and employing beam scanning mechanisms with higher speeds.

While our research work points out a new approach to signal-dependent superresolution for fluorescence microscopy, many interesting questions await further exploration. These include: how to determine optimal camera pixel sizes, whether the conventional Airy-disk PSF is the most efficient PSF for resolution enhancement, or whether and how this method can be generalized for superresolving three-dimensional objects. We hope our findings will open a new window for the science of superresolution microscopy, and eventually enable optical observations that are inaccessible through existing technologies.

Appendix A two-point object simulation experiment

We developed the simulation using MATLAB (MathWorks). Based on a vectorial-field model [25], we first simulate the image formed by each of the point sources (i.e., image system PSFs) assuming the camera pixel size to be the distance between the two point sources d, and sum these two images to create a noiseless image. Based on the noiseless image, we then generate Poisson-noise images with designated photon numbers using the function poissrnd.

To build PSF dictionaries for NNLS inversions, we simulate PSFs at 17-by-17 positions in a square-lattice pattern surrounding the two point sources. These 289 positions are visualized as the yellow grid intersections in Figs. 1(c) and 1(d). To avoid over-optimistic restorations, we place the two point sources (white dots in Fig. 1(c)) between the dictionary PSF positions without exactly overlapping with any of them. The PSF dictionary grid size d_Dict, i.e., the distance between two adjacent dictionary PSFs, is set to be d/8. For NNLS inversions, we utilize MATLAB quadratic programming quadprog with its interior-point-convex (IPC) algorithm and default stopping tolerances, and enforce that all elements in the solution vector x have to be equal to or greater than zero.

We estimate the numbers of photons required for 75% discernment rate by linearly interpolating the data points in Fig. 1(e).

Appendix B three-line object simulation experiment

In this simulation, we use 19 point sources to represent each line, and therefore the entire object consists of 57 point sources. For each line, the 19 point sources are equally spaced in a 1.5d_R length and thus the distance between two adjacent point sources is 1.5d_R/(19-1) = d_R/12. The intensity distributions for the 25 focused illumination spots (Fig. 2(b)) are simulated based on the same vectorial-field model [25] used to create imaging system PSFs. We generate a noiseless image for each of the 57 point sources assuming a d_R/2 camera pixel size, and sum up these images with their brightness shown in Fig. 2(c) as weighting factors to derive 25 noiseless images, each for a particular illumination spot position. The PSF dictionary consists of PSFs from 19-by-19 positions arranged in a square lattice pattern, with d_Dict = d_R/12. NNLS inversions are performed using the same function and settings as in the two-point object simulation.

Appendix C the imaging system

The imaging system consists of an inverted microscope stand (Axiovert 100M, Zeiss), an SCMOS camera (Zyla 5.5, Andor Technology), a laser scanning unit, and a computer. We used µManager software (Open Imaging) for camera operations, and developed a LabVIEW (National Instruments) program that synchronizes the camera and the mirror scanners. We built a laser scanning unit with a pair of galvanometric mirror scanners (6215H, Cambridge Technology). The unit has two lasers: a 488nm diode-pump solid-state laser, which is used for exciting the DNA origami sample, and a 543nm Helium-Neon laser (LHGR-0100, Research Electro-Optics, Inc.), which is used for exciting the mitochondria sample. Both laser beams are spatially filtered and magnified to a ~4mm diameter when reaching the mirror scanners. The laser scanning unit has a pair of plano-convex lenses in a 4-f configuration between the microscope objective and the galvanometric mirror scanners to magnify the scanned laser beam by two times and project the beam to the microscope objective back aperture. Several neutral density filters were used to adjust the laser power. A Zeiss Plan-NEOFLUAR 20X NA0.5 microscope objective is used for 0.5NA imaging (Figs. 3(e) and 3(f)), while a Zeiss Plan-APOCHROMAT 63X NA1.4 microscope objective is used for 1.4NA imaging (Figs. 3(a)-3(d), 3(g) and Fig. 4).

The scales of acquired images are measured using a calibration micrometer (474026-0000-000, Zeiss). When using the 0.5NA objective, the effective camera pixel size is ~325nm. For the 1.4NA objective, the effective camera pixel size is ~103nm.

Appendix D applying first-order NNLS inversions and creating final restorations for experimentally acquired images

To perform NNLS inversions for experimentally acquired images (Figs. 3 and 4), we utilize a first-order optimization algorithm from the TFOCS package [26], instead of the MATLAB IPC algorithm. This is because the numbers of acquired images in actual experiments are much larger than in the simulation experiments (Figs. 1 and 2), and, for our imaging method, the first-order approach is able to solve many images much faster than the IPC algorithm with comparable restorations. The algorithm in TFOCS has proven convergence, and can guarantee that its output is within a specified tolerance of the true solution.

For each experimentally acquired image, we first crop it such that only a relatively small square region centering its focused illumination spot remains. These small images, approximately 30-40 camera pixels in width, are then individually used for NNLS inversions described by the following optimization problem:

(1)$$\underset{x,x_i\ge 0}{{\displaystyle \text{min}}}{\Vert {\displaystyle \sum _{i=1}^{n_{\text{DictPSF}}}{x}_i}{I}_{\text{PSF}i}-I_{\text{Image}}\Vert }_2^2,$$

where n_DictPSF is the number of dictionary PSFs representing images of individual point sources at a set of positions, I_PSFi is the intensity profile of the i-th dictionary PSF, and I_Image is the intensity profile of a cropped image. After the algorithm finds the optimal dictionary PSF brightness combination vector x from each cropped image, we locate each x with respect to its illumination position, and summed up all the x’s to create the final restoration.

Because the computation time needed for the above optimization problem increases with n_DictPSF, we built PSF dictionaries using positions only within a circular range whose diameter is ~2.4λ_Ex/NA (i.e., approximately twice the diameter of the first dark fringe in a diffraction-limited focused illumination spot, Figs. 5(a)-5(c)), where λ_Ex is the excitation light wavelength. In our experiences, this range is sufficient to extract useful information in the cropped images. Further increasing the dictionary range lengthens the computation time while yielding similar restorations. For a diffraction-limited focal spot, such a range contains more than 90% of its energy.

With the restoration method described above, it is straightforward to derive that the effective pixel size in the final restoration is the PSF dictionary grid size d_Dict. Figure 5 shows a table of restorations at various SNRs and values of d_Dict, which allows us to observe how the signal level limits the minimal d_Dict that can be used for meaningful restorations. For instance, in Figs. 5(h), 5(i) and 5(l), where d_Dict is much smaller than the resolution enhancement, the restorations show highly scattered structures that do not meaningfully represent the objects. These artifacts disappear in the higher SNR restorations. We always set d_Dict as the effective camera pixel size d_CP divided by an integer. For instance, this integer is 1 in Fig. 5(a), 2 in Fig. 5(b), and 6 in Fig 5(c). Because the scanning step size is always set equal to the effective camera pixel size, such a setting greatly simplifies the procedures of creating PSF dictionaries and the assembly of individual NNLS restorations.

 

Fig. 6 Illustrations of photon number estimation. (a) The object is assumed to be a 100nm square carrying 10 fluorescent protein molecules. The white squares represent the locations of the 10 molecules. (b) The microscope objective only collects fluorescent photons emitted into its collection angle. Here, Fluorescent Photon 1 and 2 respectively exemplify photons that will or will not be collected by the microscope objective. (c) A computationally simulated widefield image of the object. (d) Widefield image of a fluorescently stained biological sample. (e) Widefield image of a DNA origami-fluorescent dye sample. These two samples have the same fluorophore with different labeling densities. The same amount of illumination is used for acquiring images in (d) and (e).

Download Full Size | PDF

Appendix E estimation of typical signal level in fluorescence microscopy

We assume the following settings to estimate maximum photon counts, n_MPC, that can be recorded in an effective widefield image before significant photobleaching occurs: 1) The object is a 100nm square that has 10 fluorescent protein molecules randomly distributed on it (Fig. 6(a)). 2) The fluorescent protein molecules are excited by a focused illumination spot moving in a confocal-laser-scanning manner. 3) For all the fluorescent molecules, the fluorescent photon emission is spherically symmetric, and the emission wavelength is ~600nm. 4) The emitted fluorescent photons are collected by a 1.4NA oil-immersion microscope objective (Fig. 6(b)), and are imaged onto a camera with a 100nm effective pixel size at a 75% quantum efficiency.

Using the data reported in [16] and the above 2) setting, we estimate the number of photons that can be emitted by a fluorescent protein molecule before photobleaching, n_Photon, to be approximately 1000×t_{1/2bleach(confocal)}. Here t_{1/2bleach(confocal)} is defined as the time needed to bleach to half emission intensity measured during confocal imaging, at an illumination level that causes each molecule to emit ~1000 photons/second initially. This half-bleach time can range from ~200 to ~7000 seconds among different fluorescent proteins [16], and for our estimation we assume t_{1/2bleach(confocal)}≈1000 seconds, thus n_Photon≈10⁶. When the photon emission is spherically symmetric, a standard geometrical calculation yields that only (1-cosα)/2 of the fluorescent photons will be collected by the microscope objective and arrive at the camera, where α is the light collection half-angle (Fig. 6(b)). For a 1.4NA oil-immersion objective, α = sin⁻¹(NA/n_Oil) ≈ 67°, where n_Oil is the refractive index of the immersion oil and is ~1.52. Therefore, the number of photons arriving at the camera, n_{Photon(camera)} is approximately n_Photon×(1-cosα)/2 ≈ 3×10⁵ from each fluorescent protein molecule.

To accurately estimate n_MPC requires computational simulation of the photon distribution on the camera. Here we provide a rule-of-thumb estimation of n_MPC: because the object size is only a fraction of d_R in this case, we simply approximate it as a point source that has emission capacity equivalent to 10 fluorescent protein molecules. Since the camera pixel size is ~d_R/3, we further simplify the photon distribution by assuming the arriving photons distribute evenly in 3-by-3 pixels surrounding the center of the distribution. Altogether, we derive n_MPC ≈ n_{Photon(camera)} × number of fluorescent molecules × camera quantum efficiency/9(pixels) ≈ 2.5×10⁵. This simple estimation is close to the simulation result showing n_MPC ≈ 3.2×10⁵ (Fig. 6(c)).

As we show in the above estimation, the maximum photon counts n_MPC are proportional to fluorescent molecule density on the labeled structure. While labeling density can vary by orders of magnitude among different samples and labeling techniques, the labeling density we assume above, i.e., ~10 fluorescent molecules on a 100nm square structure, is a relatively low density in the context of biological fluorescence microscopy. In Figs. 6(d) and 6(e), we compare the widefield images of a fluorescently stained biological sample and an engineered sample whose labeling density can be estimated. These two samples have the same fluorophore, Alexa Fluor 488, and their images are acquired with the same amount of illumination. The engineered sample is the DNA origami-fluorescent dye sample we used for characterizing restoration resolution in a previously described imaging experiment (Figs. 3(a)-3(d)). In this sample, each of the DNA origami-fluorescent dye structure carries ~20-30 Alexa Fluor 488 molecules and has a characteristic size of ~70nm, and thus its labeling density is close to the assumed density above. From the color bars in Figs. 6(d) and 6(e), we can see the maximum photon counts of the fluorescently stained sample is more than ten times greater than the engineered sample, suggesting that its fluorescent molecule density on the labeled structures is much higher than our assumption. Therefore, the above calculation should be considered as a conservative estimation of n_MPC.

Appendix F test samples and imaging experiment parameters

The DNA origami sample used for the imaging experiment shown in Figs. 3(a)-3(d) and Fig. 6(e) is GATTA-STED-70B, custom-made by GATTAquant GmbH. The mitochondria sample used for imaging experiments shown in Figs. 3(e)-3(i), Fig. 4, and Fig. 6(d) is FluoCells Prepared Slide #1, F36924, made by Thermo Fisher Scientific Inc.

For all the imaging experiments shown in Fig. 3 and Fig. 4, except for Fig. 3(g), we moved the focused illumination spot in a 200-by-200-step pattern, where the step size is approximately equal to the effective camera pixel size. For datasets used for frequency ring correlation analysis (Figs. 4(e)-4(h)), two images were acquired at each step. Otherwise, one image was acquired at each step. Fig. 3(g) is obtained through widefield illumination. To present the experimental results with efficient use of page space, Figs. 3(a), 3(b), 3(e)-3(g) and Figs. 4(a)-4(c), 4(e)-4(g) show only a small portion of the original image or restoration. The results of frequency-domain analysis shown in Fig. 3(h) and Figs. 4(d), 4(h), on the other hand, are derived using a region larger than the displayed portion.

Appendix G estimation of the signal-to-noise ratio in an effective widefield image

The signal-to-noise ratios displayed in Figs. 4(a)-4(c) are estimated using the following equation,

(2)$$\text{SNR=10}lo{g}_{10}\frac{\text{Avg}(n_{\text{MaxCounts}})}{\sqrt{\text{Avg}(n_{\text{MaxCounts}})+\text{Avg(SD}(n_{\text{DarkCounts}}){)}^2}}\text{dB,}$$

to account for photon shot noise and camera dark counts. In the above equation, Avg and SD represent the average and standard deviation operations respectively, n_MaxCounts is the maximum 100 photon counts in an effective widefield image, and n_DarkCounts are photon counts from 2000 images acquired when the camera was placed in a visually dark environment with its sensor cap in place. Here, the standard deviation is operating on the 2000 values, each from an acquired dark image, for each pixel. Consequently, the result of SD(n_DarkCounts) is an image with its size equal to a dark image.

Appendix H frequency ring average and correlation

The frequency ring average comparisons shown in Figs. 3(h) and 4(d) are derived using the following procedures: After the focused illumination spot visits all the positions, acquiring one image each (Fig. 7(a)), we obtain the effective widefield image and the restoration from our method through integration and NNLS inversions respectively (Figs. 7(b) and 7(c)). We then apply the 2D DFT to derive their spatial frequency spectra. To estimate the strength at different spatial frequencies, we divide their frequency domains into consecutive concentric ring regions. For each region, as shown in Figs. 7(b) and 7(c), we calculate the average of their absolute values. In addition, since the 1.4NA widefield image shown in Fig. 3(g) is acquired with widefield illumination, we simply apply 2D DFT and calculate its average strength accordingly. The frequency ring correlation comparisons, shown in Fig. 4(h), are derived through procedures described in [21] with slight modifications. The original procedures used in [21] are developed for single-molecule localization microscopy, where one randomly separates the entire set of acquired images into two datasets for later computations.

 

Fig. 7 Illustrations of frequency ring average calculation. (a) During image acquisition, one image is acquired at each illumination spot position. (b), (c) Frequency ring average calculations for effective widefield images and our imaging method.

Download Full Size | PDF

For our method, we acquire two images at each illumination position, and randomly assign the two images to two distinct datasets (Fig. 8(a)). After applying NNLS inversions to these two datasets individually to derive two final restorations, we again divide their frequency domains into consecutive concentric ring regions, and calculate their correlation coefficients in each of the regions (Fig. 8(b)). A Gaussian mask is applied to reduce the boundary effect in 2D DFTs.

 

Fig. 8 Illustrations of frequency ring correlation calculation. (a) During image acquisition, we acquire two images at each illumination position, and randomly assign them to two distinct data sets. (b) Frequency ring correlation calculation.

Download Full Size | PDF

Funding

National Science Foundation (1353444); Colorado Advanced Industries Accelerator Grant.

Acknowledgment

SRB acknowledges the donation of a Tesla K40c GPU from NVIDIA. This work utilized the RMACC Summit supercomputer, which is supported by the National Science Foundation (ACI-1532235, ACI-1532236), the University of Colorado Boulder, and Colorado State University.

References and links

1. L. Rayleigh, “On the theory of optical images, with special reference to the microscope,” J. R. Microsc. Soc. 23(4), 447–473 (1903). [CrossRef]  

2. J. W. Goodman, Introduction to Fourier Optics (Roberts and Company Publishers, third ed., 2005), Chap 6.5.2.

3. B. R. Frieden and J. J. Burke, “Restoring with maximum entropy, II: superresolution of photographs of diffraction-blurred impulses,” J. Opt. Soc. Am. 62(10), 1202–1210 (1972). [CrossRef]  

4. D. L. Donoho, I. M. Johnstone, J. C. Hoch, and A. S. Stern, “Maximum entropy and the nearly black object,” J. R. Stat. Soc. B 54(1), 41–81 (1992).

5. P. J. Sementilli, B. R. Hunt, and M. S. Nadar, “Analysis of the limit to superresolution in incoherent imaging,” J. Opt. Soc. Am. 10(11), 2265–2276 (1993). [CrossRef]  

6. J. W. Goodman, Introduction to Fourier Optics (Roberts and Company Publishers, third ed., 2005), Chap 6.6.5.

7. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef]   [PubMed]  

8. T. A. Klar and S. W. Hell, “Subdiffraction resolution in far-field fluorescence microscopy,” Opt. Lett. 24(14), 954–956 (1999). [CrossRef]   [PubMed]  

9. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef]   [PubMed]  

10. M. J. Rust, M. Bates, and X. Zhuang, “Stochastic optical reconstruction microscopy (STORM) provides sub-diffraction-limit image resolution,” Nat. Methods 3(10), 793–795 (2006). [CrossRef]   [PubMed]  

11. D. T. Burnette, P. Sengupta, Y. Dai, J. Lippincott-Schwartz, and B. Kachar, “Bleaching/blinking assisted localization microscopy for superresolution imaging using standard fluorescent molecules,” Proc. Natl. Acad. Sci. U.S.A. 108(52), 21081–21086 (2011). [CrossRef]   [PubMed]  

12. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proc. Natl. Acad. Sci. U.S.A. 106(52), 22287–22292 (2009). [CrossRef]   [PubMed]  

13. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef]   [PubMed]  

14. E. H. Rego, L. Shao, J. J. Macklin, L. Winoto, G. A. Johansson, N. Kamps-Hughes, M. W. Davidson, and M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy with a photoswitchable protein reveals cellular structures at 50-nm resolution,” Proc. Natl. Acad. Sci. U.S.A. 109(3), E135–E143 (2012). [CrossRef]   [PubMed]  

15. J. B. Pawley, Handbook of biological confocal microscopy (Springer, third ed., 2006).

16. N. C. Shaner, M. Z. Lin, M. R. McKeown, P. A. Steinbach, K. L. Hazelwood, M. W. Davidson, and R. Y. Tsien, “Improving the photostability of bright monomeric orange and red fluorescent proteins,” Nat. Methods 5(6), 545–551 (2008). [CrossRef]   [PubMed]  

17. L. Wiemerslage and D. Lee, “Quantification of mitochondrial morphology in neurites of dopaminergic neurons using multiple parameters,” J. Neurosci. Methods 262, 56–65 (2016). [CrossRef]   [PubMed]  

18. L. Song, E. J. Hennink, I. T. Young, and H. J. Tanke, “Photobleaching kinetics of fluorescein in quantitative fluorescence microscopy,” Biophys. J. 68(6), 2588–2600 (1995). [CrossRef]   [PubMed]  

19. L. Greenbaum, C. Rothmann, R. Lavie, and Z. Malik, “Green fluorescent protein photobleaching: a model for protein damage by endogenous and exogenous singlet oxygen,” Biol. Chem. 381(12), 1251–1258 (2000). [CrossRef]   [PubMed]  

20. A. G. York, S. H. Parekh, D. Dalle Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef]   [PubMed]  

21. N. Banterle, K. H. Bui, E. A. Lemke, and M. Beck, “Fourier ring correlation as a resolution criterion for super-resolution microscopy,” J. Struct. Biol. 183(3), 363–367 (2013). [CrossRef]   [PubMed]  

22. R. P. J. Nieuwenhuizen, K. A. Lidke, M. Bates, D. L. Puig, D. Grünwald, S. Stallinga, and B. Rieger, “Measuring image resolution in optical nanoscopy,” Nat. Methods 10(6), 557–562 (2013). [CrossRef]   [PubMed]  

23. S. van de Linde, A. Löschberger, T. Klein, M. Heidbreder, S. Wolter, M. Heilemann, and M. Sauer, “Direct stochastic optical reconstruction microscopy with standard fluorescent probes,” Nat. Protoc. 6(7), 991–1009 (2011). [CrossRef]   [PubMed]  

24. R. Swaminathan, C. P. Hoang, and A. S. Verkman, “Photobleaching recovery and anisotropy decay of green fluorescent protein GFP-S65T in solution and cells: cytoplasmic viscosity probed by green fluorescent protein translational and rotational diffusion,” Biophys. J. 72(4), 1900–1907 (1997). [CrossRef]   [PubMed]  

25. C. J. R. Sheppard and K. G. Larkin, “Vectorial pupil functions and vectorial transfer functions,” Optik (Stuttg.) 107, 79–87 (1997).

26. S. R. Becker, E. J. Candès, and M. C. Grant, “Templates for convex cone problems with applications to sparse signal recovery,” Math. Program. Comput. 3(3), 165–218 (2011). [CrossRef]