Super-resolution fluorescence blinking imaging using modified Fourier ptychography

Jingjing Wu; Bin Yu; Saiwen Zhang; Siwei Li; Xuehua Wang; Danni Chen; Junle Qu

doi:10.1364/OE.26.002740

1. Introduction

An optical imaging system images a sample by convoluting the sample and the system point spread function (PSF). However, this lead to many details of the sample being missed; in addition, the resolution of the images is limited by the diffraction limitation. To overcome these limitations, several innovative microscopy techniques have been proposed [1].One of these microscopic techniques is structured illumination microscopy (SIM) [2], which uses an extra setup to introduce special illumination patterns to modulate the PSF of the system, and it obtains high-frequency information to achieve super-resolution imaging. However, there exist some limitations in certain aspects of the SIM scheme. For example, ordinary SIM can only improve the resolution by a factor of two compared to wide-field imaging, only two-dimensional samples can be imaged, accurate information about the structured illumination patterns must already be known, and the inaccuracy of the frequency shift distance in the recovery process decreases the image resolution. In order to overcome these limitations, some alternative schemes have been proposed. To improve the resolution further, nonlinear SIM has been proposed [3,4]. Along with the development of the SIM technique, three-dimensional (3D) SIM techniques [5–7] and SIM for live cells [8] have also been proposed. Further, some blind-SIM schemes have been proposed [9–11]. In 2012, Mudry et al. proposed blind-SIM using speckle patterns obtained using a shift diffuser. This method can achieve a resolution improvement by two times compared to wide-field imaging, using an optimization algorithm [9]. In 2014, Dong et al. used Fourier ptychography (FP) imaging to realize blind-SIM [10]. In their method, the speckle patterns were also acquired through a diffuser, however, the sample needed to be shifted for each detection step during the experiment. The shifting of the sample or diffuser increases the complexity of the microscopy system; in addition, the resolution of these two types of blind-SIM is limited by the size of the speckle illuminating the surface of the sample, which is limited by the diffraction limit. Thus, one of the effective methods to further improve the image resolution would be to decrease the speckle size.

In addition to SIM, other kinds of microscopy techniques exist that use the photo switching effect, fluorescence blinking effect, or other fluorescence mechanisms such as STORM [12], PALM [13], and SOFI [14,15]. PALM and STORM are single-molecule localization microscopy techniques wherein a single fluorescence molecule is activated stochastically within the diffraction-limited region using photos witching effect and is localized with a computational method. In contrast, SOFI uses high-order statistical analysis of the fluorescence temporal fluctuation caused by the fluorescence blinking effect. Hence it is evident that PALM, STORM, and SOFI rely on fluorescence properties but have different reconstitution methods.

In this paper, we propose a new reconstitution scheme based on the modified FP to handle fluorescence blinking images; we abbreviate this scheme as SRFB. In SRFB, the time-variable fluorescence intensity fluctuations of the emitters are treated as emission intensity distributions of fluorescence molecules when speckle patterns illuminate the sample in blind-SIM. Then, the super-resolution image is recovered through a modified iterative algorithm used in FP [10,16,17]. In SRFB, because the speckle size is no longer limited by the diffraction limit, higher resolutions can be obtained. Further, the proposed method does not require an extra setup or process in the experimental process.

The remainder of this paper is structured as follows: Sec. 2 presents the basic theory and reconstitution process of the proposed scheme. In Sec. 3, with the simulated SOFI recorded image sequences, the resolution of SRFB is analyzed and compared to that of second-order SOFI. In Sec. 4, we use the SRFB algorithm to handle the raw images obtained in the SOFI and STORM experiment and analyze the resolution enhancement effect compared to that of the wide-field imaging and second-order SOFI imaging. Finally, we summarize this paper in Sec. 5.

2. Theory of SRFB

In SIM, S(r) represents the sample structure and P_n(r) represents the n^th spatially varying illumination intensity pattern. PSF(r) is the point spread function of the imaging system. According to optical imaging theory, the recorded image is

I_{n} (r) = [S (r) P_{n} (r)] \otimes P S F (r) = F^{- 1} {F [S (r) P_{n} (r)] \cdot O T F (k)},

where

\otimes

is the convolution operation, F and F⁻¹ represent the Fourier and inverse Fourier transforms, respectively, and OTF is the optical transfer function. The illumination pattern frequently used in SIM is a sinusoidal or speckle pattern. The illumination patterns are obtained through optical processes, and hence, the frequency of the illumination pattern is still limited by the diffraction limit, which limits the imaging resolution of SIM [9,10].

In super-resolution fluorescence microscopy imaging techniques such as STORM, PALM and SOFI, the “blinking” of the fluorescence molecules can be in the “on” or “off” state randomly, as shown in Fig. 1. The fluorescence intensity distribution O_n at time t_n can be treated as the product of the sample structure S and the time-varying random illumination pattern P_n. The detected image is the convolution between the fluorescence distribution and PSF of the imaging system, as expressed in Eq. (1). In the frequency domain, Eq. (1) can be rewritten as

{\tilde{I}}_{n} = {\tilde{O}}_{n} \cdot O T F, n = 1, ..., N,

where

{\tilde{I}}_{n}

and

{\tilde{O}}_{n}

are the frequency spectra of I_n and O_n respectively. O_n = S·P_n. I_n and OTF are already known, while S and P_n are unknown. The iterative algorithm used in FP is modified according to the characteristic of the fluorescence, and it is used to recover the sample structure S.

Fig. 1 Speckle pattern in fluorescence blinking.

Download Full Size | PDF

The iterative process is outlined in Fig. 2. Suppose S_g and P_n,g (n = 1, …, N) are the estimated values of S and P_n (n = 1, …, N) at the beginning of one iteration. The iterative process is as follows:

Fig. 2 Flowchart of iterative algorithm.

Download Full Size | PDF

(1)Calculate the n^th fluorescence distribution according to the estimated values as O_n,g = S_g·P_n,g. The frequency spectrum of O_n,g is represented as ${\tilde{O}}_{n, g} = F (O_{n, g})$ .
(2)Calculate the updated value of ${\tilde{O}}_{n, g}$ according to the n^th recorded image I_n: ${\tilde{O}}_{n, g}^{update} = {\tilde{O}}_{n, g} + O T F \cdot [{\tilde{I}}_{n} - O T F \cdot {\tilde{O}}_{n, g}] .$
(3)The updated value of O_n,g is obtained through the inverse Fourier transform as $O_{n, g}^{update} = F^{- 1} ({\tilde{O}}_{n, g}^{update}) .$
(4)Considering the fluorescence distribution is a real value, the updated value of S_g and P_n,g are calculated according to the FP algorithm as $S_{g}^{update} = | S_{g} + \frac{P_{n, g}}{{[\max (P_{n, g})]}^{2}} \cdot (O_{n, g}^{update} - S_{g} \cdot P_{n, g}) \cdot \bar{I} |,$ $P_{n, g}^{update} = | P_{n, g} + \frac{S_{g}}{{[\max (S_{g})]}^{2}} \cdot (O_{n, g}^{update} - S_{g} \cdot P_{n, g}) |,$
where $| \cdot |$ is the absolute value and $\bar{I} = 1 / N \sum_{n = 1}^{N} I_{n}$ .
(5)Use S_g^update to replace S_g, set n = n + 1 and go back to step (1).Calculate the updated value P_n₊₁_,g^update by using S_g^update, P_n₊₁_,g and I_{n + 1}. Repeat this process until n = N and all the updated estimated values S_g^update andP_n₊₁_,g^update (n = 1,…,N) are obtained.
(6)Use S_g^update and P_n₊₁_,g^update (n = 1,…,N) to replace S_g and P_n,g (n = 1,…,N) as the original estimated value and repeat steps (1) to (5) until the iteration is finished. S_g^update is output as the recovered image.

Although the iterative algorithm is similar to the algorithm used in FP in [10], these two schemes have relatively large differences. First, the FP algorithm used in SRFB is modified by considering the property of fluorescence. Secondly, the speckle in [10] is obtained using a diffuser which limits the resolution enhancement by a factor of two, whereas the speckle in SRFB is obtained according to the blinking behavior of the fluorescence molecule, which, in theory, allows SRFB to achieve a higher resolution. Third, in the detection process of [10], the sample or diffuser must be shifted at each detection step and an overlap of the adjacent detection regions must exist according to the FP technology as the accuracy of the shift distance can influence the imaging quality. In contrast, in the proposed method, no shift is necessary. Hence, the proposed SRFB can simplify the experimental setup and decrease the experimental steps. In the next part, the imaging resolution of the SRFB will be analyzed using the simulation and experimental results.

3. Simulation results

Because the recorded image sequences of SOFI can be handled with the proposed SRFB method, we first simulate the temporal fluctuations of blinking emitters and obtain the raw images [19]; then, we use the SOFI and SRFB algorithm to process the images. In the simulation, a line shape structure of the fluorescence molecule distribution is used as the sample. The simulation parameters are as follows: the distance of the six molecular lines are 200–600nm, fluorescence wavelength is λ = 550nm, labeling density is 50/µm, signal per frame is 2000 photons, fluorescence state duration to non-fluorescence ratio is 0.1, the effective CCD pixel size is 100nm, NA = 1.3, and number of frames N = 100.

Figures 3(a)–3(c) present the wide-field image, second-order SOFI, and proposed SRFB results with six iterations. The intensity profiles of the projection through the long lines in Figs. 3(a)–3(c) are presented in Fig. 3(d). From Fig. 3(d), we can see that the SOFI and SRFB algorithms can both realize super-resolution imaging. These two algorithms can distinguish the top two lines, which cannot be distinguished by the wide-field image. This proves that the second-order SOFI and SRFB have a higher distinguishing ability compared to the wide-field imaging. Figure 3(e) presents the profiles of the projection through the short lines in Figs. 3(a)–3(c). We use the full width of half maximum (FWHM) of these profiles to evaluate the resolution. With a Gaussian fit of the profiles in Fig. 3(e), the FWHM of the wide-field imaging, second-order SOFI and SRFB are 267, 158, and 93nm respectively. This indicates that the proposed SRFB can obtain more than twofold resolution enhancement compared to the wide-field imaging result for the above parameters.

Fig. 3 Simulation results of the line structure sample. (a) Wide-field imaging, (b) second-order SOFI, and (c) SRFB results. The scale is 500nm. Profiles of the projection through the (d) long lines and (e) short lines in (a)–(c).

Download Full Size | PDF

The number of frames affects the SOFI result [15], as well as the result of SRFB [9,10]. Figures 4(a) and 4(b) present the second-order SOFI and SRFB results when N = 50. Figure 4(c) presents the normalized intensity profiles through the lines in Figs. 4(a) and 4(b). Figures 4(d)–4(f) present the corresponding results when N = 200. Compared to Fig. 3, where N = 100, we can see that, the resolution of the second-order SOFI and SRFB increases with the number of frames, and the SRFB has a better resolution capability compared to the second-order SOFI when N = 50, 100, and 200. To demonstrate the influence of the number of frames on the SRFB result, Fig. 4(g) shows the FWHM of the intensity profiles for SOFI and SRFB as a function of the number of frames N under the same simulation condition as Fig. 3. As we can see, the FWHM of SRFB decreases with N when N<100and then tends to be a constant. For different experimental parameters such as background noise, fluorescence blinking, and different iterative times, the FWHM curve versus N must be different. Hence, the minimum number of frames to be used in SRFB is determined by the experimental conditions, iterative times, and the desired FWHM. When the SRFB algorithm is used, N can be chosen from 10 to 200, similar to the speckle blind-SIM [9,10].

Fig. 4 Comparison between second-order SOFI and SRFB when N = 50 and 200 frames. (a) Second-order SOFI and (b) SRFB with N = 50 frames. (c) Second-order SOFI and (d) SRFB with N = 200 frames. The white scale bars represent 500nm. (e) Profiles of the projection through the long lines in (a) and (b). (f) Profiles of the projection through the long lines in (c) and (d). (g) Curve of the FWHM versus number of frames N.

Download Full Size | PDF

To compare the operation time of the algorithm in SRFB and SOFI, we use the same simulated raw images and iterative times as in Fig. 3. The time taken by SOFI is 7.241s for N = 50 and 10.7s for N = 200. The time taken by SRFB is 4.51s for N = 50 and 10.55s for N = 200. Hence, the operation times of SRFB and SOFI are approximately identical when N≤200, which is the number that SRFB can use to achieve a satisfactory result.

4. Experimental results

In this part, the resolution of the SOFI and SRFB algorithms are compared using the SOFI and STORM experimental results. In the SOFI experiment, the IX71 inversion fluorescence microscope is used to obtain the fluorescence blinking image sequence. We use a 100 × 1.4-NA oil-immersion objective for image acquisition. The effective pixel size of the EMCCD is 160nm. The Hela cell microfilament labeling with semiconductor polymer dots (Pdots) [18] is used as the sample. The excitation wavelength is 405nm and the emission wavelength is 550nm. The exposure time is 10ms/frame and the number of frames is 500.

Figures 5(a)–5(c) present the results of the wide-field imaging, second-order SOFI and SRFB (20 iterations) imaging, respectively. Figures 5(d)–5(f) are the magnified views of the areas in the squares of Figs. 5(a)–5(c) respectively. Figure 5(g) presents the intensity profiles of the long solid lines in Figs. 5(d)–5(f), from which we can see that SOFI and SRFB can both obtain detailed information, while the wide-field cannot. Figure 5(h) presents the intensity profiles of the short-dotted lines in Figs. 5(d)–5(f), and the FWHM of each are 393, 276, and 229nm respectively. We can notice that an only 1.7 × improvement in resolution can be achieved by SRFB. This is because the fluorescence blinking property of the Pdots used in the SOFI experiment is not very satisfactory, which makes the fluorescence intensity fluctuations insufficiently strong, leading to a lower resolution enhancement compared to Fig. 3. However, Fig. 5 still proves that under the same experimental parameters, the proposed SRFB algorithm has a higher resolution compared to the second-order SOFI algorithm.

Fig. 5 Comparison between second-order SOFI and SRFB using SOFI experiment results. (a)–(c) Wide-field image, second-order SOFI and SRFB result, respectively. (d)–(f) Magnified view of the areas in the squares of (a)–(c), respectively. (g, h) Intensity profiles through the solid line and dotted line in (d)–(f) respectively. The scale bars in (d)–(f) represent 800nm.

Download Full Size | PDF

According to the theory of SRFB in Sec. 2, the proposed SRFB algorithm can also recover super-resolution image using the detected image sequence of the high-density STORM. To verify this viewpoint, 500 frames of STORM raw images (resolution: 128px × 128px) obtained from the EPFL website [20] are used. The parameters in this STORM experiment are as follows: NA = 1.3, fluorescence wavelength is 690nm, frame rate is 25 fps, and effective CCD pixel size is 100nm. In order to compare the SRFB with second-order SOFI, the images are enlarged to 256px × 256px by interpolation before the application of the iterative algorithm, which makes the pixel size 50nm. To obtain an image series of different-density fluorescence molecules, each L₀ frame of the recorded images is added together to obtain N = 500/L₀ frames with a higher density. Figure 6(a) presents the results of wide-field imaging. Figures 6(b) and 6(c) present the recovered image using the second-order SOFI when L₀ = 20 and 10 (N = 25 and 50), respectively. Figures 6(d) and 6(e) present the recovered image using the proposed SRFB under 40 iterations when N = 25 and 50, respectively. Figures 6(f)–6(j) show the magnified views of the square in Figs. 6(a)–6(e), respectively.

Fig. 6 Comparison between second-order SOFI and SRFB using STORM experiment results. (a) Wide-field image. Second-order SOFI with (b) N = 25 and (c) N = 50. SRFB with (d) N = 25 and (e) N = 50. (f)–(j) Magnified views of the areas in the squares of (a)–(e). (k, l) Intensity profiles through line “1” and “2” in (f)–(j), respectively. The white scale bar represents 1μm.

Download Full Size | PDF

To analyze the resolution of SOFI and SRFB in detail, Fig. 6(k) provides the normalized intensity profiles through the “1” lines in Figs. 6(f)–6(j).These lines indicate that the second-order SOFI and SRFB can both recover details that cannot be obtained from the wide-field imaging. Figure 6(l) presents the normalized intensity profiles through the “2” lines in Figs. 6(f)–6(j), which are the normalized intensity distributions of a single microtubule. The FWHMs are 328.6nm (wide-field), 215.9nm (second-order SOFI, N = 25, 1.5 × resolution enhancement), 200.3nm (second-order SOFI, N = 50, 1.6 × resolution enhancement), 163.4nm (SRFB, N = 25, 2 × resolution enhancement) and 127.7nm (SRFB, N = 50, 2.7 × resolution enhancement) respectively. Figure 6 proves that under the same experiment condition, SOFI and proposed SRFB algorithm can both obtain the super-resolution imaging result. However, SRFB can achieve a twofold resolution enhancement compared to the wide-field imaging. Figure 6 demonstrates that the proposed method can handle the recorded images of STORM and obtain a super-resolution imaging result.

5. Conclusions

In this paper, we proposed a new microscopy method based on fluorescence blinking called SRFB. In this method, the time-varying fluorescence intensity images such as STORM and SOFI can be used to recover a super-resolution image using the iterative algorithm modified from the FP algorithm. By using the simulated SOFI detected images, an image with a resolution surpassing the diffraction limit by a factor larger than two can be obtained even with less than 100 frames of recorded images. By using the experimental STORM detected images, a twofold resolution enhancement can also be reached when N is less than 100. The SRFB image resolution with the SOFI experimental image sequence is not as high as it was with the simulated SOFI image and STORM experimental image because of the chosen sample and noise; however, it was higher than the resolution of the second-order SOFI result.

Compared to the original SIM or blind-SIM scheme, the experimental setup of the SRFB is simpler, because it does not require an additional setup to obtain the illumination patterns. In contrast, the imaging resolution of SRFB can reach more than two-times the diffraction limit, which is the theoretical limit of the original blind-SIM scheme.

According to the results of this paper, we will analyze the scheme of the 3D SRFB and explore other optimization algorithms to obtain a higher resolution of SRFB in the future.

Funding

National Basic Research Program of China (2015CB352005); National Natural Science Foundation of China (NSFC) (61335001, 61178080, 61235012, 11774242); Guangdong Natural Science Foundation (2014A030312008, 2017A030310132); Shenzhen Science and Technology Planning Project (JCYJ20150324141711698, JCYJ20170412105003520); Foundation for Distinguished Young Talents in Higher Education of Guangdong, China (FDYT) (2014KQNCX131).

Acknowledgment

We thank Prof. Changfeng Wu (Department of Biomedical Engineering, Southern University of Science and Technology) for providing Pdots for SOFI.

References and links

1. B. Huang, H. Babcock, and X. Zhuang, “Breaking the diffraction barrier: Super-resolution imaging of cells,” Cell 143(7), 1047–1058 (2010). [CrossRef] [PubMed]

2. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef] [PubMed]

3. R. Heintzmann, T. M. Jovin, and C. Cremer, “Saturated patterned excitation microscopy-a concept for optical resolution improvement,” J. Opt. Soc. Am. A 19(8), 1599–1609 (2002). [CrossRef] [PubMed]

4. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U.S.A. 102(37), 13081–13086 (2005). [CrossRef] [PubMed]

5. M. G. Gustafsson, L. Shao, P. M. Carlton, C. J. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef] [PubMed]

6. L. Shao, P. Kner, E. H. Rego, and M. G. L. Gustafsson, “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods 8(12), 1044–1046 (2011). [CrossRef] [PubMed]

7. A. G. York, S. H. Parekh, D. D. Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef] [PubMed]

8. P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, and M. G. Gustafsson, “Super-resolution video microscopy of live cells by structured illumination,” Nat. Methods 6(5), 339–342 (2009). [CrossRef] [PubMed]

9. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics 6(5), 312–315 (2012). [CrossRef]

10. S. Dong, P. Nanda, R. Shiradkar, K. Guo, and G. Zheng, “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express 22(17), 20856–20870 (2014). [CrossRef] [PubMed]

11. L. H. Yeh, L. Tian, and L. Waller, “Structured illumination microscopy with unknown patterns and a statistical prior,” Biomed. Opt. Express 8(2), 695–711 (2017). [CrossRef] [PubMed]

12. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–796 (2006). [CrossRef] [PubMed]

13. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging intracellular fluorescent proteins at nanometer resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef] [PubMed]

14. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proc. Natl. Acad. Sci. U.S.A. 106(52), 22287–22292 (2009). [CrossRef] [PubMed]

15. H. Deschout, T. Lukes, A. Sharipov, D. Szlag, L. Feletti, W. Vandenberg, P. Dedecker, J. Hofkens, M. Leutenegger, T. Lasser, and A. Radenovic, “Complementarity of PALM and SOFI for super-resolution live-cell imaging of focal adhesions,” Nat. Commun. 7, 13693 (2016). [CrossRef] [PubMed]

16. L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,” Biomed. Opt. Express 5(7), 2376–2389 (2014). [CrossRef] [PubMed]

17. Y. Fang, Y. Chen, C. Kuang, P. Xiu, Q. Liu, B. Ge, and X. Liu, “Saturated pattern-illuminated Fourier ptychography microscopy,” J. Opt. 19(1), 015602 (2017). [CrossRef]

18. X. Chen, R. Li, Z. Liu, K. Sun, Z. Sun, D. Chen, G. Xu, P. Xi, C. Wu, and Y. Sun, “Small photoblinking semiconductor polymer dots for fluorescence nanoscopy,” Adv. Mater. 29(5), 1604850 (2017). [CrossRef] [PubMed]

19. M. Leutenegger, “Balanced SOFI toolbox,” (EPFL). http://documents.epfl.ch/users/l/le/leuteneg/www/BalancedSOFI/index.html

20. Biomedical Imaging Group, “Benchmarking of Single-Molecule Localization Microscopy Software” (EPFL, Lausanne, 2015). http://bigwww.epfl.ch/smlm/

Super-resolution fluorescence blinking imaging using modified Fourier ptychography

Abstract

1. Introduction

2. Theory of SRFB

3. Simulation results

4. Experimental results

5. Conclusions

Funding

Acknowledgment

References and links

Cited By

Figures (6)

Equations (6)

Optics Express