Open Access
Add to BibSonomy
Abstract
Multifocal structured illumination microscopy (MSIM) is the parallelized version of image scanning microscopy (ISM), which uses multiple diffraction limited spots, instead of a single diffraction limited spot, to increase the imaging speed. By adding pinhole, contraction and deconvolution, a twofold resolution enhancement could be achieved in theory. However, this resolution improvement is difficult to be attained in practice. In this work, without any modification of the experimental setup, we propose to use multiple measurement vector (MMV) model sparse Bayesian learning (MSBL) algorithm (MSIMMSBL) as the reconstruction algorithm of MSIM, which does not need to estimate the illumination patterns but treat the reconstruct process as an MMV signal reconstruction problem. We compare the reconstructed super-resolution images of MSIMMSBL and MSIM by using simulation and experimental raw images. The results prove that by using the MSBL algorithm, the MSIM can obtain a higher than twofold resolution enhancement compared with the wide field image. This outstanding imaging resolution combining with the primary advantages of MSIM, such as the high imaging speed, could promote the application of MSIM in super-resolution microscopy imaging technology.
© 2018 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Super-resolution fluorescence microscopy technology has been rapidly developing during the past two decades. Structured illumination microscopy (SIM) [1] is one of the technologies with the highest potential. The temporal resolution of SIM is high enough to study living systems, but the spatial resolution is hard to achieve twofold improvement in practice. In traditional implementation of SIM, the sample is illuminated by a sinusoidal modulated intensity pattern, which encode the high frequency information in low frequency and can be moved to the correct position to obtain the super-resolution image. For SIM, a twofold resolution improvement could be achieved in theory, the image speed is 1-10Hz [2–5], which can be used to observe the dynamic change of a living cell. In 2010, Muller et al. proposed a similar microscopy technology called image scanning microscopy (ISM) [6], which combined the resolving power of confocal-laser scanning microscopy with that of a wide-field imaging microscopy. ISM can achieve the twofold resolution improvement without the loss of signal-to-noise ratio (SNR). However, the imaging speed of ISM is significantly slow for monitoring many live cellular processes until now. In 2012, York et al. proposed the parallelization of ISM [3], called multifocal SIM (MSIM), by using a digital micro-mirror device (DMD) to obtain a sparse lattice of excitation foci to scan the sample. MSIM data processing contains several important steps such as digital pinholing, digital twofold contraction of each pinholed emission spot and deconvolution. The MSIM can dramatically increase the imaging speed to 1 Hz with 145 nm lateral resolution and 400 nm axial resolution.
Although the achievable gain in resolution of MSIM twofold, the resolution of MSIM in practical imaging conditions is lower than that of striped-pattern alternatives normally [7]. In recent years, many resolution improvement methods of MSIM were proposed. One way to improve the image effect is to optimize the traditional MSIM system. For example, Ingaramoa et al. proposed to implement MSIM with a microlens array to enable efficient two-photon excitation [8], which gave resolution-doubled images with better sectioning and contrast in thick scattering samples. York et al. proposed the use of optical instead of digital image-processing operations to increase the data acquisition rates [2]. The methods above can increase the effect or imaging speed of MSIM, however, the image resolution was not improved.
The alternative way to improve the resolution of MSIM is to modify the data processing algorithm. Florian et al. proposed to use joint Richardson–Lucy deconvolution algorithm [9] to processing the MSIM data, however, the improvement was still limited. Ward et al. proposed the combination of a super-resolution technique of difference microscopy with structured illumination deconvolution, called eMSIM [7], which could achieve 1.4-fold resolution improvement compared with conventional MSIM. However, this method required twice the number of images compared to a conventional MSIM. In 2017, Cao et al. proposed to use pattern-estimated Fourier ptychography on MSIM [10], which is insensitive to the rotational errors and shift errors, however, the resolution improvement was still limited. In 2018, Guo et al. combined instant SIM with total internal reflection fluorescence microscopy in a system to improving the lateral spatial resolution of TIRFM [11]. This work proved the necessity of the research to improve the resolution of MSIM.
In this work, we use multiple measurement vector model sparse Bayesian learning (MSBL) to recover the super-resolution image from MSIM raw images to improve the resolution of convolution MSIM. The MSBL algorithm [12,13] is a multiple signal recovery algorithm, which can be used to recover a signal with multiple measurements under the same support. MSBL has been used in speckle SIM [14] and STORM [15] and achieved outstanding results. In MSIM, the sample is detected by multiple measurements with scanning multifocal illumination pattern instead of speckle patterns, hence the MSBL algorithm is still can be used. On the one hand, MSBL algorithm need not to process each excitation foci like conventional MSIM, which makes the reconstruction process simpler. On the other hand, the analysis results of simulation and experimental raw images of MSIM prove that, MSIMMSBL can achieve higher resolution than MSIM.
2. Theory
In fluorescence microscopy, when a sample labeled with fluorescence probe is excited by the mth illumination pattern, the corresponding detected intensity in the image recorded plane is:
where s(r) represents the structure of the sample, em(r) is the mth illumination pattern, PSF(r) is the point spread function of the system, and is the convolution operation. In conventional SIM, em(r) is a sinusoidally modulated intensity pattern, and the precise information of illumination patterns are needed to realize the image reconstruction. In speckle SIM, em(r) is a speckle pattern, special reconstruction algorithms [14,16] can be used to obtain the super-resolution image and do not need the information of speckle patterns. Furthermore, the MSIM can be treated as one special SIM, of which the illumination pattern is a sparse lattice of excitation foci pattern. The data processing of conventional MSIM contains pinholing, twofold contraction and deconvolution [2], as shown in Fig. 1.For a raw image shown in Fig. 2(a), 1) The position of each illumination spot is located in the sample plane and the synthetic pinhole SP generated by Gaussian function is applied. This process can reject light on the surrounding pixels. The pinholed result of Fig. 2(a) is shown in Fig. 2(b); 2) Twofold contraction is used to improve the lateral resolution as follows: Firstly, the center positions of the spots are located for each frame. Then a double-size image matrix is constructed, and the raw data from the square area around every located spot is copied to the new image matrix at a position with the coordinates multiplied by a factor of two. This procedure “shifts” the information from each pixel by half its distance to the optical axis. Lastly, the final image is “shrunk” by a factor of two. The twofold contraction result corresponding to Fig. 2(b) is shown in Fig. 2(c); 3) All raw images are processed according to process 1 and 2 and then superposed together to obtain Fig. 2(d); 4. Figure 2(d) is deconvoluted with the corresponding PSF and the final MSIM result is shown in Fig. 2(e). In theory, the MSIM can achieve a twofold resolution improvement compared with a wide-field image.
Fig. 2 (a) One raw image. (b) Application of digital pinholes around each PSF point. (c) Twofold contraction result of (b). (d) Superposition of all proceed raw images. (e) Deconvolution result of (d).
We rewrite Eq. (1) as follows:
in which Y = [y1, y2, …, yN] and ym represents the column vector changed from the mth detected image Dm(r). H is the measurement matrix obtained from PSF. X = SE = [x1, x2, …, xN] and xm represents the column vector changed from s(r)em(r), S and E are the matrix form of s(r) and em(r). This process is shown in Fig. 3. Min et al. pointed out that in speckle SIM, all columns in X have the same spatial support [14] and the reconstruction algorithm under joint support constraint can be used to obtain a threefold resolution improvement. By comparing MSIM with speckle SIM, we find that this joint support constraint reconstruction algorithm is also suitable to MSIM. Hence, we proposed the use of the MSBL algorithm to recover a MSIM raw image. The detail data processing steps are as follows:- 1. All raw images and PSFs are converted into matrix form according to Fig. 3(a) and 3(b). Each raw image is transformed into a column vector of Y. The ith column of H corresponds to the acquired raw image if only one molecule emits fluoroscopic photons at the sample and its position is moved through the arrow in Fig. 3(b). Suppose the size of a raw image is m × n, the size of super-resolution image is m0 × n0 (m0, n0>m, n), the number of frames is N. We can conclude that the size of Y, H and X are mn × N, mn × m0n0, and m0n0 × N. Because X = SE, all row vectors in X have the same spatial support, which is identical to the support of S. Hence the recovery problem of X can be treated as the multiple measurement vector problem [12,14].
- 2. The MSBL algorithm [12] is used to solve Eq. (2). According to Bayesian inference proceeds, the solution of Eq. (2) X’ is given by the Maximum-A-Posterior estimate:
In MSBL algorithm, the likelihood model p(Y|X) is assumed as a Gaussian likelihood model with noise, and the parameterized weight Gaussian prior p(X) is chosen, so the estimate of X’ is given by estimating the hyperparameters. The hyperparameters can be estimated by employing the expectation maximization (EM) algorithm to maximize p(X|Y). The MSBL iteration algorithm can be summarized as follow: 1) Initialize the hyperparameters; 2) Compute the mean and covariance of the posterior density p(X|Y); 3) Update the hyperparameters using EM rule; 4) Iterate 2)-3) until convergence to a fixed hyperparameter vector.
- 3. All raw vectors in the estimated X’ are added together and transformed into the recovered super-resolution image with size m0 × n0.
Compared with conventional MSIM and MSIMMSBL process in Fig. 1, we can see that, the MSIMMSBL does not need any changes in the detection process of MSIM. Moreover, the advantages of easier implementation and higher imaging speed of MSIM are also maintained in MSIMMSBL. Concurrently, the data processing of MSIMMSBL is much simpler than MSIM. In the following section, we will use simulated and experimental data to compare the images reconstructed with the MSIMMSBL and MSIM algorithms.
3. Simulation results
Initially, we used the spoke-like structure, shown in Fig. 4(a), as the sample structure for the MSIM simulations. The size of the sample is 256 × 256 pixels, with a 30 nm/pixel dimension. The excitation and emission wavelengths are 488 nm and 520 nm respectively. The numerical aperture is NA = 1.27. We chose a periodic lattice of square for our illumination, and the pattern is moved 60 nm in the sample plane from left to right and from top to down for each detect. The wide-field image and the recovered images of MSIM and MSIMMSBL are shown in Figs. 4(b)–4(d), respectively. The intensity plots along the green and white curves in Figs. 4(c)–4(d) are shown in Figs. 4(e)–4(f). In Fig. 4(e), both MSIM and MSIMMSBL can distinguish the two adjacent spokes with a distance approximately 130 nm. However, the contrast of MSIMMSBL is well above MSIM. Figure 4(f) shows that the MSIMMSBL can distinguish the two adjacent spokes with a distance approximately 115 nm, while the conventional MSIM cannot. Figure 4 illustrates that, by using MSBL algorithm, the MSIM can distinguish the tighter spokes compared with the MSIM reconstruction algorithm.
Fig. 4 Comparison of simulated MSIM and MSIMMSBL. (a) Sample with spoke-like structure. (b) Wide-field image. (c) Simulated MSIM reconstruction. (d) Simulated MSIMMSBL reconstruction. Intensity plot along the (e) green line and (f) white line indicated on images. Scale bars: 600 nm.
To have a more comprehensive comparison between MSIM and MSIMMSBL, we simulate two sample structures, shown in Fig. 5. The size of sample is 64 × 64 pixels, the size of PSF is set as 6 pixels. Figure 5(a) show the reconstruction results of raw images without noise (in red box) and with a SNR of 20 dB (in green box) for a point emitters structure sample with point interval Δ = 2 pixels, 3 pixels, 4 pixels, and 5 pixels. Figure 5(b) show the reconstruction results of the simulated raw images with 20 dB SNR for line structure samples with line interval Δ = 2 pixels and 3 pixels. We can see that conversional MSIM can recognize the emitter pairs with Δ = 4 pixels and lines with Δ = 3 pixels, which is a nearly twofold resolution improvement compared with wide-field imaging. This result is in accord with theory results. While the MSIMMSBL can recognize the emitter pairs with Δ = 3 pixels and lines even when the interval Δ = 2 pixels. Figure 5 proves that, higher than twofold resolution improvement, compared with wide-field imaging, can be achieved by using an MSBL algorithm. Also, we can see that, for different sample structure, the resolution improvement ability is different to some extent, for both MSIM and MSIMMSBL. But the imaging quality of MSIMMSBL is always higher than MSIM in our simulation results.
4. Experiment results
Subsequently, we use experiment MSIM raw images to compare the MSIM and MSIMMSBL. Initially, publicly available MSIM data of labelled microtubules provided by Andrew York (code.google.com/p/msim/) are used. In this experiment, the excitation and emission wavelengths are 488 nm and 520 nm respectively [3]. The NA of the objective is 1.45. Figures 6(a)–6(c) show the wide-field image and the reconstruction results using MSIM and MSIMMSBL, respectively. Both algorithms performe well in enhancing the resolution. Figure 6(d) shows the intensity profile through the white short line in Figs. 6(a)–6(c). We can see that the MSIMMSBL algorithm can resolve the two-line feature with a distance approximately 90 nm, which MSIM cannot. The area in yellow box area in Figs. 6(b)–6(c) also illustrate that MSIMMSBL can obtain higher resolution than MSIM.
Fig. 6 Comparison of experiment MSIM and MSIMMSBL. (a) Wide-field image, (b) MSIM reconstruction image, and (c) MSIMMSBL reconstruction image of microtubules. (d) Intensity profile through the short white line in (a)-(c).
The optical configuration of the MSIM we used is shown in Fig. 7. The excitation beam from a solid-state laser source (Coherent, Sapphire 488-200 CW CDRH) is expanded by a factor of with a 4f system constructed from two achromatic lenses (Thorlabs, f = 10 mm and f = 60 mm). The collimated beam is subsequently directed onto a DMD (Digital Light Innovations, D4100 DLP) that is 24° off-normal, such that in its “on” position, the micromirror tilts the output beam normal to the DMD face. Subsequently, a 4f configuration (Thorlabs, f = 75 mm and f = 75 mm) images the DMD face at the back focal plane of a lens (Thorlabs, f = 300 mm), and an optical iris is placed at the Fourier plane of the 4f system to block the undesired diffraction orders of the excitation light generated by the DMD. Each DMD pixel of size is demagnified by a factor of by the telescope system consisting of a lens (f = 300 mm) and objective (Nikon,, NA = 1.27) to dimensions of in the specimen plane. Then the emitted fluorescence from the specimen is collected by the same objective (Nikon, NA = 1.27). Finally, the specimen is imaged by a scientific CMOS (sCMOS) camera (ORCA Flash 4.0 V2).
Figure 8(a) shows the sparse 2D periodic lattice of the excitation foci as generated by the DMD. A black pixel indicates the “on” state of the micromirror and a white one represents the “off” status. A single illumination spot corresponds to only one DMD mirror pixel in the “on” position. As the patterns are loaded on the DMD, the pixels of the “on” position follow the red line to move a pixel distance of 120 nm along the sample plane. The fluorescence image (at the sample plane) of the excitation foci in a uniform solution of Rhodamine 6G is shown in Fig. 8(b). Here, we note that the distance between the illumination points will affect the acquisition speed and image quality. Widely spaced foci afford less cross-talk, but correspond to more scan steps, and therefore, a suitable distribution of illumination lattice yields a good imaging result. In our experiment, the distance between the scan points in the DMD is set to 16 px (horizontal) and 14 px (vertical) to avoid overlap of two adjacent points. The resulting 224 raw exposures taken at 224 Hz correspond to a 1s 2D information acquisition rate.
Fig. 8 (a) Scanning pattern of DMD. (b) The fluorescence image of the excitation foci in a uniform solution of Rhodamine 6G at the sample plane.
In addition, we sample a section of bovine pulmonary artery endothelial (BPAE) cells (F36924, Thermo Fisher Scientific Inc., USA) to characterize the lateral resolution of the MSIM. The F-actin of the cell is labeled with Alexa Fluor 488 phalloidin, which can be efficiently excited by the laser to emit green fluorescence (λ = 520 nm, NA = 1.27). The wide-field image and the reconstruction results using MSIM and MSIMMSBL are shown in Figs. 9(a)–9(c). Figure 9(d) shows the intensity profile through the white short line in Figs. 9(a)–9(c). The detail of approximately 135 nm can be recognized by MSIM and MSIMMSBL, however, the contrast of MSIMMSBL is still higher than MSIM. From Fig. 9, we can obtain the same conclusion that the MSBL algorithm can achieve a higher super-resolution image than the convolution MSIM reconstruction algorithm.
Fig. 9 Comparison of experiment MSIM and MSIMMSBL. (a) Wide-field image, (b) MSIM reconstruction image, and (c) MSIMMSBL reconstruction image of F-actin. (d) Intensity profile through the short white line in (a)-(c).
5. Discussion
Without any modification of the MSIM data acquisition process, MSBL algorithm can obtain a more than twofold resolution enhancement compared with the widefield image, which is higher than conventional MSIM data processing methods. At the same time, the data processing of MSIMMSBL is more concise than that of MSIM. The resolution improvement of MSIM will promote its application in super-resolution microscopy.
We also can find that the resolution improvement of MSIMMSBL is smaller than speckle SIM. The difference between speckle SIM and MSIM is the illumination pattern. In speckle SIM, the illumination patterns are dynamic speckle patterns. The high and low intensity fluctuation of the fluorescent emission light intensity are random and uniform, which is more suitable for MSBL algorithm or other algorithms. While in MSIM, the illumination pattern is the array of PSF. The distribution of the excited fluorescence probes is not uniform, which makes the resolution of MSIM is less than speckle SIM when the MSBL algorithm is used.
Furthermore, MSBL algorithm has been applied in STORM, speckle SIM, and MSIM and obtain the commendable results. The data acquisition process of some other microscopy technologies, such as SIM with grating illumination [1] and super-resolution optical fluctuation imaging [17], are also can be treated as the MMV model signal detect processes. Although the difference of the illumination pattern and data acquiring process may affect the spatial resolution, the MSBL algorithm is a potential tool in super-resolution imaging technology.
Funding
National key R&D Program of China (2017YFA0700500); Guangdong Natural Science Foundation (2018A030313362, 2014A030312008, 2017A030310132); The China Postdoctoral Science Foundation (010865); Shenzhen Science and Technology Planning Project (JCYJ20170818141701667, JCYJ20170412105003520, JCYJ20170818144012025, JCYJ20170412105003520); National Basic Research Program of China (2015CB352005); National Natural Science Foundation of China (NSFC) (61775144, 61525503, 61620106016, 81727804, 61605127); Shenzhen Special Funds for the Development of Strategic Emerging Industries (SWCYL20150331010038) and China Scholarship Council (201708440486).
References
1. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef] [PubMed]
2. A. G. York, P. Chandris, D. D. Nogare, J. Head, P. Wawrzusin, R. S. Fischer, A. Chitnis, and H. Shroff, “Instant super-resolution imaging in live cells and embryos via analog image processing,” Nat. Methods 10(11), 1122–1126 (2013). [CrossRef] [PubMed]
3. A. G. York, S. H. Parekh, D. Dalle Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef] [PubMed]
4. P. Kner, B. B. Chhun, E. R. Griffis, L. Winoto, and M. G. Gustafsson, “Super-resolution video microscopy of live cells by structured illumination,” Nat. Methods 6(5), 339–342 (2009). [CrossRef] [PubMed]
5. L. Shao, P. Kner, E. H. Rego, and M. G. Gustafsson, “Super-resolution 3D microscopy of live whole cells using structured illumination,” Nat. Methods 8(12), 1044–1046 (2011). [CrossRef] [PubMed]
6. C. B. Müller and J. Enderlein, “Image scanning microscopy,” Phys. Rev. Lett. 104(19), 198101 (2010). [CrossRef] [PubMed]
7. E. N. Ward, F. H. Torkelsen, and R. Pal, “Enhancing multi-spot structured illumination microscopy with fluorescence difference,” R. Soc. Open Sci. 5(3), 171336 (2018). [CrossRef] [PubMed]
8. M. Ingaramo, A. G. York, P. Wawrzusin, O. Milberg, A. Hong, R. Weigert, H. Shroff, and G. H. Patterson, “Two-photon excitation improves multifocal structured illumination microscopy in thick scattering tissue,” Proc. Natl. Acad. Sci. U.S.A. 111(14), 5254–5259 (2014). [CrossRef] [PubMed]
9. F. Ströhl and C. F. Kaminski, “A joint Richardson-Lucy deconvolution algorithm for the reconstruction of multifocal structured illumination microscopy data,” Methods Appl. Fluoresc. 3(1), 014002 (2015). [CrossRef] [PubMed]
10. R. Cao, T. Yang, Y. Fang, C. Kuang, and X. Liu, “Pattern-illuminated Fourier ptychography microscopy with a pattern-estimation algorithm,” Appl. Opt. 56(24), 6930–6935 (2017). [CrossRef] [PubMed]
11. M. Guo, P. Chandris, J. P. Giannini, A. J. Trexler, R. Fischer, J. Chen, H. D. Vishwasrao, I. Rey-Suarez, Y. Wu, X. Wu, C. M. Waterman, G. H. Patterson, A. Upadhyaya, J. W. Taraska, and H. Shroff, “Single-shot super-resolution total internal reflection fluorescence microscopy,” Nat. Methods 15(6), 425–428 (2018). [CrossRef] [PubMed]
12. D. P. Wipf and B. D. Rao, “An empirical Bayesian strategy for solving the simultaneous sparse approximation problem,” IEEE Trans. Signal Process. 55(7), 3704–3716 (2007). [CrossRef]
13. N. Han and Z. Song, “Bayesian multiple measurement vector problem with spatial structured sparsity patterns,” Digit. Signal Process. 75, 184–201 (2018). [CrossRef]
14. J. Min, J. Jang, D. Keum, S. W. Ryu, C. Choi, K. H. Jeong, and J. C. Ye, “Fluorescent microscopy beyond diffraction limits using speckle illumination and joint support recovery,” Sci. Rep. 3(1), 2075 (2013). [CrossRef] [PubMed]
15. J. Wu, S. Li, S. Zhang, D. Lin, B. Yu, and J. Qu, “Fast analysis method for stochastic optical reconstruction microscopy using multiple measurement vector model sparse Bayesian learning,” Opt. Lett. 43(16), 3977–3980 (2018). [CrossRef] [PubMed]
16. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics 6(5), 312–315 (2012). [CrossRef]
17. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proc. Natl. Acad. Sci. U.S.A. 106(52), 22287–22292 (2009). [CrossRef] [PubMed]
