Second-order optimized regularized structured illumination microscopy (sorSIM) for high-quality and rapid super resolution image reconstruction with low signal level

Wei Yu; Yangyang Li; Stijn Jooken; Olivier Deschaume; Fei Liu; Shouyu Wang; Shouyu Wang; Carmen Bartic

doi:10.1364/OE.390745

1. Introduction

In order to overcome diffraction limit in optical imaging, various super resolution imaging techniques have been proposed, including stimulated emission depletion microscopy (STED) [1,2], photoactivated localization microscopy (PALM)/stochastic optical reconstruction microscopy (STORM) [3,4], super resolution optical fluctuation imaging (SOFI) [5] and structured illumination microscopy (SIM) [6–8], etc. Among these, SIM is extremely suited for live cell observations due to its fast imaging speed, low phototoxicity and considerable resolution enhancement [9–11]. In SIM, the non-uniform illumination on samples allows that the non-recordable high frequency components can be down modulated into recordable frequencies of objective and thus expanding the acceptable frequency range. To realize this, classical SIM image reconstruction methods first extract the high frequency components from the images obtained under structured illumination of the sample, then shift them to proper locations in the frequency domain to finally reconstruct super resolution images [7,12–15]. However, these algorithms require prior knowledge of the structured illumination patterns, and errors of pattern estimation in structured illumination decrease the reconstruction accuracy. To avoid the influence of structured illumination errors on the image reconstruction, blind SIM has been proposed, where no prior knowledge of the illumination is needed [16–18]. Moreover, Fourier ptychography (FP) [19–23] and Richardson-Lucy deconvolution [24–28] based SIM image reconstruction methods have also been proposed to pursue fast iteration convergence and rapid processing speed. These methods provide extremely high accuracy in super resolution image reconstruction in rather high signal to noise ratio (SNR) conditions. However, in order to prolong observation periods especially for live cell imaging, illumination intensity is often decreased aiming at reducing photobleaching and phototoxicity, while inevitably decreasing the signal level. Unfortunately, both classical and blind SIM image reconstruction methods cannot provide high quality results in low SNR conditions, and thus limiting their applications in long term live cell imaging.

In order to solve this problem, Chu et al. reported total variation (TV) based SIM (TV-SIM) to reduce noise effect in super resolution image reconstruction [29], while due to piecewise constant of first order derivative, TV-SIM still suffers from staircase effects that may over-sharpen sample boundaries. Additionally, Huang et al. proposed second order derivative based Hessian-SIM to reduce the staircase effects and thus better preserve sample edge details [30]. Boulanger et al. applied non-smooth convex optimization on SIM to reconstruct high-accurate super resolution images even in low signal level [31]. However, due to the ill-posed problem of the direct cost function solution in Hessian-SIM and the heavy computation load in the non-smooth convex optimization-based SIM, both methods suffer from long convergence time.

In this work, to rapidly reconstruct high-quality super resolution images from low SNR images captured with structured illumination, we have implemented a second order optimized regularized SIM (sorSIM) method. Benefiting from the introduction of a second order regularization term, noise can be effectively reduced.

While TV-SIM is using the first order partial derivative, sorSIM relies on the second order partial derivative, similar to Hessian-SIM, in order to reduce the staircase effect. On the other hand, in Hessian-SIM, only denoising is implemented during iterations, while in sorSIM, both reconstruction and denoising are implemented during each iteration.

Also different from the previously reported methods, in sorSIM the penalty factor in the regularized term can be selected according to the noise level of the captured images to balance resolution enhancement and noise resistance. The penalty factor is adapted in function of the noise level based on simulations using an image quality evaluation criterion.

Compared to the result of classical SIM image reconstruction algorithms, including those designed for low signal levels, sorSIM can provide satisfactory resolution enhancement for lower SNR conditions as demonstrated by both numerical simulations and experiments. By balancing the trade-off between resolution enhancement and noise resistance, sorSIM may become a useful tool for long term, live cell super resolution imaging with low illumination conditions.

2. Principle

In SIM, various structured illumination patterns can be used including sinusoidal [7], multi-spot [21,32] and speckle illumination [16,19]. Among these, speckle illumination often generated by a diffuser can be retrieved through iterations along with the process of sample super resolution reconstruction, however, sinusoidal and multi-spot illumination patterns can be precisely determined to accelerate image reconstruction. In this work, to speed up the image reconstruction and for compatibility with many commercial SIM systems, we used two-dimensional (2-D) sinusoidal illumination to demonstrate the sorSIM principle. Equation (1) describes the sinusoidal illumination P_θ,φ, in which r is the position vector, the angle θ quantifies the sinusoidal illumination directions (i.e., 0°, 120°, 240°), φ describes the sinusoidal illumination phase shifting angles, m denotes the modulation depth, and k_θ is the modulation frequency vector.

(1)$${P_{\theta ,\varphi }} = 1 + m\cos ({\boldsymbol{k}_{\boldsymbol{\theta }}} \cdot \boldsymbol{r} + \varphi )$$

According to SIM principle, the captured image signals, I_θ,φ, are given by Eq. (2), in which S is the sample signal, n is the noise, h is the point spread function (PSF) and ⊗ represents the convolution.

(2)$${I_{\theta ,\varphi }} = ({P_{\theta ,\varphi }}S) \otimes h + n$$

In SIM image reconstruction methods, super resolved sample image S can be extracted by deconvoluting the structured illuminated images. However, for low signal levels, S can hardly be reconstructed due to the noise influence on deconvolution. Therefore, we propose the sorSIM to solve the problem. Similar to reported SIM image reconstruction techniques [23,30,32], the structured illumination patterns, P_θ,φ, especially k_θ and φ, should be accurately determined in sorSIM before image reconstruction. However, due to the low SNR of the captured images, simple auto-correlation cannot provide accurate structured illumination information [33]. Therefore, we use the auto-correlation between F{I_θ,φ}(k)H*(k) and its shifted variant F{I_θ,φ}(k+Δk)H*(k) to estimate k_θ, in which H is the optical transfer function (OTF) as the Fourier transform (F) of h in Eq. (2), and F{I_θ,φ} is the Fourier transform of I_θ,φ as shown in Eq. (3).

(3)$${\cal{F}}\{{{I_{\theta ,\varphi }}} \}= {\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}\cdot H + {\cal{F}}{\{n \}}$$

With the help of OTF function, the high frequency noise can be effectively reduced, and k_θ can be accurately determined according to the peak value of the auto-correlation (when Δk = k_θ). Further, to determine φ, auto-correlation between the recorded images I_θ,φ and constructed sinusoidal function cos(k_θr+Δφ) is computed with varied Δφ, and only when Δφ = φ, the auto-correlation can reach the peak value [13].

Similar to other SIM image reconstruction techniques, once the structured illumination P_θ,φ has been determined, the sample image S can be extracted from the low SNR structured illuminated images by minimizing the cost function. However, in sorSIM, the cost function E given by Eq. (4), is different from the ones used in FP-SIM [19], TV-SIM [29] and Hessian-SIM [30]. As the cost function of FP-SIM does not consider noise suppression, high-quality super resolution images can hardly be extracted from low SNR structured illuminated images; while second order regularization term is introduced in sorSIM to significantly reduce the noise influence on image reconstruction. Though both TV-SIM and Hessian-SIM can reconstruct noise suppressed super resolution images in low signal level, TV-SIM suffers from staircase effect and Hessian-SIM suffers from long time processing. However, the introduced second order regularization term in sorSIM [34,35] can effectively preserve sample details compared to the total variation item in TV-SIM; moreover, sorSIM can reconstruct super resolution images faster than Hessian-SIM by avoiding the ill-posed problem of direct cost function solution.

(4)$$E(S) = \frac{1}{2}\int_{{\Omega _1}} {{{(I{^{\prime}_{\theta ,\varphi }} \otimes h - {I_{\theta ,\varphi }})}^2}} + {\alpha}\int_{{\Omega _1}} {{{(S^{\prime} - S)}^2}} + \beta \int_{{\Omega _1}} {|{{\nabla^2}S} |}$$

In Eq. (4), S’ is the updated sample super resolution image, I’_θ,φ is the updated structured illuminated image, α and β are positive values, Ω₁ represents the captured image region, and ∇²S is given by Eq. (5).

(5)$${\nabla ^2}S = \left( {\begin{array}{cc} {\frac{{{\partial^2}S}}{{\partial {x^2}}}}&{\frac{{{\partial^2}S}}{{\partial y\partial x}}}\\ {\frac{{{\partial^2}S}}{{\partial x\partial y}}}&{\frac{{{\partial^2}S}}{{\partial {x^2}}}} \end{array}} \right)$$

However, it is difficult to solve Eq. (4), and in order to simplify its solution, the alternating minimization method [36] is adopted here, which decomposes E in Eq. (4) into two sub-cost functions E₁ and E₂ as listed in Eqs. (6) and (7), respectively. It is worth noting that E₁ represents the deconvolution process while E₂ describes the denoising process; and in order to minimize E, E₁ and E₂ should be minimized simultaneously.

(6)$${E_1}(S) = \frac{1}{2}\int_{{\Omega _1}} {{{(I{^{\prime}_{\theta ,\varphi }} \otimes h - {I_{\theta ,\varphi }})}^2}} + \frac{\alpha }{2}\int_{{\Omega _1}} {{{(S^{\prime} - S)}^2}}$$

(7)$${E_2}(S) = \frac{\alpha }{2}\;\int_{{\Omega _1}} {{{(S^{\prime} - S)}^2}} + \beta \int_{{\Omega _1}} {|{{\nabla^2}S} |}$$

First, to minimize E₁, its partial derivative should be zero, from which, the updated deconvoluted sample S₁’ is derived as:

(8)$${S_1}^{\prime} = S + \frac{{{P_{\theta ,\varphi }}}}{{max{{({P_{\theta ,\varphi }})}^2}}}(I{^{\prime}_{\theta ,\varphi }} - {I_{\theta ,\varphi }})$$

In high SNR conditions, the iterative process (via Eq. (8)) is robust enough for super resolution image reconstruction. However, for low SNRs, the noise influence on deconvolution hampers the image reconstruction. To solve this, the second order regularization term is introduced in E₂, which should be minimized simultaneously. Since both α and β have positive values, Eq. (7) can be simplified as Eq. (9), where S₂’ is the updated denoised sample.

(9)$${E_2} = \frac{1}{2}\int_{{\Omega _1}} {{{({S_2}^{\prime} - {S_1}^{\prime})}^2}} + \frac{\beta }{\alpha }\int_{{\Omega _1}} {|{{\nabla^2}{S_2}^{\prime}} |}$$

To solve the constraint problem of Eq. (9), we adopted the split Bregman method [37,38] to achieve fast convergence via iterative minimization. The essence of split Bregman method is introducing a Bregman iteration parameter and calculating a Bregman distance. Here, an auxiliary splitting vector variable v is introduced to approximate the partial derivatives of the regularization term ∇²S₂’ in Eq. (9), and thus, the cost function can be rewritten as Eq. (10), where β/α is defined as μ, representing the penalty factor.

(10)$${E_2} = \frac{1}{2}\int_{{\Omega _1}} {{{({S_2}^{\prime} - {S_1}^{\prime})}^2}} + \mu \int_{{\Omega _1}} {|\boldsymbol{v} |}$$

Applying the method of Lagrange multiplier to minimize Eq. (10), the unconstrained cost function E₂’ can be expressed as in Eq. (11), in which λ is the Lagrange multiplier.

(11)$$E_2^{\prime} = {E_2} + \frac{\lambda }{2}{\int_{{\Omega _1}} {({\boldsymbol{v} - {\nabla^2}{S_2}^{\prime}} )} ^2}$$

To optimize Eq. (11), a Bregman iterative parameter b is introduced which turns Eq. (11) into Eq. (12), in which b is adopted to minimize Eq. (12) with fast convergence speed.

(12)$$E_2^{\prime} = {E_2} + \frac{\lambda }{2}{\int_{{\Omega _1}} {(\boldsymbol{v} - {\nabla ^2}{S_2}^{\prime} - \boldsymbol{b})} ^2}$$

Therefore, the new cost function E₂’ can be written as:

(13)$$E_2^{\prime} = \frac{1}{2}\int_{{\Omega _1}} {{{({S_2}^{\prime} - {S_1}^{\prime})}^2}} + \mu \int_{{\Omega _1}} {|\boldsymbol{v} |} + \frac{\lambda }{2}\int_{{\Omega _1}} {{{(\boldsymbol{v} - {\nabla ^2}{S_2}^{\prime} - \boldsymbol{b})}^2}}$$

To minimize E₂’ in Eq. (13), v is first fixed to minimize S₂’ according to Eq. (14), in which div² is the second order divergence operator.

(14)$${S_2}^{\prime} + \lambda di{v^2}({\nabla ^2}{S_2}^{\prime}) = {S_1}^{\prime} + \lambda di{v^2}({\boldsymbol{v}} - {\boldsymbol{b}})$$

To deal with this second order differential equation, Fourier transform is applied to both sides of Eq. (14), resulting in its equivalent expression:

(15)$$\left[ {1 + 8\lambda cos\left( {\frac{{\pi {f_x}}}{N} + \frac{{\pi {f_y}}}{M}} \right) - 8\lambda } \right]{\cal{F}}\{{{S_2}^{\prime}} \}= {\cal{F}}\{{{S_1}^{\prime} + \lambda di{v^2}({\boldsymbol{v}} - {\boldsymbol{b}})} \}$$

where M, N are the matrix size of image S₁’ and f_x, f_y describe the frequency indices of super resolution images, respectively.

Therefore, S₂’ in Eq. (13) can be updated using Eq. (16), and only its real part is used for later iterative minimization.

(16)$${S_2}^{\prime} = {{\cal{F}}^{ - 1}}\left\{ {\frac{{{\cal{F}}\{{{{S_1}^{\prime} + \lambda di{v^2}({\boldsymbol{v}} - {\boldsymbol{b}})} }\}}}{{[{1 + 8\lambda cos({{{\pi {f_x}} \mathord{\left/ {\vphantom {{\pi {f_x}} N}} \right.} N} + {{\pi {f_y}} \mathord{\left/ {\vphantom {{\pi {f_y}} M}} \right.} M}} )- 8\lambda } ]}}} \right\}$$

After S₂’ extraction with fixed v, next S₂’ is fixed to optimize the splitting vector variable v through solving the Euler equation, and v can be computed as:

(17)$${\boldsymbol{v}} = \left\{ \begin{array}{cl} (|{{\nabla^2}{S_2}^{\prime} + {\boldsymbol{b}}} |- \frac{\mu }{\lambda })\frac{{{\nabla^2}{S_2}^{\prime} + {\boldsymbol{b}}}}{{|{{\nabla^2}{S_2}^{\prime} + {\boldsymbol{b}}} |}} &if \,\, |{{\nabla^2}{S_2}^{\prime} + {\boldsymbol{b}}} |> \frac{\mu }{\lambda }\\ 0 &if \,\, \textrm{e}lse \end{array} \right.$$

Finally, the Bregman iteration parameter b can be calculated as:

(18)$${\boldsymbol{b}} = {\boldsymbol{b}} + {\nabla ^2}{S_2}^{\prime} - {\boldsymbol{v}}$$

During the E₂’ minimization process, the value of the penalty factor μ in regularization term has a critical impact on the image reconstruction: when μ is large, noise can be significantly suppressed, while however, the resolution is sacrificed; otherwise, the high resolution can be maintained but the noise may still affect the reconstructed image. Therefore, in order to balance the trade-off between resolution enhancement and noise resistance, μ should be adapted in sorSIM according to the SNR of the captured images. In addition, to achieve high processing speed and rapid convergence in sorSIM image reconstruction, after updating the sample super resolution image according to the above procedure, the structured illuminated images, I_θ,φ, are also updated in the frequency domain. This additional updating process not only speeds up the convergence, but also further suppresses the noise by filtering out the noise beyond the system OTF.

Similar to Eq. (6), the cost function E₃ in frequency domain is constructed as Eq. (19), in which Ω₂ represents the frequency regions of the structured illuminated images, $\tau$ is a positive value.

(19)$${E_3}({\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}) = \frac{1}{2}\int_{{\Omega _2}} {{{({\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}\cdot H - {\cal{F}}\{{{I_{\theta ,\varphi }}} \})}^2}} + \frac{\tau }{2}\int_{{\Omega _2}} {{{({\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}- {\cal{F}}\{{{I_{\theta ,\varphi }}} \})}^2}}$$

In order to minimize E₃, its partial derivative should be zero, therefore, the structured illuminated image is updated as:

(20)$${\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}= {\cal{F}}\{{{I_{\theta ,\varphi }}} \}+ H \cdot ({\cal{F}}\{{I{^{\prime}_{\theta ,\varphi }}} \}\cdot H - {\cal{F}}\{{{I_{\theta ,\varphi }}} \})$$

To use the proposed sorSIM method for low signal to noise ratio super resolution image reconstruction, first, the sample super resolution image S₁’ is updated according to Eq. (8). Then, sample super resolution image S₂’ is updated according to Eq. (16) with v and b given by Eqs. (17) and (18) to minimize the cost function E₂ for denoising. Finally, with the updated sample super resolution image, the structured illuminated images I_θ,φ are updated according to Eq. (20). In sorSIM, one iteration represents all these three updating procedures, and when the relative error (RE) between two adjacent updated sample super resolution images is less than a preset value, ε, or the iteration reaches the preset maximal iteration number, the iteration process stops to provide the final sample super resolution image. The pseudocode is listed in Algorithm 1, and the flowchart of sorSIM is shown in Fig. 1. According to the principle of sorSIM, there is only one parameter described as penalty factor μ that needs to be introduced (see Line 4 in the Code 1).

Fig. 1. Flowchart of the proposed sorSIM.

Download Full Size | PDF

The second order regularization term is introduced not only to reconstruct super resolution image from low SNR structured illuminated images, but also to preserve high-quality image reconstruction and to accelerate processing speed.

oe-28-11-16708-i001

3. Simulations

First, the proposed sorSIM was tested by numerical simulations not only for identifying the relation between the penalty factor in the regularization term and the noise level of captured images, but also for verifying its super resolution image reconstruction performance in low SNR situations. As in the theoretical analysis and experiment, in simulations, a 2-D sinusoidal illumination pattern was applied with three direction angles and three phase shifting angles (both angles were 0°, 120° and 240°, respectively). The first order Bessel function was used to generate PSF. The value of k_θ was set as 0.8 times of the OTF cutoff frequency. The modulation contrast-to-noise ratio (MCNR) implemented in SIMCheck toolbox [39] was used to quantify the quality of recorded images: the higher the MCNR, the better the image quality, and a MCNR higher than 4, corresponds to a high SNR image. Samples as Siemens star and microtubules were used to mimic standard and actual samples, respectively. A computational platform with CPU of i7-3770 and memory of 16 GB was employed in both numerical simulations and experiments for SIM super resolution image reconstruction.

Inappropriate penalty factor selection leads to insufficient noise suppression or over-smoothing of structure preservation, which corresponds to poor retrieved image quality or reduced retrieval resolution respectively. To find out the relation between the MCNR value and the penalty factor, simulations were conducted for different noise degrees. To quantify the reconstructed image quality, we defined a new criterion, Q, which combines the structural similarity index (SSIM) and the peak SNR (PSNR), since a single criterion based on SSIM or PSNR may not effectively reflect the image quality. The value ranges of PSNR and SSIM are quite different. PSNR ranges from 20 to 40 dB for the retrieved images, while the SSIM, value changes from 0 to 1. Higher values of each criterion mean better retrieval image quality. In order to make the weight of both indices equal, their values were divided by different factors to define the image quality evaluation criterion Q as:

(21) $$Q = PSNR/80 + SSIM/2$$

where Q can range from 0 to 1, with higher value Q charactering better retrieved image quality.

Figures 2(A) and (B) illustrate the reconstructed image quality for different MCNR values and different penalty factor values. In agreement with the theoretical analysis, for high SNR cases, even a small value of μ enables high quality image reconstruction. However, for low SNR situations, larger values of μ are required for weak signal extraction and noise suppression. Figure 2(C) gives the quantitative comparison of image quality for different penalty factors and MCNR values, while Fig. 2(D) illustrates the optimal penalty factor values for different image quality situations (as characterized by MCNR). The definition of the quality criterion, Q, allows evaluating the reconstruction performance based on the similarity between the reconstructed image and the ground truth as well as on the image signal to noise ratio. However, the Q value does not evaluate the high frequency performance. Hence, an additional resolution enhancement ratio, R, is considered, which quantifies the resolution enhancement of the reconstructed super resolution image compared to the wide field image in the same MCNR conditions. R is calculated by Eq. (22) as the ratio between the cut-off frequency of reconstructed image f_c^recon and its corresponding widefield image’s cut-off frequency f_c^w within the same MCNR conditions [40,41].

(22)$$R = {{f_c^{recon}} \mathord{\left/ {\vphantom {{f_c^{recon}} {f_c^w}}} \right.} {f_c^w}}$$

Fig. 2. Relation between the quality of the recorded images (i.e., MCNR values) and the penalty factor μ for Siemens star and microtubule samples.

Download Full Size | PDF

In Fig. 2(A), when the penalty factor μ becomes higher, R decreases; while when the penalty factor μ becomes lower, R increases. However, pursuing only higher resolution enhancement ratios will induce more background artifacts. Therefore, the Q factor is a good criterion for optimizing reconstruction results to balance both resolution enhancement and noise resistance. According to Eq. (21), the weight of SSIM equals that of PSNR, however, by adjusting the coefficients, the evaluation criterion can focus more on SSIM or PSNR.

After analysis on the optimized penalty factor selection in sorSIM, its super resolution image reconstruction performance was tested. To simulate different SNR conditions, different degrees of additive white Gaussian noise (i.e., 1%, 5%, 10%, 20% and 30%) were introduced to the images, corresponding to different MCNR values (i.e., 6.4, 5.3, 3.8, 3.0 and 2.8 for Siemens star sample and, respectively, 6.0, 5.4, 4.0, 3.2 and 2.9 for microtubules images). Figure 3 shows the wide-field and corresponding reconstructed images of both Siemens star and microtubules samples retrieved by different methods: traditional SIM (Wiener), FP-SIM, TV-SIM, Hessian-SIM and sorSIM. Note that in sorSIM, the penalty factor was optimized selected according to the MCNR estimate of the captured image. In order to fairly compare the proposed sorSIM with TV-SIM and Hessian-SIM, all the input parameters in TV-SIM and Hessian-SIM are optimized, which means that the reconstruction results via TV-SIM and Hessian-SIM are all optimized. Although there are no quantitative analyses on the influence of input parameters on the reconstruction quality in TV-SIM and Hessian-SIM, in order to obtain the optimized input parameters as well as reconstruction results, many trials with different input parameters were performed and optimized input parameters were selected along with the best reconstruction results (i.e., with sufficient resolution enhancement and noise resistance) as shown in Fig. 3. In addition, to quantify the image reconstruction performance, structural similarity index (SSIM) and peak SNR (PSNR) values were used. Moreover, the processing time of these SIM image reconstruction methods was also counted, both are listed in Fig. 3.

Fig. 3. Comparisons of super resolution images reconstructed with Wiener, FP-SIM, TV-SIM, Hessian-SIM and sorSIM methods, for different captured image quality (MCNR values). Corresponding SSIMs, PSNRs, and processing times of the 512×512×9 matrix are indicated under each image.

Download Full Size | PDF

According to both visual and quantitative evaluations, when the introduced noise was rather low (e.g. 1% noise), all SIM image reconstruction methods could retrieve high-quality super resolution images (i.e., with high SSIM and PSNR values). When the noise level is higher, the quality of images reconstructed by Wiener and FP-SIM methods decreases significantly, as revealed by the lower SSIM and PSNR values. On the other hand, the reconstruction results of TV-SIM, Hessian-SIM and sorSIM still maintain high resolution. However, TV-SIM retrieved images often have lower SSIM values compared to Hessian-SIM and sorSIM retrieved images since TV-SIM suffers from staircase effect. Although both Hessian-SIM and sorSIM methods can provide high-quality super resolution images, Hessian-SIM requires more computational time compared to sorSIM as listed in Fig. 3. Additionally, due to the optimized penalty factor selection, sorSIM can balance the trade-off between resolution enhancement and noise resistance for different MCNR conditions. Although TV-SIM, Hessian-SIM and sorSIM can work well for low signal level, when MCNR was less than 2.8, high-quality images can hardly be retrieved due to the severe noise influence on the deconvolution.

According to the numerical simulation, by selecting the proper penalty factor according to the MCNR of the captured images, sorSIM can rapidly reconstruct high-quality super resolution images with high resolution, SSIM and PSNR when MCNR is no less than 2.8. Moreover, compared to Wiener and FP-SIM, sorSIM has a better performance for low signal levels; since, when compared to TV-SIM, sorSIM significantly reduces the staircase effect; and, when compared to Hessian-SIM, sorSIM requires less image reconstruction time.

4. Experiments

To evaluate the performance of the proposed method, experiments were conducted on a commercial Nikon SIM microscope, which was equipped with an oil immersion micro-objective (100x magnification and numerical aperture of 1.49) and 2-D sinusoidal illumination was used in this commercial SIM system. First, the optimized penalty factor selection was tested on samples consisting of 100 nm fluorescent beads (TetraSpeck Microspheres, Thermo Fisher). The excitation source was a laser with the wavelength of 488 nm, and the fluorescent bead emission signal around the wavelength 515nm was collected by an Andor Xion Ultra 897 EMCCD camera. In order to simulate different SNR conditions, the exposure time of the EMCCD camera was kept the same (100 ms), while the laser power was varied as 40 mW, 30 mW, 20 mW and 10 mW corresponding to estimated MCNRs of captured images as 3.9, 3.3, 3.0 and 2.8, respectively. Figure 4 shows the super resolution images reconstructed by sorSIM with different penalty factors as well as the wide-field images which are summations of all 9 captured structured illuminated images. According to the simulation result (Fig. 2), optimal penalty factors corresponding to these MCNRs should be 0.03, 0.06, 0.08, and 0.10, respectively. Besides reconstruction results using the optimal penalty factors, other results using different penalty factors are also shown in Fig. 4. Compared to the wide-field images, the retrieved images can significantly enhanced resolution, as revealed by closely positioned fluorescent beads marked with arrows. More importantly, when the penalty factors were selected depending on the MCNRs according to the optimized relation, super resolution images with the highest quality can be obtained in specific MCNR conditions.

Fig. 4. Wide-field and sorSIM reconstructed super resolution images with different penalty factors values and in different MCNR conditions. Scale bar indicates 1 μm. Corresponding penalty factors are listed in each sub-figure. The subplots describe marked closely positioned fluorescent beads (upper panel) and background artifacts (lower panel). In addition, processing time T of reconstructing a 1024×1024×9 matrix is also listed.

Download Full Size | PDF

As shown by the images in Fig. 4, the penalty factor value plays an important role in the quality of image reconstruction. Different from numerical simulations where PSNR and SSIM are used to quantitatively evaluate the reconstruction quality, in experiments, direct profile plots of closely positioned fluorescent beads and background artifacts are used to estimate the resolution enhancement and the noise resistance, respectively, [19,26,30]. According to these subplots in Fig. 4, with large penalty factors, the noise can be significantly suppressed, but the image resolution is inevitably reduced as the closely positioned fluorescent beads can hardly be distinguished; otherwise, with low penalty factors, high image resolution can be obtained as closely positioned fluorescent beads can easily be distinguished but more noise occurs in reconstructed images. Therefore, in order to balance the trade-off between resolution enhancement and noise resistance, the penalty factor should be optimized selected in sorSIM according to the MCNR of the captured images by following the optimization relation.

After verification on the optimized penalty factor selection in sorSIM, its super resolution image reconstruction performance was tested first using the same 100 nm fluorescent beads. Moreover, the laser power was varied as 40 mW, 30 mW, 20 mW, 10 mW and 5 mW, respectively, while the exposure time of the EMCCD camera was kept at 100 ms. Imaging reconstructed using the Wiener, FP-SIM, TV-SIM, Hessian-SIM and sorSIM methods are shown in Fig. 5. The reconstruction results via TV-SIM and Hessian-SIM were obtained with optimized input parameters. In sorSIM, since the MCNRs of the captured structured illuminated images were estimated to be 3.9, 3.3, 3.0, 2.8 and 2.5, their corresponding penalty factors were 0.03, 0.06, 0.08, 0.1 and 0.15 by following the optimized penalty factor selecting relation proved by both simulations and experiments. As shown in Fig. 5, the resolution of the reconstructed super resolution images can be enhanced compared to wide-field images, especially revealed by the analysis of closely positioned fluorescent beads marked with arrows. In order to analyze the reconstruction quality, direct profile subplots of closely positioned fluorescent beads and background artifacts in Fig. 5 were used to evaluate the resolution enhancement and noise resistance. For lower MCNR values, the image reconstruction quality via Wiener and FP-SIM methods significantly deteriorated, with poor resolution and obvious artifacts, since these method do not consider the noise suppression in deconvolution, while TV-SIM, Hessian-SIM and sorSIM methods especially designed for low SNR conditions, still generate high quality images with satisfactory resolution enhancement and noise resistance, proving that sorSIM can reconstruct high quality super resolution images even in low signal level situations.

Fig. 5. Comparison of images reconstructed using Wiener, FP-SIM, TV-SIM, Hessian-SIM and sorSIM methods of samples containing 100 nm fluorescent beads for different MCNR conditions. Scale bar indicates 1 μm. Penalty factors used in sorSIM are listed in each sub-figure. The subplots describe marked closely positioned fluorescent beads (upper panel) and background artifacts (lower panel). In addition, processing time T of reconstructing a 1024×1024×9 matrix is also listed.

Download Full Size | PDF

However, it is still difficult to evaluate the quality of the reconstructed super resolution images via TV-SIM, Hessian-SIM and sorSIM only using fluorescent beads. We have also imaged microtubules in Vero cells to evaluate the image reconstruction quality as shown in Fig. 6. Vero cells (ATCC CCL-81) were cultured in Dulbecco’s modified Eagle medium (DMEM, Gibco), supplemented with 10% fetal calf serum (FCS, Biological Industry) and antibiotics (100 U/mL penicillin and 100 μg/mL streptomycin). For microtubule staining, rabbit anti-alpha-tubulin IgG (1:200 dilutions, Proteintech Group) as primary antibody and CoraLite 488-conjugated goat anti-rabbit IgG (1:400 dilutions, Proteintech Group) as secondary antibody, were used to generate fluorescence signals with the emission wavelength of ∼515 nm. A laser with the wavelength of 488 nm and 10 mW power was used for fluorescence excitation, and the EMCCD camera was used for structured illuminated image recording with the fixed exposure time of 100 ms. The MCNR of the captured images was estimated to be 3.0, and, therefore, the optimal penalty factor value should be 0.08.

Fig. 6. Comparison on images of Vero cells with microtubules staining reconstructed using different methods. Captured images have an estimated MCNR of 3.0. Scale bar indicates 3 μm. Corresponding processing times and different penalty factors (for sorSIM) used in reconstructing a 1024×1024×9 matrix are listed in the respective image. The subplots describe marked closely positioned microtubules (right) and background artifacts (left).

Download Full Size | PDF

The super resolution images reconstructed by Wiener, FP-SIM, TV-SIM, Hessian-SIM and sorSIM are shown in Fig. 6, as well as the wide-field images obtained as the summation of all 9 structured illuminated images. The reconstruction results via TV-SIM and Hessian-SIM were obtained with optimized input parameters upon multiple trials. For comparison, reconstruction results using other penalty factor values in sorSIM are also provided in Fig. 6. All the SIM methods could significantly enhance the image resolution compared to wide field image. In order to analyze the reconstruction quality, direct profile subplots of closely positioned microtubules and background artifacts in Fig. 6 can be used to estimate the resolution enhancement and the noise resistance, respectively. For the MCNR value as 3.0, the image reconstruction quality using Wiener and FP-SIM is worse than of TV-SIM, Hessian-SIM and sorSIM (with optimized penalty factor), since both Wiener and FP-SIM suffer from obvious artifacts. Compared to Wiener and FP-SIM, the artifacts in the background are fewer in the results reconstructed via sorSIM (when μ is optimized as 0.08); and compared to TV-SIM and Hessian-SIM, the reconstructed microtubules via sorSIM (when μ is 0.08) have better spatial resolution. Moreover, when μ is 0.01 (lower than the optimized value), the separation of the microtubules is more obvious, but with more artifacts; and when μ is 0.15 (higher than the optimized value), the artifacts are reduced, while microtubules can hardly be distinguished. Therefore, careful selection of the penalty factor in sorSIM can result in the super resolution images with highest quality.

The sorSIM method is slower than Wiener and TV-SIM, close to FP-SIM, but faster than Hessian-SIM. However, as shown in Figs. 5 and 6, sorSIM can reconstruct super resolution images with much less artifacts compared to Wiener and FP-SIM and performs better than TV-SIM by avoiding the staircase effect. Moreover, sorSIM has a similar reconstruction performance compared to Hessian-SIM, while using less computational time (processing speed is around 35% higher, because sorSIM avoids the ill-posed problem of direct cost function solution in Hessian-SIM). Considering both the reconstruction performance and processing time, the proposed sorSIM provides a new way for super resolution image reconstruction, especially useful for low signal to noise ratio conditions.

As also shown in experiments, optimized penalty factor selection allows sorSIM achieve both resolution enhancement and noise resistance. Moreover, sorSIM can accurately and rapidly reconstruct super resolution images even from low SNR images. Therefore, the proposed sorSIM is a potential tool for super resolution imaging especially in low signal level.

5. Conclusion

In this paper, we designed and evaluated a method that allows to rapidly reconstruct high-quality super resolution images from captured structured illuminated images especially relevant for low signal level situations. This is particularly relevant for live cell imaging in order to reduce photobleaching and phototoxicity. With the introduction of a second order regularization term, noise can be effectively reduced through iterative cost function minimization. Additionally, penalty factor in the regularization term can be optimized selected according to MCNR index of captured images thus to balance resolution enhancement and noise resistance. As proven by both numerical simulations and experiments, sorSIM can reconstruct super resolution images with high quality from low SNR structured illuminated images. Moreover, compared to methods not considering noise suppression, sorSIM performs better especially in low signal conditions; and compared to methods designed for noise suppression, sorSIM not only preserves the accurate image reconstruction, but also accelerates the image reconstruction process. Therefore, with the advantages of good noise suppression capability, fast speed and high accuracy, the proposed sorSIM can be used in long term live cell imaging under low illumination for photobleaching and phototoxicity reduction.

Funding

Fonds Wetenschappelijk Onderzoek (G0947.17N); KU Leuven (C14/18/061, OT/14/084, C14/16/063).

Disclosures

The authors declare no conflicts of interest.

References

1. S. W. Hell and J. Wichmann, “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy,” Opt. Lett. 19(11), 780–782 (1994). [CrossRef]

2. T. A. Klar, S. Jakobs, M. Dyba, A. Egner, and S. W. Hell, “Fluorescence microscopy with diffraction resolution barrier broken by stimulated emission,” Proc. Natl. Acad. Sci. U. S. A. 97(15), 8206–8210 (2000). [CrossRef]

3. E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, “Imaging Intracellular Fluorescent Proteins at Nanometer Resolution,” Science 313(5793), 1642–1645 (2006). [CrossRef]

4. M. J. Rust, M. Bates, and X. Zhuang, “Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM),” Nat. Methods 3(10), 793–796 (2006). [CrossRef]

5. T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, “Fast, background-free, 3D super-resolution optical fluctuation imaging (SOFI),” Proc. Natl. Acad. Sci. U. S. A. 106(52), 22287–22292 (2009). [CrossRef]

6. M. G. L. Gustafsson, L. Shao, P. M. Carlton, C. J. R. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-Dimensional Resolution Doubling in Wide-Field Fluorescence Microscopy by Structured Illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef]

7. M. G. L. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef]

8. M. G. L. Gustafsson, “Nonlinear structured-illumination microscopy: Wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U. S. A. 102(37), 13081–13086 (2005). [CrossRef]

9. L. M. Hirvonen, K. Wicker, O. Mandula, and R. Heintzmann, “Structured illumination microscopy of a living cell,” Eur. Biophys. J. 38(6), 807–812 (2009). [CrossRef]

10. Y. Guo, D. Li, S. Zhang, Y. Yang, J.-J. Liu, X. Wang, C. Liu, D. E. Milkie, R. P. Moore, U. S. Tulu, D. P. Kiehart, J. Hu, J. Lippincott-Schwartz, E. Betzig, and D. Li, “Visualizing Intracellular Organelle and Cytoskeletal Interactions at Nanoscale Resolution on Millisecond Timescales,” Cell 175(5), 1430–1442.e17 (2018). [CrossRef]

11. A. Markwirth, M. Lachetta, V. Mönkemöller, R. Heintzmann, W. Hübner, T. Huser, and M. Müller, “Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction,” Nat. Commun. 10(1), 4315 (2019). [CrossRef]

12. M. Müller, V. Mönkemöller, S. Hennig, W. Hübner, and T. Huser, “Open-source image reconstruction of super-resolution structured illumination microscopy data in ImageJ,” Nat. Commun. 7(1), 10980 (2016). [CrossRef]

13. A. Lal, C. Shan, and P. Xi, “Structured Illumination Microscopy Image Reconstruction Algorithm,” IEEE J. Sel. Top. Quantum Electron. 22(4), 50–63 (2016). [CrossRef]

14. L. J. Young, F. Ströhl, and C. F. Kaminski, “A Guide to Structured Illumination TIRF Microscopy at High Speed with Multiple Colors,” J. Visualized Exp. 30(111), e53988 (2016). [CrossRef]

15. J. Demmerle, C. Innocent, A. J. North, G. Ball, M. Müller, E. Miron, A. Matsuda, I. M. Dobbie, Y. Markaki, and L. Schermelleh, “Strategic and practical guidelines for successful structured illumination microscopy,” Nat. Protoc. 12(5), 988–1010 (2017). [CrossRef]

16. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics 6(5), 312–315 (2012). [CrossRef]

17. R. Ayuk, H. Giovannini, A. Jost, E. Mudry, J. Girard, T. Mangeat, N. Sandeau, R. Heintzmann, K. Wicker, K. Belkebir, and A. Sentenac, “Structured illumination fluorescence microscopy with distorted excitations using a filtered blind-SIMalgorithm,” Opt. Lett. 38(22), 4723–4726 (2013). [CrossRef]

18. A. Jost, E. Tolstik, P. Feldmann, K. Wicker, A. Sentenac, and R. Heintzmann, “Optical Sectioning and High Resolution in Single-Slice Structured Illumination Microscopy by Thick Slice Blind-SIM Reconstruction,” PLoS One 10(7), e0132174 (2015). [CrossRef]

19. S. Dong, P. Nanda, R. Shiradkar, K. Guo, and G. Zheng, “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express 22(17), 20856–20870 (2014). [CrossRef]

20. S. Dong, J. Liao, K. Guo, L. Bian, J. Suo, and G. Zheng, “Resolution doubling with a reduced number of image acquisitions,” Biomed. Opt. Express 6(8), 2946–2952 (2015). [CrossRef]

21. N. Chakrova, B. Rieger, and S. Stallinga, “Deconvolution methods for structured illumination microscopy,” J. Opt. Soc. Am. A 33(7), B12–B20 (2016). [CrossRef]

22. R. Cao, T. Yang, Y. Fang, C. Kuang, and X. Liu, “Pattern-illuminated Fourier ptychography microscopy with a pattern-estimation algorithm,” Appl. Opt. 56(24), 6930–6935 (2017). [CrossRef]

23. Q. Liu, Y. Chen, W. Liu, Y. Han, R. Cao, Z. Zhang, C. Kuang, and X. Liu, “Total internal reflection fluorescence pattern-illuminated Fourier ptychographic microscopy,” Opt. Lasers. Eng. 123, 45–52 (2019). [CrossRef]

24. V. Perez, B.-J. Chang, and E. H. K. Stelzer, “Optimal 2D-SIM reconstruction by two filtering steps with Richardson-Lucy deconvolution,” Sci. Rep. 6(1), 37149 (2016). [CrossRef]

25. F. Ströhl and C. F. Kaminski, “Speed limits of structured illumination microscopy,” Opt. Lett. 42(13), 2511–2514 (2017). [CrossRef]

26. F. Ströhl and C. F. Kaminski, “A joint Richardson—Lucy deconvolution algorithm for the reconstruction of multifocal structured illumination microscopy data,” Methods Appl. Fluoresc. 3(1), 014002 (2015). [CrossRef]

27. Y. Zhang, S. Lang, H. Wang, J. Liao, and Y. Gong, “Super-resolution algorithm based on Richardson–Lucy deconvolution for three-dimensional structured illumination microscopy,” J. Opt. Soc. Am. A 36(2), 173–178 (2019). [CrossRef]

28. M. Ingaramo, A. G. York, E. Hoogendoorn, M. Postma, H. Shroff, and G. H. Patterson, “Richardson–Lucy Deconvolution as a General Tool for Combining Images with Complementary Strengths,” ChemPhysChem 15(4), 794–800 (2014). [CrossRef]

29. K. Chu, P. J. McMillan, Z. J. Smith, J. Yin, J. Atkins, P. Goodwin, S. Wachsmann-Hogiu, and S. Lane, “Image reconstruction for structured-illumination microscopy with low signal level,” Opt. Express 22(7), 8687–8702 (2014). [CrossRef]

30. X. Huang, J. Fan, L. Li, H. Liu, R. Wu, Y. Wu, L. Wei, H. Mao, A. Lal, P. Xi, L. Tang, Y. Zhang, Y. Liu, S. Tan, and L. Chen, “Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy,” Nat. Biotechnol. 36(5), 451–459 (2018). [CrossRef]

31. J. Boulanger, N. Pustelnik, L. Condat, L. Sengmanivong, and T. Piolot, “Nonsmooth convex optimization for structured illumination microscopy image reconstruction,” Inverse Probl. 34(9), 095004 (2018). [CrossRef]

32. A. G. York, S. H. Parekh, D. D. Nogare, R. S. Fischer, K. Temprine, M. Mione, A. B. Chitnis, C. A. Combs, and H. Shroff, “Resolution doubling in live, multicellular organisms via multifocal structured illumination microscopy,” Nat. Methods 9(7), 749–754 (2012). [CrossRef]

33. K. Wicker, “Non-iterative determination of pattern phase in structured illumination microscopy using auto-correlations in Fourier space,” Opt. Express 21(21), 24692–24701 (2013). [CrossRef]

34. S. Lefkimmiatis, A. Bourquard, and M. Unser, “Hessian-Based Norm Regularization for Image Restoration With Biomedical Applications,” IEEE Trans. on Image Process. 21(3), 983–995 (2012). [CrossRef]

35. S. Lefkimmiatis and M. Unser, “Poisson Image Reconstruction With Hessian Schatten-Norm Regularization,” IEEE Trans. on Image Process. 22(11), 4314–4327 (2013). [CrossRef]

36. Y. Wang, J. Yang, W. Yin, and Y. Zhang, “A New Alternating Minimization Algorithm for Total Variation Image Reconstruction,” SIAM J. Imaging Sci. 1(3), 248–272 (2008). [CrossRef]

37. J. Duan, Z. Pan, X. Yin, W. Wei, and G. Wang, “Some fast projection methods based on Chan-Vese model for image segmentation,” J. Image Video Proc. 2014(1), 7 (2014). [CrossRef]

38. T. Goldstein and S. Osher, “The Split Bregman Method for L1-Regularized Problems,” SIAM J. Imaging Sci. 2(2), 323–343 (2009). [CrossRef]

39. G. Ball, J. Demmerle, R. Kaufmann, I. Davis, I. M. Dobbie, and L. Schermelleh, “SIMcheck: a Toolbox for Successful Super-resolution Structured Illumination Microscopy,” Sci. Rep. 5(1), 15915 (2015). [CrossRef]

40. A. Stemmer, M. Beck, and R. Fiolka, “Widefield fluorescence microscopy with extended resolution,” Histochem. Cell Biol. 130(5), 807–817 (2008). [CrossRef]

41. E. Stelzer, “Contrast, resolution, pixelation, dynamic range and signal-to noise ratio: fundamental limits to resolution in fluorescence light microscopy,” J. Microsc. 189(1), 15–24 (1998). [CrossRef]

Second-order optimized regularized structured illumination microscopy (sorSIM) for high-quality and rapid super resolution image reconstruction with low signal level

Abstract

1. Introduction

2. Principle

3. Simulations

4. Experiments

5. Conclusion

Funding

Disclosures

References

Cited By

Figures (6)

Equations (22)

Optics Express