1. Introduction

In the family of super-resolution fluorescence microscopy imaging, there are three main categories including structured illumination microscopy (SIM) [1,2], stimulated emission depletion (STED) [3], and single molecule localization microscopy (SMLM) [4–6]. Compared to the other two methods, SIM is time-saving, requires low light intensity, and does not impose photophysics, so it is popular in the biological research fields especially live-cell imaging.

Simple to say, SIM achieves super-resolution by shifting high-frequency information of the sample into the frequency support of microscope’s optical transfer function (OTF), and a super-resolution image can be reconstructed by extracting the sample’s high-frequency components from the mixed frequency domain accordingly. As for traditional linear two-dimensional SIM, nine sinusoidal fringes consisting of 3 phases × 3 orientations are essential to obtain nine raw SIM images for the reconstruction of a super-resolution image. The fringes enable the shifting of uncollectable high-frequency information to microscope’s OTF, and their contrasts are directly linked to the amount of retrievable high-frequency information so that a higher fringe contrast will result in a higher resolution improvement. Commonly, high-contrast fringes can be produced by interference occurring between the ±1st diffraction lights from a physical diffraction grating [1,2,7] or a liquid crystal on silicon spatial light modulator (LCoS SLM) [8–10], and a high-coherence laser source is required to introduce destructive interference in fringe patterns for maximum contrast. However, both of the two interferometric SIM implementations have limitations in practice, the former is limited in speed and accuracy, and the latter is polarization-dependent and expensive. For these reasons, digital micromirror device (DMD), an amplitude-dependent SLM with lower cost and higher switching speed, becomes a promising alternative in many interferometric SIM implementations including single-color [11,12], multi-color [13,14] imaging applications, and a combination with localization technique [15].

The DMD is a micro-electro-mechanical system with a micromirror array, it works by flipping the orientation of each micromirror between +12 degrees (referred to as “on” state) and -12 degrees (referred to as “off” state) individually, so it allows for continuous display of various patterns. This unique characteristic contributes to the generation of a fringe-projection DMD-SIM [16], where the DMD is no longer used as a diffractive optical element but to image pre-generated sinusoidal fringe patterns into the sample plane directly. An incoherent LED light is used for illumination in this projection DMD-SIM, the LED is speckle-noise-free but low-brightness which will introduce a compromise in signal-to-noise ratio (SNR). Therefore, a high-brightness laser source is preferred, despite its high-coherence characteristic will lead to inhomogeneous (Gaussian) illumination and speckle. For the projection scheme, the two phenomenons are harmful and need to be suppressed, because inhomogeneous illumination cannot match the projected patterns perfectly and speckle may cause reconstruction artifacts that to be misinterpreted as biologically relevant features. Substantially, speckle can be quantified as a so-called speckle contrast ratio (SCR), which is the ratio of the standard deviation of the sensor’s pixel intensity to mean intensity [17]. Accordingly, speckle noise can be eliminated by reducing the standard deviation of the imaging camera’s pixel intensity. An effective method is to generate sequences of uncorrelated speckle patterns of equal mean intensity over the integration time of the camera such that they are averaged. To date, many speckle-suppression methods have been exploited in the field of projection display, such as rotating light pipe, diffuser and microlens array, or vibrating Hadamard binary phase matrix, multimode fiber, and microlens array beam shaper [18], but the achievement of both speckle suppression and energy preservation is still hard, and the limited vibration frequency of these methods often set a limitation to the image acquisition speed. In the field of coherent SLM-based SIMs which apply LCoS or DMD, there are also speckle-reduction devices that commonly use multimode fiber in conjunction with shaker [9,13], so that to produce a simulated spatially incoherent light source at the fiber output. Compared with coherent SIM implementations, projected SIM implementations have more stringent requirements for speckle reduction.

However, although despeckled and homogeneous illumination is obtained, the projection scheme will bring about low-contrast projected patterns owing to the low-passing filtering of the microscope, and a higher resolution comes at the expense of lower SNR which results in an inferior reconstructed result finally. This trade-off cannot be remedied completely by conservative system hardware settings alone unless combined with an effective reconstruction algorithm. Towards the situation of low pattern contrast and raw SIM images with low SNR, linear Wiener filter deconvolution is challenged because it cannot compensate for low contrast in the input data [19], and its quantitative nature has a low tolerance for low SNR [20]. However, an iterative algorithm like Richardson-Lucy (RL) deconvolution often performs better in reconstructing low-contrast data, and it takes into account statistical fluctuations in the signal substantially, therefore it also can reconstruct low-SNR data [21,22].

Here, we propose a novel fringe-projection DMD-SIM system that uses a high-brightness laser for illumination and a high-frequency phase-randomization deformable mirror (PRDM) for speckle removal with high energy utilization meanwhile. The aluminum-coated PRDM can be considered as a plane reflective mirror when it is in an inactive state. Once in an active state, its thin reflective flexible membrane is vibrated like a wave by an underside piezo actuator which is driven at a frequency up to 500 kHz, so that reflected lights at random angles are generated to be summed over the exposure time of the imaging camera to reduce SCR. The speckle-free laser reflected from the PRDM is still Gaussian whose intensity should be homogenized further, here we choose to couple it into a square-core multimode fiber (SCMF) which is so long (2 m) that multiple reflections inside it can be ensured for better homogenization. Regarding the problem of low SNR, at the end of an inverse-matrix-based SIM reconstruction algorithm presented in [23], we add a parallel iterative RL (PIRL) deconvolution algorithm which iterates the reconstructed image and the point spread function (PSF) that contains the effects of image manipulations parallelly. Moreover, given previously reported erroneous resolution measurement because of the scattering characteristic of gold beads in projection LED-DMD-SIM [24], all our experiments are conducted on fluorescent samples, including nanoparticles, Argo-HM calibration slide, and biological structures.

2. Optical setup

The homemade projection laser-DMD-SIM system is mounted on an inverted fluorescence microscope (Nikon, Ellipse Ti-E), as demonstrated in Fig. 1(a). In the illumination path, the beam emitted from a laser source (Oxxius, LBX-488-100-CSB-OE) is first converged by a lens (L1) to illuminate the 5-mm-diameter circle working area of the PRDM (Dyoptyka, µDM) for maximum energy efficiency. The mirror surface of the active-state PRDM is fluctuating but smooth (Fig. 1(b)) so that scattering losses can be avoided. Then, after passing through a pair of achromatic doublet lenses (L2 and L3), the laser beams are coupled into the homogenization element SCMF to reflect multiple times inside (Fig. 1(c)). The SCMF (CeramOptec, Optran NCC) with a core diameter of 600 µm and a numerical aperture of 0.22. Subsequently, the speckle-free and homogeneous laser beam from the end of the SCMF, which is a simulated spatially incoherent light source, is expanded by an objective lens (OL1, Nikon, 10×/0.45) and a lens (L4) to the size that matches the aspect ratio of a DMD (Texas Instruments, DLP6500FYE). The DMD chip with a micromirror array size of 1920 × 1080 pixels, a pitch of 7.6 µm, and a fill factor of 92%, respectively. The refresh frequency of our DMD can be up to about 9.5 kHz, which is well above the deform frequency of the PRDM, i.e., 500 kHz so that the high-frequency advantage of the DMD will not be affected. Two mirrors (M1 and M2) are placed before the DMD to direct the incident beam to it at twice the tilt angle of +12 degrees (i.e., 24 degrees), as a result, laser beams reflected from the “on” state micromirrors of the DMD can be perpendicularly directed into the input port of the microscope, and stray beams reflected from the “off” state micromirrors can be shielded from the optical path at four times the tilt angle of -12 degrees (i.e., -48 degrees), as illustrated in Fig. 1(d)). The DMD, which is located at the front focal plane of a tube lens (TL1, Thorlabs, TTL200-A), allows being tilted at a small angle for the adjustment of the light reflected from a dichroic mirror (DM, Chroma, ZT488rdc-UF1) so that the central diffraction order can live in the center of the back aperture of a high-NA imaging objective lens (OL2, Nikon, 100×/1.49 TIRF) to obtain maximum light efficiency and fringe contrast. The DMD chip plane is also conjugated to the focal plane of the OL2, and the TL1 can be slid slightly to change the divergence of the illumination beam so that the fringe patterns can be projected on the sample precisely. In the detection path, the fluorescence emitted from the sample is separated from the excitation light by the DM and then focused by a tube lens (TL2) onto a scientific complementary metal-oxide-semiconductor (sCMOS) camera (Tucsen, Dhyana 400BSI V2.0) after reflecting from a mirror (M3). Thereinto, the TL2 and M3 are two components integrated into the inverted fluorescence microscope.

 

Fig. 1. The diagram of the projection laser-DMD-SIM system. (a) Experimental setup. L1-L4, lens; M1-M3, mirror; PRDM, phase randomization deformable mirror; SCMF, square core multimode fiber; OL1-OL2, objective lens; TL1-TL2, tube lens; DM, dichroic mirror. (b), (c) and (d) are working principles of PRDM, SCMF, and DMD, respectively.

Download Full Size | PDF

During the experiment, nine 1-bit binary gratings consisting of 3 phases × 3 orientations are stored on-board the DMD and selected in sequence by sending transistor-transistor logic (TTL) trigger pulses to the DMD control board. To prevent crosstalks between two adjacent frames captured by the imaging camera, synchronization of the DMD and the camera should be controlled, which is achieved by custom software programmed in C++ in our system. When the rising edge of a TTL trigger pulse from the DMD arrives, the camera starts exposure, after a preset exposure time, it stops to collect the current frame and waits for the next pulse signal.

3. Principles

The binary grating patterns loaded on the DMD in our experiment are shown in Figs. 2(a)-(c). They are rectangular wave functions, and sinusoidal functions can be generated after Fourier decomposition and a low-pass filter. The rectangular wave function is as follows,

 

Fig. 2. Fringe patterns and speckle their intensities. (a) Nine binary grating patterns loaded onto the DMD, and the 0° (b) and nominally 60° (c) orientations are shown with on and off pixels in white and black respectively. In (b), the 0° pattern is four-pixel-linespacing. In (c), the periodicity is indicated by coloring one pixel in each unit cell red which repeats along the two yellow-colored vectors: v_a= (8,0) and v_b= (5,3), so it has an actual orientation angle of arctan(5/3) ≈ 59.04° and a linespacing of 8 × cos(59.04°) ≈ 4.12 pixels, very close to the target values of 60° and 4 pixels. (d) and (e) are the six-pixel and four-pixel illumination patterns captured by the imaging camera when the deformable mirror is in the active state, respectively, (f) is the four-pixel illumination patterns when the deformable mirror is in the inactive state. The line-scans drawn at the same position of (d) and (e) are plotted in (g), where the modulation contrast is 0.55 and 0.36, respectively. (h) and (i) are profiles of the SCMF output without and with speckle reduction for the 488-nm laser source, respectively, and the speckle contrast is indicated at the bottom of each image. (j) The lateral intensity profiles of (h) (black curve) and (i) (red curve) along the dashed line of (h).

Download Full Size | PDF

We obtain experimental illumination patterns when the PRDM is in the active state (Figs. 2(d) and (e)) or the active state (Fig. 2(f)), and the intensity difference shown in Fig. 2(g) between the six-pixel pattern (Fig. 2(d)) and the four-pixel pattern (Fig. 2(e)) demonstrates the reduction of pattern contrast (calculated by taking the ratio of the difference between the maximum and minimum of the pattern intensity and their sum) when finer patterns are used. And we can see that the illumination pattern is almost sinusoidal when the PRDM is in the active state but is distorted by speckle if the PRDM is in the inactive state.

We can also see speckle is a high-contrast granular pattern in appearance, whose formation can be regarded as a random process. To be specific, in an imaging system, a non-smooth surface is equivalent to many randomly distributed scatterers. When the laser passes through it, scattering components with random phases are introduced to the laser and then superimposed coherently on the imaging plane, and bright spots and dark spots appear at the position of constructive and destructive interferences respectively, therefore the image captured by the imaging camera is corrupted by speckle noise. Static speckle is difficult to eliminate but a dynamic (time-varying) one can be eliminated effectively for its statistical characteristic. The intensity distribution function of an uncorrelated time-varying speckle is a temporal uncorrelated function, whose autocorrelation function is the Dirac function. Uncorrelated speckle intensity distribution functions of multiple time points are independent of each other at any different time point, and a sum of them tends to be a uniformly distributed function as the number of time points approaches positive infinity, which can be written as

Besides, another problem that needs to be solved in our projection laser-DMD-SIM system is the trade-off between spatial resolution and SNR, that is a smaller fringe period leads to higher spatial resolution but lower fringe contrast and thus lower SNR. In our experiments, a fringe period as low as four DMD pixels is adopted and the corresponding contrast is relatively low (0.36) as shown in Figs. 2(e) and (g). Under the circumstances, an inferior super-resolution image will be obtained if traditional Wiener filter deconvolution is applied. Instead, we apply a robust PIRL deconvolution algorithm which is modified from a blind RL deconvolution algorithm presented in [26] in this article for SIM reconstruction. The PIRL deconvolution requires two RL iterations as follows,

 

Fig. 3. Overview of the working flow of the PIRL deconvolution algorithm. The initial image f₀ is derived from the frequency of the sample after order combination, the initial PSF p₀ is the inverse Fourier transform of the extended frequency support of the OTF, i.e., OTF_SIM. At each iteration, the body of the algorithm needs two RL deconvolution steps to prepare the image and PSF for the next iteration until acceptable results are obtained.

Download Full Size | PDF

4. Experimental results

To demonstrate the imaging performance of our system, we conducted a series of experiments that were all performed with the excitation light of 488 nm and the corresponding 520 nm fluorescence. As SIM is a widefield imaging method whose excitation intensity arriving at the sample is relatively low, we kept the laser density at the sample plane at ∼1–10 W/cm² to guarantee sufficient brightness and little photobleaching, and the exposure time of the camera is 50 ms.

To measure the spatial resolution of our system, we used 100-nm fluorescent nanoparticles deposited on 170-mm-thickness coverslips as test samples, and the experimental results are shown in Fig. 4. Quantificationally, the theoretical resolution limit is ∼213 nm calculated for the 100×/1.49NA objective lens at the wavelength of 520 nm according to Rayleigh criterion, and the expected resolution enhancement at the 4 DMD pixels fringe spacing is ∼1.7. The resultant full width at half maximum (FWHM) of the measured nanoparticle in the widefield image (Fig. 4(a)) is 219 nm, which is much better than that of the SIM image (Fig. 4(a)), i.e., 100 nm, as shown in Fig. 4(c). Besides, two particles separated by a distance of 162 nm in the SIM image are unresolved in the widefield image, as shown in Fig. 4(d).

 

Fig. 4. Experimental results of 100-nm-diameter fluorescent nanoparticles. (a) Widefield image. (b) SIM image. The bottom-right insets are zoom regions of the white outlined regions. (c) Intensity profiles of measurement of 15 nanoparticles in (a) (red) and (b) (blue) respectively. The line-scans of two adjacent nanoparticles in (a) (red) and (b) (blue) are plotted in (d). Scalebars, 1.5 µm in (a) and (b) and 0.5 µm in insets.

Download Full Size | PDF

To demonstrate the super-resolution and speckle-suppression capabilities of our system, we imaged a ground-truth fluorescent calibration slide (Argo-HM V2, Argolight) and microtubules in a fixed COS-7 cell, and the experimental results are shown in Fig. 5. The calibration slide has gradually spaced lines whose spacing increases from 100 nm to 700 nm in steps of 50 nm. Figures 5(a)-(c) shows its widefield, despeckled SIM, and speckled SIM images, respectively. We can see that lines are clearer and thinner in Figs. 5(b) and (c) compared to the ones in Fig. 5(a). Lines are more continuous in the despeckled SIM image compared to ones in the SIM image without speckle suppression, and this difference is more visualized in their intensity profiles, which is shown in Figs. 5(d) and (e) respectively. It is obvious that the 100-nm and 150-nm-spacing lines are both resolvable and without false bifurcations in Fig. 5(d), but the situation in Fig. 5(e) is different, where lines should be single but become bifurcate and there is only one peak at the position of the 150-nm-spacing line. As a further demonstration, we imaged biological samples. In the fixed COS-7 cell, two adjacent microtubules filaments which cannot be resolved in the widefield image (Fig. 5(f)) are resolvable in despeckled and speckled SIM images (Figs. 5(g) and (h)), but the filaments in Fig. 5(g) are more continuous than Fig. 5(h). As for their respective intensity profiles, the difference is shown in Fig. 5(i). There are two peaks in SIM results but one in widefield result, and one of the peaks is degraded by speckle in the SIM intensity profile without speckle suppression which confirms that speckle indeed reduces the quality of the SIM image and may introduce some deviations to mistake real biological structures. However, our PRDM can solve this problem effectively, as shown in Fig. 5(i), and the obtained FWHM of isolated microtubules is 138 nm which is smaller than the theoretical resolution limit of ∼213 nm.

 

Fig. 5. Experimental verification of the super-resolution and the speckle-suppression capabilities with (a)-(e) gradually spaced lines of fluorescent calibration slide and (f)-(i) microtubules in a fixed COS-7 cell. (a)-(c) Widefield, Despeckled SIM, and speckled SIM images of the gradually spaced lines, respectively. Intensity profiles along the white lines at the same position in (b) and (c) are shown in (d) and (e) respectively. (f)-(h) Widefield, speckled SIM, and despeckled SIM images of microtubules in the fixed COS-7 cell, respectively. The bottom-left insets are zoom regions of the white outlined regions. The line-scans of two adjacent microtubules filaments in (f) (red), (g) (blue), and (h) (black) are plotted in (i), and the full width at half maximum (FWHM) is indicated by the yellow arrow. Scalebars, 4 µm in (f, g,h) and 0.5 µm in insets.

Download Full Size | PDF

Furthermore, to verify that the effectiveness of our PIRL deconvolution algorithm in reconstructing low-SNR raw SIM data compared to the traditional Wiener filter deconvolution, we imaged the outer membrane of mitochondria in fixed COS-7 cell, as shown in Fig. 6. In the widefield image (Fig. 6(a)), the signal-to-background ratio (SBR) is low and some fine mitochondrial structures are unresolvable. In the SIM image reconstructed from the Wiener filter deconvolution as shown in Fig. 6(b), the SBR increases obviously but the SNR is low. However, in the SIM image reconstructed from the PIRL deconvolution as shown in Fig. 6(c), the SBR and SNR are the highest, and the structural detail is the most distinguishable as indicated by the white arrowheads in Figs. 6(d)-(f). The differences between these three results can be seen more intuitively in their respective intensity profiles, as shown in Fig. 6(g).

 

Fig. 6. Experimental results of mitochondria in a fixed COS-7 cell. (a) widefield image. (b) and (c) are SIM reconstruction images via traditional Wiener filter deconvolution and PIRL deconvolution respectively. (d)-(f) are zoom regions of the yellow outlined regions in (a), (b), and (c), respectively, and the white arrowheads indicate the structural detail differences between them. (g) Intensity profiles along the white lines in (a) (red), (b) (black) and (c) (blue), respectively. The yellow arrows indicate the intensity differences at some positions. Scalebars, 4 µm in (a, b, c) and 1 µm in (d, e, f).

Download Full Size | PDF

5. Conclusion and outlook

In conclusion, we have designed and built a sinusoidal-fringe-projection laser-DMD-SIM system. To attain resolution as high as possible, a binary grating with a period of four DMD pixels is applied, which results in low-contrast illumination patterns and final inferior super-resolution images as in the existing projection LED-DMD-SIM. To address this problem, we apply a robust PIRL deconvolution for SIM reconstruction in the algorithm and use a laser source for illumination in hardware to attain higher SNR. The speckle-removal and illumination-homogenization of the laser are achieved by a combination of a PRDM and SCMF which possess more excellent effect, higher frequency, and fewer energy losses compared to the existing methods. Although our resolution improvement is moderate compared to the coherent DMD-SIM, the projection DMD-SIM scheme still has some unique advantages. For example, the phase-shifting and orientation-change are determinate for fringe patterns, besides, various illumination modalities like lattice, multi-spot, and random patterns can be obtained easily. In conjunction with specified reconstruction strategies, the scheme can be a practical tool in the field of SIM imaging. Furthermore, our projection laser-DMD-SIM is easy to be extended to a versatile system that is compatible with projection, optical sectioning, and interferometric functions, which is exactly what we plan to do next.