Open Access
Add to BibSonomy
Abstract
We investigated the capabilities of deconvolution for image enhancement in scatter-plate microscopy. This lensless imaging technique enables the investigation of microstructures through scattering media by cross-correlating the scattered light intensity with a previously recorded point spread function (PSF) of the scattering medium. The autocorrelation function of the PSF appears as the transfer function of the imaging process. Deconvolution methods use the knowledge of this transfer function to enhance the image quality by reducing the blur and strengthening the contrast with the objective to achieve diffraction-limited resolution. We obtained significant image enhancement both with means of inverse filtering and by applying iterative deconvolution algorithms.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Imaging through scattering media is a challenging task and has a wide range of applications e.g. in biological and medical imaging [1], in imaging through fog and smoke [2], in astronomical imaging and in imaging through turbid media [3,4]. In recent times, many imaging techniques exploiting the memory effect of the scattering media to image through the visually opaque materials have been presented [5]. The optical memory effect describes the shift-invariance of speckle patterns when an illuminating source is slightly tilted (e.g. in the case of plane-wave illumination) or shifted (e.g. in the case of point source illumination) [6]. Some of the mentioned techniques retrieve images by applying Fienup-type algorithms [7] on the random patterns generated by illuminated objects hidden behind scattering media [8,9]. Other techniques investigate the scattering behaviour by recording patterns generated with known objects [10]. A special case of a known object is represented by a point source. In former publications, we demonstrated digital holographic imaging [11, 12] and incoherent 3D imaging [13] through scattering media using a point source as reference. Furthermore, we demonstrated the possibilities of scatter-plate microscopy [14], a technique that regards the scattering medium not as image disturbing but as the actual imaging element. The scatter-plate microscopy is a diffraction-limited imaging method for microscopic investigation that enables imaging with variable magnification, variable numerical aperture (NA) and variable working distance just by a single low-cost ground glass diffuser replacing the objective in conventional microscopes. Here we demonstrate how the images acquired by scatter-plate microscopy can be processed by means of deconvolution to improve the image quality significantly [15, 16]. Deconvolution uses the knowledge of the transfer function of an optical system for image enhancement: acquired images can be regarded as the object structure convolved with the transfer function. Deconvolution techniques try to solve this mathematical relationship for the object structure. The methods can correct aberrations, reduce occurring blur and they can in principle enable imaging below the diffraction limit. Imaging in scatter-plate microscopy is based on cross-correlating random intensity patterns with a previously recorded PSF. As the transfer function of the imaging process appears, depending on the perspective, either this PSF or its autocorrelation. In contrast to conventional microscopy where the acquisition of the transfer function is accompanied by additional effort, in scatter-plate microscopy the acquisition of the transfer-function is therefore anyway a part of the image acquisition. Hence, the application of deconvolution microscopy is an obvious and straightforward procedure in scatter-plate microscopy. Here we present great image enhancement with diffraction-limited resolution realized by the application of the Wiener filter, the iterative Gold algorithm and by the iterative Janssson-Van Cittert algorithm.
2. Principles
The image formation of an optical system is characterized by the point spread function S. In the case of incoherent imaging, the PSF is the intensity distribution in the image plane generated by a point source emitting in the object plane. For imaging systems with extensively shift invariant PSF the image formation process can be expressed as convolution:
where Iob is the intensity distribution recorded in the image plane, O is the intensity distribution emitted from the object plane and S denotes the PSF. z and ẑ describe the axial position of the image plane respectively of the object plane (in the case of a scatter-plate microscope reasonably measured as the distance between the planes and the imaging scatter-plate). r describes the lateral position in the recorded intensity distribution whereas r̂ means the lateral position in the object plane (see Fig. 1).
Fig. 1 Experimental setup for the sample pattern acquisition. To record the PSF the illumination optics, the rotating ground glass diffuser and the sample have to be removed and the laser beam has to be focused in the object plane by a microscope objective.
For scattering media, Eq. (1) can still be applied although the shift-invariance of the PSF is restricted to a small area in which the so-called memory effect [6] holds true. Nevertheless, if the field of view (FOV) is part of this memory effect area, Eq. (1) can describe the image formation and the recorded intensity distribution in the image plane is a convolution of the object’s intensity distribution and the PSF. For a scatter-plate, the intensity distribution Iob turn out to be a random intensity pattern (in the following called sample pattern SP) and the PSF to be a speckle pattern. The object structure can be retrieved by cross-correlating (★) the SP with the PSF:
C(r, z) = S(r) ★ S(r) is the autocorrelation of a speckle pattern and appears as transfer function of the imaging process. Imaging is realized since C(r, z) = S(r) ★ S(r) appears as a sharply peaked function and except for a constant offset ∊ very similar to the PSF of conventional imaging systems (airy disc). The magnification is given by and since the positions of object and image plane can be chosen in a wide sense arbitrarily, the scatter-plate microscope realizes variable magnification with a single simple imaging element. Ground glass diffusers prove to be very suitable as scatter-plates. Since they can be fabricated very extended with both unvarying surface roughness parameters and unvarying surface correlation they can be used for imaging with high numerical aperture.In order to reach both high resolution and high magnification it is crucial to record the patterns with a sensor of high resolution and with large sensor format. To avoid unsuitable computational efforts, convolution theorem is used to calculate the cross-correlation:
with ^ denoting the two-dimensional Fourier transform, ℱ−1 the operator of the inverse Fourier transform, p the spatial frequency vector and * the complex conjugate. In digital image processing, the Fourier transform and its inverse is efficiently realized by the FFT-algorithm.3. Experimental setup and image acquisition
The experimental setup is shown in Fig. 1. We used a frequency-doubled Nd:YAG-laser with a wavelength of 532 nm as light source. To realize spatially incoherent sample illumination, the laser beam was let through a rotating ground glass diffuser. To achieve optimized results, a condenser consisting of a collimating and a focusing lens (focal length 50 mm and 30 mm) was installed between the rotating diffuser and the sample. To approximate a point source, we focused the laser beam by a microscope objective with 40X magnification into the object plane. The patterns were recorded with a CCD-Camera from VISTEK-GMBH with 3280 × 4896 pixels and a pixel size of 7.45 μm × 7.45 μm. The FOV was restricted by a circular pinhole with a diameter of 400 μm. High resolution over the whole pinhole area proves that in this way the FOV is smaller than the memory effect range. A diaphragm between the scatter-plate and the object determines the NA. Our scatter-plate was a THORLABS DG-600 ground glass diffuser.
The recorded patterns carry a so-called scattered light envelope. This low frequency intensity variation affects the calculated cross- and autocorrelations and disturb the imaging. A common way to remove the envelope is to perform a shading correction by dividing the recorded patterns by low-pass filtered versions of themselves. However, recently it was shown that image enhancement can be achieved by applying a Zernike polynomial fitting on the patterns and subtract these fits from the recorded patterns [17]. A fit with twenty Zernike Polynomials prove to be sufficient for our purpose.
The images obtained by calculating the cross-correlation were normalized by subtracting the minimum pixel value followed by a division through the maximum pixel value. It might happen that also after the shading correction due to dust on the CCD or due to defect pixels a bright spot-like feature remains in the centre of the image. However, this spot does not really disturb and just has not to be taken into account for the determination of the maximum in the normalization procedure.
We demonstrate the possibilities of scatter-plate microscopy by imaging a 1951 USAF high-resolution test target and onion cells. The smallest element of the test target (group 9 element 3) indicates a resolution of 645.1 lp/mm or 0.78 μm. The ground glass diffuser was mounted on a step rotator stage from PI GmbH & Co.KG enabling imaging for different orientations of the diffuser. Averaging over images recorded with different diffuser orientations improved the image quality significantly. Figure 2 shows images obtained with a single diffuser orientation and the images we obtained by averaging over five (USAF-target) respectively eight (onion cells) images obtained with different diffuser orientations.
Fig. 2 Images acquired with the scatter-plate microscope with 20X magnification, NA = 0.3 and without deconvolution: (a) and (b) show the USAF test target, (c) and (d) onion cells. (a) and (c) were acquired with a single pattern recorded for a single diffuser orientation. (b) and (d) are the result of averaging over five (USAF) respectively eight (onion) images acquired with different diffuser orientations. The field of view was restricted by a pinhole having a diameter of 400 μm.
4. Deconvolution in scatter-plate microscopy
Deconvolution uses the knowledge of the transfer function of shift-invariant imaging systems (Eq. (1)) to improve resolution and contrast and to reduce the blur caused by aberrations of the imaging system.
In scatter-plate microscopy, two deconvolution approaches are possible. Either the deconvolution is applied on the sample pattern Iob and the PSF is regarded as the transfer function or the deconvolution is applied on the reconstructed image according to equation Eq. (2). In the latter case the autocorrelation of the PSF C takes the role of the transfer function [15, 16]. When deconvolution was applied on the retrieved image and not on the pattern, best results were achieved by deconvolving an averaged image with an averaged PSF autocorrelation (averaging over different diffuser orientations, see Sec. 3). The averaging procedure increases the signal to noise ratio (SNR) both of the image and of the autocorrelation. Here we present three different deconvolution methods (Wiener filter, Gold algorithm, Jansson Van-Cittert algorithm) which proved to improve the quality of the images acquired with our scatter-plate microscope significantly.
4.1. Wiener filter with constant regularization
The Wiener filter [18] is best described in Fourier space and can be applied on both the sample pattern (PSF is the transfer function) and the reconstructed image (C is the transfer function):
with N(p) being the mean power spectral density of the noise. Since N(p) is unknown, some suitable regularization term has to be selected. Here we will present results we obtained with a constant regularization N(p) = α. The specific choice of an optimal α depends on the noise being present in the image. The application of the Wiener filter estimates the object in the sense of a mean square optimization.4.2. Gold algorithm
Both the Gold algorithm [18,19] and the Jansson Van-Cittert algorithm [20,21] described in the next paragraph develop a deconvolved solution in an iterative process. This solution provides the acquired image when it is convolved by the known filter function. Since the success of the iterative process strongly depends on a good initial guess (meaning that initial guess and the true object should look similar) the application of iterative deconvolution on the sample pattern does not lead to satisfying results. The Gold algorithm follows the rule
We applied the Gold algorithm to the normalized averaged image I(r) with an averaged transfer function C(r). As an initial estimate, we set . In order to make comparable with I(r) we normalized the convolution term in each iteration step. To avoid numerical problems caused by division through values close to zero an offset δ must be added both in the numerator and in the denominator. The algorithm always ensures a non-negativity condition () for transparent objects (i.e. and C(r) need to be positive). However, the transfer function C(r) has also some negative values since we subtracted the mean intensity from the speckle pattern before calculating the autocorrelation. Applying the algorithm in the form of Eq. (5), a disturbing influence of these negative values could not be noted. A disadvantage of iterative deconvolution algorithms is that they will give poor results if there is too much noise in the initial estimate . It might happen that in the iterative process the noise is enhanced leading to disturbed images. We overcome this problem by filtering the estimate with a 2-D Gaussian smoothing kernel with standard deviation σ repeatedly after a certain number of iteration steps. The specific standard deviation of the smoothing kernel is a compromise between the reduction of disturbing noise and the preservation of a good resolution.4.3. Jansson Van-Cittert algorithm
The Jansson Van-Cittert algorithm [20,21] is like the Gold algorithm an iterative deconvolution method. The algorithm was also applied to a normalized image and again was chosen as an initial guess. The iteration followed the rule
withAs explained in the former paragraph, the normalization of the convolution term is necessary to enable a comparison with the normalized image I(r). As in the case of the Gold algorithm, noise enhancement was an image-disturbing factor. A repeatedly applied 2-D Gaussian filter could reduce the problem once more.
5. Results
We realized imaging of a USAF test target and of a piece of onionskin positioned on a microscope slide, both with 20X magnification. The object plane was at a distance of 5 cm, the image plane at a distance of 100 cm from the scatter-plate. A diaphragm with a diameter of 30 mm close to the scatter-plate set an NA of 0.3. Hence diffraction limited imaging would have a resolution of 0.89 μm.
5.1. USAF-target
Images were acquired for five different orientations of the scatter-plate. The averaged and normalized image is shown in Fig. 2(b), a detailed view in Fig. 3(a). According to the descriptions in 4.1, we applied the Wiener deconvolution both on the patterns and on the retrieved image. While deconvolution of the image was done on the averaged image with an averaged transfer function, the deconvolution of the patterns had to be done on each single pattern. At the end, the deconvolved patterns were averaged. For applying the Wiener method on the patterns (Fig. 3(b)) the noise parameter was set to α = 7 × 104, for the application on the image (Fig. 3(c)) the parameter was set to α = 5.1 × 107.
Fig. 3 Image enhancement by deconvolution: (a) detailed view of the averaged USAF-target shown in Fig. 2(b), deconvolution was not applied, (b) Wiener filter with α = 7 × 104 applied to the patterns, (c) Wiener filter with α = 5.1 × 107 applied on the averaged image, (d) Gold algorithm applied with 210 iterations, Gaussian smoothing with σ = 0.5 pixel in every 15th iteration, (e) Jansson van-Cittert algorithm applied with 300 iterations, Gaussian smoothing with σ = 0.5 pixels in every 25th iteration. (f)–(h) intensity profiles along the white lines in Fig. 3(a).
The Gold algorithm was applied on the averaged image with 210 iterations and a Gaussian filtering with σ = 0.5 pixels in every 15th iteration step (Fig. 3(d)). The Jansson Van-Cittert was also applied on the averaged image with 300 iterations and a Gaussian filtering with the same standard deviation in every 25th iteration step (Fig. 3(e)). For both iterative algorithm, an averaged transfer function was used. The intensity distribution plots in Figs. 3(f)–3(h) show clearly the image enhancing effect of deconvolution on both contrast and resolution. Plotted are intensity distributions along the white paths in Fig. 3(a). Without deconvolution, the USAF target elements are resolved down to group 8 element 4 indicating a resolution of 1.38 μm. On the other hand, the applied deconvolution approaches enabled to resolve group 9 element 2 of the target. This indicates a resolution better than 0.87 μm, which is approximately the diffraction limit of optical systems with NA=0.3 (0.89 μm). In the Figs. 3(c) and 3(e) (Wiener filter applied on the image and Jansson van-Cittert algorithm) even parts of group 9 element 3 (indicating a resolution better than 0.78 μm) are resolved. We suppose that the reason why this element is not completely resolved in these cases is an undersampling due to a pixel pitch too large for the reached magnification.
To estimate the performance of the iterative deconvolution algorithms the convergence of
can be regarded, with Ω the region of interest (ROI). The summation is performed over the pixel values within the ROI. In Fig. 4 the performances of the Gold and the Jansson van-Cittert algorithm are compared.Fig. 4 Performance of the iterative deconvolution algorithms applied on the image shown in Fig. 2(b). Δ was calculated according to Eq. 8, Ω was a region on the USAF-target cantaining the groups 8 and 9.
5.2. Onion cells
The microscope slide with a piece of onionskin was positioned in the object plane. The normalized average over eight images acquired with 20X magnification and an NA of 0.3 is shown in Fig. 5(a). We applied a Wiener filter on the image using an averaged transfer function and a regularization parameter α = 1015 (Fig. 5(b)). For deconvolving the patterns (Fig. 5(c)) the regularization parameter was set to α = 3 × 107. The Gold algorithm (Fig. 5(d)) was applied with 210 iterations and again a Gaussian smoothing with σ = 0.5 pixels in every 15th iteration step. The Jansson van-Cittert algorithm (Fig. 5(e)) was applied without any smoothing filter. Instead, the iteration was ended already after 11 steps just before the noise amplification became significant. Figure 6 shows that a continued iteration with repeatedly applied Gaussian smoothing does not lead to a better result. All four methods emphasize the details of the cell structures. Nevertheless, by applying deconvolution on this biological sample the great drawback of these methods also become obvious: a remarkable noise amplification can accompany these image processing methods.
Fig. 5 Image enhancement by deconvolution: (a) detail view of the averaged onion cells in Fig. 2(d), deconvolution was not applied, (b) Wiener filter with 3 × 107 applied on the patterns, (c) Wiener filter with α = 1015 applied on the averaged image, (d) Gold algorithm applied with 210 iterations, Gaussian smoothing with σ = 0.5 pixels in every 15th iteration, (e) Jansson van-Cittert algorithm applied with 11 iterations and without Gaussian smoothing, (f) shows an image of the cells recorded with a conventional microscope objective with the same NA (NA = 0.3) and the same magnification (20X).
Fig. 6 Application of the Jansson van-Cittert algorithm: (a) algorithm applied with 100 iterations and without Gaussian smoothing, enhanced noise disturbs the image. (b) Algorithm applied with 100 iterations and a Gaussian smoothing with σ = 2 pixels in every 20th iteration. (c) Best results were achieved with 11 iterations and without applying any Gaussian smoothing.
For comparison Fig. 5(f) shows an image of the cells recorded with a conventional microscope objective with the same NA (NA = 0.3) and the same magnification (20X) and a Ximea XiQ camera with a pixel size of 5.5 μm × 5.5 μm. Compared to the result recorded with the scatter-plate microscope this conventional image is better resolved and less noisy but on the other hand, the conventional image shows less contrast. We suppose the reason for the high contrast of the images acquired with scatter-plate microscopy is that the removal of the scattered light envelope (see Sec. 3) works as a high-pass filter.
6. Conclusions
Scatter-plate microscopy proves to be a promising technique for imaging microstructures with high resolution and enhanced contrast. The low-cost lensless method realizes variable magnification and NA without the necessity to apply various objective lenses. Moreover since the recording of the PSF is anyway necessary in scatter-plate microscopy the tool for deconvolution and therefore to improve the image quality significantly in both resolution and contrast is naturally at hand.
We applied the Wiener filter, the Gold algorithm and the Jansson van-Cittert algorithm to both technical and biological samples and were able to enhance the resolution with these methods below the diffraction limit.
Funding
German Research Association (DFG) (Os 111/49-1); Sino-German Center for Research Promotion (CDZ) (GZ 1391).
References
1. J. M. Dela Cruz, I. Pastirk, M. Comstock, V. V. Lozovoy, and M. Dantus, “Use of coherent control methods through scattering biological tissue to achieve functional imaging,” Proc. Natl. Acad. Sci. U.S.A. 101, 16996–17001 (2004). [CrossRef] [PubMed]
2. J. Fade, S. Panigrahi, A. Carré, L. Frein, C. Hamel, F. Bretenaker, H. Ramachandran, and M. Alouini, “Long-range polarimetric imaging through fog,” Appl. Opt. 53, 3854–3865 (2014). [CrossRef] [PubMed]
3. S. Sudarsanam, J. Mathew, S. Panigrahi, J. Fade, M. Alouini, and H. Ramachandran, “Real-time imaging through strongly scattering media: seeing through turbid media, instantly,” Sci. Rep. 6, 25033 (2016). [CrossRef] [PubMed]
4. I. M. Vellekoop and A. P. Mosk, “Focusing coherent light through opaque strongly scattering media,” Opt. Lett. 32, 2309–2311 (2007). [CrossRef] [PubMed]
5. I. Freund, “Looking through walls and around corners,” Phys. A 168, 49–65 (1990). [CrossRef]
6. I. Freund, M. Rosenbluh, and S. Feng, “Memory effects in propagation of optical waves through disordered media,” Phys. Rev. Lett. 61, 2328–2331 (1988). [CrossRef] [PubMed]
7. J. R. Fienup, “Phase retrieval algorithms: a comparison,” Appl. Opt. 21, 2758–2769 (1982). [CrossRef] [PubMed]
8. O. Katz, P. Heidmann, M. Fink, and S. Gigan, “Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations,” Nat. Photonics 8, 784 (2014). [CrossRef]
9. J. Bertolotti, E. G. van Putten, C. Blum, A. Lagendijk, W. L. Vos, and A. P. Mosk, “Non-invasive imaging through opaque scattering layers,” Nature 491, 232 (2012). [CrossRef] [PubMed]
10. X. Xu, X. Xie, H. He, H. Zhuang, J. Zhou, A. Thendiyammal, and A. P. Mosk, “Imaging objects through scattering layers and around corners by retrieval of the scattered point spread function,” Opt. Express 25, 32829–32840 (2017). [CrossRef]
11. A. K. Singh, D. N. Naik, G. Pedrini, M. Takeda, and W. Osten, “Looking through a diffuser and around an opaque surface: A holographic approach,” Opt. Express 22, 7694–7701 (2014). [CrossRef] [PubMed]
12. M. Takeda, A. K. Singh, D. N. Naik, G. Pedrini, and W. Osten, “Holographic correloscopy - unconventional holographic techniques for imaging a three-dimensional object through an opaque diffuser or via a scattering wall: A review,” IEEE Trans. Ind. Inf. 12, 1631–1640 (2016). [CrossRef]
13. A. K. Singh, D. N. Naik, G. Pedrini, M. Takeda, and W. Osten, “Exploiting scattering media for exploring 3d objects,” Light. Sci. Appl. 6, e16219 (2017). [CrossRef] [PubMed]
14. A. K. Singh, G. Pedrini, M. Takeda, and W. Osten, “Scatter-plate microscope for lensless microscopy with diffraction limited resolution,” Sci. Rep. 7, 10687 (2017). [CrossRef] [PubMed]
15. S. Ludwig, A. K. Singh, G. Pedrini, and W. Osten, “Scatter-plate microscope: improved image acquisition,” Proc.SPIE 10677, 1067717 (2018).
16. S. Ludwig, B. L. Teurnier, G. Pedrini, and W. Osten, “Image processing for scatter-plate microscopy,” in Forum Bildverarbeitung 2018, T. Längle, F. Puente León, and M. Heizmann, eds. (KIT Scientific Publishing, Karlsruhe, 2018), pp. 105–116.
17. M. Liao, D. Lu, W. He, G. Pedrini, W. Osten, and X. Peng, “Improving reconstruction of speckle correlation imaging by using a modified phase retrieval algorithm with the number of nonzero-pixels constraint,” Appl. Opt. 58, 473–478 (2019). [CrossRef] [PubMed]
18. J.-B. Sibarita, “Deconvolution microscopy,” in Microscopy Techniques, vol. 95 of Advances in Biochemical Engineering J. Rietdorf, ed. (Springer, Heidelberg Berlin, 2005), chap. 6, pp. 201–243, 1st ed. [CrossRef]
19. R. Gold, An iterative unfolding method for response matrices (Argonne National Laboratory, U.S. Atomic Energy Commission, Argonne, Ill., 1964). [CrossRef]
20. P. A. Jansson, ed., Deconvolution of Images and Spectra (2Nd Ed.) (Academic Press, Inc., Orlando, FL, USA, 1996).
21. P. H. van Cittert, “Zum Einfluß der Spaltbreite auf die Intensitätsverteilung in Spektrallinien. II,” Z. Phys. 69, 298–308 (1931). [CrossRef]
