Open Access
Add to BibSonomy
Abstract
Fourier ptychographic microscopy (FPM) is used to achieve high resolution and a large field of view. However, traditional FPM image reconstruction methods often yield poor image quality when encountering out-of-focus issues during reconstruction. Therefore, this study proposes a defocus-distance regression network based on convolutional neural networks. In an experimental validation, the root-mean-square error calculated from 1000 sets of predicted and true values was approximately 6.2 µm. The experimental results suggest that the proposed method has good generalization, maintains high accuracy in predicting defocus distances even for different biological samples, and extends the imaging depth-of-field of the FPM system by a factor of more than 3.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Microscopic imaging technology helps scientists to understand cell structure and function and provides doctors with accurate images for disease diagnosis. Thus, there is an urgent need for high-resolution (HR) microscopic imaging techniques for biomedical research and medical diagnosis [1–6]. Fourier ptychographic microscopy (FPM) is an emerging imaging technique [7–13] that illuminates a sample with transmittance from an array of light-emitting diodes (LEDs) at different positions, which is captured by a camera after passing through an objective lens to produce a series of low-resolution (LR) images. These LR images are then stitched together at the corresponding positions in the frequency domain to achieve super-resolution imaging. Compared with conventional microscopic imaging techniques, the FPM technique does not require sophisticated instrumentation to increase the space–bandwidth product of the optical system and acquire images with a larger field of view and higher resolution [14]. However, uniformity of the sample thickness and correct focusing of the instrument are key to high-quality imaging using FPM.
An uneven sample thickness or a nonideal focusing position may result in part of the field of view being out of focus. When the thickness of the sample is not uniform, the light passing through areas of different thicknesses undergoes varying degrees of phase delay, which can result in blurring or defocusing of portions of the image [15]. Similarly, if the focusing position of the sample is not ideal, that is, the light is focused on different positions in front of and behind the sample, a portion of the image may be out of focus. Thus, a clear image is obtained only within a very narrow focus range during image reconstruction, whereas the out-of-focus region is blurred, limiting the depth-of-field of the conventional FPM system to a very small range, which adversely affects the practical applicability of the FPM system.
To solve these problems, conventional methods usually use mechanical focusing to adjust the focus position and align the focal plane with the sample surface or region of interest. Although the mechanical focusing method of conventional FPM systems is relatively simple and easy to implement, it can be time-consuming and inaccurate when focusing over a wide area because of the need to manually adjust the position of the objective lens or sample stage. In addition, focusing accuracy can be subjective because of differences in operator skill and experience levels.
With continuous innovation in technology, new automatic correction methods and techniques have been introduced into FPM systems. The existing optimization methods are broadly divided into two aspects. The first is to directly optimize the structure of the FPM reconstruction algorithm to achieve image aberration correction without the need for a priori error estimation. Ou et al. proposed an embedded pupil function reconstruction algorithm (EPRY-FPM) [17] that adds a pupil function updating strategy to correct the system aberration without the need for a priori compensation. In addition, our recently proposed adaptive aberration correction method (AA-P) [18] can achieve fast correction and reconstruction of high-quality images, even under multiple mixed aberration interferences, by simultaneously optimizing the spectral updating strategy and the optical pupil function updating strategy. However, these two aberration correction algorithms focus on the overall aberration correction, and the degree of correction for defocused items alone remains insufficient. This significantly limits the application of FPM in cases where the sample thickness is not uniform, and large defocus aberrations are generated by cell movement observed over a prolonged period.
The aforementioned methods are based on pre-estimating the system defocus error and gradually correcting the defocus aberration in subsequent iterations or directly correcting the instrument to achieve the focus. Digital refocusing algorithms can be implemented by embedding an optimization search module into the subsequent reconstruction process. The goal of the optimization search module is to determine the optimal defocus distance for image refocusing. Henry et al. used a fully connected network to learn the Fourier-domain information of an out-of-focus image; however, it may produce large errors when encountering different types of data [19]. Claveau et al. implemented refocusing by propagating the reconstructed HR image to different defocus distances, allowing defocus distance correction and imaging depth-of-field extension for thin samples [20]. However, when the defocus distances of different sections are different, many computation and image processing steps are required, which can result in long processing times. In addition, Zhang et al. used a symmetric illumination technique to improve image quality by reducing the lateral shift of the defocused image [21]. This method can effectively reduce image blur but does not eliminate it—particularly in the case of large defocus distances. Recently, Zhang et al. obtained the defocus distance of each subregion of a sample by calculating the relative lateral shifts under different oblique angle illuminations. Subsequently, a digital refocusing strategy based on the angular spectrum (AS) propagation method was integrated into the FPM framework to achieve HR and phase information reconstruction for each subregion [15]. This method solves the problem that the traditional optimization search module has a long computation time, but it still needs to perform an image alignment algorithm to compute the lateral shifts, which requires highly accurate image alignment. Furthermore, this method may face computational and storage challenges when dealing with complex samples and large-scale datasets. Therefore, further research may be needed to optimize the refocusing method to extend the FPM system imaging depth-of-field problems.
In summary, in the traditional approach to FPM optimization, appropriate algorithms and parameter settings must be selected according to the needs of the specific application and the actual situation to achieve efficient and accurate digital refocusing. In addition, traditional FPM defocus aberration correction algorithms have limitations in terms of speed and accuracy.
To further increase the speed and accuracy of the FPM refocusing algorithm, this study proposes a fast digital refocusing method based on convolutional neural networks (CNNs) for FPM imaging. The method is inspired by the theory of lateral shift between out-of-focus images proposed by Zhang et al [15] and uses CNNs to learn this image feature for achieving defocus distance prediction of the image region without an image alignment algorithm. The digital refocusing strategy is then fused into an aberration correction iterative algorithm for reconstruction to achieve fast refocusing of images with large defocus aberrations. Experimental results indicated that our method is highly accurate and fast in calculating the defocus distance and can meet the depth-of-field requirements of most low-magnification objectives.
The remainder of this paper is organized as follows. First, the principle of the standard FPM framework is presented in Section 2.1. The lateral offset characteristics of a defocused sample under tilted illumination with the proposed CNN framework are presented in Section 2.2. The flow of the proposed digital refocusing method is described in Section 2.3. To verify the ability of the proposed method to achieve digital refocusing, simulations and real experiments were performed to demonstrate the effectiveness of our method in digital refocusing and depth-of-field extension, as discussed in Section 3. The results are summarized and discussed in Section 4.
2. Principles
2.1 FPM principle
The FPM is an advanced microscope system that utilizes the principle of the Fourier transform to achieve HR intensity and phase imaging. Its architecture is shown in Fig. 1, where an array of adjustable LEDs is used to provide illumination at different angles, and the FPM illuminates a sample using a monochromatic plane wave and records the spectral information generated in the Fourier domain. Illumination at different angles causes the spectrum in the Fourier domain to shift, providing high-frequency information regarding the sample. The FPM records LR images directly in the spatial domain. By applying the Fourier transform and splicing multiple LR images, the FPM can recover the high-frequency information of the sample, allowing it to overcome the diffraction-limit problem in conventional microscopy and effectively increase the space–bandwidth product of the optical imaging system without the need for precision mechanical scanning [14] to realize a large field-of-view and HR imaging. The FPM can also acquire the phase information of the sample through a phase-recovery algorithm, thus providing richer image details, which can be described by the following equations:
2.2 Defocus aberration and convolutional neural networks
In conventional FPM strategies, several types of aberrations occur in optical imaging systems, which degrade the final image quality. The lateral motion of LR images [22] and defocus aberrations produce lateral shifts [15,16]. Although the AA-P algorithm used in this study can correct for the effects of these aberrations to a certain extent, as the defocus distance increases, the AA-P algorithm has difficulty correcting defocus aberrations owing to variations in the focusing height or nonuniformity in the sample thickness. When the sample is imaged outside the focal plane of the objective lens, the lateral shift of the bright-field (BF) image is proportional to the tangent of the defocus distance and the illumination angle. When the defocus distances are different, different lateral offsets occur between the BF images with different illumination angles. However, according to the Fresnel transfer function, an LR image captured by vertical illumination does not have a lateral offset, even if it is not in the focal plane [11]. Therefore, the central BF image can be used as a reference image. Figures 2(a) and (b) show the LR images under the focus plane, and Figs. 2(c) and (d) show the lateral offset between different LR images under an out-of-focus distance of 100 µm. The comparison between the two images reveals that the defocused imaging introduces a lateral offset, as clearly seen from the position of the red circle.
Fig. 2. Lateral shift effect comparison chart: (a) focused center image, (b) focused angled image, (c) defocused center image, and (d) defocused angled image.
To further increase the speed and accuracy of the digital refocusing algorithm and exclude the influence of the image calibration algorithm on the accuracy of defocus distance prediction, we propose a digital refocusing algorithm that uses a CNN to learn the lateral offset features of a set of BF images illuminated at different angles, so that the defocus distance corresponding to this set of defocused images is predicted via regression. The neural network structure is shown in Fig. 3.
Our neural network architecture for predicting image defocus is called a defocus-distance regression network (DRN), which is different from the CNN typically used in image processing tasks [23,24]. Because the input comprises nine time-domain images in the center and around the LED, the defocus aberration correlates not only with the image offset but also directly with image sharpness. Deep CNNs primarily learn image contours and often ignore features such as offsets and image edges.
Moreover, when handling out-of-focus images, the convolution and pooling layers play a pivotal role in extracting features. However, as the number of network layers increases, the adaptability of the feature map diminishes, resulting in the loss of offset features among images [25]. Conversely, concerning feature extraction, local feature extraction through convolution operations may induce inconsistency between the target position in the feature layer and the position in the original image. Prior studies have indicated that CNNs encounter issues with translational invariance, particularly in simple down-sampling scenarios that fail to comply with Nyquist's sampling law, potentially resulting in a substantial loss of translational invariance [26].
To consider the relationships between the defocus distance, image offset, and image sharpness, we designed a DRN that uses eight convolutional layers and two max-pooling layers to extract shallow image features, while saving computing resources, and has sufficient depth to extract deep image features. In addition, the rectified linear unit (ReLU) layer is used to reduce the amount of training operations and facilitate network convergence. Thus, the network has high accuracy, speed, and generalization.
2.3 Digital refocusing method
According to the FPM imaging model with system defocus aberration, we implement the forward imaging model using the traditional FPM algorithm [14]. Leveraging the a priori knowledge provided by the DRN for computing the defocus distance in Section 2.2, the AS method can be embedded into the FPM phase restoration algorithm [18], which aims to realize the digital refocusing and extend the imaging depth-of-field capability of the FPM system. In this method, deep learning is used to solve the out-of-focus distance z by inputting a BF offset image, and the AS method is inserted into the iteration of the FPM frame to correct the defocusing aberration and achieve digital refocusing. The complex amplitude distribution of the refocused image can be expressed as follows:
The initial HR time-domain image, ${s_1}({u,v} )$, is initialized at a predetermined focal plane by inputting the BF-tilted LR image into the pre-trained deep learning model to predict the defocus distance, and its frequency-domain distribution is denoted as ${S_1}({u,v} )$. The center LED-positive incidence LR image is selected as the initial iterative image ${O_0}$. According to the FPM reconstruction strategy, we obtained the sub-region information of the frequency-domain letters of the ith sub-image and relay ${O_i}({u,v} )$ to the predicted focal plane location according to Eq. (3) based on the obtained focal plane location. The obtained spectral information was used to determine the frequency-domain constraints for the interception of HR spectral information after each iteration. We calculated the difference between the average pixel values of the actual image and the target image as a threshold and then subtracted the LR image from this threshold to remove noise. The noise-reduced image was subsequently used as intensity information to replace the amplitude information of the target complex amplitude. The spectral function used for updating is optimized by introducing an adaptive modulation factor α. The updated complex amplitude distribution is propagated to the focal plane, and the spectral and optical pupil functions are updated according to the AA-P algorithm. The AA-P-specific algorithm description was presented in [18] and will not be repeated here.
A flowchart of the proposed digital refocusing algorithm, which consists of the following three steps, is shown in Fig. 5.
Step 1: Obtain the defocused BF LR image dataset input into the CNN DRN model to be trained several times and arrive at the best model.
Step 2: Input the defocused BF LR image to be predicted into the pre-trained network model to obtain the defocus distance z. A clear LR image at the predicted focus position is then obtained using the AS method.
Step 3: The detailed digital refocusing FPM reconstruction algorithm is given in Fig. 4, from which the amplitude and phase information of the defocused sample image can be recovered to achieve digital refocusing, expanding the imaging depth-of-field of the FPM system.
3. Experiments and results
First, 64 mouse kidney section samples from different locations and 32 mouse skin section samples were photographed as the training and validation sets of the neural network in the constructed conventional FPM imaging system using a high-magnification objective lens of 20X/0.4 under LED illumination at a wavelength of 630 nm. This set of 4088 × 3072 × 64 mouse kidney HR images was then randomly cropped into 250 512 × 512 images as high-definition (HD) real images, followed by simulation of a low-magnification objective (4X/0.1) to generate out-of-focus image stacks. Ultimately, 1000 defocused images of size 128 × 128 × 9 within ±100 µm were generated with these simulated center LEDs, as well as tilted illumination of surrounding LEDs. Images corresponding to different out-of-focus distances were tested in real situations using a publicly available dataset of plain and immunohistochemical (IHC)-stained blood smears [14].
The CNN model was implemented using Keras with a convolutional layer kernel size of 3 × 3 and a pooling layer window size of 2 × 2, both set with strides of 1. The network was trained using a Nvidia RTX 3090 graphics processing unit (GPU). The learning rate, “batch_size,” and number of epochs were set to 1e-5, 16, and 150, respectively, and the optimization was performed using the Adam optimizer.
3.1 Simulation experiments
In this study, 1000 images generated by randomly adding defocus errors at different locations of mouse kidney section samples were used as the dataset for the neural network, which was divided into training and validation sets at a ratio of 7:3. The network was trained using the mean absolute error (MAE) as the loss function and the root-mean-square error (RMSE) as the evaluation metric. Figures 6(a) and (b) show the curves of MAE and RMSE during the training process of the CNN, respectively. As the error of the validation set decreased, the CNN can gradually resolve the defocus aberration issue due to the spatial variation of the FPM system by learning the lateral offset between the BF images and the detailed information of the edge of the image, effectively extracting the relevant features of the defocus distance and accurately predicting the defocus distance.
Fig. 6. Training progress and test results of the DRN for biological samples: (a) progression of change during training batches using the MAE as a loss function, (b) variation in RMSE as an evaluation metric across the training batches, (c) comparison between the predicted output results from the pre-trained model and the actual results obtained using mouse kidney samples as the test set, and (d) comparison between the predicted output of the pre-trained model and the actual results obtained using mouse skin samples as the test set.
After 150 training iterations, the network model underwent testing using 1000 mouse kidney sections and 1000 mouse skin sections, each encompassing a randomly added defocus range of ±100 µm. The model required an average of 3 ms per image for determining the corresponding defocus distance. Given that the model was exclusively trained on mouse kidney slice sample images, its generalization performance (i.e., its accuracy when used for other sample images) was verified by successive testing with mouse kidney and epithelial slice samples. Figures 6(c) and (d) show the test results of the trained network applied to mouse kidney sections and mouse skin sections, respectively. Notably, both figures indicate RMSEs of around 6 µm, with MAEs averaging around 3 µm. Additionally, the confidence bands in dark red and dark purple comprise smaller areas, indicating the reliability and accuracy of the model's predictions. Furthermore, most points within the light pink prediction band align closely with the confidence line, indicating that the convolutional neural network has strong generalization and sustained accuracy in predicting different sample types. These results collectively demonstrate the network model's robustness and accuracy in regression prediction of out-of-focus distances.
Here, δ is the depth-of-field distance of the image captured by the microscope objective, λ is the wavelength of the illumination source, and NA is the numerical aperture of the objective lens.
By comparing the experimental results of the different methods in Table 1, we can see that the method of defocus-distance prediction using the CNN model has significant advantages for digital refocusing and depth-of-field extension. Using the trained CNN model, the defocus distance was accurately predicted.
Table 1. Comparison of Experimental Results of Different Methods
The RMSE reached 6 μm, although the CNN model prediction method has a lower accuracy than Henry’s method. The present method was applied to the FPM method using a low-magnification objective lens with a numerical aperture of 0.1 and a magnification of 4×, and the depth-of-field of this objective lens under the illumination condition of a light source with a wavelength of 630 nm can be calculated using Eq. (5) to be approximately ±30 µm. The depth-of-field range of this objective exceeds the predicted defocus error, and the proposed digital refocusing method can expand the imaging depth-of-field of the FPM system to ±100 µm, which is more than three times higher. In practical applications, high-quality images can be recovered through iterations. In addition, this method has a stronger generalization ability than Henry’s method owing to the use of the CNN DRN for extracting the displacement features of the image. When two randomly sampled types of out-of-focus images were inputted into the trained model for predictive regression, accurate results were obtained, indicating that the model has better generalization performance than Henry’s method [19] when applied to different biological samples without additional training. To test the efficiency of the network model in predicting the defocus distance, we inputted 1000 sets of images of size 128 × 128 × 9 into the model for prediction and found that on average, the model only required 3 ms to predict the defocus distances, making it faster than Zhang’s computational method [15]. This suggests that the predictive regression method of the CNN model also has a faster computational speed to achieve fast digital refocusing and depth-of-field expansion.
After predicting the corresponding defocus distance z using the CNN model, we reconstructed the image using a phase reconstruction algorithm that incorporates the AS method. First, we simulated the data distribution used for training, using the captured HD images as intensity information and zero-filled images of the corresponding size as phase information for the simulation as a preliminary verification. Intensity images were randomly selected from different regions of the sample for predicting the defocus distance z with iterative reconstruction. The reconstructed super-resolution images of mouse kidney and skin sections are shown in Figs. 7 and 8, respectively.
Fig. 7. Comparison of reconstructed images of mouse kidney sections: (a1, a2) original images of mouse kidney section samples acquired using a high magnification objective with zero-filled phase images; (b1) – (b6) amplitude and phase images reconstructed via the AA-P method under defocus distances of 0, 50, and 100 µm; and (c1) – (c6) amplitude and phase images reconstructed via the proposed method at defocus distances of 0, 50, and 100 µm.
Fig. 8. Comparison of reconstructed images of mouse skin sections: (a1, a2) original images of mouse skin section samples acquired using a high-magnification objective with zero-filled phase images; (b1) – (b6) amplitude and phase images reconstructed via the AA-P method at defocus distances of 0, 50, and 100 µm; and (c1) – (c6) amplitude and phase images reconstructed via the proposed method at 0, 50, and 100 µm.
In a detailed comparison with the AA-P method [18], the proposed method demonstrated a significant improvement in image reconstruction with a defocus distance of approximately 50 µm. The image quality was also significantly improved when the out-of-focus distance reached 100 µm and the image was reconstructed using the proposed method. Our method can better restore the details and sharpness of the image when processing the defocused image, allowing more accurate reconstruction of defocused images at approximately 50 µm. Furthermore, when the out-of-focus distance was increased to 100 µm, our method effectively enhanced the quality of the reconstructed image and extended the imaging depth-of-field of the FPM system.
The light transmitted in real experiments contains both intensity and phase information of the sample. Thus, relying solely on the LR maps generated from pure intensity HR images for digital refocusing might lack a certain level of confidence. Instead, to further validate the feasibility of our method, we conducted tests mixing intensity and phase LR images. Two sets of simulated experiments were created using mouse kidney slices and “Cameraman” images as intensity information, along with mouse skin slices and “Westconcordorthophoto” images as ideal phase “Ground truth” input. To account for the light intensity transmission effect in complex amplitude samples, defocusing translates the “phase” to the captured images [27].The reconstructed images are shown in Figs. 9 and 10. The proposed method reconstructed high-quality intensity and phase images, even at an out-of-focus distance of 100 μm.
Fig. 9. Comparison of other image reconstruction results: (a1) Cameraman image as amplitude information, (a2) Westconcordorthophoto image as phase information; (b1) – (b6) amplitude and phase images reconstructed via the AA-P method at defocus distances of 0, 50, and 100 µm; and (c1) – (c6) amplitude and phase images reconstructed via the proposed method at 0, 50, and 100 µm.
Fig. 10. Comparison of hybrid image reconstruction results for mouse section samples: (a1) original image of a mouse kidney section sample acquired using a high-magnification objective as amplitude information; (a2) original image of a mouse skin section sample acquired using a high-magnification objective as phase information; (b1) – (b6) amplitude and phase images reconstructed via the AA-P method at defocus distances of 0, 50, and 100 µm; and (c1) – (c6) amplitude and phase images reconstructed via the proposed method at 0, 50, and 100 µm.
We then compared the structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) values of the reconstructed intensity and phase maps, as shown in Table 2, and found that the SSIM of the reconstructed image after refocusing increased from 0.3560 to 0.9788, and the PSNR improved from 14.3398 to 38.7438. This indicates that the proposed method is effective for digital refocusing and extending the FPM system to reconstruct the depth-of-field.
Table 2. Calculation of SSIM and PSNR for z = 100 µm Reconstructed Images Compared with Focused Images
In summary, the method of predicting the defocus distance using a CNN model has the advantages of high accuracy, fast calculation, and wide applicability in digital refocusing and can effectively expand the imaging depth-of-field of the FPM system. It provides a new solution for FPM digital refocusing and is expected to play a key role in practical applications.
3.2 Real experiments
To verify the effectiveness of the proposed method in applications, we conducted experiments using real samples. We selected blood smears corresponding to different staining methods and defocus distances contained in the public dataset of Zheng [14] as samples for the experiments. The sample plane at the time of this data acquisition was 90.88 mm from the LED array, the numerical aperture NA of the objective lens was 0.1, and the LR images were captured with the defocus range of the data within ±100 µm. For each out-of-focus plane, 225 (15 × 15) LR images were acquired. We then calculated the defocus distances of the regions corresponding to the data via the method described in Section 2.
Conventional FPM methods assume that the sample is in the focal plane; therefore, conventional reconstruction methods produce poor-quality images under the assumed conditions. However, using the proposed method, we can efficiently recover HR images on defocused planes for effective digital refocusing and depth-of-field expansion. To verify the effectiveness of the proposed method, we used conventionally stained and IHC-stained images at different defocus distances. The experimental results are shown in Fig. 11, which indicates that even for blood smears with different staining conditions, our method could still obtain high-quality images. As indicated by Table 3, the accuracy of the AA-P method gradually decreased with an increase in the defocus distance, but the reconstructed images of our method still maintained high SSIM and PSNR values. For the images reconstructed using our method, the average SSIM was improved from 0.8265 to 0.9917, and the average PSNR was improved from 26.7050 to 35.0649. The results confirm the robustness of the proposed method for digital refocusing and depth-of-field expansion, as well as its practical applicability.
Fig. 11. Comparison of reconstruction results of blood smear samples tested under different defocusing conditions in real experiments: (a1, a2) images obtained from conventionally stained blood smears photographed in blue light, (a3, a4) images obtained from conventionally stained blood smears photographed in red light, and (a5, a6) images obtained from IHC-stained blood smears photographed in green light. In (a1) – (a6), reconstructed images at the respective corresponding focusing surfaces are used as the real result maps for comparison of amplitude and phase maps. (b1) – (b6) Amplitude and phase images reconstructed via the AA-P method at defocus distances of −15, −55, and −75 µm. (c1) – (c6) Amplitude and phase images reconstructed via the proposed method at −15, −55, and −75 µm.
Table 3. Calculation of SSIM and PSNR of Reconstructed Results with Different Defocus Distances from Actual Image
4. Discussion and conclusions
In this study, we addressed the speed and time problems of existing algorithms for solving different defocus distances corresponding to different lateral offsets of images under tilted illumination conditions in BFs. By redesigning the model using a CNN, we developed a network called DRN that can capture the details and structural information of images, analyze the lateral offset features between defocused images, and resolve the defocus aberration introduced by the spatial change of the objective lens of the FPM system to accurately predict the corresponding defocus distance. The network model was tested on mouse skin cell samples, and the prediction results indicated that it could maintain high accuracy when used for different samples, suggesting that it has a good generalization ability. Tests confirmed the effectiveness of the digital refocusing method for reconstructing the images produced by the FPM within a defocus distance range of ±100 µm. The method enlarged the depth-of-field of FPM imaging by a factor of more than 3, which extends the applicability range of the aberration correction algorithm. A comparison with the pre-compensation prediction revealed that the proposed method can improve the SSIM and PSNR of the reconstructed image with an out-of-focus distance of 100 µm by 174% and 170%, respectively, indicating that it can effectively improve the reconstruction effect of the FPM technique under the condition of a large defocus error. In addition, regarding the efficiency of acquiring the defocus distance, the proposed DRN has a higher computational speed than other methods (it can predict the defocus distance of each image in 3 ms on average) and acquires the defocus distance more efficiently [15,19,20]. This makes the process of digital refocusing and depth-of-field expansion more efficient and faster, and the defocus distance of the image can be calculated in real time, significantly saving time and computational resources. To verify the applicability of the proposed digital refocusing method to different types of biological samples, we tested other blood smear samples with different staining methods through real experiments [14] and tested the reconstruction results of the same sample under different defocus distances through ablation experiments. The results indicated that the proposed method can be used in practical applications with different samples, achieving fast and effective refocusing, broadening the scope of applicability of the error correction, and broadening the scope of application of the error correction algorithm. Therefore, the proposed method effectively expands the imaging depth-of-field of the FPM system and provides a new research idea and technical means to solve the digital refocusing problem. Applying this technique to other non-biological sample images and expanding the depth-of-focus of the FPM system are directions for future research.
Funding
Guangdong Provincial Key Field R&D Plan Project (NO. 2020B1111120004); Natural Science Foundation of Hubei Province (2022CFB099); Foshan University ((Grant No. XSJJ202205KJB07), 2023 Annual Student Academic Fund).
Disclosures
The authors declare no conflicts of interest.
Data availability
We acknowledge the open-source dataset of the biological sample provided by Zheng et al. [14].
References
1. J. Chung, X. Ou, R. Kulkarni, et al., “Counting white blood cells from a blood smear using Fourier ptychographic microscopy,” PLoS One 10(7), e0133489 (2015). [CrossRef]
2. L. Tian, Z. Liu, L.-H. Yeh, et al., “Computational illumination for high-speed in vitro Fourier ptychographic microscopy,” Optica 2(10), 904 (2015). [CrossRef]
3. J. Sun, C. Zuo, J. Zhang, et al., “High-speed Fourier ptychographic microscopy based on programmable annular illuminations,” Sci. Rep. 8, 7669 (2018). [CrossRef]
4. J. Zhang, T. Xu, Z. Shen, et al., “Fourier Ptychographic microscopy reconstruction with multiscale deep residual network,” Opt. Express 27(6), 8612 (2019). [CrossRef]
5. T. Aidukas, R. Eckert, A. Harvey, et al., “Low-cost, sub-micron resolution, wide-field computational microscopy using opensource hardware,” Sci. Rep. 9(1), 7457 (2019). [CrossRef]
6. K. Guo, S. Dong, and G. Zheng, “Fourier ptychography for brightfield, phase, darkfield, reflective, multi-slice, and fluorescence imaging,” IEEE J. Sel. Top. Quantum Electron. 22(4), 77–88 (2016). [CrossRef]
7. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photonics 7(9), 739–745 (2013). [CrossRef]
8. L. Bian, J. Suo, G. Zheng, et al., “Fourier Ptychographic reconstruction using Wirtinger flow optimization,” Opt. Express 23(4), 4856–4866 (2015). [CrossRef]
9. C. Zuo, J. Sun, and Q. Chen, “Adaptive step-size strategy for noise-robust Fourier ptychographic microscopy,” Opt. Express 24(18), 20724–20744 (2016). [CrossRef]
10. S. Jiang, K. Guo, J. Liao, et al., “Solving Fourier ptychographic imaging problems via neural network modeling and TensorFlow,” Biomed. Opt. Express 9(7), 3306–3319 (2018). [CrossRef]
11. J. Sun, Q. Chen, J. Zhang, et al., “Single-shot quantitative phase microscopy based on color-multiplexed Fourier ptychography,” Opt. Lett. 43(14), 3365 (2018). [CrossRef]
12. P. Song, S. Jiang, H. Zhang, et al., “Super-resolution microscopy via ptychographic structured modulation of a diffuser,” Opt. Lett. 44(15), 3645–3648 (2019). [CrossRef]
13. A. Pan, M. Zhou, Y. Zhang, et al., “Adaptive-window angular spectrum algorithm for near-field ptychography,” Opt. Commun. 430, 73–82 (2019). [CrossRef]
14. G. Zheng, C. Shen, S. Jiang, et al., “Concept, implementations and applications of Fourier ptychography,” Nat. Rev. Phys. 3(3), 207–223 (2021). [CrossRef]
15. S. Zhang, G. Zhou, C. Zheng, et al., “Fast digital refocusing and depth of field extended Fourier ptychography microscopy,” Biomed. Opt. Express 12(9), 5544–5558 (2021). [CrossRef]
16. G. Zhou, S. Zhang, Y. Zhai, et al., “Single-shot through-focus image acquisition and phase retrieval from chromatic aberration and multi-angle illumination,” Front. Phys. 9, 648827 (2021). [CrossRef]
17. X. Ou, G. Zheng, and C. Yang, “Embedded pupil function recovery for Fourier ptychographic microscopy,” Opt. Express 22(5), 4960–4972 (2014). [CrossRef]
18. R. Wu, J. Luo, J. Li, et al., “Adaptive correction method of hybrid aberrations in Fourier ptychographic microscopy,” J. Biomed. Opt. 28(03), 036006 (2023). [CrossRef]
19. H. Pinkard, Z. Phillips, A. Babakhani, et al., “Deep learning for single-shot autofocus microscopy,” Optica 6(6), 794–797 (2019). [CrossRef]
20. R. Claveau, P. Manescu, M. Elmi, et al., “Digital refocusing and extended depth of field reconstruction in Fourier ptychographic microscopy,” Biomed. Opt. Express 11(1), 215–226 (2020). [CrossRef]
21. M. Zhang, L. Zhang, D. Yang, et al., “Symmetrical illumination based extending depth of field in Fourier ptychographic microscopy,” Opt. Express 27(3), 3583–3597 (2019). [CrossRef]
22. L. Bian, G. Zheng, K. Guo, et al., “Motion-corrected Fourier ptychography,” Biomed. Opt. Express 7(11), 4543–4553 (2016). [CrossRef]
23. X. Qin, Z. Zhang, C. Huang, et al., “U2-Net: Going deeper with nested U-structure for salient object detection,” Pattern. Recognit. 106, 107404 (2020). [CrossRef]
24. A. Aghabiglou and E. M. Eksioglu, “MR image reconstruction using densely connected residual convolutional networks,” Comput. Biol. Med. 139, 105010 (2021). [CrossRef]
25. A. Azulay and Y. Weiss, “Why Do Deep Convolutional Networks Generalize so Poorly to Small Image Transformations?” arXiv, arXiv:1805.12177 (2018). [CrossRef]
26. J. C. Myburgh, C. Mouton, and M.H. Davel, “Tracking Translation Invariance in CNNs,” arXiv, arXiv:2104.05997 (2021). [CrossRef]
27. C. Zuo, J. Li, J. Sun, et al., “Transport of Intensity Equation: A Tutorial,” Opt. Lasers Eng. 135, 106187 (2020). [CrossRef]
