Open Access
Add to BibSonomy
Abstract
Intravital microscopy in small animals growingly contributes to the visualization of short- and long-term mammalian biological processes. Miniaturized fluorescence microscopy has revolutionized the observation of live animals’ neural circuits. The technology’s ability to further miniaturize to improve freely moving experimental settings is limited by its standard lens-based layout. Typical miniature microscope designs contain a stack of heavy and bulky optical components adjusted at relatively long distances. Computational lensless microscopy can overcome this limitation by replacing the lenses with a simple thin mask. Among other critical applications, Flat Fluorescence Microscope (FFM) holds promise to allow for real-time brain circuits imaging in freely moving animals, but recent research reports show that the quality needs to be improved, compared with imaging in clear tissue, for instance. Although promising results were reported with mask-based fluorescence microscopes in clear tissues, the impact of light scattering in biological tissue remains a major challenge. The outstanding performance of deep learning (DL) networks in computational flat cameras and imaging through scattering media studies motivates the development of deep learning models for FFMs. Our holistic ray-tracing and Monte Carlo FFM computational model assisted us in evaluating deep scattering medium imaging with DL techniques. We demonstrate that physics-based DL models combined with the classical reconstruction technique of the alternating direction method of multipliers (ADMM) perform a fast and robust image reconstruction, particularly in the scattering medium. The structural similarity indexes of the reconstructed images in scattering media recordings were increased by up to 20% compared with the prevalent iterative models. We also introduce and discuss the challenges of DL approaches for FFMs under physics-informed supervised and unsupervised learning.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Fluorescence microscopy has allowed a great deal of research and discoveries in neuroscience over the years, and remains a reference point among all fluorescence imaging techniques [1,2]. Confocal, light-sheet, and two-photon fluorescence microscopy are well-known approaches capable of dealing with light scattering for obtaining single cell-resolution neuronal recordings, but they offer limited speed for real-time brain imaging in freely moving animals. Head-mounted miniaturized fluorescence microscopes are used to track the neuronal activity in live animals [3–5], alongside other techniques, such as fiber photometry [6–8]. These miniature microscopes, also known as Miniscopes, have significantly advanced over the recent decade based on open-source developments [9]. One of the latest Miniscope releases enables 200 µm electrical focal adjustment instead of having a constant working distance [10]. The size and the weight of the optical components are the main limitations of this application. Moreover, despite all the progress, these tools are not optimized for in-depth imaging in scattering media, such as in brain tissues.
Flat microscopy promises to enable the next generation of miniaturized microscopes by avoiding the utilization of bulky lenses and providing in-depth imaging using artificial intelligence-based light scattering compensation techniques. To this end, flat on-chip microscopes have been built for bright-field and fluorescence imaging applications [11–15]. These designs are widely referred to as lens-free or lensless systems. However, the increasing use of microlens array masks in recent demonstrations motivates the utilization of the broader term flat microscopy. Fluorescence imaging relies on fluorophores that incoherently illuminate the sample and can transmit each other's light and the background illumination [15,16]. Accordingly, digital holography and shadow-based techniques cannot be used for fluorescence flat imaging [17–21]. These methods also require two-sided access to the sample, making them unsuitable for freely moving experiments.
Flat fluorescence microscope (FFM) technology is based on the utilization of a thin mask placed between the sample and the image sensor. The incoming light is encoded while traveling through the mask till it reaches the image sensor. Although this requires a computational reconstruction of an image scene, three-dimensional (3D) information can also be decoded in the process. 3D recovery of a two-dimensional (2D) single-shot recording has been reported in flat photography [22–24] and flat microscopy [14,15,25].
Miniature fluorescence microscopes are susceptible to background noise and scattering because no physical component is used to eliminate out-of-focus light propagation. In the last 5 years, most computational fluorescence microscopy (mask-based designs with or without other lenses) studies have focused on 3D information extraction out of a single-shot recording as a primary application of such a modality. Previous studies present 2D or 3D reconstruction results with clear or nearly transparent tissues, without scattering, as if the technology has been directly adapted from flat photography models. The resolution and accuracy of these prototypes have been reported with clear or semi-clear tissues. The images captured with these systems in fixed biological scattering media samples are blurry [14,15,25]. Tissue scattering is a major obstacle to 2D or volumetric brain imaging with FFM in freely-moving mice. The Miniscope3D has achieved a significant quality improvement using an embedded microlens mask and reported imaging results in a scattering brain slice, but no performance assessment was provided [26]. Adams et al. [27] reported in-vivo imaging results with a head-fixed mouse using their microscope. They were able to extract $C{a^{2 + }}$ signals from regions of 150 $\mathrm{\mu }{\textrm{m}^2}$ to 0.01 $\textrm{m}{\textrm{m}^2}$ and compared them with an epifluorescence reference microscope. The clear tissue resolution reported for this prototype was 9 µm, which is highly superior to its in-vivo imaging quality. This fact suggests that the scattering phenomena in in-vivo imaging significantly influence the reconstruction algorithm. Therefore, a critical gap in flat microscopy is dealing with scattering effects.
Recent Deep Learning (DL) inverse problem models outperform classical iterative solvers in many image reconstruction and restoration applications [28–30]. A variety of DL architectures have been used in flat photography to enhance image reconstruction. These techniques have provided significant improvement using trained and untrained networks [31–34]. Moreover, imaging through scattering media [35–38] was performed using convolutional neural networks (CNN). Accordingly, a diffuser’s scattering profile can be learned by a CNN to denoise and reconstruct images with coherent and incoherent illuminations. These studies have exposed the capability of using convolutional neuronal networks to deal with unseen scattering media. To date, semi-supervised deep learning techniques in optical image reconstruction and enhancement are state-of-the-art. While classical iterative techniques are susceptible to model and calibration mismatches, semi-supervised methods usually compensate for modeling mismatch using trained networks and rely on the classical schemes as the initial image predictor based on the system’s physics [31,33,37–39]. Unsupervised DL networks are recently developed in inverse imaging problems such as lensless photography and hold great promise for many lensless applications where dataset generation is impossible or challenging [34,40].
In this paper, we test and report the performance of supervised and unsupervised DL approaches in scattering media for the first time in an FFM model. This work presents supervised physics-informed deep neural networks as a great solution for fast and robust image reconstruction in scattering media. Moreover, different FFM scattering calibration strategies were studied to suggest effective calibration and Point Spread Function (PSF) compression methods for similar studies. We also addressed the training data dependency of DL approaches and discussed unsupervised methods’ capability to improve flat microscopy in the lack of a dataset. For this purpose, Zemax Optic Studio is used to simulate ray propagation through the microscope components (mask, etc.) and the scattering medium using a ray-tracing and Monte Carlo model, similar to other scattering models in the literature of biological tissues [41–43], to generate our datasets. In the subsequent sections, we present a comprehensive model of a flat fluorescence microscope that incorporates a microlens mask (known for its superior performance compared to other masks) and investigate the sensitivity of such an FFM to varying degrees of scattering phenomena. We later streamline this design to focus on the scattering effect, allowing for the creation of datasets and the exploration of DL reconstruction models.
2. System overview
2.1 Holistic FFM modeling
Holistic system modeling provides an essential tool to exemplify scattering phenomena in FFM and assess novel approaches for scattering sensitivity compensations. Our holistic model consists of key components of FFM systems, including a mask, a detector, an excitation module, emission filters, and fluorescence samples embedded within a scattering medium, as presented in Fig. 1(a). Available FFMs use various types of excitation modules such as optical fiber, light sheet illumination, collimated miniaturized LEDs, or laser diodes. Here we employed an excitation module that comprises LEDs and corresponding collimating lenses. Four tilted collimated light sources are diagonally placed atop the sample, ensuring a nearly uniform illumination across the field of view, a technique similar to the one used in Xue et al. study [44]. The emission filters are placed between the mask and detector to block excitation light based on the Thorlabs Ltd standard spectral response. Emission filtering in mask-based microscopy often encounters challenges due to the presence of non-collimated beams in the filtering plane, which decreases the efficiency of the filters. Therefore, secondary emission filters can be necessary to increase the signal-to-noise ratio [45].
Fig. 1. Overview and performance of the holistic FFM model in the presence of scattering. a) The proposed flat fluorescence microscope configuration for experimental applications, consisting of the excitation module and Monte Carlo -based developed scattering medium and embedded fluorescence samples. The FFM uses a multifocal random microlens array for light encoding. b) Measurement and reconstructed images of samples with various medium properties, providing insights into the scattering sensitivity. The reported values in the images indicate the peak irradiance ($W/c{m^2}$).
Today, various types of amplitude and phase masks are tested in masked-based imaging. Amplitude masks are often used in mask-based imagers; however, phase masks increase the mask’s light transmission compared with amplitude masks [14,15,26]. Microlens array masks are a great example of phase masks that are also advantageous as their response patterns show higher contrast and larger dark areas. Random microlens arrays, made up of several lenslets randomly positioned across the mask surface, break uniformity in the pattern and improve the reconstruction [15,46]. Yanny et al. [26] custom-designed a multifocal random microlens mask using Nanoscribe two-photon polymerization 3D printing. Similarly, we use a multifocal random microlens array with three focal lengths randomly distributed between arbitrarily located plano-convex lenslets with square sides ranging from 222 to 370 µm, Fig. 1(a).
Microlens masks maximize contrast and dark areas in the PSFs across the volume. Therefore, each lenslet should ideally converge the diverging rays of a field-of-view’s point to a detector’s point. We experimentally adjusted the focal lengths of the lenslets for planes in 50, 200, and 350 µm depths of volume and assigned the calculated curvature radii of 0.75, 0.82, and 0.9 mm to the lenslets in an arbitrary sequence. Besides, discrete sectioning of lenslets in this design supports standard etching microlens fabrication processes and Nanoscribe 3D printing, which can be critical for microlens array fabrication.
The objective plane is positioned 3.35 mm away from the effective surface of the mask. A 10802 (3.45-µm pixel size) detector is positioned 3.2 mm away from the mask. This design can adequately capture the mask response for sources across the volume. The mask–detector (${d_{md}}$) and the mask–objective plane (${d_{mo}}$) distances define the system’s magnification of $M = \frac{{{d_{md}}}}{{{d_{mo}}}} \approx{-} 0.95$ [15].
Monte Carlo modeling of biological tissue is essential for realistic FFM modeling due to light diffusion in biological tissues. Ray tracing modeling of computational fluorescence microscopes is widely adopted to simulate Point Spread Function (PSF) with clear medium assumptions [4,15,26]. However, previous studies did not include biological tissue interaction in their mask-based microscopy models. To the best of our knowledge, we included the scattering medium in an FFM simulation for the first time. The fluorescence samples consist of 3D models of neurons, which are imported as 3D volumes embedded in the medium measuring 720 × 720 × 400 µm. Within the holistic model, a rigorous model of Mie scattering is used to simulate the fluorescence (enhanced green fluorescent protein) and scattering response in the neuron and medium [47]. This profile is loaded based on the publicly available Dynamic Link Library (DLL) developed by Carles et al. [42]. Our continuous fluorescence and excitation wavelengths are centered at 510 and 480 nm, similar to their miniaturized fluorescence microscopy tests. The emitted light passes through the medium and undergoes encoding by passing through the diffuser mask. Finally, an image of the objective can be predicted using backward models (detailed in Section 3).
Figure 1(b) showcases the outcomes of a simulated fluorescence activity featuring nine neurons after altering the scattering coefficients. The samples are placed in the 350-µm depth of the medium. The scattering coefficients (µs) adopted for the simulation are 0, 0.5, 5, and 10 $m{m^{ - 1}}$, corresponding to zero, mild, average, and high scattering conditions, respectively. It should be noted that the scattering coefficient and depth of view can have interchangeable effects. For instance, a thicker sample with lower scattering properties can yield similar outcomes as a thinner sample with higher scattering characteristics. Here, the depth of view remained constant while the scattering coefficient was altered. Further details regarding the scattering properties can be found in Supplementary Note S1.
After introducing the scattering, the reconstructed images clearly exhibit an instant image quality and contrast reduction. Even with a low scattering coefficient, the accuracy of computational models based on clear media calibration is compromised (described in detail in Section 2.3). This case can apply to 2D in vivo imaging, mainly when the targeted neurons are located beneath a scattering tissue. In applications involving in vivo brain-stem and cortex imaging, the outer neurons can be located beneath 10ths of micrometers of scattering tissues [27,48,49]. As a result, a 2D distribution of beads within a scattering tissue, regardless of its thickness, can closely resemble these real-world scenarios. Hence, conducting scattering effects investigations on FFM and proposing novel calibration and reconstruction techniques provides valuable insights for achieving dependable in vivo imaging capabilities.
2.2 Dataset generation using a simplified FFM model
This study focuses on the challenges posed by scattering in lensless imaging systems and aims to explore the potential of deep learning (DL) approaches to overcome these challenges. Supervised learning relies on datasets of input-output image pairs, and developing an experimental FFM dataset encompassing all the desired scenarios requires extensive experiments and resources. Therefore, computational datasets are valuable for assessing the benefits and drawbacks of supervised learning. In addition, such datasets help to verify and compare different reconstruction techniques statistically.
Due to the computational time constraints of the original model, generating large-scale datasets by this model is too complicated and, to some extent, unnecessary. To address this issue, we simplified the model by excluding excitation illumination and focusing solely on simulating the system after the emission occurs, as presented in Fig. 2(a). Although we neglect the excitation noise by this assumption, the simulation time substantially improves, making dataset generation feasible. This simplification does not impact our further analyses and conclusions for two reasons. Firstly, the simplification equally affects all tested approaches, ensuring a fair comparison. Secondly, as scattering phenomena exhibit characteristics of a noise source in the system, deep learning achieved outcomes are expected to apply to other noise sources not covered in this simulation. To summarize, microlens-based flat microscopes have proven their capability to compensate for substantial noise sources in clear media applications effectively; Therefore, we evaluate their performance by specifically addressing the hurdle of scattering, which significantly affects the high-frequency information of their response.
Fig. 2. Simplified FFM Model in dataset generation scheme. a) The simplified FFM excludes excitation illumination and filtering for dataset generation. The bead sources embedded in 350 µm depth of the clear and scattering media model the fluorescence activity within the media. b) Numerical dataset generation scheme for generating ground truth and measurement data pairs. The extracted phase-contrast images of the LIVECell dataset were preprocessed to highlight cell morphologies and extract our selected locations for placing the beads. The cross-section of the beads at the selected locations is superposed to create ground truth labels. The system responses from 7-µm objective scanning with a 10-µm radius bead are registered and superposed to calculate the measurement images based on the selected locations.
The fluorescence activities in the simplified model are simulated by 10-µm radius spherical light sources embedded in 720 × 720 × 400 µm transparent and turbid media for generating clear and scattering datasets, respectively, as shown in Fig. 2(a). Within the simplified model, we use the Henyey-Greenstein distribution, with a mean-free path of 0.05 mm, defining the characteristics of the turbid medium [50,51]. For further details regarding the simplified model scattering profile, please refer to Supplementary Note S1.
Our numerical dataset generation scheme, depicted in Fig. 2(b), creates pairs of ground truth and measurement data for both clear and scattering datasets. The distributions of the fluorescence beads determine the label images and are preferred to be derived from realistic biological patterns. Accordingly, we utilized the LIVECell dataset, including diverse images of cell morphologies, to generate the fluorescence activity labels [52]. LIVECell data underwent preprocessing to choose some potential locations for placing the beads, which we refer to as “selected locations”.
In the next phase, we optimize the simulation process by scanning the objective with a 10-µm radius sphere source at 10,201 (1012) uniformly distributed points. This scanning size provides 7-µm resolution in objective space. The beads are distributed at a depth of 350 µm of the medium representing a specific case of volumetric imaging. The detector measurement of each scanning point is simulated and stored for subsequent superposition based on the selected locations. It is important to note that selected beads are distributed within ${101^2}$ pixels, matching our objective scanning steps.
Unlike volumetric imaging systems, FFM systems exhibit depth-dependent responses. Therefore, reconstructing 3D distributed samples requires the use of 3D reconstruction models that is computationally intensive. Preserving the meaningful distribution in the samples is critical for DL model performances, and randomly assigning depth information may lead to suboptimal performance in DL models. In our approach, the LIVECell dataset provided 2D locations of a thin layer sample, and we placed all the beads in the deepest targeted plane. This choice was motivated by the fact that the system experiences the most significant high-frequency information loss at this depth, which makes it difficult to distinguish between two beads. We compare the measurement and reconstruction results of two scattering trials with similar selected locations distributed in 2D and 3D and discuss their differences in Supplementary Note S2 and Fig. S1.
The ground truth (label) images consist of 10-µm radius beads centered on the location of the selected locations. The intentional use of larger bead sizes than the scanning step size (7 µm) allows overlapping bead sources which causes variable intensities in the objective instead of a binary intensity profile. This can be expressed as the superposition of selected beads in the objective space. The effect of overlapping beads in software and generated labels are compared in Fig. S2(a).
Unlike supervised DL models, which do not rely on system magnification, classical approaches should have label images resized to object dimensions. The calculated M magnification implies that labels should have 220 pixels. However, the reconstructed image size can vary depending on the specific algorithm employed for image reconstruction. Therefore, a reliable approach in computational microscopy for image comparison is to maximize the correlation between the reconstructed and ground truth images by rescaling the photos [15]. We observed that classical models result in slightly smaller, by a few pixels, and varying among models. To ensure compatibility with various approaches, we stored the label images with a size of 224 pixels, which is a popular pixel size used in deep learning methods [53]. This size is larger than the approximated dimensions of classical reconstructions. The label images are later downsampled to different extents to determine the optimal matching to the reconstructed images.
Instead of simultaneously placing all the beads in the software and simulating the measurements (software-only method), we employ a numerical approach that superposes the effects of each selected bead in the detector space. Once the objective scanning is completed, the responses of the selected locations are added for both clear and scattering media datasets. By the numerical approach, we could significantly accelerate the simulation, achieving a speedup of up to 300 times compared with the software-only technique. The acquired measurements of these two approaches are compared in the Fig. S2(b). The measurement images are generated by an 8802-pixel detector size, emphasizing the image sensor crop effect. Subsequently, they were digitized to 8-bit representation, simulating an extreme image sensor digitization error in the generated data.
This process is repeated to generate 10,000 labels and recorded images for clear and scattering media. Our computational datasets have similar labels, which facilitates performance comparison of the system with different media. Afterward, these datasets are divided into 9000 training sets and 1000 testing sets. Within the training set, 2 percent of the data was allocated for validation.
2.3 Field-varying scattering PSFs
The intensity impulse response of our optical system, commonly referred to as PSF, is captured by placing a point-like light source in the center of the objective. The PSF pattern determines the amount of information transferred via the optical system in response to the impulse. The convolution of the PSF and the actual scene image determines the response of a linear space-invariant optical system for all incoherent inputs. We can express the forward model of our system with the following equation:
Fig. 3. The impact of scattering on PSFs. a) Illustration of the system's calibration grid and the PSFs captured in clear and scattering media. The PSF image on the right displays the cropping effect caused by the obstructer and image sensor. The reported values in the images indicate the peak irradiance ($W/c{m^2}$). b) Logarithmic magnitude spectra of the PSFs and magnitude degradation of cross-sectional frequency response normalized by the center (0,0) of the spectrum. c) The ratio of weights to the summation of the weights. The number of significantly larger weights is highlighted. (To improve clarity, a smoothing filter with a window size of 15 and a polynomial order of 2 is applied to the cross-sectional responses. 81-step calibration done by a 5-µm bead for clear medium and a 10-µm bead in scattering medium.)
Measuring PSF(s) via a calibration process, is crucial for classical inverse problem algorithms. Calibration of field-varying PSFs is typically executed in discrete steps, as depicted in Fig. 3(a). A local convolution model can use all the recorded PSFs with their weighting factor representing the masking area where the measurement is valid [15]. However, precise recording requires hundreds of PSFs, especially for 3D reconstructions.
Unlike previous studies, our hybrid model allowed us to extensively explore and compare the recorded PSFs with clear and scattering media. Our scattering simulation shows that the peak of PSF irradiation is 0.003 times smaller than the clear medium PSFs, Fig. 3(a). This significant PSF contrast reduction challenges computational imaging in the presence of scattering, as the clear medium calibration differs from the system's behavior in the turbid medium.
The difference between clear and scattered PSFs extends further than just intensity degradation. Spectral analysis of the PSFs can provide valuable insights into the information transition after introducing scattering to the system. Figure 3(b) illustrates the PSFs in their frequency domain and compares the spectral magnitude of the central PSFs. Clear mode PSF includes high-frequency information with a decline in amplitude as the frequency rises. On the other hand, the PSF of the system under a scattering medium does not contain significant high-frequency features. This noisy-like high-frequency information is unreliable for backward models since it can vary in different attempts for a given target.
As expected, the Nyquist resolution of the system in scattering application decreases. Our conducted comparison of the frequency information within the system using beads of sizes 5 and 10 µm validates that our 10-µm bead size for calibration of scattering PSFs respects the Nyquist resolution of the system in scattering calibration (see Fig. S3).
Singular Value Decomposition (SVD) compresses the data by keeping only the essential information of the measurements, considering the memory and processing limits. We applied the SVD analyses on the PSFs matrix of $N$-measured ${M_p}$-pixel images, $h({N,{M_p}} )$, to extract its kernels (${g_r}$) and weights (${w_r}$) [26,54], which can be written as
In our subtle field-varying systems, only a small number of weights, also known as singular values, significantly contribute to capturing important information about the PSFs matrix (see Fig. S4). In other words, weights with values close to zero correspond to kernels that contain less or noisy information about the PSFs matrix. Therefore, h* can estimate h by considering the K most significant weights and their corresponding kernels (${h_{({N,Mp} )}} \approx {h^\mathrm{\ast }}_{({K,Mp} )}$), where K can be much smaller than N. The PSF matrix and the forward equation can be rewritten for the K-significant kernels and their corresponding weights as follows:
Model architecture plays a vital role in the field-varying characteristics of the system. Miniscope3D has achieved approximately two times greater resolution by field-varying PSF considerations [26]. In their work, the GRIN lens and high magnification resulted in significant variation across the recorded PSFs. Although our system magnification ratio is much lower than that in Miniscope3D with a GRIN lens (5.2×), the field-varying consideration is expected to assist our reconstruction models.
Our SVD comparative analysis between clear and scattering PSFs reveals a decrease in the number and value of significant weights of the scattering PSFs. Figure 3(c) depicts the ratio of SVD weights with 81-step calibration. In clear media, three weights contribute more than 0.05 to the matrix transformation, whereas in scattering media, only one kernel is significantly more important than others. This causes the estimation only relies on the significant kernel unless all other kernels are included. Fig. S4 presents our SVD analyses applied with varying PSF recordings and illustrates the most important kernels of the 81-step calibrations.
The substantial difference between two PSFs of the same system poses an important question: Should scattered medium images be reconstructed with clear medium PSF(s)? The reconstruction qualities of each technique have been detailed and discussed in Section 5.1 to answer this question.
3. Reconstruction methods
3.1 Metrics
We use image comparison measures to evaluate reconstruction accuracies quantitatively. The least absolute deviation error (L1) and structural similarity index (SSIM) are two image comparison metrics used to validate the distance between the images. An L1 error of 0 is expected for an identical label and reconstructed image, whereas the SSIM metric is 1 for this case. Our DL training processes combine these two metrics in the loss functions.
3.2 Iterative classical backward models
Classical backward models recover the actual scene image by solving an iterative optimization problem. These techniques minimize the distance between the actual and the predicted images in a stepwise manner. A sparsity-constrained inverse problem model can solve convex problems only by knowing the physics of the system, h. The inverse problem for estimating the fluorescence activity of the scene (v), where $\tau $ is the tuning parameter, can be expressed as
Fast iterative shrinkage-thresholding algorithm (FISTA) [55] and the alternating direction method of multipliers (ADMM) [56] models efficiently solve the proposed minimization problem. Both techniques are reliable solvers that have been extensively tested in similar optimization studies. FISTA converges with more iterations, while ADMM is much faster, requiring only a few iterations. ADMM has more tunable parameters and is more susceptible to noise than FISTA. These models should be implemented with small PSF data, estimated with h*, due to computational challenges [26].
We implemented both techniques as iterative reconstruction references for the developed DL reconstruction models, focusing on quality and speed. Hyperparameters were fine-tuned to provide the best possible outcome for both models. We tested different K values (1,3,5 and 9) in our grid search process and its results supported our observations in Fig. 3(c). Optimum iteration numbers were found in the range of 200 to 1500 for FISTA and 1 to 5 for ADMM in different implementations. The optimum number of iterations varies depending on the values of tuneable parameters. So, we included the iteration number in our grid search parameter tuning processes.
3.3 Deep learning UNet backward model
UNet deep architecture has achieved significant results in image prediction in lensless photography and scattering media imaging [33,35,57,58]. The network consists of several steps of double convolution, downsampling, and up-convolution in its contracting and expanding pathways. The skip connections in this structure profoundly observe previous layers without vanishing gradient issues.
We implement a UNet model based on the method in [59], Fig. 4(a), and compare its performance with classical and physics-informed DL models. The training process of the UNet backward model is presented in Fig. 4(b), where it transfers train data into the label domain and measures the resulting image distance from the actual labels. Our loss function calculates L1 and SSIM distances between the predicted and actual labels, here called label loss, which penalizes the network in training. The label loss is equal to $a({L1} )+ b({1 - SSIM} )$, where $a + b = 1$ and $a,b \in ({0,1} ]$. The hyperparameters of the network were fine-tuned to prevent overtraining while convergence was guaranteed.
Fig. 4. UNet backward model. a) The training scheme of the UNet backward model. b) The operation scheme of the backward model. c) The architecture of the UNet.
Similar to many fluorescence microscopy applications, the developed fluorescence labels have large zero areas. CNNs work well with dense data and can face challenges once introduced to sparse patterns. The UNet can remain stuck in local minimums during the optimizations for reconstructing sparse fluorescence labels. This problem can be solved by providing more information about the label image or physics of the system, which we discover in the next section. Another solution is to apply gaussian kernels on the training labels. We notice that UNet properly trains on the gaussian added labels. Although adding the gaussian effect highly affects the prediction quality, still in scattering media recording UNet model can provide fast and comparable results to the classical models. We gradually increased the gaussian kernel size and sigma value to find the optimum filter that supports successful training with the least possible effect on the labels. The UNet model operation scheme uses the trained network to predict test labels based on the test data, as shown in Fig. 4(c).
Our label and measurement images have different dimensions and should be interpolated to have the same size before being used in the implemented UNet network. At first, we upsampled the labels to the measurement data size to keep all the measurement information. However, the network only recovers the gaussian added patterns of labels, and it suggests that measurement downsampling should not affect the network performance in this case. Therefore, we trained the network with downsampled measurement data to the label size, which could achieve similar results with a faster operation time.
3.4 Physics-informed deep learning backward models
In this section, we present our physics-informed deep learning models. The UNet backward model predicts the images without prior knowledge of the system's physics. The sparse nature of the simulated fluorescence images and generated microlens measurements make a high-resolution image prediction difficult for the UNet, due to its convolutional structure. Therefore, knowing the label layout redefines the UNet as a denoiser instead of a predictor.
DL schemes can highly enhance iterative reconstruction models (Section 3.2). A fast-iterative technique such as ADMM, which requires a few iterations, can provide the label layout as a DL model's input. This technique has a drastically faster performance compared with the FISTA method. Several studies have implemented neural network-based ADMM models in a supervised unrolled learning scheme followed by a UNet denoiser [33,58,60]. Similarly, we implement two DL-assisted ADMM backward models for an FFM application for the first time.
3.4.1 Denoised-ADMM (dADMM) backward model
ADMM-reconstructed images can show high noise levels, especially in scattering media reconstruction. A UNet denoiser network has demonstrated promising performance in improving ADMM results in lensless photography [33,58,60]. The denoised ADMM (dADMM) model, shown in Fig. 5(a), receives ADMM outputs and predicts the enhanced version of the ADMM labels [61]. The label loss is measured between the actual and predicted labels, similar to the UNet backward model. The dADMM’s operation scheme is similar to the one in the UNet model (Fig. 4(b)) but with the inputs of ADMM-reconstructed test labels instead of the test data.
Fig. 5. Supervised physics-informed deep learning models. a) The dADMM training scheme. The UNet enhances the ADMM-reconstructed label inputs with the label loss that updates the network in backpropagation. b) The dADMM+ training scheme. The UNet receives Train data and ADMM-reconstructed labels as input to predict the labels. The A forward model projects the predicted labels to the detector domain and predicts the input train data using the calibrated PSF information. The label and fidelity losses are measured to determine the total loss that updates the network in backpropagation.
3.4.2 Updated denoised-ADMM (dADMM+) backward model
Although the dADMM network takes advantage of the physics of the system, it never observes the measurement data. The sparse nature of microlens-generated measurements suggests that details in the measurements and labels are directly correlated. Therefore, observing the measurements while training on ADMM-reconstructed labels ensures that the network is optimized using all available system information.
The measurements (train data) can be added to the dADMM scheme in two ways, presented as dADMM+ backward model. First, the network can receive both the measurement data and ADMM-reconstructed labels in its input, as if we combine the UNet and dADMM backward models. In this case, the network can predict high-quality labels using the ADMM inputs while trying to extract more details from the measurement data. Compared with the UNet backward model, the network starts the training while a reconstructed version of the label is already observed, accelerating the prediction convergence. Second, the measurement data can be used as an additional fidelity loss similar to other physics-informed DL models [31,39,62]. The training scheme of the dADMM+ backward model is presented in Fig. 5(b). By adding the fidelity loss in this model, unsupervised implementation of the dADMM+ is also achievable, which we discuss its differences and results in the Section 5.3.
In this model, the UNet network predicts the label images, which provides the label loss, as presented in the previous methods. Next, the forward model projects the predicted labels to the detector domain and measures the fidelity loss. The forward model computes the sensor recording using the main PSF kernel based on Eq. (3). The fidelity loss, ${l_{Fidelity}} = \; a({L1} )+ b({1 - SSIM} )$, has fine-tuned a and b to have relatively comparable data and label losses. The overall training loss is the summation of the label loss and the fidelity loss, which updates the parameters of the backward network through backpropagation. Once the network is trained, it receives the test data and ADMM-reconstructed test labels in its operation step to predict the test labels. The dADMM+ model not only measures the quality of the predicted label but also validates if a microscope’s forward model acknowledges such a prediction. The dADMM+ network, instead of image prediction from scratch, uses all system information to enhance ADMM reconstructed label.
4. Results
This section compares the image reconstruction performance of each presented method, including the classical iterative and deep learning models. Since system calibration in scattering media is not reviewed in previous works, comparing and reporting how scattered media calibration affects existing classical lensless imaging analyses is essential. The PSF calibration also involves our physics-informed DL approaches. Therefore, all physics-based models are tested with clear and scattered calibration assumptions. Table 1 presents the full image reconstruction report of our computational datasets where L1 loss error and SSIM similarity index measure the prediction accuracy of the test sets.
Table 1. Clear and scattering tissue reconstruction report
Parameter tuning and the number of iterations affect the image quality in classical models. The ADMM and the FISTA methods were thoroughly tested to determine the best parameters for each implementation. We optimized these models based on the introduced metrics and visual comparisons to preserve more information after reconstruction. Although the classical ADMM and the FISTA models perform well in the clear media recording, their performance in scattering media recordings severely worsens, particularly once calibrated in the clear medium. Developing calibration measurements in scattering media highly assists scattering media recordings in classical approaches.
Unlike lensless photography applications, the sparse nature of the fluorescence datasets and the field-varying nature of the FFM limit the performance of the UNet backward model, particularly in clear media recording. In scattering media recording, it operates close to the best of ADMM and FISTA models with around 14× faster reconstruction time than ADMM, making this model an excellent solution for real-time reconstruction applications.
The dADMM and dADMM+ models with state-of-the-art results outperform all other techniques in both propagation scenarios. These models have achieved almost identical reconstruction quality for clear media recordings with average structural similarity of $0.995$ and $0.983$. They have scattering media performances close to the best of other models in the clear media application, with a structural similarity of $0.861$ and 0.887, respectively. The dADMM+ still has the time advantage of ADMM over FISTA while compensating for scattering media behavior better than other models. The dADMM and dADMM+ show an excellent performance when the system is not calibrated for the scattering profile of the medium compared with the classical models.
Image comparison metrics do not always align well with human visual perceptual metrics. Accordingly, a visual report in Fig. 6 presents the image reconstruction of all models in both propagation scenarios. The presented test images are chosen to show the performance of the models for different label patterns. Both calibration strategies are depicted in scattering media reconstruction for all models except the UNet, which uses no calibration information. Similar to the metrics’ suggestion, ADMM-assisted DL approaches result in the best reconstructions.
Fig. 6. A visual reconstruction report of the clear and scattering media recordings. The microscope’s measurements and labels (ground truths) are shown on the left and right columns. The computational reconstructions of the presented models are illustrated in the middle part. Scattering media recording is reported for clear and scattered calibration scenarios. The calibration status does not apply to the UNet model since it does not require any calibrations.
5. Discussion
Physics-informed DL models could significantly increase FFM performance in turbid medium recordings with a fast reconstruction time, even with clear medium calibration. On the other hand, the classical models showed high noise levels when working based on the clear media calibration. The main limitation of the supervised DL approaches is the need for prior training data. Therefore, the stability of ADMM-based DL models was examined with different training data sizes. Besides, we tested the dADMM+ model in an unsupervised setup to assess the potential of untrained network architectures in a field-varying microlens-based FFM application.
5.1 Calibration in scattering vs. clear media
All reviewed flat or mask-based fluorescence microscopy studies have used clear tissue PSFs when reconstructing scattering tissue recordings. Clear tissue calibration is initially preferred because it provides the system response regardless of the unpredicted changes in different imaging applications. Second, scattering calibration requires a scattering profile assumption of an imaging application. But, limiting an FFM application only to a particular scattering profile is advisable when it significantly contributes to the imaging quality, especially for in-vivo applications, where the FFMs are designed to operate in a known scattering media.
We tested clear and scattered PSF calibrations in turbid medium imaging with classical and physics-informed DL models. Our results show that the scattered PSFs calibration improves the image reconstruction quality in both tested iterative models by a significant margin. The scattered PSFs, as expected, better represent the scattering phenomena and strengthen the ADMM and FISTA models. This analysis answers the raised question in Section 2.3: scattered media calibration helps scattered media image reconstruction. The dADMM and dADMM+ models can denoise the ADMM-reconstructed labels regardless of the calibration strategy. These DL models learn and remove the scattering effect while training on the provided dataset.
Therefore, we suggest that mask-based microscopes developed for fluorescence imaging within scattering media can be improved by providing a scattering media calibration or dataset development through DL approaches.
5.2 What amount of training data is required?
Supervised learning methods are highly dependent on the training data quantity and compatibility with the reality of the application. In almost all DL applications, the short answer to the raised question is the more, the better. Developing a comprehensive dataset for all possible applications of FFM requires extensive experiments and resources. However, mass production of a low-cost and implantable FFM can justify the dedicated effort for training data development. Meanwhile, application-specific projects can benefit from supervised deep learning reconstruction with a feasible training data preparation process. Studies suggest that fluorescence imaging requires less training data than other DL applications [63].
Physics-informed deep learning models have a much faster training cycle and decrease performance dependency on the data size. We have quantitively analyzed how the training set size affects the performance of dADMM and dADMM+, respectively. The train set size effect was studied with 9, 90, and 900 data. The L1 and SSIM metrics are reported in Fig. 7 based on the train dataset size for the scattering and clear media datasets. Reduction of the training data size results in uncertainty in the network performance. Accordingly, we randomly selected ten different subsets for each implementation to report the performance with varying training inputs.
Fig. 7. Analysis of training data size. The dADMM and dADMM+ models were trained 10× with 9,90 and 900 train data. Clear media imaging with dADMM works well with 900 train data and has an equally lower average for 90 and 9 train sizes. The scattering media recording with scattered PSFs shows high-quality reconstruction with 900 data and still has reliable performance with 90 and 9 data assigned for training.
Although decreasing training data size increases the L1 error and reduces the SSIM measure, even with 10× less data size ($900$), its performance is relatively close to the initial $9000$ data size. The smaller dataset sizes still have an advantage over classical models, especially when the system is only calibrated for clear imaging. This notion approves our expected behavior of physics-informed models in less training data dependency. Besides, the results show that manual image annotation techniques can significantly enhance scattering media imaging by providing a few manually corrected ADMM labels. Therefore, experimental application-specific FFM potentially would not require a huge training data size to be leveraged by the physics-informed supervised learning models.
5.3 Leveraging unsupervised deep learning in FFMs
Novel networks and training strategies can help reduce the FFM training data dependency. Untrained deep network (UDN) imaging is a novel unsupervised method for image reconstruction in applications such as lensless photography and lensless phase imaging [34,62]. UDN models are slower than supervised models since they require thousands of iterations while training for each data individually. Physics-informed UDN models have a similar structure to the developed dADMM+ (Fig. 5(b)), consisting of a forward model that compares the generated data with the actual measurement. The label loss of the dADMM+ model should be zeroed in untrained implementation. Therefore, the model never observes the actual label while training.
We tested several configurations to find a stable UDN model capable of improving FFM datasets. Our tests showed that networks such as UNet and ResNet fail in consistent image enhancement in untrained implementations. This can happen because of the sparsity in fluorescence labels (and microlens patterns in clear tissue), the field-varying structure of the imager, and the nature of CNN-based models. The UDN model only successfully reconstruct a few of our data. Figure 8 reports the operation results of the untrained dADMM+ model.
Fig. 8. The operation examples of a successful and a failed untrained dADMM+ reconstruction in scattering tissue.
Our tests show that the UDN methods, unlike the supervised DL models, are less robust for FMM applications and need further investigation. We suggest that using networks capable of dealing with sparsity or adding a feature extractor network before the primary network can overcome this problem in the following developments.
6. Conclusion
To the best of our knowledge, this work is the first study investigating the performance of deep learning-based image reconstruction in scattering media in a field-varying PSF system. This work has demonstrated deep learning image reconstruction models capable of dealing with the rarely explored tissue scattering challenge in flat fluorescence microscopes (FFMs). Our hybrid ray-tracing and Monte Carlo microscope models have been used to compare different PSF calibration strategies in the scattering tissues for the first time. We examined the system calibration in scattering medium and recommended that scattered PSFs calibration improves image reconstruction compared with clear medium calibration. The dADMM and dADMM+ have taken advantage of the physics of the system to achieve a fast and robust reconstruction compared with the classical models, capable of state-of-the-art scattering media imaging even without scattering media calibration. We show that the physics-informed DL architectures perform well in applications with limited training data. Besides, we explored the potential and pitfalls of unsupervised implementations of deep network approaches in the FFM, which is critical for applications where dataset generation is impractical. These results strongly suggest that deep learning-empowered FFMs can play a significant role in the future of miniaturized microscopy to pave the way for the realization of an implantable microscopy system. We plan to experimentally develop an application-specific FFM to investigate the advantages and challenges of different physics-informed deep network models in image reconstruction.
Funding
Natural Sciences and Engineering Research Council of Canada (RGPIN-2020-06361, RGPIN-2022-03984); CMC Microsystems.
Acknowledgments
M.F. thanks Alireza Geravand for their valuable comments. The authors would like to thank CMC Microsystems for providing the design and testing tools. The authors acknowledge support from the Canada Research Chair in Smart Biomedical Microsystems, NSERC operating grants to B. G. and Y. D. K., SMAART NSERC Collaborative Research and Training Experience (CREATE) Program, Microsystems Strategic Alliance of Quebec (ReSMiQ) Doctoral Scholarship program, and Brockhouse Canada Prize.
Disclosures
The authors declare no conflicts of interest related to this article.
Data availability
Modeling files underlying the results presented in this work will be available at [64]. The datasets also can be shared upon reasonable requests.
Supplemental document
See Supplement 1 for supporting content.
References
1. L. Thibon, L. E. Lorenzo, M. Piché, and Y. De Koninck, “Resolution enhancement in confocal microscopy using Bessel-Gauss beams,” Opt. Express 25(3), 2162 (2017). [CrossRef]
2. F. Ferrini, J. Perez-Sanchez, S. Ferland, L. E. Lorenzo, A. G. Godin, I. Plasencia-Fernandez, M. Cottet, A. Castonguay, F. Wang, C. Salio, N. Doyon, A. Merighi, and Y. De Koninck, “Differential chloride homeostasis in the spinal dorsal horn locally shapes synaptic metaplasticity and modality-specific sensitization,” Nat. Commun. 11(1), 3935 (2020). [CrossRef]
3. K. K. Ghosh, L. D. Burns, E. D. Cocker, A. Nimmerjahn, Y. Ziv, A. El Gamal, and M. J. Schnitzer, “Miniaturized integration of a fluorescence microscope,” Nat. Methods 8(10), 871–878 (2011). [CrossRef]
4. O. Skocek, T. Nöbauer, L. Weilguny, F. Martínez Traub, C. N. Xia, M. I. Molodtsov, A. Grama, M. Yamagata, D. Aharoni, D. D. Cox, P. Golshani, and A. Vaziri, “High-speed volumetric imaging of neuronal activity in freely moving rodents,” Nat. Methods 15(6), 429–432 (2018). [CrossRef]
5. A. de Groot, B. J. G. G. van den Boom, R. M. van Genderen, J. Coppens, J. van Veldhuijzen, J. Bos, H. Hoedemaker, M. Negrello, I. Willuhn, C. I. De Zeeuw, and T. M. Hoogland, “NINscope, a versatile miniscope for multi-region circuit investigations,” eLife 9, e49987 (2020). [CrossRef]
6. M. N. Khiarak, E. Martianova, C. Bories, S. Martel, C. D. Proulx, Y. De Koninck, and B. Gosselin, “A Wireless Fiber Photometry System Based on a High-Precision CMOS Biosensor with Embedded Continuous-Time Σ Δ Modulation,” IEEE Trans. Biomed. Circuits Syst. 12(3), 495–509 (2018). [CrossRef]
7. V. K. Lazarjan, A. B. Gashti, M. Feshki, A. Garnier, and B. Gosselin, “Miniature Fiber-Spectrophotometer for Real-Time Biomarkers Detection,” IEEE Sensors J. 21(13), 14822–14837 (2021). [CrossRef]
8. M. Feshki, M. S. Monfared, and B. Gosselin, “Development of a Dual-Wavelength Isosbestic Wireless Fiber Photometry Platform for Live Animals Studies,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC (2020). [CrossRef]
9. D. Aharoni and T. M. Hoogland, “Circuit investigations with open-source miniaturized microscopes: Past, present and future,” Front. Cell. Neurosci. 13(April), 1–12 (2019). [CrossRef]
10. “Miniscope V4,” http://miniscope.org/index.php/Miniscope_V4.
11. C. Oh, S. O. Isikman, B. Khademhosseinieh, and A. Ozcan, “On-chip differential interference contrast microscopy using lensless digital holography,” Opt. Express 18(5), 4717 (2010). [CrossRef]
12. B. Khademhosseinieh, G. Biener, I. Sencan, and A. Ozcan, “Lensfree color imaging on a nanostructured chip using compressive decoding,” Appl. Phys. Lett. 97(21), 211112 (2010). [CrossRef]
13. G. Kuo, N. Antipa, R. Ng, and L. Waller, “3D fluorescence microscopy with diffusercam,” in Imaging and Applied Optics 2018, OSA Technical Digest (Optica Publishing Group, 2018), paper CM3E.3.
14. J. K. Adams, V. Boominathan, B. W. Avants, D. G. Vercosa, F. Ye, R. G. Baraniuk, J. T. Robinson, and A. Veeraraghavan, “Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope,” Sci. Adv. 3(12), 1–10 (2017). [CrossRef]
15. G. Kuo, F. Linda Liu, I. Grossrubatscher, R. Ng, and L. Waller, “On-chip fluorescence microscopy with a random microlens diffuser,” Opt. Express 28(6), 8384 (2020). [CrossRef]
16. A. Greenbaum, W. Luo, T. W. Su, Z. Göröcs, L. Xue, S. O. Isikman, A. F. Coskun, O. Mudanyali, and A. Ozcan, “Imaging without lenses: Achievements and remaining challenges of wide-field on-chip microscopy,” Nat. Methods 9(9), 889–895 (2012). [CrossRef]
17. O. Mudanyali, D. Tseng, C. Oh, S. O. Isikman, I. Sencan, W. Bishara, C. Oztoprak, S. Seo, B. Khademhosseini, and A. Ozcan, “Compact, light-weight and cost-effective microscope based on lensless incoherent holography for telemedicine applications,” Lab Chip 10(11), 1417–1428 (2010). [CrossRef]
18. W. Bishara, T.-W. Su, A. F. Coskun, and A. Ozcan, “Lensfree on-chip microscopy over a wide field-of-view using pixel super-resolution,” Opt. Express 18(11), 11181 (2010). [CrossRef]
19. G. Zheng, S. A. Lee, Y. Antebi, M. B. Elowitz, and C. Yang, “The ePetri dish, an on-chip cell imaging platform based on subpixel perspective sweeping microscopy (SPSM),” Proc. Natl. Acad. Sci. 108(41), 16889–16894 (2011). [CrossRef]
20. X. Cui, L. M. Lee, X. Heng, W. Zhong, P. W. Sternberg, D. Psaltis, and C. Yang, “Lensless high-resolution on-chip optofluidic microscopes for Caenorhabditis elegans and cell imaging,” Proc. Natl. Acad. Sci. 105(31), 10670–10675 (2008). [CrossRef]
21. Y. Wu and A. Ozcan, “Lensless digital holographic microscopy and its applications in biomedicine and environmental monitoring,” Methods 136, 4–16 (2018). [CrossRef]
22. M. S. Asif, A. Ayremlou, A. Sankaranarayanan, A. Veeraraghavan, and R. G. Baraniuk, “FlatCam: Thin, Lensless Cameras Using Coded Aperture and Computation,” IEEE Trans. Comput. Imaging 3(3), 384–397 (2017). [CrossRef]
23. N. Antipa, G. Kuo, R. Heckel, B. Mildenhall, E. Bostan, R. Ng, and L. Waller, “DiffuserCam: lensless single-exposure 3D imaging,” Optica 5(1), 1 (2018). [CrossRef]
24. V. Boominathan, J. K. Adams, J. T. Robinson, and A. Veeraraghavan, “PhlatCam: Designed Phase-Mask Based Thin Lensless Camera,” IEEE Trans. Pattern Anal. Mach. Intell. 42(7), 1618–1629 (2020). [CrossRef]
25. A. K. Singh, G. Pedrini, M. Takeda, and W. Osten, “Scatter-plate microscope for lensless microscopy with diffraction limited resolution,” Sci. Rep. 7(1), 10687 (2017). [CrossRef]
26. K. Yanny, N. Antipa, W. Liberti, S. Dehaeck, K. Monakhova, F. L. Liu, K. Shen, R. Ng, and L. Waller, “Miniscope3D: optimized single-shot miniature 3D fluorescence microscopy,” Light: Sci. Appl. 9(1), 171–7538 (2020). [CrossRef]
27. J. K. Adams, D. Yan, J. Wu, V. Boominathan, S. Gao, A. V. Rodriguez, S. Kim, J. Carns, R. Richards-Kortum, C. Kemere, A. Veeraraghavan, and J. T. Robinson, “In vivo lensless microscopy via a phase mask generating diffraction patterns with high-contrast contours,” Nat. Biomed. Eng. 6(5), 617–628 (2022). [CrossRef]
28. G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, “Deep Learning Techniques for Inverse Problems in Imaging,” IEEE J. Sel. Areas Inf. Theory 1(1), 39–56 (2020). [CrossRef]
29. Y. Chen, Y. Zhao, W. Jia, L. Cao, and X. Liu, “Adversarial-learning-based image-to-image transformation: A survey,” Neurocomputing 411, 468–486 (2020). [CrossRef]
30. K. De Haan, Y. Rivenson, Y. Wu, and A. Ozcan, “Deep-Learning-Based Image Reconstruction and Enhancement in Optical Microscopy,” Proc. IEEE 108(1), 30–50 (2020). [CrossRef]
31. J. D. Rego, K. Kulkarni, and S. Jayasuriya, “Robust lensless image reconstruction via PSF Estimation,” in Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, WACV 2021 (2021), pp. 403–412.
32. H. Zhou, H. Feng, Z. Hu, Z. Xu, Q. Li, and Y. Chen, “Lensless cameras using a mask based on almost perfect sequence through deep learning,” Opt. Express 28(20), 30248 (2020). [CrossRef]
33. K. Monakhova, J. Yurtsever, G. Kuo, N. Antipa, K. Yanny, and L. Waller, “Learned reconstructions for practical mask-based lensless imaging,” Opt. Express 27(20), 28075 (2019). [CrossRef]
34. K. Monakhova, V. Tran, G. Kuo, and L. Waller, “Untrained networks for compressive lensless photography,” Opt. Express 29(13), 20913 (2021). [CrossRef]
35. Y. Li, Y. Xue, and L. Tian, “Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media,” Optica 5(10), 1181 (2018). [CrossRef]
36. M. Lyu, H. Wang, G. Li, S. Zheng, and G. Situ, “Learning-based lensless imaging through optically thick scattering media,” Adv. Photonics 1(03), 1 (2019). [CrossRef]
37. K. Yamazaki, R. Horisaki, and J. Tanida, “Imaging through scattering media based on semi-supervised learning,” Appl. Opt. 59(31), 9850 (2020). [CrossRef]
38. S. Zhu, E. Guo, J. Gu, L. Bai, and J. Han, “Imaging through unknown scattering media based on physics-informed learning,” Photonics Res. 9(5), B210 (2021). [CrossRef]
39. S. Lim and J. C. Ye, “Blind Deconvolution Microscopy Using Cycle Consistent CNN with Explicit PSF Layer,” in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (2019), pp. 173–180.
40. A. Qayyum, I. Ilahi, F. Shamshad, F. Boussaid, M. Bennamoun, and J. Qadir, “Untrained Neural Network Priors for Inverse Imaging Problems: A Survey,” IEEE Trans. Pattern Anal. Mach. Intell. 45(5), 1–20 (2022). [CrossRef]
41. I. Hassaninia, R. Bostanabad, W. Chen, and H. Mohseni, “Characterization of the Optical Properties of Turbid Media by Supervised Learning of Scattering Patterns,” Sci. Rep. 7(1), 15259 (2017). [CrossRef]
42. G. Carles, P. Zammit, and A. R. Harvey, “Holistic Monte-Carlo optical modelling of biological imaging,” Sci. Rep. 9(1), 15832 (2019). [CrossRef]
43. M. Kim, J. An, K. S. Kim, M. Choi, M. Humar, S. J. J. Kwok, T. Dai, and S. H. Yun, “Optical lens-microneedle array for percutaneous light delivery,” Biomed. Opt. Express 7(10), 4220 (2016). [CrossRef]
44. Y. Xue, Q. Yang, G. Hu, K. Guo, and L. Tian, “Deep-learning-augmented computational miniature mesoscope,” Optica 9(9), 1009 (2022). [CrossRef]
45. K. Sasagawa, A. Kimura, M. Haruta, T. Noda, T. Tokuda, and J. Ohta, “Highly sensitive lens-free fluorescence imaging device enabled by a complementary combination of interference and absorption filters,” Biomed. Opt. Express 9(9), 4329 (2018). [CrossRef]
46. K. Yanny, N. Antipa, R. Ng, and L. Waller, “Miniature 3D fluorescence microscope using random microlenses,” in Biophotonics Congress: Optics in the Life Sciences Congress (2019), (Optica Publishing Group), paper BT3A.4.
47. C. F. Bohren, Absorption and Scattering of Light by Small Particles (Wiley,1983).
48. K. J. Sekiguchi, P. Shekhtmeyster, K. Merten, A. Arena, D. Cook, E. Hoffman, A. Ngo, and A. Nimmerjahn, “Imaging large-scale cellular activity in spinal cord of freely behaving mice,” Nat. Commun. 7(1), 11450 (2016). [CrossRef]
49. A. Klioutchnikov, D. J. Wallace, J. Sawinski, K. M. Voit, Y. Groemping, and J. N. D. Kerr, “A three-photon head-mounted microscope for imaging all layers of visual cortex in freely moving mice,” Nat. Methods 20(4), 610–616 (2023). [CrossRef]
50. L. G. Henyey and J. L. Greenstein, “Diffuse radiation in the galaxy,” Nature 93(3733), 70 (1941). [CrossRef]
51. W. M. Cornette and J. G. Shanks, “Physically reasonable analytic expression for the single-scattering phase function,” Appl. Opt. 31(16), 3152 (1992). [CrossRef]
52. C. Edlund, T. R. Jackson, N. Khalid, N. Bevan, T. Dale, A. Dengel, S. Ahmed, J. Trygg, and R. Sjögren, “LIVECell—A large-scale dataset for label-free live cell segmentation,” Nat. Methods 18(9), 1038–1045 (2021). [CrossRef]
53. J. Deng, W. Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “ImageNet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 248–255.
54. G. Kuo, N. Antipa, R. Ng, and L. Waller, “DiffuserCam: Diffuser-based lensless cameras,” in In Computational Optical Sensing and Imaging (2017), (Optica Publishing Group), paper CTu3B-2.
55. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009). [CrossRef]
56. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn. 3(1), 1–122 (2010). [CrossRef]
57. J. Wu, L. Cao, and G. Barbastathis, “DNN-FZA camera: a deep learning approach toward broadband FZA lensless imaging,” Opt. Lett. 46(1), 130 (2021). [CrossRef]
58. O. Kingshott, N. Antipa, E. Bostan, and K. Akşit, “Unrolled primal-dual networks for lensless cameras,” Opt. Express 30(26), 46324 (2022). [CrossRef]
59. O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (2015), 9351, pp. 234–241.
60. T. Zeng and E. Y. Lam, “Robust Reconstruction with Deep Learning to Handle Model Mismatch in Lensless Imaging,” IEEE Trans. Comput. Imaging 7, 1080–1092 (2021). [CrossRef]
61. M. Feshki, Y. De Koninck, and B. Gosselin, “Deep Learning Empowered Fresnel-based Lensless Fluorescence Microscopy,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, (to be published).
62. F. Zhang, X. Liu, C. Guo, S. Lin, J. Jiang, and X. Ji, “Physics-based Iterative Projection Complex Neural Network for Phase Retrieval in Lensless Microscopy Imaging,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2021), pp. 10518–10526.
63. C. Belthangady and L. A. Royer, “Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction,” Nat. Methods 16(12), 1215–1225 (2019). [CrossRef]
64. M. Feshki, S. Martel, Y. De Koninck, and B. Gosselin, “Deep learning empowered flat fluorescence microscopy,” https://github.com/mohamadfeshki.
