Open Access
Add to BibSonomy
Abstract
Effectively imaging through dynamic scattering media is of great importance and challenge. Some imaging methods based on physical or learning models have been designed for object reconstruction. However, with an increase in exposure time or more drastic changes in the scattering medium, the speckle pattern superimposed during camera integration time undergoes more significant changes, resulting in a modification of the collected speckle structure and increased blurring, which brings significant challenges to the reconstruction. Here, the clearer structural information of blurred speckles is unearthed with a presented speckle de-blurring algorithm, and a high-throughput imaging method through rapidly changing scattering media is proposed for reconstruction under long exposure. For the problem of varying blur degrees in different regions of the speckle, a block-based method is proposed to divide the speckle into distinct sub-speckles, which can realize the reconstruction of hidden objects. The imaging of hidden objects with different complexity through dynamic scattering media is demonstrated, and the reconstruction results are improved significantly for speckles with different blur degrees, which verifies the effectiveness of the method. This method is a high-throughput approach that enables non-invasive imaging solely through the collection of a single speckle. It directly operates on blurred speckles, making it suitable for traditional speckle-correlation methods and deep learning (DL) methods. This provides a new way of thinking about solving practical scattering imaging challenges.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Imaging through scattering media is crucial to many practical applications and becomes one of the focuses in optical imaging fields [1–5]. After the random modulation by scattering media, the object information carried by the light beam will be seriously degraded, and the light beam will be diffused into speckles without original spatial structure information [6–8]. Over the years, several methods have been proposed for imaging through scattering media, and the typical techniques include wavefront shaping methods [9–11], transmission matrix (TM) methods [12,13], point spread function (PSF) methods [14–16], speckle-correlation methods [17–20], and deep learning (DL) methods [21–24]. The scattering medium is variable in most practical scenarios, and its dynamic characteristics will lead to the rapid decorrelation of speckles carrying object information in time. Therefore, the speckle information obtained by integral in exposure time is blurred, which greatly increases the difficulty of imaging [7].
It is challenging to reconstruct hidden objects in a dynamic scattering environment, and some methods have been proposed to recover degraded objects. The image reconstruction can be realized by calculating the difference between two frames of speckles recorded in sequence [25]. However, the experimental results offered by this approach are relatively scarce, warranting further validation of its reconstruction capabilities for rapidly changing scattering media. On a different front, after capturing multiple speckle patterns of blinking point sources, a stochastic optical scattering localization imaging (SOSLI) technology is proposed for object restoration [26]. While this method can improve imaging resolution, its suitability is limited in rapidly changing scattering environments. In order to achieve imaging through rapidly changing scattering media, the imaging model of the shower-curtain effect is introduced into the speckle correlation technology, and the static object can be reconstructed [27]. However, it should be noted that this method is based on the Fourier-domain shower-curtain effect, which needs to meet the far-field condition under speckle illumination and is not suitable for scenes where the object is close to the scattering medium.
The DL methods have great capability in data mining and optimal solution for the inverse problems [22–24,28]. Consequently, researchers have explored the application of neural networks to address the challenge of object reconstruction in dynamic scattering scenes. Using the DL method to classify speckles based on the scattering degrees can recover objects hidden behind the media successfully [29]. Nevertheless, this method solely relies on data-driven approaches and lacks the incorporation of relevant physical priors. Therefore, it necessitates a substantial volume of data to enhance the network’s generalization capabilities [30,31]. An adaptive inverse mapping method is introduced, enabling the correction of the inverse mapping of dynamic scattering media through unsupervised learning [32]. However, the method requires the collection of $10^{3}-10^{4}$ speckles to correct the inverse mapping, and the acquisition process requires a quasi-static scattering system. Therefore, the speckle image acquisition time should be significantly shorter than the speckle decorrelation time, which restricts the practical application of this method.
If the scattering medium is changing during the speckle collection process, the formed speckle patterns will also vary accordingly. The superposition of different speckles within the camera integration time results in a blurred speckle pattern [7,33,34]. In light of the optical memory effect (OME) theory, when the exposure time is shorter than the speckle decorrelation time, the speckle can be considered translation invariant, and the collected speckle can be called motion blurred speckle [17,18,20].
In actual dynamic scattering scenarios, the scattering media change randomly, leading to varying degrees of blur degradation in different regions of the speckle. In this case, it is not appropriate to process the entire speckle. In addition, the quality of captured speckles is influenced by various factors, e.g., the exposure time and the velocity of change in the scattering medium. If the exposure time is longer or the changing speed of the scattering media is faster, the superimposed speckle changes during the camera integration time will be more pronounced. Therefore, the collected speckle is blurrier, and the degradation of object information is more serious. While reducing the exposure time can mitigate the degree of speckle blurring, it also results in a decrease in the total luminous flux. If the flux is low, the detector may struggle to respond to sufficient information, leading to a reduced amount of captured speckle information. In high-flux conditions, the detector can effectively respond to an adequate amount of information, thereby obtaining complete structural information of the object. Moreover, even with reduced exposure time, rapid changes in the scattering medium can still result in the collection of blurred speckles. Therefore, to achieve the high-quality reconstruction of hidden objects in dynamic scattering environments, it is necessary to solve the problem of speckle structure blurring caused by rapidly changing scattering media.
In this paper, an effective high-throughput method is proposed, which can extract clearer structural information from blurred speckles and achieve high-fidelity imaging of objects with different complexity through dynamic scattering media. To address the issue of varying degrees of blur degradation in different speckle regions within actual dynamic scattering environments, a block-based approach is employed to partition the speckle into multiple regions for individual reconstruction. Patch-wise minimal pixel (PMP) is an obvious difference between blurred speckle and clear speckle [34]. Employing a blind anti-motion blur (BAMB) algorithm with PMP priori to de-blur the collected speckle can enhance the speckle structure information, thereby improving the reconstruction result of the object. Using the relevant physical priors of the actual scattering scene, the generalization and reconstruction ability of the DL model under small data volume can be effectively enhanced. Furthermore, the high-throughput imaging method proposed in this paper can restore blurred speckles into speckles with clear structural information. Therefore, this method can alleviate the limitation of other speckle imaging approaches on the change speed of scattering medium and expand their effectiveness in dynamic scattering scenes.
2. Method
2.1 Physical basic of the high-throughput method based on speckle de-blurring
Feng et al. deduce and experimentally verify the existence of the OME [17,18]. The obtained speckles are invariant to small tilts or shifts in an incident light, and the outgoing light field still retains some information carried by the incoming beam [20,35,36]. In the dynamic scattering environment, the dynamic characteristics of the scattering medium lead to the decorrelation of the speckle carrying the object information. The change speed and scattering times of the scattering medium; the storage time, temperature, and experimental duration of the actual living biological tissue will affect the decorrelation time. Within the OME range and decorrelation time, the scattering system can be considered as an imaging system with shift-invariant PSF [25,37,38].
As shown in Fig. 1, the imaging object is a number of "3", the speckles through a rotating scattering medium with different speeds and exposure times are collected, and the autocorrelation and reconstruction results are calculated. The peak signal-to-noise ratio (PSNR) is calculated as the quantitative indicator using the autocorrelation of the first row in Fig. 1(a) as a reference, and the results are marked in the lower right corner of autocorrelations. It can be found that under low exposure, the object information collected by the camera is insufficient, and the speckle autocorrelation structure is missing. When the rotational speed of the scattering medium is low (5 r/min), the complete autocorrelation structure can be calculated by increasing the exposure time to 1 ms, and the speckle ambiguity is low. In the speckle decorrelation time, when the exposure time increases, the camera superimposes more moving speckles, the superimposed speckle changes become more pronounced, and the blur degree of the speckle increases. The blur degree of speckles collected at 1ms exposure time increases at higher rotational speeds, and the calculated autocorrelations structural features overlap, which makes it difficult to achieve accurate reconstructions of objects based on such blurred speckles. Therefore, in order to break through the limitation of dynamic imaging on the rotational speed of the scattering medium, it is necessary to solve the problem of blur speckle imaging.
Fig. 1. Imaging through a rotating scattering medium. (a) Speckles, autocorrelations and reconstruction results in a static scattering environment. (b) The speckles and the corresponding reconstruction results under different exposure times. DC, decorrelation time. The exposure times and the corresponding DC are marked in the lower right corner of the reconstruction results. (c) The speckles and the corresponding reconstruction results with different rotating speeds. The rotating speeds and the corresponding DC are marked in the lower right corner of the reconstruction results.
The motion of scattering media will lead to the blurring of the speckle. Within the speckle decorrelation time, the camera accumulates shifting speckle patterns, leading to a blurring effect analogous to motion blur. Upon surpassing the speckle decorrelation time, the camera captures speckle patterns with larger variations, resulting in a more irregular and unpredictable blurring. The foundation of this paper is speckle correlation techniques, necessitating the fulfillment of the requirement that the exposure time remains within the speckle decorrelation time.
The blurred image $B$ has a significant feature as the ’smearing’ of the scattering particles, which is manifested in the smoothness of the pixels on the speckle pattern. Let an image $B_p$ be divided into $P$ non-overlapped patches with a patch size of $r \times r$, for which $P = [\frac {v}{r}\cdot \frac {w}{r}]$, $v$ and $w$ are the size of the image $B_p$. PMP is the collection of local minimal pixels over non-overlapping patches, which can be defined as:
where $i = 1,2,\ldots,P$ , and $\Omega _i$ denotes the index set of the pixel locations of the $i$-th patch.The blurring process has a smoothing effect on the image pixels, consequently leading to an increase in the intensity of a local minimal pixel. As a result, the PMP of clear speckles is much sparser than those of blurred speckles. This sparsity property of PMP provides a natural criterion for distinguishing between clear and blurred speckles. Capitalizing on this feature, the BAMB algorithm is proposed to de-blur the collected speckle patterns, and the flow chart of the algorithm is shown in Fig. 2(b). In the de-blurring process, PMP is sparsely induced to obtain more accurate blur kernels and object estimation results. If the speckle is highly correlated during the exposure time, the speckle has translation invariance, and the BAMB algorithm can be used for de-blurring [20,35,37].
Fig. 2. The schematic diagram of high-throughput imaging method based on speckle de-blurring in the dynamic scattering environment. (a) The schematic diagram of the imaging method. (b) Speckle de-blurring step. (c) Speckle-correlation step.
De-burring processing is performed on a single ’motion blurred’ speckle pattern. It is assumed that the blurring degree of different speckle regions is the same since the speckle has spatial translation invariance, the blur process can be modeled as a convolution operation [33], and the blurred speckle pattern $B$ captured by the camera can be expressed as:
where $k$ , $I$ , $m$ , and $\ast$ respectively denote blur kernel, latent speckle pattern, additive noise, and convolution operator.The de-blurring problem within the maximum a posteriori (MAP) framework can be represent as [39]:
where $p(k)$ and $p(I)$ are the priors of the blur kernel and the latent image, respectively. $\gamma$ and $\mu$ are the corresponding weight parameters. The first term in Eq. (3) is the data fidelity term, which restricts $I\ast k$ to be consistent with the blurred image $B$.The sparse characteristics of PMP can effectively distinguish between clear speckle and blurred speckle. In this paper, the sparsity regularization of PMP is combined into the conventional MAP framework, and the objective function for image de-blurring can be represented as [34,40,41]:
We use an alternating minimization algorithm based on the half-quadratic splitting algorithm [42]. While keeping one parameter fixed, the intermediate latent image $I$ and the blur kernel $k$ are estimate iteratively as follows:
andAfter several rounds of iterations, the ultimately estimated blur kernel is obtained. Finally, utilizing the blurred image $B$ and blur kernel $k$, the speckle pattern can be estimated with total variation (TV) prior. The above is the principle of the BAMB algorithm. The BAMB algorithm draws inspiration from the frameworks of de-blurring algorithms [34]. Building upon this foundation, it incorporates preprocessing steps for the input speckle data, such as normalization and grayscale stretching. Additionally, it changes the constraint on the data fidelity term in Eq. (4), making the modified algorithm better suited for speckle data.
The high-throughput method requires the speckle with uniform blurring, however, the actual dynamic scattering media (such as onion epidermis, chicken breast, turbid solution, fog) may be non-uniform and random, and the degree of blur degradation in different scattering regions will be different, it is not appropriate to apply the same level of de-blurring to the whole speckle pattern. In view of this situation, the method of block-based de-blurring is adopted. According to the blurring of the speckle pattern, the speckle is divided into multiple sub-speckles and then each sub-speckle is homogenized and de-blurred.
The collected whole speckle pattern $B$ can be expressed as a combination of stitching the sub-speckles together [43]:
where $\cup$ is a stitching operator, $n$ is the number of sub-speckles, $B_i$ is the $i$-th sub-speckle. As shown in Fig. 3, if the actual collected speckle is not uniform, it is homogenized to obtain a speckle with uniform intensity shown in Fig. 3(b). After that, the whole uniform speckle is divided into $n$ sub-speckles. According to Eq. (2), the blurred sub-speckle can be approximated as the convolution of the clear sub-speckle and the blur kernel, therefore, Eq. (7) can be written as:Fig. 3. Block-based processing of the non-uniform speckle collected through onion epidermis. (a) The non-uniform speckle collected through onion epidermis. (b) The uniform speckle obtained by homogenization pretreatment of Fig. 3(a). (c) The block-based processing. The whole speckle can be regarded as the splicing of n sub-speckles in space.
As shown in Fig. 2, the speckle collected is blurred in the dynamic scattering environment, and the reconstructed object is also blurred. The clear speckle pattern $I$ is obtained by the BAMB algorithm, which is close to the speckle pattern collected in the static scattering environment.
The relationship between speckle autocorrelation and object autocorrelation can be established based on speckle correlation method. According to the Wiener-Khinchin theorem, the Fourier amplitude information of the object can be deduced from its autocorrelation. The object image can then be reconstructed using hybrid input-output (HIO) algorithm [44,45].
The HIO algorithm has limited ability to reconstruct objects with high complexity, therefore, in section 2.2, a physics-aware DL framework is proposed. The object can be reconstructed only with HIO or only with DL method according to actual needs.
2.2 Framework of physics-aware learning
The solely data-driven neural networks have strong optimization ability. However, they do not combine the actual physical processes during training and learning, which easily lead to overfitting of existing data and restricted generalization ability of unknown scattering scenarios, and limited application in dynamic scattering environments. Although the speckle correlation methods based on physical model have limited reconstruction ability, the non-invasive imaging technology approach has better generalization ability for the scene. For the problem of object reconstruction through dynamic scattering media, the introduction of relevant physical prior information can effectively improve the generalization ability of the data model.
As shown in Fig. 2(a), a physically-aware DL framework is proposed for blurred speckles. By utilizing the physical priors with speckle-correlation and speckle de-blurring theory, the potential of neural networks can be fully exploited for object imaging through dynamic scattering media. The clear and uniform speckle patterns are obtained by processing the collected speckles, then the speckle autocorrelation $R(x,y)$ is calculated, the mathematical operations can be expressed as:
where the symbol $\bigstar$ denotes the autocorrelation operation [19], the symbol $\mathscr {F}\{\cdot \}$ denotes the Fourier transform, and the symbol $\mathscr {F}^{-1}\{\cdot \}$ denotes the inverse Fourier transform. Then the calculated speckle autocorrelation physical prior is introduced into the learning framework. After that, the intrinsic correlation mining and object reconstruction are carried out through the U-net neural network [30]. To train the learning models, an Adam optimizer is selected to optimize the convolution neural networks (CNN) weights in the training process. Each CNN is trained with 400 epochs by the Adam optimizer, the learning rate of 1E-4 is used for the first 300 epochs, and 1E-5 for the next 100 epochs. The CNN is performed on PyTorch 1.4.0 with a Titan RTX graphics unit and i9-9940X CPU under Ubuntu 16.04.2.3 Criteria for judging the clarity of speckle patterns
A quantitative evaluation method is used to judge the clarity of speckle patterns, which is called the gradient judgment method in this paper. Ambiguity and clarity are two relative concepts, and the two indicators are inversely proportional in value: as the image becomes sharper, the clarity value increases while the ambiguity value decreases.
The gradient judgment method first calculates the Laplacian operator to obtain the intensity values of each pixel in the speckle pattern, and then the median of the first 0.1% part of the intensity value is calculated to obtain the clarity value of the speckle. The Laplace operator is used to measure the second derivative of the image and can highlight areas of the image that contain rapid gradient changes [46,47]. If the variance of an image is high, then the image has a wide range of responses. However, if the variance is very low, there will be a small response spread, which indicates that there is almost no edge in the image. The edges of the image are fewer if the image is more blur. Therefore, the Laplace operator can be used to detect the blur degree of the image. The gradient judgment method is verified using speckle patterns with different blur degree.
3. Experimental results and analysis
3.1 Experimental configuration
The optical configuration designed for the speckle de-blurring imaging method is shown in Fig. 4. A pseudo-thermal spatially-incoherent source composed of a 650 nm laser and rotating diffuser (RD) is used as the illumination source. The beam is collimated by a collimating lens (CL) and transmitted through a total internal reflection (TIR) prism. The beam is then modulated by a DMD (pixel count: 1024$\times$768, pixel pitch: 13.68 $\mu$m) and carries object information. The pattern is obtained using an industrial complementary metal-oxide-semiconductor (CMOS) camera (MV-CA023-10UM, pixel count: 1920$\times$1200, pixel pitch: 5.86$\mu$m). Three different scattering media are employed, which include 220 grit ground glass, RD and onion epidermis. D1 represents ground glass with a stage that can be moved laterally, and D2 represents a rotating diffuser with speed control. D1 performs lateral motion, D2 performs rotational motion. D1 and D2 are used to simulate the change of dynamic scattering media to verify the effectiveness of the de-blurring algorithm. D3 signifies onion epidermis affixed to a glass slide, with pure water dropped onto the onion epidermis from a fixed height during experimentation. The distance between the target and the scattering medium is 500 mm, and the distance between the scattering medium and the camera is 100 mm.
Fig. 4. The experimental setup for imaging through dynamic diffusers. The speckle patterns of different scattering scenes are obtained by using different scattering media. DMD, digital micro-mirror device; TIR, total internal reflection; CL: collimating lens; RD, rotating diffuser.
The high-throughput imaging method based on speckle-correlation needs to ensure both OME conditions and decorrelation conditions. In order to measure the range of OME, a series of speckles are collected while the point object on the target surface is horizontally displaced with the step of one pixel. The cross-correlation coefficients between speckle patterns and the system PSF is calculated, and 0.5 is selected as the threshold of the cross-correlation coefficient to determine the range of OME. According to the measurement results and experimental parameters, the OME range of the system on the DMD is 120$\times$120 pixels [48–50]. The decorrelation time of speckles collected through the onion epidermis is measured to be approximately 130 s [51–54]. The variation of the scattering medium is accelerated when dropping pure water over the onion epidermis, leading to a significantly reduction in decorrelation time. Since the introduction of water droplets onto the onion epidermis induces continuously changing medium conditions, the decorrelation time of speckle patterns also undergoes continuous alterations. For the subsequent reconstruction process, the speckle patterns acquired within a certain time interval (approximately 1 s-1.5 s) after the water droplet impact are selected. During this interval, the estimated decorrelation time of the speckle patterns ranges from 20 ms to 100 ms.
The increase of the rotational speed $v$ of the scattering medium and the increase of the exposure time $t$ will increase the blur degree of the speckle. When the correlation coefficient between the speckle patterns falls below 0.5, the speckle patterns can be considered de-correlated [18,20]. When the translation speed of the scattering medium is set to 1 mm/s, 3 mm/s, and 5 mm/s, the corresponding decorrelation times between speckle patterns are 200 ms, 70 ms, and 45 ms, respectively. For scattering mediums rotating at 5 r/min and 20 r/min, the decorrelation times are 40 ms and 10 ms, respectively. The high-throughput imaging method in the paper requires a certain degree of correlation between the speckle patterns, the de-blurring effect is improved by a stronger correlation.
3.2 Imaging through dynamic scattering media
To demonstrate the effectiveness of the high-throughput method proposed in this paper, comprehensive experiments are provided and analyzed. By moving or rotating the static ground glass to simulate the change of biological tissue, the effectiveness of the imaging method and the threshold of imaging ability are explored. Then, the onion epidermis is used as the scattering medium to verify the effectiveness of the method in the actual dynamic scattering scene. Finally, the DL method based on physical perception is used to improve the imaging effect and verify that the method is suitable for objects with different complexity.
3.2.1 Imaging through scattering medium which moves transversely
When the scattering medium moves laterally, the speckle patterns under different moving speeds and exposure times are collected. The 400$\times$400 pixels of the speckle center is intercepted for de-blurring processing, and then the autocorrelation and object reconstruction results of the original speckle and the de-blurred speckle are calculated, the results are shown in Fig. 5. The first to fourth lines of Fig. 5(b) are the results under different exposure times when the movement speed of the scattering medium is 1mm/s. The second, fifth and sixth lines are the results of changing the movement speed of the scattering medium when the exposure time is 30ms. Figure 5(c) is the results obtained by de-blurring the speckles in Fig. 5(b). As shown in Fig. 5(b), the correlation coefficient is calculated by selecting two frames of speckles with the acquisition time interval corresponding to the speckle exposure time in Fig. 5(b), and the results are shown in Fig. 5(d). Based on the gradient judgment method, the clarity of the original speckle and the de-blurred speckle under six different speeds and exposure times in Fig. 5(b) and (c) are obtained, which are shown in Fig. 5(e). It should be noted that judging the clarity of the image is to verify the partial effect of the de-blurring method. The reconstruction quality of an image primarily depends on the speckle structure, and there is no direct correlation between speckle clarity and the quality of the reconstructed object.
Fig. 5. Object reconstruction through scattering medium moving transversely. (a) The speckles with their autocorrelations and reconstruction results collected in the static scattering environment. The upper left corner of the speckle is GT, and the left side of the image block diagram is the movement speed of the scattering medium. (b) Speckles collected in a dynamic scattering environment with their autocorrelations and reconstruction results. (c) De-blurred speckles with their autocorrelations and reconstruction results. The lower left corner of the speckle is the exposure time, the lower left corner of the reconstruction result is the PSNR between the reconstruction result and GT. (d) The correlation coefficients of speckles in Fig. 5(b). (e) The clarity value of speckles with six different speeds and exposure times in Fig. 5(b) and (c). The black line represents the clarity value of the raw speckles in Fig. 5(b), and the red line represents the clarity value of the de-blurred speckles in Fig. 5(c). Scale bar: 260$\mu$m.
As shown in Fig. 5, with the increase of exposure time, the speckles become more blurred, and there will be a more pronounced ’smearing’ phenomenon. The structure of speckle autocorrelation deteriorates, the reconstruction result of the HIO algorithm is blurred, and it is difficult to distinguish the approximate shape of the object. De-blurring the speckle pattern can reduce the motion blur of the speckle, resulting in a significant improvement in the clarity of the speckle compared to the raw pattern. However, if the blur degree between the raw speckles is very different, the de-blurred speckles will still with large difference blur degrees. The speckle autocorrelation structure of the first four rows in Fig. 5(c) is complete, and the approximate shape of the object can be reconstructed. However, with increasing speckle blur, although the reconstructed object shape remains distinguishable, there is more deviation in its structure. The PSNR between different reconstruction results and ground truth (GT) is presented in Fig. 5, it can be seen that the quality of the reconstructed results improves after the de-blurring operation on the speckle patterns.
The effectiveness of the high-throughput method is limited by the correlation of speckle patterns and the capability of the de-blurring reconstruction algorithm. The increase of exposure time or speed will lead to the decrease of speckle correlation, and effect of the de-blurring method will also decrease accordingly. If the correlation coefficient falls below 0.5, the precondition for the implementation of the de-blurring method is not satisfied, and the speckle cannot be de-blurred. As shown in Fig. 5(d), it is demonstrated that when the correlation coefficient is higher than 0.8, the reconstruction effect is satisfactory, but when the correlation coefficient is reduced to 0.7, the effect of the de-blurring method is significantly limited.
3.2.2 Imaging through scattering medium which moves rotationally
When the scattering medium is rotating, the speckle patterns under different moving speeds and exposure times are collected. The PSNR between different reconstruction results and GT is presented in Fig. 6, the speckle de-blurring method can improve the performance. For slightly blurred speckles, as shown in the first, second, and fifth lines of Fig. 6(b), direct imaging of blurred speckles can simply distinguish the objects, but the results of the objects have some structural defects. After using the de-blurring operation, the structural information of the reconstructed objects is well restored. For the speckle with high ambiguity, as shown in the third line of Fig. 6(b) and (c), it is difficult to obtain effective objects information by imaging the original speckles directly, the imaging results are blurred and the shapes of objects are difficult to distinguish. In this case, the de-blurring operation can reduce the rotational motion blur of speckles, and the structural information of the reconstructed objects is basically restored. Under the exposure time and rotation speed of the fourth and sixth lines of Fig. 6(b) and (c), the speckle correlation is less than 0.7, and the effect of the de-blurring method is significantly limited. Therefore, the reconstruction result can only barely distinguish the object, and the shape is missing or has a large deviation.
Fig. 6. Object reconstruction through rotating scattering medium. (a) The speckles with their autocorrelations and reconstruction results collected in the static scattering environment. (b) Speckles collected in dynamic scattering environment with their autocorrelation and reconstruction results. (c) De-blurred speckles with their autocorrelations and reconstruction results. (d) The correlation coefficients of speckles in Fig. 6(b). (e) The clarity value of speckles with six different speeds and exposure times in Fig. 6(b) and (c). Scale bar: 260$\mu$m.
The parameter settings for the BAMB algorithm corresponding to each blurred speckle pattern in Fig. 5 and Fig. 6 are summarized in Table 1. The de-blurring efficacy and strength of the blind deconvolution algorithm, BAMB, employed in this paper are influenced by the specified parameters. These parameters encompass the size of the scattering kernel, the overall iteration count, the weight parameters involved in the optimization process ($\gamma$, $\mu$, $\alpha$ and the weight parameter $\beta$ of the TV regularization term used in the final step). Typically, the optimal values of these parameters depend on the distribution of the clear speckle pattern, the model of the blur kernel, and the optimization algorithm used for the non-convex de-blurring problem. Hence, the selection of these parameter values is challenging, and, akin to most blind deconvolution methods, these parameters are often chosen empirically in practical.
3.3 Imaging through onion epidermis which moves randomly
Considering that the movement of ground glass is regular and cannot fully simulate the actual scattering media, the onion epidermis is selected as the scattering medium to verify the performance of the high-throughput method. The decorrelation time of the onion epidermis is long, and the change of speckle is not obvious. In this experiment, water droplets are dripped onto the onion epidermis, and fast-changing speckles are collected when the epidermis moves. The objects reconstruction results are shown in Fig. 7.
Fig. 7. Object reconstruction results through dynamic scattering media. (a) Object reconstruction results through the onion epidermis are shown in the first line. As shown in lines 2-4, water droplets are added to the onion epidermis, the autocorrelation and reconstruction results are calculated. (b) When N=4, the results before and after de-blurring of the four-block sub-speckle shown in the third row of Fig. 7(a) are calculated, and the clarity values are marked in the upper right corner of the speckles.
In the actual scene, the distribution of onion tissue is non-uniform, and the movement has certain randomness after adding water droplets. As shown in Fig. 7(a), the blur degree of different regions of the collected speckle is different. A total of 1, 4, or 9 sub-speckles with dimensions of 400$\times$400 pixels are selected from the center of the entire speckle pattern. These sub-speckles experience sequential filtering and de-blurring processes, and the autocorrelation of each de-blurred sub-speckle is calculated. The autocorrelation shown in the first line of Fig. 7(a) are used as the reference, and the PSNR values of different sub-speckles autocorrelations are computed. As shown in the second column of Fig. 7(a), based on the autocorrelation with the highest PSNR value among N sub-speckle autocorrelations, and the reconstruction results are calculated. From these reconstruction results, it can be observed that the selection of a single sub-speckle is capable of recovering the fundamental structure of the object. Moreover, as the number of selected sub-speckles increases, there is an improvement in the overall reconstruction quality. To analyze the reasons for the quality improvement, as shown in Fig. 7(b), the speckle patterns and reconstruction results of the four selected sub-speckles in the third row of Fig. 7(a) before and after de-blurring are compared. By evaluating the clarity values of the sub-speckles, it becomes apparent that the blurring degree of the four sub-speckles is different. It is easier to obtain better reconstruction results by using sub-speckles with lower blur and uniformity. Consequently, increasing the number of sub-speckles facilitates the selection of regions characterized by lower and more uniform blurring, which enhances the likelihood of achieving superior reconstruction outcomes. It should be noted that selecting a larger number of regions does not necessarily guarantee better reconstruction results. The quality of the reconstruction primarily depends on the blurring degree of the selected speckle and the choice of BAMB algorithm parameters. Additionally, selecting too many sub-speckles can lead to increased computational demand. Generally, selecting four sub-speckles is sufficient to achieve high-quality reconstruction. If an algorithm is designed to automatically identify regions with lower blurring and greater uniformity, direct de-blurring reconstruction can be performed without the need of block-based method.
More reconstruction results of the high-throughput imaging method based on speckle de-blurring are shown in Fig. 8. The four sub-regions of different objects undergo a filtering de-blurring preprocessing, resulting in significantly improved reconstruction results. The shapes of the objects are mostly reconstructed, although some details may be slightly deviated, overall the results are similar to speckle reconstruction in a static scattering environment.
Fig. 8. Reconstruction results of different objects through a dynamic scattering medium. (a) Different object reconstruction results through the onion epidermis. (b) The blurred speckles and reconstruction results are obtained by dropping water droplets on the onion epidermis. (c) The clear speckles and reconstruction results are obtained by de-blurring the speckles in Fig. 8(b). Scale bars: 260$\mu$m.
In practical scenarios, the motion of the scattering medium is typically random, and the degree of blur in the collected speckles can vary greatly. The de-blurring algorithm needs to select different parameters for calculation, which is not suitable for processing speckles of different blur degrees in large quantities. This will be the content of our next plan.
3.4 Imaging through scattering media based on learning method
The traditional HIO algorithm can only reconstruct object with low complexity and have limited ability to reconstruct objects with high ambiguity speckles. The DL network can improve the scattering imaging ability. Considering the dynamic characteristics of the scattering medium, the speckle-related physical prior and data model are jointly optimized and modeled to obtain a CNN network suitable for the dynamic scattering environment. Figure 9 shows the reconstruction results through scattering medium moving transversely based on the CNN network with four different inputs. The first method uses speckles as the input, while the second method uses speckle autocorrelations. The third method uses the BAMB algorithm to de-blur the collected speckles and then inputs the speckle autocorrelations into the network. The fourth method is a small improvement on the basis of the third method. Considering that the performance of the BAMB algorithm is related to the set blur parameters, method 4 uses four different parameters and the corresponding four sets of speckle autocorrelations are input into the network. The object data set used in this experiment is 600 single characters. The first 500 are trained and the last 100 are tested. The test sets are all unseen objects.
Fig. 9. Imaging through scattering medium which moves transversely based on CNN network. AC, autocorrelation; BAMB, blind anti-motion blur. GT, ground truth. M1: using blur speckles as the input. M2: using the autocorrelations of blur speckles as the input. M3: using the autocorrelations of de-blurred speckles as the input. M4: uses four different parameters and the corresponding four sets of de-blurred speckle autocorrelations as the input.
It is observed that, for stationary scattering media, the first two methods yield superior reconstruction results. However, in dynamic scattering environments, the pure data-driven approach exhibits limited generalization ability, leading to challenges in accurately reconstructing the object. Therefore, method 1 is unable to reconstruct the object. In the case of low speckle ambiguity, method 2 can basically reconstruct the object shape, but there are some defects and significant structural deviations. When the level of speckle ambiguity increases, the autocorrelation structure of untreated blurred speckles is poor, and it is difficult to reconstruct the object. The autocorrelation structure can be partially restored by de-blurring the speckle, which allows method 3 to outperform the first two methods in terms of reconstruction accuracy. Method 4 uses four groups of speckle de-blurring parameters, resulting in better object detail reconstruction than method 3. However, if the degree of speckle blurring increases, the deviation of object reconstruction also increases. The reconstruction results through the scattering medium moving rotational is shown in Fig. 10, which use the same object data set as Fig. 9 in the experiment, and similar conclusions can be obtained.
The quickdraw data set and two-character data set with more complex element composition and information details are selected as hidden objects for the reconstruction ability test, in which the two-character object is composed of two single characters randomly selected. Both datasets contain 1100 objects, 1000 training sets, and 100 unseen objects as test sets. The reconstruction results are shown in Fig. 11. When the CNN network is used to recover the objects in the static scattering environment, the reconstruction results have some missing structures and detail deviations because the network is relatively simple and the training set is small. When imaging through dynamic scattering media, for the quickdraw data set, method 1 and method 2 can distinguish a few objects, but the reconstruction results are not so good. For the more complex two-character data set, the objects cannot be reconstructed based on method 1, the reconstruction results of method 2 are blurred, and difficult to effectively distinguish different objects. By comparing the reconstruction results of method 2 and method 4, it can be found that the de-blurring algorithm can significantly improve the reconstruction results of complex objects, and the shapes of the objects are basically restored. However, there is still some deviation in the details of the objects. The details of the unknown complex object reconstruction results need to be further studied and improved, which is influenced by factors such as data volume and object information richness.
Fig. 11. Reconstruction results of different objects through scattering media moving transversely based on CNN network.
4. Discussion
According to the theoretical analysis and experimental demonstration, several discussions are clarified as follows:
- (i) To address the challenge of object reconstruction based on blur speckles in dynamic scattering environments, a non-invasive high-throughput imaging method based on speckle de-blurring is proposed. Through theoretical analysis and experimental demonstration, the feasibility and superiority of speckle de-blurring technology are proved. If the exposure time is within the speckle decorrelation time, the reconstruction result can be significantly improved even if the speckle is highly blurred.
- (ii) The actual dynamic scattering medium may be non-uniform and random, and the degree of blur degradation in different regions of the speckle might be different. Therefore, it is necessary to divide the speckle into multiple sub-speckles according to the degree of blur, and perform de-blurring reconstruction respectively.
- (iii) The DL method based on physical perception is necessary. The traditional HIO algorithm can reconstruct the simple object. For the object with high complexity, the DL method can better fit the hidden object. For the problem of object reconstruction through dynamic scattering environment, the introduction of relevant physical prior information can effectively improve the generalization ability of the data model.
- (iv) This paper focuses on the effectiveness of the de-blurring method. The de-blurring method is limited by the coherence of the speckle and the BAMB algorithm used. The performance of the BAMB algorithm is related to the set parameters, which cannot process speckles with different ambiguities in large quantities. The development of more effective de-blurring algorithms still needs further exploration to achieve higher fidelity object reconstruction.
5. Conclusion
In summary, a high-throughput imaging method based on speckle de-blurring is proposed, which can reconstruct hidden objects by using the blurred speckles collected in a dynamic scattering environment. The high-throughput method provides a new idea for dynamic scattering reconstruction, the blurred speckle can be effectively reconstructed by de-blurring processing without reducing the exposure time of the camera. This method is suitable for fast-changing scattering media, and the reconstruction results are improved significantly for hidden objects with different complexity. The high-throughput imaging method is generally applicable to traditional physical methods and DL methods, which effectively improves the imaging ability of these methods in dynamic scattering environment.
Funding
National Natural Science Foundation of China (61971227, 62031018, 62101255); Jiangsu Provincial Key Research and Development Program (BE2022391); China Postdoctoral Science Foundation (2021M701721, 2022M721620, 2023T160319).
Acknowledgment
The authors thank Chenyin Zhou, Yi Wei, Jinye Miao, Chenyang Huang, and Silei Wu for technical supports and experimental discussion.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. W. Goodman, Speckle phenomena in optics: theory and applications (Roberts and Company, 2007).
2. Y. Li, H. Lu, K.-C. Li, et al., “Non-uniform de-scattering and de-blurring of underwater images,” Mobile Netw. Appl. 23(2), 352–362 (2018). [CrossRef]
3. G. Satat, M. Tancik, and R. Raskar, “Towards photography through realistic fog,” 2018 IEEE International Conference on Computational Photography (ICCP) pp. 1–10 (2018).
4. L. Wang and H. Wu, Biomedical Optics: Principles and Imaging (Wiley, 2012).
5. W. Zhang, S. Zhu, K. Bai, et al., “Locating through dynamic scattering media based on speckle correlations,” Appl. Opt. 61(35), 10352–10361 (2022). [CrossRef]
6. A. Ishimaru, “Wave propagation and scattering in random media and rough surfaces,” Proc. IEEE 79(10), 1359–1366 (1991). [CrossRef]
7. H. Ruan, Y. Liu, J. Xu, et al., “Fluorescence imaging through dynamic scattering media with speckle-encoded ultrasound-modulated light correlation,” Nat. Photonics 14(8), 511–516 (2020). [CrossRef]
8. X. Li, A. Stevens, J. A. Greenberg, et al., “Single-shot memory-effect video,” Sci. Rep. 8(1), 13402 (2018). [CrossRef]
9. I. M. Vellekoop and A. P. Mosk, “Focusing coherent light through opaque strongly scattering media,” Opt. Lett. 32(16), 2309–2311 (2007). [CrossRef]
10. A. P. Mosk, A. Lagendijk, G. Lerosey, et al., “Controlling waves in space and time for imaging and focusing in complex media,” Nat. Photonics 6(5), 283–292 (2012). [CrossRef]
11. S. Rotter and S. Gigan, “Light fields in complex media: Mesoscopic scattering meets wave control,” Rev. Mod. Phys. 89(1), 015005 (2017). [CrossRef]
12. S. M. Popoff, G. Lerosey, R. Carminati, et al., “Measuring the transmission matrix in optics: An approach to the study and control of light propagation in disordered media,” Phys. Rev. Lett. 104(10), 100601 (2010). [CrossRef]
13. A. Drémeau, A. Liutkus, D. Martina, et al., “Reference-less measurement of the transmission matrix of a highly scattering material using a dmd and phase retrieval techniques,” Opt. Express 23(9), 11898–11911 (2015). [CrossRef]
14. D. Lu, M. Liao, W. He, et al., “Imaging dynamic objects hidden behind scattering medium by retrieving the point spread function,” in Speckle: International Conference on Speckle Metrology, (2018).
15. X. Xu, X. Xie, A. Thendiyammal, et al., “Imaging of objects through a thin scattering layer using a spectrally and spatially separated reference,” Opt. Express 26(12), 15073–15083 (2018). [CrossRef]
16. H. He, X. Xie, Y. Liu, et al., “Exploiting the point spread function for optical imaging through a scattering medium based on deconvolution method,” Journal of Innovative Optical Health Sciences (2019).
17. S. Feng, C. Kane, P. A. Lee, et al., “Correlations and fluctuations of coherent wave transmission through disordered media,” Phys. Rev. Lett. 61(7), 834–837 (1988). [CrossRef]
18. I. Freund, M. Rosenbluh, and S. Feng, “Memory effects in propagation of optical waves through disordered media,” Phys. Rev. Lett. 61(20), 2328–2331 (1988). [CrossRef]
19. J. Bertolotti, E. G. van Putten, C. Blum, et al., “Non-invasive imaging through opaque scattering layers,” Nature 491(7423), 232–234 (2012). [CrossRef]
20. O. Katz, P. Heidmann, M. Fink, et al., “Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations,” Nat. Photonics 8(10), 784–790 (2014). [CrossRef]
21. S. Li, M. Deng, J. Lee, et al., “Imaging through glass diffusers using densely connected convolutional networks,” Optica 5(7), 803–813 (2018). [CrossRef]
22. Y. Li, Y. Xue, and L. Tian, “Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media,” Optica 5(10), 1181–1190 (2018). [CrossRef]
23. M. Lyu, H. Wang, G. Li, et al., “Learning-based lensless imaging through optically thick scattering media,” Adv. Photonics 1(03), 1–036002 (2019). [CrossRef]
24. E. Guo, Y. Sun, S. Zhu, et al., “Single-shot color object reconstruction through scattering medium based on neural network,” Opt. Lasers Eng. 136, 106310 (2021). [CrossRef]
25. H. Liu, X. Wang, J. Gao, et al., “Seeing through dynamics scattering media: Suppressing diffused reflection based on decorrelation time difference,” Journal of Innovative Optical Health Sciences (2019).
26. D. Wang, S. K. Sahoo, X. Zhu, et al., “Non-invasive super-resolution imaging through dynamic scattering media,” Nat. Commun. 12(1), 3150 (2021). [CrossRef]
27. E. Edrei and G. Scarcelli, “Optical imaging through dynamic turbid media using the fourier-domain shower-curtain effect,” Optica 3(1), 71–74 (2016). [CrossRef]
28. S. Zhu, E. Guo, Q. Cui, et al., “Locating and imaging through scattering medium in a large depth,” Sensors 21(18), C1 (2021). [CrossRef]
29. Y. Sun, J. Shi, L. Sun, et al., “Image reconstruction through dynamic scattering media based on deep learning,” Opt. Express 27(11), 16032–16046 (2019). [CrossRef]
30. S. Zhu, E. Guo, J. Gu, et al., “Imaging through unknown scattering media based on physics-informed learning,” Photonics Res. 9(5), B210–B219 (2021). [CrossRef]
31. S. Zhu, E. Guo, J. Gu, et al., “Efficient color imaging through unknown opaque scattering layers via physics-aware learning,” Opt. Express 29(24), 40024–40037 (2021). [CrossRef]
32. X. Hu, J. Zhao, J. E. Antonio-Lopez, et al., “Adaptive inverse mapping: a model-free semi-supervised learning approach towards robust imaging through dynamic scattering media,” Opt. Express 31(9), 14343–14357 (2023). [CrossRef]
33. T. F. Chan and C.-K. Wong, “Total variation blind deconvolution,” IEEE Trans. on Image Process. 7(3), 370–375 (1998). [CrossRef]
34. F. Wen, R. Ying, Y. Liu, et al., “A simple local minimal intensity prior and an improved algorithm for blind image deblurring,” IEEE Trans. Circuits Syst. Video Technol. 31(8), 2923–2937 (2021). [CrossRef]
35. J. A. Newman, Q. Luo, and K. J. Webb, “Imaging hidden objects with spatial speckle intensity correlations over object position,” Phys. Rev. Lett. 116(7), 073902 (2016). [CrossRef]
36. L. Zhu, Y. Wu, J. Liu, et al., “Color imaging through scattering media based on phase retrieval with triple correlation,” Opt. Lasers Eng. 124, 105796 (2020). [CrossRef]
37. M. Hofer, C. Soeller, S. Brasselet, et al., “Wide field fluorescence epi-microscopy behind a scattering medium enabled by speckle correlations,” Opt. Express 26(8), 9866–9881 (2018). [CrossRef]
38. S. Zhu, E. Guo, K. Bai, et al., “Displacement-sensible imaging through unknown scattering media via physics-aware learning,” Opt. Lasers Eng. 160, 107292 (2023). [CrossRef]
39. L. Xu, S. Zheng, and J. Jia, “Unnatural l0 sparse representation for natural image deblurring,” 2013 IEEE Conference on Computer Vision and Pattern Recognition pp. 1107–1114 (2013).
40. W. Ren, X. Cao, J. shan Pan, et al., “Image deblurring via enhanced low-rank prior,” IEEE Trans. on Image Process. 25(7), 3426–3437 (2016). [CrossRef]
41. Y. Bai, G. Cheung, X. Liu, et al., “Graph-based blind image deblurring from a single photograph,” IEEE Trans. on Image Process. 28(3), 1404–1418 (2019). [CrossRef]
42. L. Xu, C. Lu, Y. Xu, et al., “Image smoothing via l0 gradient minimization,” Proceedings of the 2011 SIGGRAPH Asia Conference (2011).
43. X. Wang, H. Liu, M. Chen, et al., “Imaging through dynamic scattering media with stitched speckle patterns,” Chin. Opt. Lett. 18(4), 042604 (2020). [CrossRef]
44. J. R. Fienup, “Phase retrieval algorithms: a comparison,” Appl. Opt. 21(15), 2758–2769 (1982). [CrossRef]
45. Y. Wang, J. Cao, C. Xu, et al., “Moving target tracking and imaging through scattering media via speckle-difference-combined bispectrum analysis,” IEEE Photonics J. 11(6), 1–14 (2019). [CrossRef]
46. J. Shi, L. Xu, and J. Jia, “Discriminative blur detection features,” 2014 IEEE Conference on Computer Vision and Pattern Recognition pp. 2965–2972 (2014).
47. J. Shi, L. Xu, and J. Jia, “Just noticeable defocus blur detection and estimation,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pp. 657–665 (2015).
48. E. Guo, S. Zhu, Y. Sun, et al., “Learning-based method to reconstruct complex targets through scattering medium beyond the memory effect,” Opt. Express 28(2), 2433–2446 (2020). [CrossRef]
49. D. Tang, S. K. Sahoo, V. Tran, et al., “Single-shot large field of view imaging with scattering media by spatial demultiplexing,” Appl. Opt. 57(26), 7533–7538 (2018). [CrossRef]
50. C. Guo, J. Liu, W. Li, et al., “Imaging through scattering layers exceeding memory effect range by exploiting prior information,” Optics Communications (2019).
51. G. Maret and P.-E. Wolf, “Multiple light scattering from disordered media. the effect of brownian motion of scatterers,” Z. Phys. B: Condens. Matter 65(4), 409–413 (1987). [CrossRef]
52. F. C. MacKintosh and S. John, “Diffusing-wave spectroscopy and multiple scattering of light in correlated random media,” Phys. Rev. B 40(4), 2383–2406 (1989). [CrossRef]
53. M. M. Qureshi, J. Brake, H.-J. Jeon, et al., “In vivo study of optical speckle decorrelation time across depths in the mouse brain,” Biomed. Opt. Express 8(11), 4855–4864 (2017). [CrossRef]
54. J. Brake, M. Jang, and C. Yang, “Analyzing the relationship between decorrelation time and tissue thickness in acute rat brain slices using multispeckle diffusing wave spectroscopy,” J. Opt. Soc. Am. A 33(2), 270–275 (2016). [CrossRef]
