Weighted multi-scale denoising via adaptive multi-channel fusion for compressed ultrafast photography

Chengzhi Jin; Chengzhi Jin; Dalong Qi; Dalong Qi; Jiali Yao; Yilin He; Pengpeng Ding; Zihan Guo; Zhengqi Huang; Yu He; Yunhua Yao; Zhiyong Wang; Zhenrong Sun; Shian Zhang; Shian Zhang; Shian Zhang

doi:10.1364/OE.469345

1. Introduction

Aiming at capturing transient scenes in space and time with an ultra-high temporal resolution or frame rate, ultrafast optical imaging (UOI) techniques [1,2] have been widely applied in the fields of photochemistry [3], plasma physics [4] and biophotonics [1], and played irreplaceable roles in the analysis and understanding of their phenomena and mechanisms. Based on the imaging strategy, existing UOI techniques can be generally divided into two categories: multi-shot and single-shot. Multi-shot UOI techniques mainly include various pump-probe imaging methods via temporal scanning by ultrashort probe pulses or spatial scanning using zero-dimensional and one-dimensional ultrafast detectors. However, due to the requirement of repeated measurements, these techniques are not suitable for observing irreducible or destructive events. Complementarily, single-shot UOI techniques [5] can obtain spatiotemporal information of the scene in a snapshot without multiple measurements, which can be further divided into active illumination and passive acquisition types. Active illumination types, such as chirped spectral mapping ultrafast photography (CSMUP) [6], sequentially timed all-optical mapping photography (STEAM) [7] and frequency recognition algorithm for multiple exposures (FRAME) [8], require modulations on the illumination, and thus cannot measure self-emission ultrafast events (i.e., fluorescence lifetime). In contrast, passive acquisition types directly receive the photons emitted by the object without the need of illumination. Representative methods include ultrafast all-optical solid-state framing camera (UASFC) [9], time and spatial-frequency multiplexing (TSFM) [10], and compressed ultrafast photography (CUP) [11]. Among them, CUP stands out due to the characteristics of both ultrahigh frame rate and high sequence depth (number of frames per exposure) in a receive-only manner.

CUP is an ultrafast computational imaging technique based on compressed sensing (CS) [12]. It records 3D spatiotemporal information in a 2D compressed image by combining aperture-encoded imaging [13] and deflection imaging techniques, and subsequently recover the scene with reconstruction algorithms. The frame rate of 10 trillion frames per second and sequence depth of 300 frames have been demonstrated in previous studies [11]. Therefore, CUP has great advantages for measuring self-luminous or non-repeatable ultrafast phenomena [14]. However, the large sequence depth also brings the problem of high compression ratio and further results in low image quality in image reconstruction, preventing CUP from observing transient scenes with complex spatial information. Many efforts have been made to improve CUP’s imaging quality, mainly from the two aspects of hardware and algorithm. In terms of hardware, adding channels to increase the sampling rate is an effective strategy, and the most representative methods are complementary dual-channel CUP proposed by Liang et al. [15], and multi-channel coupled CUP (MC-CUP) originated by Yao et al. [16]. Both of them obtain multiple measurements of the same scene by using different encoding masks during a single acquisition. In terms of algorithms, a lot of model-based or learning-based methods are proposed to improve the reconstruction performance to a higher level, such as TV-BM3D [17], DeSCI [18], ALDL [19], U-net [20], alternating direction method of multipliers (ADMM)-based FFDNet [21]. However, all the existing methods in hardware or algorithm aspect are independent, there has not emerged a method making the joint use of hardware and algorithms yet.

Due to its flexibility and efficiency, the plug-and-play (PnP) framework [22,23] has been widely used in the field of computational imaging, including deblurring [24], super-resolution [25,26], hyperspectral imaging [27,28], CUP, and so on. The main idea of the PnP framework is to combine proximal algorithms with advanced image denoisers, which applies the sparse factorizable prior implicitly embedded in a Gaussian denoiser. However, the work by Zhang et al. [29] showed that the noise to be handled in PnP does not follow a fixed Gaussian distribution, and the general-purpose Gaussian denoiser prior and the manual selection of hyper-parameters comes at the cost of loss of efficiency and specialization. The choice of denoiser with optimal denoising parameters is the key to the elevation of image reconstruction performance of PnP-based algorithms, since unsuitable denoising parameters will lead to an increase in the iteration number or even non-converging results. Previous studies [23,30] have shown that different denoisers and different hand-crafted parameter settings have a large impact on the results of the final recovery. Fortunately, Athavale et al. [31] give us a new approach to remove the non-Gaussian distribution noise by Gaussian denoiser with weighted multi-scale (MS) denoising.

Here, we introduce weighted MS denoising into the PnP-based ADMM framework via adaptive MC fusion for CUP reconstruction, named MCMS-PnP algorithm, in which adaptive MS denoising with optimal parameters are obtained. In order to optimize the weights of different scale denoising during each ADMM iteration, MC-CUP technique is fully exploited. By extracting the coupling data from different channels with a least-squares optimization method, adaptive parameters optimization without manual adjustment and complete fusion of hardware and algorithms is achieved. Considering the high flexibility of the PnP framework, a deep learning-based DRUNet [29] is employed as a denoiser to form an ADMM-DRUNet algorithm for demonstration in this work. The performance of the proposed algorithm is validated by simulations and real experiments from MC-CUP. Compared with the traditional MC processing methods [15,16], the simulated and experimental results demonstrate that the proposed adaptive MS denoising ADMM-DRUNet algorithm outperforms all existed CUP reconstruction algorithms in terms of image quality, and has strong noise robustness as well. Therefore, the proposed MCMS-PnP method is able to achieve the state-of-the-art (SOTA) reconstruction results in CUP by fully extracting the coupling information in different channels.

2. Principles

As a typical computational optical imaging strategy, the overall procedures of MC-CUP are composed of two parts, i.e., the data acquisition and image reconstruction, which integrally collects transient scenes in a coded aperture manner and recovers them with the aid of algorithms for inverse problems, respectively. Figure 1(A) shows the schematic diagram of the data acquisition in MC-CUP. As can be seen, a 3D transient scene, ${\boldsymbol I}({x,y,t} )$, is divided into M replicas, and each replica is first spatially encoded by an independent random binary pattern generated by a spatial light modulator, e.g., digital micromirror device (DMD) [11] or printed transmissive mask [32]. Successively, the coded replicas are temporally sheared along one of the spatial axes with a temporal deflector, e.g., streak camera [33,34] or electro-optical deflector [35], and integrally measured by an image detector, e.g., complementary metal oxide semiconductor (CMOS) or charged coupled device (CCD). Finally, 2D compressed images ${{\boldsymbol E}_i}({x^{\prime},y^{\prime}} )({i\textrm{ } = 1,2, \cdots ,M} )$ are obtained. Mathematically, the data acquisition through different channels can be expressed as follows

(1)$$\left\{ \begin{array}{c} {{\boldsymbol E}_1}({x^{\prime},y^{\prime}} )= {\textbf {TS}}{{\textbf C}_\textrm{1}}{\boldsymbol I}(x,y,t) + {n_\textrm{1}}\\ {{\boldsymbol E}_2}({x^{\prime},y^{\prime}} )= {\textbf {TS}}{{\textbf C}_2}{\boldsymbol I}(x,y,t) + {n_2}\\ \vdots \\ {{\boldsymbol E}_M}({x^{\prime},y^{\prime}} )= {\textbf {TS}}{{\textbf C}_M}{\boldsymbol I}(x,y,t) + {n_M} \end{array} \right., $$

where ${{\textbf C}_i}({i\textrm{ } = 1,2, \cdots ,M} )$ represents the spatial encoding for each acquisition channel with M denoting the number of channels, S represents the temporal shearing, T represents the spatiotemporal integration, ${n_i}(i\textrm{ } = 1,2, \cdots ,M)$ denotes the measurement and detector noise in the image acquisition for each channel, and ${{\boldsymbol E}_i}({x^{\prime},y^{\prime}} )$ is the measured 2D image in the ith channel. For simplicity, it is set that ${{\mathbf \Phi }_{\boldsymbol i}} = {\textbf {TS}}{{\textbf C}_{\boldsymbol i}}$, and ${\boldsymbol I}({x,y,t} )$ and ${{\boldsymbol E}_i}({x^{\prime},y^{\prime}} )$ are abbreviated to ${\boldsymbol x}$ and ${{\boldsymbol y}_i}$, respectively, thus the image acquisition for each channel can be concatenated as

(2)$${{\boldsymbol y}_i} = {{\mathbf \Phi }_i}{\boldsymbol x} + {n_i},{\kern 1pt} {\kern 1pt} {\kern 1pt} \forall i = 1,2, \cdots ,M, $$

where ${\boldsymbol y} = \textrm{Vec}({\textbf Y} )\in {\mathrm{\mathbb{R}}^{{N_x}{N_y}}}$, $n = \textrm{Vec}({\textbf N} )\in {\mathrm{\mathbb{R}}^{{N_x}{N_y}}}$, ${\boldsymbol x} = \textrm{Vec}({\textbf X} )\in {\mathrm{\mathbb{R}}^{{N_x}{N_y}{N_t}}}$ and ${{\mathbf \Phi }_{\boldsymbol i}} \in {\mathrm{\mathbb{R}}^{{N_x}{N_y} \times {N_x}{N_y}{N_t}}}$. Here, X, Y, and N represent higher order matrix representation of the corresponding data, and N_x, N_y and N_t denote the numbers of discretized pixels in the x, y and t coordinates, respectively. The sampling rate of each channel is $1/{N_t}$, so the overall sampling rate of MC-CUP is $M/{N_t}$. In general, MC-CUP with multiple independent acquisition channels can effectively increase the sampling rate of the transient scene. The influence of number of channels on the quality of the reconstructed images is investigated by simulations, please refer to Supplemental 1 for the detailed result.

Fig. 1. (A) Schematic diagrams of MC-CUP; (B) Data flowchart of weighted MS denoising via adaptive MC for CUP reconstruction, where each stage is composed of the projection G representing the operation in Eq. (9), the projection D representing the operation in Eq. (10) and the projection W representing the operation in Eqs. (16) and (17).

Download Full Size | PDF

Based on the model derived in Eq. (2), we develop an adaptively weighted MS denoising method for MC-CUP reconstruction using the PnP-ADMM framework. In this process, the origin 3D transient scene needs to be reconstructed from the captured 2D compressed image for each channel, which employs an iterative algorithm via the CS theory to solve the inverse problem given by

(3)$${\hat{\boldsymbol {x}}} = \textrm{arg}\mathop {\textrm{min}}\limits_{\boldsymbol x} \frac{1}{2}\parallel {\boldsymbol y} - {\mathbf \Phi }{\boldsymbol x}\parallel _2^2 + \lambda g\left( {\boldsymbol x} \right),$$

where $g({\boldsymbol x})$ is an employed prior, and λ is a regularization parameter to balance the fidelity term and prior. To decouple the fidelity term and prior, an auxiliary parameter, v, is introduced, and Eq. (3) is modeled as

(4)$${\hat{\boldsymbol {x}}} = \textrm{arg}\mathop {\textrm{min}}\limits_{{\boldsymbol x},{\boldsymbol v}} \frac{1}{2}\parallel {\boldsymbol y} - {\mathbf \Phi }{\boldsymbol x}\parallel _2^2 + \lambda g\left( {\boldsymbol v} \right),\textrm{ subject to }{\boldsymbol x}\textrm{ = }{\boldsymbol v}.$$

In the following, we further introduce an updatable auxiliary parameter, u, and a manually set penalty parameter, ρ, and construct the augmented Lagrangian function

(5)$$\mathrm{{\cal L}}({{\boldsymbol x},{\boldsymbol v},{\boldsymbol u}} )= \frac{1}{2}\parallel {\boldsymbol y} - {\mathbf \Phi }{\boldsymbol x}\parallel _2^2 + \lambda g({\boldsymbol v} )+ {{\boldsymbol u}^\mathrm{\ \top }}({{\boldsymbol v} - {\boldsymbol x}} )+ \frac{\rho }{2}\parallel {\boldsymbol v} - {\boldsymbol x}\parallel _2^2. $$

By employing the ADMM framework, the minimization of $\mathrm{{\cal L}}$ in Eq. (5) can be split into the following three sub-problems written as

(6)$${{\boldsymbol x}^{({k + 1} )}} = \textrm{arg}\mathop {\textrm{min}}\limits_{\boldsymbol x} \frac{1}{2}\parallel {\boldsymbol y} - {\mathbf \Phi }{\boldsymbol x}\parallel _2^2 + \frac{\rho }{2}{\parallel}{\boldsymbol x} -({ {{\boldsymbol v}^{(k )}}} - \frac{1}{\rho }{{\boldsymbol u}^{(k )}})\parallel _2^2, $$

(7)$${{\boldsymbol v}^{({k + 1} )}} = \textrm{arg}\mathop {\textrm{min}}\limits_{\boldsymbol v} \lambda g({\boldsymbol v} )+ \frac{\rho }{2}{\parallel}{\boldsymbol v} - ({{\boldsymbol x}^{({k + 1} )}} + \frac{1}{\rho }{{\boldsymbol u}^{(k )}})\parallel _2^2, $$

(8)$${{\boldsymbol u}^{({k + 1} )}} = {{\boldsymbol u}^{(k )}} + \rho ({{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol v}^{({k + 1} )}}} ). $$

Here, the superscript k denotes the iteration number within the maximal iterations, O. According to Ref. [18], the solution of x sub-problem in Eq. (6) represents a constraint on the estimated results, and it can be efficiently implemented by

(9)$${{\boldsymbol x}^{({k + 1})}} = {\left({{{\mathbf \Phi }^T}{\mathbf \Phi } + \rho {\textbf I}} \right)^{ - 1}}\left[{{{\mathbf \Phi }^T}{\boldsymbol y} + \rho \left({{{\boldsymbol v}^{(k )}} + {{\boldsymbol u}^{(k )}}} \right)} \right]. $$

According to Ref. [23], the v sub-problem in Eq. (7) can be regarded as a denoising problem by using deep denoising networks such as FFDNet [36] and DRUNet [29]. Defining $\sigma \; = \sqrt {\lambda /\rho } $, the solution of Eq. (7) can be denoted as

(10)$${{\boldsymbol v}^{({k + 1} )}} = {\mathrm{{\cal D}}_\sigma }({{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol u}^{(k )}}} ), $$

where $\sigma $ is the estimated noise level of denoiser $\mathrm{{\cal D}}$, and the resulting algorithm is called PnP-ADMM.

Considering the efficiency and specialization problems in conventional PnP-ADMM algorithms, the MS denoising method is further derived for MC-CUP to reduce the difficulty of adjusting the denoising parameters and make full use of the coupling information between different channels. Inspired by Ref. [30], we assume that there exists a distribution of the denoiser hyperparameter, $\sigma $, in each iteration of PnP, which is denoted as $p\left({\sigma |{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol u}^{(k )}}} \right)$. With the distribution of $\sigma $ in consideration, we eliminate the hyperparameter via integral as follows

(11)$$p\left({{{\boldsymbol v}^{({k + 1} )}}|{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol u}^{(k )}}} \right)= \int {p\left({{{\boldsymbol v}^{({k + 1} )}}|{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol u}^{(k )}},\sigma } \right)p\left({\sigma |{{\boldsymbol x}^{({k + 1} )}} - {{\boldsymbol u}^{(k )}}} \right)d\sigma }. $$

Therefore, a very straightforward method is to discretize $\sigma $ for calculation purposes. To discretize $\sigma $, the hyperparameter is considered obeying the discrete distribution of $p({\sigma |{\boldsymbol x} - {\boldsymbol u}} )$ with $\sigma \in {\boldsymbol S}$, where S is a set with finite elements of noise level. For convenience, the superscript (k+1) is omitted, therefore, Eq. (11) can be rewritten as

(12)$$p\left( {{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u}} \right) = \mathop \sum \limits_{\sigma \in {\boldsymbol S}} p\left( {{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u},\sigma } \right)p\left( {\sigma |{\boldsymbol x} - {\boldsymbol u}} \right).$$

It is easy to see that Eq. (11) is the posterior we get from the denoiser ${\mathrm{{\cal D}}_\sigma }$ and

(13)$$p\left({{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u},\sigma } \right)= {\mathrm{{\cal D}}_\sigma }\left({{\boldsymbol x} - {\boldsymbol u}} \right). $$

By rearranging the above optimization problem and denoting ${w_\sigma } = p\left({\sigma |{\boldsymbol x} - {\boldsymbol u}} \right),\sigma \in {\boldsymbol S}$ with , the (k+1)th iterative optimal solution can be obtained by

(14)$${{\mathbb{E}}_{p\left( {{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u}} \right)}}\textrm{[}{\boldsymbol v}\textrm{]} = \mathop \sum \limits_{\sigma \in {\boldsymbol S}} {w_\sigma }{{{\cal D}}_\sigma }\left( {{\boldsymbol x} - {\boldsymbol u}} \right).$$

Since the distribution of noise and computational error during the ADMM iteration is unknown, the distribution of ${w_\sigma }$ is also unknown, which leads to the fact that Eq. (14) cannot be solved. However, the MC acquisition technique gives us the possibility of solving the distribution of ${w_\sigma }$. For each channel, the collected data contains two parts of real scene data and noise data, and the real scene data of all channels are coupled with each other while the noise data are irrelevant. It can be simply deduced that after weighted MC denoising, the extracted data should be the same. On the basis of this, we can calculate the distribution of ${w_\sigma }$, and then extract the coupling data of different channels, which is the ${\mathrm{\mathbb{E}}_{p({{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u}} )}}\textrm{[}{\boldsymbol v}\textrm{]}$ we need. Considering ${{\boldsymbol x}_i}$, ${{\boldsymbol v}_i}$ with $i \in \left\{{1,2, \cdots ,M} \right\}$, Eq. (14) can be formulated as

(15)$${{\mathbb{E}}_{p\left( {{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u}} \right)}}\textrm{[}{\boldsymbol v}\textrm{]} = \mathop \sum \limits_{\sigma \in {\boldsymbol S}} {w_{i,\sigma }}{{{\cal D}}_\sigma }\left( {{{\boldsymbol x}_i} - {{\boldsymbol u}_i}} \right).$$

In the case of an ideal situation without noise, all channels should get the same ${\mathrm{\mathbb{E}}_{p({{\boldsymbol v}|{\boldsymbol x} - {\boldsymbol u}} )}}[{\boldsymbol v}]$ and ${w_\sigma }$. However, considering the noise, we can obtain ${w_\sigma }$ by solving the following optimization problem

(16)$$\begin{array}{c} \mathop {\textrm{min}}\limits_{{w_i},{w_j}\; } \parallel \mathop \sum \limits_{\sigma \in {\boldsymbol S}} {w_{i,\sigma }}{{{\cal D}}_\sigma }\left( {{{\boldsymbol x}_i} - {{\boldsymbol u}_i}} \right) - \mathop \sum \limits_{\sigma \in {\boldsymbol S}} {w_{j,\sigma }}{{{\cal D}}_\sigma }\left( {{{\boldsymbol x}_j} - {{\boldsymbol u}_j}} \right)\parallel _2^2\\ \forall i,j \in \left\{ {1,2, \cdots ,{\boldsymbol M}} \right\},i \ne j \end{array}$$

We use an iterative approach based on the least-squares optimization method to solve the problem via the CVXPY toolbox [37]. Throughout the ADMM calculation, S is kept constant while ${w_\sigma }$ is solved automatically, which omits the process of manually setting the denoising parameters. Finally, the MCMS result is

(17)$${\boldsymbol v} = \frac{1}{M}\mathop \sum \limits_{i = 1}^M \mathop \sum \limits_{\sigma \in {\boldsymbol S}} {w_{i,\sigma }}{{{\cal D}}_\sigma }\left( {{{\boldsymbol x}_i} - {{\boldsymbol u}_i}} \right)$$

Algorithm 1 exhibits the overall workflow of the proposed adaptive MCMS-PnP algorithm for CUP, and the entire computational flow is shown in Fig. 1(B).

oe-30-17-31157-i001

3. Results and Discussion

The adaptive MCMS-PnP algorithm is implemented in python, and Pytorch is used to invoke a pretrained network, i.e., DRUNet in this work, as the denoising module on a server with Intel Core i7-12700K CPU and NVIDIA Geforce GTX 3090 GPU. To validate the reconstruction performance of the algorithm, we create two different types of dynamic scenes for simulation in the MC-CUP modality, and take the dual-channel sampling with complementary encoding patterns, i.e., lossless encoding CUP [15], as an example. The first type is composed of 12 kinds of moving image scenarios from publicly available datasets. Each moving image scenario contains 10 pictures with the size of 256×256 pixels, and each picture is shifted right by ten pixels relative to the previous one. The second type is composed of 7 video scenarios captured by ultrahigh-speed cameras, and each of them contains 10 consecutive frames with the same size as it in the former. To create compressed 2D images by the MC-CUP for reconstruction, all scene data of different types are processed following the data acquisition indicated in Fig. 1(A). Specifically, all frames of the dynamic scene are duplicated as a pair of replicas, and each pair is encoded by the same couple of pseudo-random binary masks with elements {0, 1} via element-wise multiplication. Then, the encoded frames are sequentially shifted with each frame moving down one pixel relative to the previous one to simulate the temporal deflection. Finally, both frames are spatially integrated to obtain the final 2D measurements in different channels. To be closer to the actual situation, additive white Gaussian noise (AWGN) with the noise level of σ₀= 10, is also considered in the simulations of video scenarios. It is noted that although only AWGN is added to the simulated video scenario, the requirement of removing non-Gaussian distributed noise also exists at the beginning of the iteration [29]. In order to demonstrate the superiority of the proposed adaptive MCMS-PnP algorithm, the pre-trained SOTA DRUNet is integrated into three kinds of PnP-ADMM algorithms (ADMM is omitted in all subsequent contents), who are denoted as DRUNet for the single-channel and single-scale (SCSS) framework, DRUNet-MC for the conventional MC method [16], and DRUNet-MCMS for our proposed one, respectively. Moreover, several kinds of mainstream CUP reconstruction algorithms, including GAP-TV [38], TV-BM3D [17], DeSCI [18] and FFDNet [21], are employed for the comparison with DRUNet in simulations. Besides, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are introduced and averaged by all of the shifted frames as image quality assessment (IQA).

In the moving image scenario reconstruction, the initial variables ${\boldsymbol x}_i^0$, ${{\boldsymbol u}^0}$ are set to zero, ${{\boldsymbol v}^0}$ is set according to the GAP-TV result, ${\boldsymbol S} = \{ 40,30,20,10,5\} $ and the maximal iteration number of 100 are also set. Table 1 lists the averaged PSNR and SSIM values for reconstruction results of all images, and the best results are highlighted in bold font. Among them, the first five columns in Methods are for SCSS denoising. It can be seen that in this case, DRUNet can achieve very close first-class results to DeSCI, which illustrates the superior performance of DRUNet compared with other regularization methods (i.e., TV, BM3D and FFDNet). Taking MC sampling in consideration, DRUNet-MC demonstrates that enhancing the data sampling rate solely by MC acquisition can significantly improve the reconstruction quality. However, it is still restricted to individual acquisition channels during reconstruction. In contrast, our proposed DRUNet-MCMS algorithm outperforms all the other methods by fully exploiting the coupling of information among multiple channels. According to the obtained IQA values, DRUNet-MCMS shows an improvement of up to 4.68 dB (0.107) and 2.04 dB (0.0457) than that of DRUNet and DRUNet-MC in averaged PSNR (SSIM), respectively.

Table 1. The averaged results of PSNR (dB) and SSIM by different methods on moving image scenarios.

View Table | View all tables in this article

Figure 2 shows the fifth reconstructed images of Sunflower, Man, Boat, Castle, Monarch, and Parrot scenarios reconstructed by the algorithms displayed in Table 1, respectively, together with the corresponding ground truths for comparison. As can be seen from the reconstructed images and the magnified image blocks in the red boxes, none of the SCSS methods is favorable for recovering image details, and their reconstructions all produce various degrees of image blurring and artifacts. Even with DRUNet-MC, the reconstructed results still show obviously degraded structures in Sunflower and Monarch scenarios. In contrast, DRUNet-MCMS produces sharper borders and better image details, especially for shadow structures with insignificant contrast (e.g., in Sunflower) and irregular structures (e.g., in Parrot). The main reason for the SOTA results is that the information of the image in each ADMM iteration can be fully extracted in MS denoising in DRUNet-MCMS. More importantly, the adaptive solver of denoising weights on the basis of MC-CUP can ensure that the most suitable parameters are selected at each iteration to optimize the image toward the optimal solution.

Fig. 2. Selected image reconstruction results by GAP-TV, TV-BM3D, DeSCI, FFDNet, DRUNet, DRUNet-MC, DRUNet-MCMS, together with the ground truths for comparison. The sub-image at the bottom right corner is the enlarged area in the corresponding red box.

Download Full Size | PDF

In video reconstruction, all the parameters in DRUNet-MCMS are set as the same as that in moving image scenarios. Similarly, Table 2 lists the averaged PSNR and SSIM values for reconstruction results of all videos, and the best results are also labeled in bold font. One can obtain the same conclusion as indicated in moving image scenarios, which is that DRUNet exceeds all the involved methods in the SCSS framework, and the reconstruction quality is further greatly improved in MC-CUP. It is worth noting that the proposed adaptively weighted MS denoising method, DRUNet-MCMS, can elevate 1.11 dB (0.0062) in averaged PSNR (SSIM) on the basis of DRUNet-MC. In addition, Fig. 3 displays 3 out of 10 frames (i.e., frames 3, 6 and 9) in the videos of Welding (A) and Detonator (B) reconstruction results by DRUNet, DRUNet-MC, DRUNet-MCMS, respectively, together with the ground truths. The values under each subfigure represent the PSNR (in dB) and SSIM corresponding to that frame. From the demonstrated results, it can be concluded that the DRUNet-MCMS method has the best performance in terms of recovery of background and details for CUP. Please refer to Supplemental 1 for the variation of multi-scale denoising weights with the increase of iteration for the Welding video scenario.

Fig. 3. Selected video reconstruction frames of (A) Welding, (B) Detonator by DRUNet, DRUNet-MC, DRUNet-MCMS, together with the ground truths for comparison. A reconstruction movie of Welding and Detonator corresponding to the ground truths is provided in Visualization 1 (see Supporting Information).

Download Full Size | PDF

Table 2. The averaged results of PSNR (dB) and SSIM by different methods on video scenarios.

View Table | View all tables in this article

In the DRUNet and DRUNet-MC methods, when the collected data contains noise with noise level of ${\sigma _0}$, the denoising parameters in Eq. (10) need to be set larger than ${\sigma _0}$ in each iteration, otherwise the recovery images will be contaminated by noise leading to a worse result. On the other hand, the oversized denoising parameter in the ADMM iteration will cause the recovered results become over smooth and blurred, as shown in the results obtained by DRUNet and DRUNet-MC algorithms in Fig. 3. In contrast, the range of denoising parameters S still does not need to be changed in the proposed DRUNet-MCMS algorithm, which confirms that the MCMS-PnP algorithm has stronger robustness to noise. It is worth noting that this is more favorable in experimental data reconstruction.

As shown above, it has been proven from numerical simulations that the MCMS-PnP algorithm brings higher image quality and possesses stronger noise immunity in the image reconstruction for CUP than the conventional algorithms. Next, we apply the proposed DRUNet-MCMS algorithm to recover transient scenes obtained by our home-built MC-CUP system for experimental validation. As indicated in Fig. 4(A), an ultrafast optical scene as the object is collected by an objective lens (OL) and imaged onto a DMD (Texas Instruments, DLP Light Crafter 3000), on which a static pseudo-random binary pattern is loaded. Then the ultrafast scene is split into two beams by the DMD along different directions of ±12 degrees, and both of the replicas are encoded by a pair of complementary masks, thus producing a dual-channel encoding. After that, the encoded ultrafast scene of each channel is transferred through a 4f imaging system composed of L1 and L2 or L3 and L4, and reflected by a reflecting prism. Finally, both replicas are sent into a streak camera (Hamamatsu, C7700) with the slit fully opened, which is a scientific instrument that is able to capture ultrafast scenes based on the photo-electric effect and temporal shearing technique [33,34]. The streak camera and transient scenes are precisely synchronized in time by a digital delay generator (Stanford Research Systems, DG645) to temporally shear and spatiotemporally integrate them. Combining our proposed MCMS-PnP algorithm with the home-built MC-CUP system, we measured the spatiotemporal evolution of a spatially modulated E-shaped picosecond laser pulse and achieved volumetric (i.e., x-y-z) imaging of a 3D ladder structure by the time of flight (ToF) detection experimentally.

Fig. 4. (A) Experimental system configuration of MC-CUP. OL, objective lens; DMD, digital micromirror device; M1-M4, mirror; L1-L4, lens; RP, reflecting prism; SC, streak camera. (B) Static image of 2D measurements overlay the intensity evolution of 1D measurements by streak camera as reference snapshot and reconstructed spatiotemporal images by DRUNet, DRUNet-MC, DRUNet-MCMS of E-shaped laser pulse; (C) PSNR and SSIM distribution calculated by DRUNet, DRUNet-MC, DRUNet-MCMS with ground truth, respectively; (D) the normalized intensities for ground truth (black line), DRUNet (green line), DRUNet-MC (yellow line) and DRUNet-MCMS (green line) along the white dotted line in the reconstructed images with the time of 60 ps in (B). A spatiotemporal evolution movie of E-shaped laser pulse corresponding to the reference snapshot is provided in Visualization 2 (see Supporting Information).

Download Full Size | PDF

In the first scene, a single laser pulse from a mode-locked Ti:Sapphire laser amplifier (Spectra-Physics, 50 fs, 0.8 mJ) was stretched up to 200 ps by a pulse stretcher. The stretched pulse was spatially expanded to illuminate a hollow letter ‘E’ fabricated in a black nylon plate. Photons inside the shape could pass through the plate, while those on the outside were blocked. The resulted E-shaped laser pulse was projected onto a thin sheet of white paper for scattering, and further observed by the MC-CUP system. The three DRUNet-based algorithms are used to reconstruct the spatiotemporal evolution of the E-shaped laser pulse, and corresponding normalized results are shown in Fig. 4(B). 10 representative frames are selected from the reconstructed scenes with a time interval of 30 ps, where the reconstructed results for DRUNet, DRUNet-MC and DRUNet-MCMS are displayed for comparison. It needs to declare that the snapshot in the first row is generated by combining the static 2D image and dynamic 1D intensity evolution obtained by the streak camera as reference, and the DRUNet results in the second row are from one of the channels in MC-CUP. From Fig. 4(B), one can see that the DRUNet results show obvious incompleteness in shape, and the results of DRUNet-MC and DRUNet-MCMS are closer to the ground truth in terms of spatial structure. Moreover, the MS strategy has an absolute advantage in improving the dynamic range of reconstructed images as shown in frames at 90 and −90 ps. To quantitatively compare the reconstruction performance, we calculated the PSNR and SSIM values for the middle 8 frames of Fig. 4(B), and the results are shown in Fig. 4(C). It can be easily recognized from Fig. 4(C) that DRUNet-MCMS outperforms the other two methods in both PSNR and SSIM, especially under the condition of low signal to noise ratio (SNR). To further illustrate the improvement in spatial details, we plotted the normalized intensities along the white dotted lines for the frames at 60 ps in Fig. 4(B), and the results are given in Fig. 4(D). As expected, the spatial intensity distribution of DRUNet-MCMS still holds the highest SNR for the three peaks. Besides, the root-mean-square error (RMSE) of DRUNet, DRUNet-MC and DRUNet-MCMS calculated from Fig. 4(D) are 0.1930, 0.1822 and 0.1215, respectively, indicating the closest fidelity to the ground truth for DRUNet-MCMS.

In the second scene, we measured the surface profile of a volumetric object by capturing the backscattered photons from the surface via time-of-flight (ToF) detection. As shown in Fig. 5(A), a 3D ladder structure with alternating lengths is fabricated using white nylon and each step has a width and height of 3 mm, respectively. A 50-fs laser pulse with spatial expansion was employed to illuminate the 3D ladder structure. The MC-CUP system illustrated in Fig. 4(A) was placed perpendicular to the x-y plane of the ladder structure and collected the backscattered photons from the surface. The depth information of the object can be further obtained by measuring the round-trip ToF signal of the fs laser pulse, and it is given by $z = \textrm{ }c \times {t_{ToF}}/2$, where c and ${t_{ToF}}$ denote the speed of light and propagation time, respectively. Similarly, the reconstructed images via the three DRUNet-based algorithms are shown in Fig. 5(B). For comparison, 6 representative images out of 60 frames at the time delays of 0, 20, 40, 60, 80 and 100 ps are given, and the white dotted boxes identify the accurate sizes and locations of the steps. It can be visually observed that the DRUNet-MCMS algorithm results in more uniform intensity distribution. Moreover, the 3D morphology of the object based on the DRUNet-MCMS result with a temporal resolution of 2ps is reconstructed as shown in Fig. 5(C), and the obtained 3D morphology shows a perfect match with the actual model. In order to quantitatively analyze the volumetric detection error, we compared the overlap degree between the corresponding images of different steps obtained from the three algorithms and the ground truth in Fig. 5(B) by calculating the Intersection over Union (IoU) [39] indices, and the results are displayed in Fig. 5(D). It is clearly seen that the IoU values of DRUNet-MCMS are much higher than those of the other two methods, and the mean IoU values of DRUNet, DRUNet-MC, DRUNet-MCMS is 0.46, 0.51, 0.69, respectively. In other words, The IoU value of DRUNet-MCMS have more than 0.18 improvement over the other two methods. Obviously, DRUNet-MCMS can provide an effective method to further improve the accuracy of volumetric detection.

Fig. 5. (A) The real size and photograph of the 3D stepped sample. (B) Reconstructed images at the time points of 0, 20, 40, 60, 80 and 100 ps by DRUNet, DRUNet-MC, DRUNet-MCMS, the white dotted boxes identify the accurate sizes and locations of the step; (C) the retrieved 3D morphology from DRUNet-MCMS result in (B); (D) IoU distribution calculated from each step of (B).

Download Full Size | PDF

4. Conclusion

In summary, we have developed a MS denoising PnP algorithm, shorten as MCMS-PnP, for MC-CUP to reconstruct ultrafast scenes, which is capable of realizing adaptive parameter tuning without manual settings. To test the reconstruction capability of this PnP algorithm, we first performed two numerical simulations, and the results show that the proposed method has the best performance compared to previous algorithms used in CUP, and the method can provide both higher image reconstruction quality and stronger noise immunity. Furthermore, two kinds of scenes are measured with our home-built MC-CUP system. The proposed method performs very well on the image details and shadow textures of the experimental data reconstruction results, which can effectively improve the spatial details. Moreover, more advanced denoisers or algorithms can be easily combined in this PnP framework to further improve the image reconstruction performance in the future. It can be prospected that the proposed MCMS-PnP algorithm would enable CUP to visualize ultrafast scenes with complicated spatial details, thus extending the applications in various areas, including shockwave diagnosis in inertial confined fusion [40] and implosions detection in Z-pinches [41].

Funding

National Natural Science Foundation of China (11727810, 12034008, 12074121, 62105101, 62175066, 91850202, 92150301); Science and Technology Commission of Shanghai Municipality (20ZR1417100, 21JM0010700, 21XD1400900).

Disclosures

The authors declare no conflict of interest.

Data availability

Data and source code underlying the results presented in this paper are available in Ref. [42].

Supplemental document

See Supplement 1 for supporting content.

References

1. K. E. Sheetz and J. Squier, “Ultrafast optics: Imaging and manipulating biological systems,” J. Appl. Phys. 105(5), 051101 (2009). [CrossRef]

2. J. Liang and L. V. Wang, “Ultrafast optical imaging,” in Handbook of Laser Technology and Applications3, 315–328 (CRC, 2021).

3. A. Rosspeintner, B. Lang, and E. Vauthey, “Ultrafast photochemistry in liquids,” Annu. Rev. Phys. Chem. 64(1), 247–271 (2013). [CrossRef]

4. C. Campbell, X. Tang, Y. Sechrest, K. Fezzaa, Z. H. Wang, and D. Staack, “Ultrafast x-ray imaging of pulsed plasmas in water,” Phys. Rev. Res. 3(2), L022021 (2021). [CrossRef]

5. J. Liang and L. V. Wang, “Single-shot ultrafast optical imaging,” Optica 5(9), 1113–1127 (2018). [CrossRef]

6. Y. Yao, Y. He, D. Qi, F. Cao, J. Yao, P. Ding, C. Jin, X. Wu, L. Deng, and T. Jia, “Single-shot real-time ultrafast imaging of femtosecond laser fabrication,” ACS Photonics 8(3), 738–744 (2021). [CrossRef]

7. K. Nakagawa, A. Iwasaki, Y. Oishi, R. Horisaki, A. Tsukamoto, A. Nakamura, K. Hirosawa, H. Liao, T. Ushida, K. Goda, F. Kannari, and I. Sakuma, “Sequentially timed all-optical mapping photography (STAMP),” Nat. Photonics 8(9), 695–700 (2014). [CrossRef]

8. Z. Li, J. Borggren, E. Berrocal, A. Ehn, M. Aldén, M. Richter, and E. Kristensson, “Simultaneous multispectral imaging of flame species using Frequency Recognition Algorithm for Multiple Exposures (FRAME),” Combust. Flame 192, 160–169 (2018). [CrossRef]

9. G. Gao, K. He, J. Tian, C. Zhang, J. Zhang, T. Wang, S. Chen, H. Jia, F. Yuan, and L. Liang, “Ultrafast all-optical solid-state framing camera with picosecond temporal resolution,” Opt. Express 25(8), 8721–8729 (2017). [CrossRef]

10. J. Moon, S. Yoon, Y.-S. Lim, and W. Choi, “Single-shot imaging of microscopic dynamic scenes at 5 THz frame rates by time and spatial frequency multiplexing,” Opt. Express 28(4), 4463–4474 (2020). [CrossRef]

11. L. Gao, J. Y. Liang, C. Y. Li, and L. H. V. Wang, “Single-shot compressed ultrafast photography at one hundred billion frames per second,” Nature 516(7529), 74–77 (2014). [CrossRef]

12. Y. C. Eldar and G. Kutyniok, Compressed sensing: theory and applications (Cambridge university, 2012).

13. P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, “Coded aperture compressive temporal imaging,” Opt. Express 21(9), 10526–10545 (2013). [CrossRef]

14. D. Qi, S. Zhang, C. Yang, Y. He, F. Cao, J. Yao, P. Ding, L. Gao, T. Jia, and J. Liang, “Single-shot compressed ultrafast photography: a review,” Adv. Photonics 2(1), 014003 (2014). [CrossRef]

15. J. Y. Liang, C. Ma, L. R. Zhu, Y. J. Chen, L. Gao, and L. H. V. Wang, “Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse,” Sci. Adv. 3(1), e1601814 (2017). [CrossRef]

16. J. L. Yao, D. L. Qi, C. S. Yang, F. Y. Cao, Y. L. He, P. P. Ding, C. Z. Jin, Y. H. Yao, T. Q. Jia, Z. R. Sun, and S. A. Zhang, “Multichannel-coupled compressed ultrafast photography,” J. Opt. 22(8), 085701 (2020). [CrossRef]

17. J. L. Yao, D. L. Qi, Y. H. Yao, F. Y. Cao, Y. L. He, P. P. Ding, C. Z. Jin, T. Q. Jia, J. Y. Liang, L. Z. Deng, Z. R. Sun, and S. Zhang, “Total variation and block-matching 3D filtering-based image reconstruction for single-shot compressed ultrafast photography,” Opt. Laser Eng. 139, 106475 (2021). [CrossRef]

18. Y. Liu, X. Yuan, J. L. Suo, D. J. Brady, and Q. H. Dai, “Rank Minimization for Snapshot Compressive Imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2990–3006 (2019). [CrossRef]

19. C. S. Yang, Y. H. Yao, C. Z. Jin, D. L. Qi, F. Y. Cao, Y. L. He, J. L. Yao, P. P. Ding, L. Gao, T. Q. Jia, J. Y. Liang, Z. R. Sun, and S. A. Zhang, “High-fidelity image reconstruction for compressed ultrafast photography via an augmented-Lagrangian and deep-learning hybrid algorithm,” Photonics Res. 9(2), B30–B37 (2021). [CrossRef]

20. A. K. Zhang, J. M. Wu, J. L. Suo, L. Fang, H. Qiao, D. D. U. Li, S. A. Zhang, J. T. Fan, D. L. Qi, Q. H. Dai, and C. Q. Pei, “Single-shot compressed ultrafast photography based on U-net network,” Opt. Express 28(26), 39299–39310 (2020). [CrossRef]

21. C. Z. Jin, D. L. Qi, Y. H. Yao, F. Y. Cao, L. Deng, S. X. Xu, Z. R. Sun, and S. A. Zhang, “Single-shot real-time imaging of ultrafast light springs,” Sci. China Phys. Mech. Astron. 64(11), 1–8 (2021). [CrossRef]

22. S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, “Plug-and-play priors for model based reconstruction,” in 2013 IEEE Global Conference on Signal and Information Processing (IEEE, 2013), pp. 945–948.

23. S. H. Chan, X. R. Wang, and O. A. Elgendy, “Plug-and-Play ADMM for Image Restoration: Fixed-Point Convergence and Applications,” IEEE Trans. Comput. Imaging 3(1), 84–98 (2017). [CrossRef]

24. M. Ljubenovic and M. A. T. Figueiredo, “Plug-and-play approach to class-adapted blind image deblurring,” IJDAR 22(2), 79–97 (2019). [CrossRef]

25. K. Zhang, W. Zuo, and L. Zhang, “Deep plug-and-play super-resolution for arbitrary blur kernels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, 2019), pp. 1671–1681.

26. Y. L. He, Y. H. Yao, D. L. Qi, Z. Y. Wang, T. Q. Jia, J. Y. Liang, Z. R. Sun, and S. A. Zhang, “High-speed super-resolution imaging with compressive imaging-based structured illumination microscopy,” Opt. Express 30(9), 14287–14299 (2022). [CrossRef]

27. P. P. Ding, Y. H. Yao, D. L. Qi, C. S. Yang, F. Y. Cao, Y. L. He, J. L. Yao, C. Z. Jin, Z. Q. Huang, L. Deng, L. Z. Deng, T. Q. Jia, J. Y. Liang, Z. R. Sun, and S. A. Zhang, “Single-shot spectral-volumetric compressed ultrafast photography,” Adv. Photonics 3, 045001 (2021). [CrossRef]

28. S. M. Zheng, Y. Liu, Z. Y. Meng, M. Qiao, Z. S. Tong, X. Y. Yang, S. S. Han, and X. Yuan, “Deep plug-and-play priors for spectral snapshot compressive imaging,” Photonics Res. 9(2), B18–B29 (2021). [CrossRef]

29. K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

30. H. Qiu, Y. Wang, and D. Meng, “Effective snapshot compressive-spectral imaging via deep denoising and total variation priors,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (IEEE, 2021), pp. 9127–9136.

31. P. Athavale, R. Xu, P. Radau, A. Nachman, and G. A. Wright, “Multiscale properties of weighted total variation flow with applications to denoising and registration,” Med. Image Anal. 23(1), 28–42 (2015). [CrossRef]

32. J. Liang, L. Zhu, and L. V. Wang, “Single-shot real-time femtosecond imaging of temporal focusing,” Light: Sci. Appl. 7(1), 1–10 (2018). [CrossRef]

33. A. Takahashi, M. Nishizawa, Y. Inagaki, M. Koishi, and K. Kinoshita, “New femtosecond streak camera with temporal resolution of 180 fs,” in Generation, amplification, and measurement of ultrashort laser pulses, (SPIE, 1994), pp. 275–284.

34. Hamamatsu Photonics, K.K. Guide to Streak Cameras (2008).

35. C. Yang, D. Qi, F. Cao, Y. He, J. Yao, P. Ding, X. Ouyang, Y. Yu, T. Jia, and S. Xu, “Single-shot receive-only ultrafast electro-optical deflection imaging,” Phys. Rev. Appl. 13(2), 024001 (2020). [CrossRef]

36. K. Zhang, W. M. Zuo, and L. Zhang, “FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising,” IEEE Trans. on Image Process. 27(9), 4608–4622 (2018). [CrossRef]

37. S. Diamond and S. Boyd, “CVXPY: A Python-embedded modeling language for convex optimization,” The Journal of Machine Learning Research 17, 2909–2913 (2016).

38. X. Yuan, “Generalized alternating projection based total variation minimization for compressive sensing,” in 2016 IEEE International Conference on Image Processing (IEEE, 2016), pp. 2539–2543.

39. H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (IEEE, 2019), pp. 658–666.

40. Y. Yang, Y. Li, Z. Guan, C. Yang, S. Zhang, F. Wang, and T. Li, “A diagnostic system toward high-resolution measurement of wavefront profile,” Opt. Commun. 456, 124554 (2020). [CrossRef]

41. Y. Li, L. Sheng, D. Hei, X. Li, J. Zhang, M. Li, and A. Qiu, “Dynamical analysis of surface-insulated planar wire array Z-pinches,” Phys. Plasmas 25(5), 052708 (2018). [CrossRef]

42. C. Z. Jin,"Code for weighted multi-scale denoising via adaptive multi-channel fusion for compressed ultrafast photography”, https://github.com/Jocooob/MCMS-PnP-CUP.

Data	IQA	Methods
Data	IQA	GAP-TV	TV-BM3D	DeSCI	FFDNet	DRUNet	DRUNet-MC	DRUNet-MCMS
Baboon	PSNR	21.22	21.65	24.35	23.97	24.66	26.03	26.66
Baboon	SSIM	0.5637	0.5715	0.6083	0.5678	0.5877	0.7079	0.7607
Barbara	PSNR	21.99	22.43	26.29	23.97	26.31	29.53	32.12
Barbara	SSIM	0.6052	0.6245	0.7883	0.6958	0.7878	0.8867	0.9356
Boats	PSNR	23.89	24.98	26.94	25.74	27.1	29.93	32.42
Boats	SSIM	0.6862	0.7286	0.7857	0.7489	0.7998	0.8736	0.924
Cameraman	PSNR	22.3	22.8	25.09	25.33	25.84	28.98	30.11
Cameraman	SSIM	0.6061	0.6384	0.7911	0.7894	0.8121	0.8594	0.8878
Castle	PSNR	23.81	24.19	25.09	24.47	25.3	28.08	29.46
Castle	SSIM	0.6977	0.7131	0.7803	0.748	0.7776	0.8558	0.8975
Face	PSNR	30.24	30.86	32.62	29.57	30.89	32.56	34.98
Face	SSIM	0.7848	0.7851	0.8383	0.7478	0.7756	0.8204	0.8848
House	PSNR	26.57	27.95	30.29	29.23	31.47	33.9	36.68
House	SSIM	0.7369	0.7762	0.8406	0.8157	0.8543	0.8711	0.9214
Man	PSNR	23.28	23.57	25.95	24.82	25.75	27.94	29.61
Man	SSIM	0.6508	0.6577	0.7379	0.6908	0.7165	0.7974	0.8558
Monarch	PSNR	21.79	22.19	24.77	24.32	25.67	29.12	31.48
Monarch	SSIM	0.6947	0.714	0.8184	0.8146	0.8591	0.9121	0.9427
Parrots	PSNR	24.61	26.07	27.1	26.9	28.4	31.27	33.34
Parrots	SSIM	0.7563	0.7884	0.8621	0.8382	0.8733	0.8962	0.9299
Peppers	PSNR	23.34	25.49	26	25.75	27.57	30.55	32.92
Peppers	SSIM	0.6794	0.7641	0.8117	0.8021	0.8419	0.8828	0.919
Sunflower	PSNR	25.36	26.33	27.36	26.37	27.44	30.16	32.83
Sunflower	SSIM	0.7228	0.7512	0.8131	0.7871	0.8159	0.8732	0.9265
Average	PSNR	24.03	24.88	26.82	25.87	27.20	29.84	31.88
Average	SSIM	0.6821	0.7094	0.7897	0.7539	0.7918	0.8531	0.8988

Data	IQA	Methods
Data	IQA	GAP-TV	TV-BM3D	DeSCI	FFDNet	DRUNet	DRUNet-MC	DRUNet-MCMS
Welding	PSNR	26.4	26.76	28.74	27.71	28.22	30.63	31.31
Welding	SSIM	0.6592	0.7056	0.7623	0.7593	0.7708	0.8212	0.8283
Energy	PSNR	28.97	31.51	34.59	33.82	34.29	36.19	37.73
Energy	SSIM	0.5024	0.7108	0.8925	0.968	0.9665	0.9635	0.9769
Filament	PSNR	25.16	25.93	28.85	29.19	30.64	34.63	36.11
Filament	SSIM	0.5896	0.7645	0.8172	0.9554	0.964	0.9633	0.9764
Detonator	PSNR	28.1	29.35	32.31	29.75	30.42	32.92	33.62
Detonator	SSIM	0.697	0.8053	0.9169	0.9074	0.9098	0.934	0.9246
Blowtorch	PSNR	26.49	28.47	32.18	31.12	32.61	35.5	36.23
Blowtorch	SSIM	0.4651	0.661	0.7156	0.9431	0.9554	0.9593	0.9694
Match	PSNR	27.94	28.99	32.07	33.1	34.19	37.99	38.55
Match	SSIM	0.7159	0.8351	0.9091	0.891	0.9223	0.9488	0.9524
Water	PSNR	27.27	28.51	30.7	30.35	31.64	34.54	36.63
Water	SSIM	0.6546	0.777	0.8717	0.9343	0.9426	0.9591	0.9645
Average	PSNR	27.19	28.50	31.35	30.72	31.72	34.63	35.74
Average	SSIM	0.6120	0.7513	0.8408	0.9084	0.9188	0.9356	0.9418

Data	IQA	Methods
Data	IQA	GAP-TV	TV-BM3D	DeSCI	FFDNet	DRUNet	DRUNet-MC	DRUNet-MCMS
Baboon	PSNR	21.22	21.65	24.35	23.97	24.66	26.03	26.66
Baboon	SSIM	0.5637	0.5715	0.6083	0.5678	0.5877	0.7079	0.7607
Barbara	PSNR	21.99	22.43	26.29	23.97	26.31	29.53	32.12
Barbara	SSIM	0.6052	0.6245	0.7883	0.6958	0.7878	0.8867	0.9356
Boats	PSNR	23.89	24.98	26.94	25.74	27.1	29.93	32.42
Boats	SSIM	0.6862	0.7286	0.7857	0.7489	0.7998	0.8736	0.924
Cameraman	PSNR	22.3	22.8	25.09	25.33	25.84	28.98	30.11
Cameraman	SSIM	0.6061	0.6384	0.7911	0.7894	0.8121	0.8594	0.8878
Castle	PSNR	23.81	24.19	25.09	24.47	25.3	28.08	29.46
Castle	SSIM	0.6977	0.7131	0.7803	0.748	0.7776	0.8558	0.8975
Face	PSNR	30.24	30.86	32.62	29.57	30.89	32.56	34.98
Face	SSIM	0.7848	0.7851	0.8383	0.7478	0.7756	0.8204	0.8848
House	PSNR	26.57	27.95	30.29	29.23	31.47	33.9	36.68
House	SSIM	0.7369	0.7762	0.8406	0.8157	0.8543	0.8711	0.9214
Man	PSNR	23.28	23.57	25.95	24.82	25.75	27.94	29.61
Man	SSIM	0.6508	0.6577	0.7379	0.6908	0.7165	0.7974	0.8558
Monarch	PSNR	21.79	22.19	24.77	24.32	25.67	29.12	31.48
Monarch	SSIM	0.6947	0.714	0.8184	0.8146	0.8591	0.9121	0.9427
Parrots	PSNR	24.61	26.07	27.1	26.9	28.4	31.27	33.34
Parrots	SSIM	0.7563	0.7884	0.8621	0.8382	0.8733	0.8962	0.9299
Peppers	PSNR	23.34	25.49	26	25.75	27.57	30.55	32.92
Peppers	SSIM	0.6794	0.7641	0.8117	0.8021	0.8419	0.8828	0.919
Sunflower	PSNR	25.36	26.33	27.36	26.37	27.44	30.16	32.83
Sunflower	SSIM	0.7228	0.7512	0.8131	0.7871	0.8159	0.8732	0.9265
Average	PSNR	24.03	24.88	26.82	25.87	27.20	29.84	31.88
Average	SSIM	0.6821	0.7094	0.7897	0.7539	0.7918	0.8531	0.8988

Data	IQA	Methods
Data	IQA	GAP-TV	TV-BM3D	DeSCI	FFDNet	DRUNet	DRUNet-MC	DRUNet-MCMS
Welding	PSNR	26.4	26.76	28.74	27.71	28.22	30.63	31.31
Welding	SSIM	0.6592	0.7056	0.7623	0.7593	0.7708	0.8212	0.8283
Energy	PSNR	28.97	31.51	34.59	33.82	34.29	36.19	37.73
Energy	SSIM	0.5024	0.7108	0.8925	0.968	0.9665	0.9635	0.9769
Filament	PSNR	25.16	25.93	28.85	29.19	30.64	34.63	36.11
Filament	SSIM	0.5896	0.7645	0.8172	0.9554	0.964	0.9633	0.9764
Detonator	PSNR	28.1	29.35	32.31	29.75	30.42	32.92	33.62
Detonator	SSIM	0.697	0.8053	0.9169	0.9074	0.9098	0.934	0.9246
Blowtorch	PSNR	26.49	28.47	32.18	31.12	32.61	35.5	36.23
Blowtorch	SSIM	0.4651	0.661	0.7156	0.9431	0.9554	0.9593	0.9694
Match	PSNR	27.94	28.99	32.07	33.1	34.19	37.99	38.55
Match	SSIM	0.7159	0.8351	0.9091	0.891	0.9223	0.9488	0.9524
Water	PSNR	27.27	28.51	30.7	30.35	31.64	34.54	36.63
Water	SSIM	0.6546	0.777	0.8717	0.9343	0.9426	0.9591	0.9645
Average	PSNR	27.19	28.50	31.35	30.72	31.72	34.63	35.74
Average	SSIM	0.6120	0.7513	0.8408	0.9084	0.9188	0.9356	0.9418

Weighted multi-scale denoising via adaptive multi-channel fusion for compressed ultrafast photography

Abstract

1. Introduction

2. Principles

3. Results and Discussion

4. Conclusion

Funding

Disclosures

Data availability

Supplemental document

References

Supplementary Material (3)

Data availability

Cited By

Figures (5)

Tables (2)

Equations (17)

Optics Express

Name	Description
Supplement 1	Supplemental 1
Visualization 1	Reconstructed videos of the Weld (top) and Detonator (bottom) scenarios in simulation by DRUNet, DRUNet-MC, DRUNet-MCMS, together with the ground truths for comparison.
Visualization 2	Reconstructed spatiotemporal evolution of the spatially modulated ps laser pulse in experiment by DRUNet, DRUNet-MC, DRUNet-MCMS, together with the ground truths for comparison.