Open Access
Add to BibSonomy
Abstract
Single-photon counting imaging is a novel computational imaging technique that counts every photon collected by reflective light; it has target detection capability under extremely low-light conditions and thus has elicited increasing research interest. However, a low single-photon counting number and considerable noise will significantly affect image quality under low-light conditions. To improve the quality of single-photon counting image efficiently, we propose a robust single-photon counting imaging method with spatially correlated and total variation (TV) constraints. A robust Poisson negative log-likelihood function is introduced as a data fidelity term, which is robust to some spatial points that have extremely large background count in real situations. The TV regularization constraint is adopted to reduce noise. Considering that the reflectivity of several spatially correlated points may be similar, we suggest adding another constraint based on the counting information from these points rather than a single point for estimating reflectivity in each pixel. This approach will be helpful in reducing truncation errors. The proposed imaging model is formulated on the basis of the aforementioned factors. The alternative direction multiplier method is used to solve the optimization problem. The superiority of the proposed method over state-of-the-art techniques is verified on simulated and real captured experimental datasets under different conditions.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Photon counting imaging can be used to detect targets under extremely low-light conditions in which other imaging techniques experience difficulty in capturing efficient data. Therefore, this method has elicited considerable research interests in many fields, including space monitoring [1], biological imaging [2], fluorescence [3] and microscopy [4]. In contrast to most imaging techniques that measure the reflective light intensity of a target, photon counting imaging counts every photon collected by reflective light from different spatial points. Therefore, photodetectors that can detect individual photons play an important role in photon counting imaging. Historically, photomultiplier tubes are the primary photodetectors. In recent years, Geiger-mode avalanche photodiode (GM-APD) [5,6], also called single-photon avalanche photodiode (SPAD), which exhibits excellent sensitivity for single-photon detection, has been the focus of studies [7–9]. For GM-APD, two types of single-photon counting imaging system are used: array-based photon counting imaging system (APCIS) [10] and scanning-based photon counting imaging system (SPCIS) [11]. In the APCIS, the photon counting image is recovered from a detector array with a global illumination; its advantages include fast imaging, small size and low power consumption, which is the developing trend of single-photon counting imaging. Given the difficulty of integrating SPADs and readout circuits, the spatial resolution of commercial detector arrays is constantly low (e.g. $32\times 32$ [12]) and the cost is high, which brings obstacles for research purpose of APCIS. Therefore, SPCIS, in which the target is illuminated with a scanning laser and then detected point-by-point using only one SPAD, is always used to analyze the performance for single-photon counting imaging of APCIS in recent studies [13–15].
A typical single-photon counting imaging method based on SPCIS involves calculating the maximum likelihood (ML) of target reflectivity by using the number of reflective photons from every spatial point. Background noise is high under low-light conditions. Therefore, the ML imaging method cannot generate a satisfactory result. In 2016, Altmann et al. proposed a new Bayesian model and developed an adaptive Markov chain Monte Carlo algorithm to estimate the unknown model parameters, which exhibits evident advantages in low-photon-count imaging [16]. Considering the influence of noise under low-light conditions, Poisson distribution is always used to measure the counting process of reflective photons [17]. To remove the Poisson noise, variance-stabilizing transform (VST) can be used to approximate the Poisson distribution by Gaussian distribution, and a conventional denoising algorithm, such as block-matching and three-dimensional (3D) filtering (BM3D) [18], is further adopted for noise removal [19]. However, this approximation is inaccurate in most single-photon counting imaging applications as the received number of photons is very low [17,20], leading to high-frequency artifacts. Although an iterative algorithm [21] that can improve the effectiveness of VST is proposed for Poisson denoising, its efficiency remains limited. Non-local principal component analysis (PCA), which combines dictionary learning and the sparse patch-based representations of images, is suitable for Poisson noise reduction [22], but it always blurs the edges. In a recent work on single-photon counting imaging [23], reconstruction from counting data is always accomplished by minimizing a negative Poisson log-likelihood term. Furthermore, sparsity is constantly added as a prior constraint to reduce noise [24]. In [25], sparsity based on discrete wavelet transform (DWT) is applied to reduce noise in single-photon counting images. In 2014, Kirmani et al. proposed a first-photon imaging (FPI) method that uses only the first photons received by a single-photon detector [26]. The acquisition of high-quality 3D images is achieved by suppressing Poisson noise in an image by using DWT-based sparsity regularization. In recent years, total variation (TV) has been widely used as a prior constraint, demonstrating its efficiency in image denoising [27,28]. In 2015, Dongeek Shin et al. considerably improved imaging quality on the basis of the FPI system by providing a fixed dwell time for each pixel [29] where TV is introduced. In 2017, Jeffrey H. Shapiro et al. proposed a constrained optimization-based framework that uses low photon counts to achieve super-resolution depth features, and thus, solving the problems of photon scarcity and blurring in the forward imaging kernel [30]. In recent years, some researchers have started to study deep learning (DL)-based method [31,32], wherein a sensor fusion approach is taken directly on the output of a single-photon detector. However, the performance of the DL method highly depends on the consistency of the training and test data, which limits its applications; thus, it is not of the focus of the present study.
Although previous studies have achieved various superior experimental results in single-photon counting imaging, their methods still suffer from the following limitations that degrade image quality: (1) The background count may not be a fixed value for all spatial points in single-photon counting images. For example, hot pixel always exists in APCIS as the dark count of some pixels is significantly larger than the average dark count of all pixels [33,34], thereby introducing several outliers (e.g. extremely bright pixels with large counting) into the single counting images. Therefore, the potential influence of the non-uniform property of background count should be considered. (2) The counting number is always a small integer value under low-light conditions (e.g. several or a dozen); thus, the reflectivity level in single-photon counting imaging will be insufficient to represent the true reflectivity of a target. This difference is called truncation error in this work. For example, the maximum single-photon counting number of 16 leads to a reflectivity level in a single-photon counting image of 4 bits/pixel. The insufficient reflectivity level will result in several false edges in single-photon counting imaging.
In this study, we propose a robust single-photon counting imaging method with spatially correlated and TV constraints. The major contributions of this work can be summarized as follows:
(1) To reduce the influence of the non-uniform property of background count, a robust Poisson negative log-likelihood function is proposed as a data fidelity term, wherein the background count is constituted of a constant representing the average background count and a sparse variable denoting the background count of some pixels deviated to the average value a lot. TV is incorporated as one prior constraint term to reduce the influence of noise whilst preserving edges.
(2) To reduce truncation error, we suggest adding another constraint based on the counting information from spatially correlated points rather than a single point. This constraint will be helpful in increasing reflectivity levels in single-photon counting imaging. The rationale behind this strategy is that targets always have similar patterns or structures with similar reflectivity.
(3) On the basis of the above mentioned factors, the proposed imaging model is formulated by trading off the importance among the aforementioned terms. The alternative direction multiplier method (ADMM) is adopted to solve the optimization problem. In the experiment, we verify the efficiency of the proposed methods on experimental data under different conditions in terms of image quality metrics and visual effects.
2. Methods
2.1 SPCIS and APCIS setups
Figure 1 shows the schematics of SPCIS and APCIS. In SPCIS, a periodic pulsed laser is used to illuminate a scene. After passing through a polarization beam splitter (PBS), the emitted lights are initially reflected to the target by a two-axis galvonometer with raster scanning. Then, these lights are reflected by the target and finally detected by the single-photon detector. Scanning is uniform in SPCIS. Therefore, each spatial point (an extremely small region along the horizontal direction) is illuminated by the same $p$ number of laser pulses. The single-photon counting module is used to measure the number of detected photons from each spatial point. In APCIS, a periodic pulsed laser is used to illuminate a scene with a diffuser. The reflected lights are detected by a single-photon detector array. Therefore, each pixel in APCIS correlates to one single-photon detector, and the image resolution is determined by the size of the single-photon detector array. The dark count of pixels in APCIS is different due to the integration difficulty of the single-photon detector array. This phenomenon results from many reasons, such as doping concentrations and the number of impurities. Consequently, some ‘hot pixels’, which have extremely large dark count, are manifested as white points (called as outliers) in the single-photon counting images, thereby degrading the image’s quality. In this study, SPCIS is built to analyze the performance of single-photon counting imaging similar to that in [13–15], and the result can be easily extended to APCIS.
2.2 Imaging model formulation
For the convenience of description in the remaining parts of the manuscript, scalars are denoted by lowercase letters, and matrices are denoted by bold uppercase letters. Lowercase bold letters denote the stacked column vectors of a matrix. $\mathbf {F}\in \mathbb {R}^{m \times n}$ denotes a single-photon counting image with spatial resolutions of $m$ and $n$. $\mathbf {F}{(i,j)}$ denotes the reflectivity of the spatial point with coordinate $(i,j)$.
In general, the proposed photon counting imaging model can be formulated as follows:
where $J(\mathbf {F})$ is the data fidelity term and $K(\mathbf {F})$ represents the prior constraints to $\mathbf {F}$. In the following section, we present the construction details of $J(\mathbf {F})$ and $K(\mathbf {F})$.(1) Construction of the data fidelity term $J(\mathbf {F})$
In general, the Poisson negative log-likelihood [29] of reflectivity $\mathbf {F}{(i,j)}$ given the counting number $\mathbf {G}(i,j)$ is expressed as
(2) Construction of the prior constraint term $K(\mathbf {F})$
The prior constraint term $K(\mathbf {F})$ is composed of two constraints: TV $K_1(\mathbf {F})$ and spatially correlated $K_2(\mathbf {F})$ constraints.
(2.1) TV constraint
TV constraint is based on the fact that images with excessive and possibly spurious details will have considerable TV values. This condition implies that reducing the TV of an image will be helpful in removing noises whilst preserving edges [35,36]. Therefore, TV regularization is adopted as one prior constraint term in our work and is formulated as
where $\mathbf {D}_h$ and $\mathbf {D}_v$ are the gradient operators in the horizontal and vertical directions, respectively, and are represented as(2.2) Spatially correlated constraint
The counting number of each spatial point is low under low-light conditions, leading to insufficient reflectivity levels in single-photon counting images. Considering that similar patterns or structures can always be found in a target, we are motivated to find spatially correlated points from similar patterns or structures. Given that the reflectivity of these points are similar, we propose to use their counting information as a constraint to estimate reflectivity. This strategy will be helpful in increasing the number of reflectivity levels (reducing truncation error) and improving image quality. Figure 2 illustrates this motivation. As shown in Fig. 2(a), the counting number of single-photon counting image is no more than $16$, thus the reflectivity level of current point is $4$ bits/pixel. Compared with the corresponding grayscale image ($8$ bits/pixel) of the same target illustrated in Fig. 2(b), the reflectivity level of single-photon counting image is insufficient to describe the accurate information of the target. In Fig. 2(a), some spatially correlated points with similar reflectivity can be easily found, thereby the counting information from these points can be used to reduce the truncation error. To reduce the computation cost, the search area is defined as a relatively large search region around each current point, rather than the whole image. It is difficult to find spatial-correlated points based on the difference of these points directly due to the influence of the noise. We notice that the structures of blocks centered in these spatially correlated points are similar. Therefore, block-based operation can be used to find these spatial-correlated points, which is a general operation based on the calculation of the similarity of different blocks [18,22]. Blocks with a small size $M \times N$ are utilized to find spatially correlated points for a current point by using the block matching algorithm [37,38]. Large block size will be helpful to reduce the influence of noise, but it will increase the computation cost. Subsequently, the $K$ most similar spatially correlated points for each current point are collected for computation to reduce the truncation error of reflectivity. As a result, the number of reflectivity level can be increased to approximate $4+\log _2{K}$ bits/pixel. The selection of $K$ depends on the similarity of blocks of the single-photon counting image.
Fig. 2. Motivation of the spatially correlated constraint. (a) single-photon counting image. (b) grayscale image.
The similarity of blocks is calculated on the basis of the mean absolute difference (MAD) and defined as
In (3), the $L0$ norm is difficult to solve such that the $L1$ norm is always used as an approximation of the $L0$ norm. Thus, by integrating the data fidelity term $J(\mathbf {F})$ and the prior constraint terms $K_1(\mathbf {F})$ and $K_2(\mathbf {F})$, the proposed model after substituting $\mathbf {B}$ by $b+\mathbf {D}$ is formulated as
2.3 Solution to optimization
To solve (8), ADMM is adopted to divide the original optimization problem into small subproblems that can be easily handled. By introducing auxiliary variables $\mathbf {Z}=\eta \mathbf {S}\mathbf {F}+b+\mathbf {D}$, the Lagrangian form $L(\mathbf {F},\mathbf {D},\mathbf {Z},\mathbf {Y})$ of (8) can be expressed as
1) $\mathbf {F}$ subproblem:
The equivalent form of (10) can be described as
2) $\mathbf {D}$ subproblem:
(11) has a closed-form solution based on the soft thresholding operator [41], the result of which is as follows:
where $\mathbf {V}=\mathbf {Z}_t-(\eta \mathbf {S}\mathbf {F}_{t+1}+b+\frac {\mathbf {Y}_t}{\rho })$ and $\odot$ is a point-by-point multiplication operation.3) $\mathbf {Z}$ subproblem:
By supposing the derivative (with respect to $\mathbf {Z}$) of (12) to be $0$, we have
4) $\mathbf {Y}$ subproblem:
$\mathbf {Y}$ can be directly updated from (13).
On the basis of the aforementioned inference, the optimization algorithm involved in the proposed method is summarized in Algorithm 1.
|
3. Results
In this section, we perform a series of experiments on simulated and real captured data to verify the efficiency of the proposed method.
3.1 Simulation results
To quantify the performance of the methods in the simulation, we introduce two quantitative metrics: peak signal-noise-ratio (PSNR), which describes the difference in each pixel between two images, and structural similarity (SSIM), which measures the similarity to the ground truth image. PSNR and SSIM are defined as follows:
A simulation process similar to that in [42] is adopted in our simulation. We use the ‘Male’ image with a resolution of $256\times 256$ from the image database of the University of Southern California Signal and Image Processing Institute [43] as the ground truth image. Poisson noise, which is caused by the photoelectric conversion of the sensor, is generally the most common background noise under low-light conditions. In addition, outliers due to the non-uniform property of background count, such as some hot pixels in APCIS, are simulated as impulse noise added into the background count. Lastly, simulation is performed using the following steps:
(1) The ground truth reflectivity image is generated from the normalized intensity ‘Male’ image.
(2) First, we use the ground truth reflectivity image to calculate the counting probability, and then impulse noise with $0.5\%$ possibility is added to simulate the outliers of the background count. Poisson noise is further added.
(3) Finally, we multiply it with the number of $p$ emitted pulses for each pixel. The results, which are the simulated noiseless photon counting images, are rounded to integral values.
We compare our method with four state-of-the-art algorithms, namely, BM3D with VST in iterative Poisson denoising (VST+BM3D) [21], Poisson non-local sparse PCA (NLSPCA) [22], sparse Poisson intensity reconstruction algorithm (SPIRAL)-orthonormal basis (SPIRAL-ONB) [26], and binomial SPIRAL-TV [29]. SPIRAL minimizes a regularized negative log-likelihood objective function with various penalties. SPIRAL-ONB utilizes the $L1$ norm of the DWT coefficients, and binomial SPIRAL-TV, which is under the Poisson approximation to the binomial distribution, indicates the TV penalty. In the two SPIRAL methods, we set a minimum of $10$ and maximum of $100$ iterations with a convergence tolerance $tol=10^{-8}$. In NLSPCA and VST+BM3D, the parameters are set using the suggestions in code distributed with [21] and [22]. In our method, the search region is defined as a square with a size of $21 \times 21$ and the block size is $5 \times 5$. The regularization parameters are $\lambda =100$ and $\rho =0.0018$. The maximum iteration number is set to $10$. For fair comparison, other parameters of each approach is adjusted to achieve their best performance.
First, we simulate the situation that the numbers of emitted pulses per for each pixel are $p=200$ and $p=100$. The scanning process is supposed to be uniform; as a result, the signal count $\mathbf {S}$ for each pixel is a constant. Table 1 lists the objective comparison results. By adopting TV as a constraint, binomial SPIRAL-TV performs better than SPIRAL-ONB based on an evaluation by PSNR and SSIM. In general, the performance of NLSPCA and binomial SPIRAL-TV is close. The reason is that the truncation error can be reduced by using the spatially correlated points in NLSPCA, but binomial SPIRAL-TV has better denoising capability than NLSPCA. VST+BM3D and the proposed method outperform other methods in terms of PSNR and SSIM. When $p=200$, the proposed method is improved by approximately $1.6$ dB in PSNR compared with VST+BM3D, indicating that the proposed method is close to the ground truth. However, VST+BM3D has better structure information than the proposed method; thus, its SSIM value is higher. When SNR is reduced by changing $p$ to $100$, the influence of noise will increase, thereby degrading the image’s quality. As the proposed method is more robust to noise, the noise influence is smaller in the proposed method than in other methods. For example, the reduction of PSNR in the proposed method is approximately $0.66$ dB. VST+BM3D and the proposed method still perform better than other methods. In comparison with VST+BM3D, the improvement of PSNR is increased to around $2$ dB while the difference of SSIM is reduced. The reason is that VST works inefficiently when the received number of photons is very low.
Table 1. Objective comparison results with different numbers of emitted pulses for each pixel: $\mathbf {p=200}$ and $\mathbf {p=100}$.
The aforementioned conclusion can be also verified from the visual comparison results ($p=100$), presented in Fig. 3. Figure 3(b) shows the simulated photon counting image without noise. From the feather part, the image quality is lower than that of the original image. This difference is generated by the truncation error, which is in the rounding process. Figure 3(c) displays the simulated photon counting image with additional noise. Noise seriously degrades the image quality. In particular, the impulse noise will induce some outliers, which have extremely bright pixels with large counting or dark pixels with small counting in the single-photon counting image. Figures 3(d)–3(h) show the results of our method compared with the four traditional algorithms. From the enlarged section in the lower right area, the influence of noise is apparent in the reconstructed images using the SPIRAL-ONB method. Blocky artifacts can be also observed. Binomial SPIRAL-TV achieves better denoising results than SPIRAL-ONB by adopting the Poisson approximation of the binomial distribution with TV constraint. Although binomial SPIRAL-TV can efficiently suppress noise, it considerably amplifies truncation errors at the cost of reducing the influence of noise. Therefore, several false edges and oversmoothed details are always found in this method. As the spatially correlated points are used in calculating the reflectivity, the truncation errors can be efficiently reduced in NLSPCA, VST+BM3D and our method. Therefore, more gray levels can be observed in these three methods. Although NLSPCA can reduce noise, its edges are blurry. VST+BM3D demonstrates excellent denoising capability. However, some parts in its result are oversmoothed. Importantly, some high-frequency artifacts, such as false stripes, are evident, which will affect the scientific analysis in some applications. In our method, the influences of truncation errors and noise are efficiently controlled by trading off the importance of different terms in the proposed model. In particular, truncation errors can be reduced by considering the counting information from spatially correlated points. Meanwhile, sufficient details can be provided as the influence of noise is reduced by adopting the robust Poisson negative log-likelihood function and incorporating TV as a constraint. Then, we can conclude that our method achieves the best performance from the aspects of objective metrics and visual effects.
Fig. 3. Visual comparison results of different algorithms ($p=100$). (a) Ground truth. (b) Simulated photon counting image without noise. (c) Simulated photon counting image with noise. (d) SPIRAL-ONB. (e) Binomial SPIRAL-TV. (f) NLSPCA. (g) VST+BM3D. (h) Proposed method.
3.2 Experimental results
In this section, real data are captured from SPCIS to verify the efficiency of the proposed method further. The illumination source is a $640$ nm-wavelength pulsed laser diode (NPL64B, Thorlabs) with a tunable pulse width and repetition rate. Thorlabs GVS002, a dual-axis galvo scanning system, is used to scan the target. The field of view is limited by the maximum mechanical scan angle of the galvo system ( $\pm 12.5$°). A Tektronix AFG3152C signal generator supplies two-axis input analogue voltages to the galvo. After scanning the target, the scattered light is reflected by PBS and finally sent back to the single-photon counting module Thorlabs SPCM20A to obtain photon data. The detector has a quantum efficiency of $35\%$, an effective area of $180$ $\mu$m, and less than $25$ dark counts per second. We set the two analogue inputs as sawtooth signals with an amplitude of $3$ V, ensuring a line-by-line scanning mode and a square scanning area. The maximum total photon count of the single-photon counting module SPCM20A is $10000$, i.e., $10000$ points are acquired, and we eventually obtain an image with $100\times 100$ pixels. The raw data are required to have a short dwell time $T_a = 0.1$ ms at each pixel. To verify the generality of the method, we obtain a series of real datasets for two targets, namely, a Chinese character and a ceramic cup. The targets are placed at position $d = 1$ m from the detector. In our experiment, outliers are mainly induced by the light leakage of the PBS in the scanning process. These lights are received by the detector after multiple reflections, leading some pixels to have a larger background count than other pixels.
3.2.1. Experimental results at different repetition rates
First, we set the repetition rate at $5$ MHz. The reconstruction results of all methods are presented in Fig. 4. The results show that our method can remove environmental noise more effectively than the SPIRAL-ONB method. The reconstruction results of our method have more gray levels and clearer edges than those of binomial SPIRAL-TV. To suppress the noise, especially the outliers, the reconstruction result of NLSPCA is blurry. In VST+BM3D, some outliers can be still observed. Although VST+BM3D provides clear edges, some visible artifacts (e.g. false stripes) exist in the image. Our method can obtain more image details and present a better visual effect than SPIRAL-ONB, binomial SPIRAL-TV and NLSPCA. Less visible artifacts can be found in the proposed method, which will be closer to the real target, than in VST+BM3D. Subsequently, we reduce the repetition rate to $1$ MHz. That is, the number of laser-emitting pulses decreases during dwell time $T_a$, whereas the photon count received by the detector at each pixel declines. The quality of the raw data images in Fig. 5(b) and Fig. 5(i) is significantly lower than that in Fig. 4(b) and Fig. 4(i), and the proportion of outliers is increased in the data. As shown in Fig. 5, the SPIRAL-ONB and NLSPCA methods do not perform well in noise suppression. Although binomial SPIRAL-TV reduces the influence of noise, it amplifies the effects of truncation errors, leading to oversmoothed edges and a limited number of reflectivity levels. Given the inaccurate approximation of VST in a lower number of photons, several visible artifacts (e.g. the false netted texture on the ceramic cup) can be found in VST+BM3D. Thus, this phenomenon will be further serious when the number of received photons is furtherly reduced. Our method is robust to different types of noise even when the number of received photons is relatively small. As a result, clear edges can still be found in the proposed method.
Fig. 4. Reconstruction results of different methods at a repetition rate of 5 MHz. (a) Photograph of the target (a Chinese character). (b) Photon counting raw data of (a). (c) SPIRAL-ONB of (b). (d) Binomial SPIRAL-TV of (b). (e) NLSPCA of (b). (f) VST+BM3D of (b). (g) Proposed method of (b). (h) Photograph of the target (a ceramic cup). (i) Photon counting raw data of (h). (j) SPIRAL-ONB of (i). (k) Binomial SPIRAL-TV of (i). (l) NLSPCA of (i). (m) VST+BM3D of (i). (n) Proposed method of (i).
Fig. 5. Reconstruction results of different methods at a repetition rate of 1 MHz. (a) Photograph of the target (a Chinese character). (b) Photon counting raw data of (a). (c) SPIRAL-ONB of (b). (d) Binomial SPIRAL-TV of (b). (e) NLSPCA of (b). (f) VST+BM3D of (b). (g) Proposed method of (b). (h) Photograph of the target (a ceramic cup). (i) Photon counting raw data of (h). (j) SPIRAL-ONB of (i). (k) Binomial SPIRAL-TV of (i). (l) NLSPCA of (i). (m) VST+BM3D of (i). (n) Proposed method of (i).
3.2.2. Quantitative metric results under different SNR conditions
To evaluate the performance of different methods further, we calculated PSNR and SSIM to assess the performance of different methods under different SNR conditions. First, the ground truth data are generated by setting the dwell time to be extremely long with a large repetition rate, such as more than $5$ MHz, in our experiment. Then, the repetition rate is fixed at $1$ MHz with a common dwell time. We test the performance of different methods on fixed targets by placing an attenuator with different extinction ratios to produce high, medium and low SNR conditions. Some single-photon counting images and the corresponding photographs of the targets are shown in Fig. 6. The testing data should be well registered with the ground truth data before evaluation [44,45]. The objective comparison results are calculated based on the average of metrics of all targets, as shown in Fig. 7. As the denoising capability of SPIRAL-ONB and NLSPCA is always limited, these methods do not perform well. Binomial SPIRAL-TV can not manage the truncation errors; thus, its PSNR and SSIM are unsatisfactory. VST+BM3D and the proposed method outperform other methods. Generally, the performance of VST+BM3D and the proposed method is similar in high SNR condition. When the SNR is changed to medium condition, the performance of the proposed method evaluated by PSNR will be considerably better than that of VST+BM3D, and the SSIM values of these two methods are close. In low SNR condition, the influence of noise increases. The proposed method is robust to noise; hence, it demonstrates superior imaging capability and has the best performance in terms of PSNR and SSIM compared with other methods. Therefore, the proposed method is an effective single-photon counting imaging method, which will have extensive application prospect in single-photon counting imaging applications.
Fig. 6. Targets for quantitative analysis. (a) Single-photon counting image. (b) Corresponding photograph.
4. Conclusions
In this study, we propose a robust photon counting imaging method with spatially correlated and TV constraints. Under low-light conditions, truncation errors during image acquisition can lead to several false edges due to insufficient photon counts. To solve this problem, we consider using the correlation of adjacent spatial points and utilize the counting data from a small region to reduce truncation errors. Simultaneously, we propose a robust Poisson negative log-likelihood function as the fidelity term and introduce TV as the prior constraint term to reduce the influence of noise. ADMM is adopted to solve the optimization problem. In the simulation, we verify the efficiency of the proposed method in terms of visual effect and objective metrics, including PSNR and SSIM. Furthermore, different methods are compared on real data captured from a built SPCIS. The proposed method is robust to noise and provides clear details. Therefore, the proposed method is an effective single-photon counting imaging method, especially in low-light imaging applications.
Our future work will include the following: (1) In the proposed method, the parameters for balancing the importance of different terms used in our imaging model are determined by adopting a cross-validation strategy. Automatically finding the optimal parameters will be a topic of our future work. (2) In the present study, we use photon count to obtain reflectivity estimation without depth information. Therefore, our future work will be to study a 3D single-photon imaging model from single-photon counting information and time-of-flight data. (3) DL-based single-photon counting imaging method is also one of our future interests.
Funding
National Natural Science Foundation of China (61971315); Natural Science Foundation of Hubei Province (2018CFB435); Fundamental Research Funds for the Central Universities (2042018kf1009).
Acknowledgments
The authors wish to thank the editor and the anonymous reviewers for their valuable suggestions.
Disclosures
The authors declare no conflicts of interest.
References
1. S. Nie, C. Wang, X. Xi, S. Luo, G. Li, J. Tian, and H. Wang, “Estimating the vegetation canopy height using micro-pulse photon-counting LiDAR data,” Opt. Express 26(10), A520–A540 (2018). [CrossRef]
2. X. Michalet, O. Siegmund, J. Vallerga, P. Jelinsky, J. Millaud, and S. Weiss, “Photon-counting H33D detector for biological fluorescence imaging,” Nucl. Instruments Methods Phys. Res. Sect. A: Accel. Spectrometers, Detect. Assoc. Equip. 567(1), 133–136 (2006). [CrossRef]
3. H. Ikoma, M. Broxton, T. Kudo, and G. Wetzstein, “A convex 3D deconvolution algorithm for low photon count fluorescence imaging,” Sci. Rep. 8(1), 11489 (2018). [CrossRef]
4. S. Liu, Z. Zhang, J. Zheng, L. Xu, C. Kuang, and X. Liu, “Parallelized fluorescence lifetime imaging microscopy (FLIM) based on photon reassignment,” Opt. Commun. 421, 83–89 (2018). [CrossRef]
5. H. Luo, X. Yuan, and Y. Zeng, “Range accuracy of photon heterodyne detection with laser pulse based on Geiger-mode APD,” Opt. Express 21(16), 18983–18993 (2013). [CrossRef]
6. S. Li, Z. Zhang, Y. Ma, H. Zeng, P. Zhao, and W. Zhang, “Ranging performance models based on negative-binomial (NB) distribution for photon-counting lidars,” Opt. Express 27(12), A861–A877 (2019). [CrossRef]
7. D.-U. Li, J. Arlt, J. Richardson, R. Walker, A. Buts, D. Stoppa, E. Charbon, and R. Henderson, “Real-time fluorescence lifetime imaging system with a 32 × 32 0.13µm CMOS low dark-count single-photon avalanche diode array,” Opt. Express 18(10), 10257–10269 (2010). [CrossRef]
8. D. Shin, J. H. Shapiro, and V. K. Goyal, “Computational single-photon depth imaging without transverse regularization,” in Proceedings of IEEE International Conference on Image Processing, (IEEE, 2016), pp. 973–977.
9. G. Gariepy, N. Krstajic, R. Henderson, C. Li, R. R. Thomson, G. S. Buller, B. Heshmat, R. Raskar, J. Leach, and D. Faccio, “Single-photon sensitive light-in-fight imaging,” Nat. Commun. 6(1), 6021 (2015). [CrossRef]
10. M. O’Toole, F. Heide, D. B. Lindell, K. Zang, S. Diamond, and G. Wetzstein, “Reconstructing transient images from single-photon sensors,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2017), pp. 1539–1547.
11. L. Ye, G. Gu, W. He, H. Dai, J. Lin, and Q. Chen, “Adaptive target profile acquiring method for photon counting 3-D imaging lidar,” IEEE Photonics J. 8(6), 1–10 (2016). [CrossRef]
12. M. Itzler, G. Salzano, M. Entwistle, X. Jiang, M. Owens, B. Piccione, S. Wilton, K. Slomkowski, S. C. Roszko, and E. Wei, “Asynchronous Geiger-mode APD cameras with free-running InGaAsP pixels,” Proc. SPIE 10212, 102120K (2017). [CrossRef]
13. D. Shin, A. Kirmani, V. K. Goyal, and J. H. Shapiro, “Computational 3D and reflectivity imaging with high photon efficiency,” in Proceedings of IEEE International Conference on Image Processing, (IEEE, 2014), pp. 46–50.
14. D. Shin, F. Xu, F. N. C. Wong, J. H. Shapiro, and V. K. Goyal, “Computational multi-depth single-photon imaging,” Opt. Express 24(3), 1873–1888 (2016). [CrossRef]
15. F. Heide, S. Diamond, D. B. Lindell, and G. Wetzstein, “Sub-picosecond photon-efficient 3D imaging using single-photon sensors,” Sci. Rep. 8(1), 17726 (2018). [CrossRef]
16. Y. Altmann, X. Ren, A. McCarthy, G. S. Buller, and S. McLaughlin, “Lidar waveform-based analysis of depth images constructed using sparse single-photon data,” IEEE Trans. on Image Process. 25(5), 1935–1946 (2016). [CrossRef]
17. Z. T. Harmany, R. F. Marcia, and R. M. Willett, “This is SPIRAL-TAP: Sparse Poisson intensity reconstruction algorithms-theory and practice,” IEEE Trans. on Image Process. 21(3), 1084–1096 (2012). [CrossRef]
18. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. on Image Process. 16(8), 2080–2095 (2007). [CrossRef]
19. M. Makitalo and A. Foi, “Optimal inversion of the Anscombe transformation in low-count Poisson image denoising,” IEEE Trans. on Image Process. 20(1), 99–109 (2011). [CrossRef]
20. E. D. Kolaczyk, “Wavelet shrinkage estimation of certain Poisson intensity signals using corrected thresholds,” Stat. Sinica 9, 119–135 (1999).
21. L. Azzari and A. Foi, “Variance stabilization for noisy+estimate combination in iterative Poisson denoising,” IEEE Signal Processing Letters 23(8), 1086–1090 (2016). [CrossRef]
22. J. Salmon, Z. Harmany, C.-A. Deledalle, and R. Willett, “Poisson noise reduction with non-local PCA,” J. Math. Imaging Vis. 48(2), 279–294 (2014). [CrossRef]
23. D. Shin, F. Xu, D. Venkatraman, R. Lussana, F. Villa, F. Zappa, V. K. Goyal, F. N. Wong, and J. H. Shapiro, “Photon-efficient imaging with a single-photon camera,” Nat. Commun. 7(1), 12046 (2016). [CrossRef]
24. J. Jiang, J. Ma, C. Chen, X. Jiang, and Z. Wang, “Noise robust face image super-resolution through smooth sparse representation,” IEEE Trans. Cybern. 47(11), 3991–4002 (2017). [CrossRef]
25. A. Kirmani, D. Venkatraman, A. Colaço, F. N. Wong, and V. K. Goyal, “High photon efficiency computational range imaging using spatio-temporal statistical regularization,” Proceedings of IEEE Conference on Lasers and Electro-Optics, (IEEE, 2013), pp. 1–2.
26. A. Kirmani, D. Venkatraman, D. Shin, A. Colaço, F. N. C. Wong, J. H. Shapiro, and V. K. Goyal, “First-photon imaging,” Science 343(6166), 58–61 (2014). [CrossRef]
27. Y. Wen, M. K. Ng, and Y. Huang, “Efficient total variation minimization methods for color image restoration,” IEEE Trans. on Image Process. 17(11), 2081–2088 (2008). [CrossRef]
28. H. Fan, C. Li, Y. Guo, G. Kuang, and J. Ma, “Spatial–spectral total variation regularized low-rank tensor decomposition for hyperspectral image denoising,” IEEE Trans. Geosci. Remote Sensing 56(10), 6196–6213 (2018). [CrossRef]
29. D. Shin, A. Kirmani, V. K. Goyal, and J. H. Shapiro, “Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors,” IEEE Trans. Comput. Imaging 1(2), 112–125 (2015). [CrossRef]
30. D. Shin, J. H. Shapiro, and V. K. Goyal, “Photon-efficient super-resolution laser radar,” Proc. SPIE 10394, 1039409 (2017). [CrossRef]
31. T. Remez, O. Litany, R. Giryes, and A. M. Bronstein, “Deep convolutional denoising of low-light images,” arXiv preprint arXiv:1701.01687 (2017).
32. D. B. Lindell, M. O’Toole, and G. Wetzstein, “Single-photon 3D imaging with deep sensor fusion,” ACM Trans. Graph. 37(4), 1–12 (2018). [CrossRef]
33. I. Gyongy, A. Davies, N. A. Dutton, R. R. Duncan, C. Rickman, R. K. Henderson, and P. A. Dalgarno, “Smart-aggregation imaging for single molecule localisation with SPAD cameras,” Sci. Rep. 6(1), 37349 (2016). [CrossRef]
34. P. W. Connolly, X. Ren, R. K. Henderson, and G. S. Buller, “Hot pixel classification of single-photon avalanche diode detector arrays using a log-normal statistical distribution,” Electron. Lett. 55(18), 1004–1006 (2019). [CrossRef]
35. D. Strong and T. Chan, “Edge-preserving and scale-dependent properties of total variation regularization,” Inverse Probl. 19(6), S165–S187 (2003). [CrossRef]
36. J. Ma, C. Chen, C. Li, and J. Huang, “Infrared and visible image fusion via gradient transfer and total variation minimization,” Inf. Fusion 31, 100–109 (2016). [CrossRef]
37. F. Dufaux and F. Moscheni, “Motion estimation techniques for digital TV: A review and a new contribution,” Proc. IEEE 83(6), 858–876 (1995). [CrossRef]
38. J. Lu and M. L. Liou, “A simple and efficient search algorithm for block-matching motion estimation,” IEEE Trans. Circuits Syst. Video Technol. 7(2), 429–433 (1997). [CrossRef]
39. I. Markovsky, Low Rank Approximation: Algorithms, Implementation, Applications (Springer, 2012).
40. A. Beck and M. Teboulle, “A fast iterative shrinkage-thresholding algorithm for linear inverse problems,” SIAM J. Imaging Sci. 2(1), 183–202 (2009). [CrossRef]
41. N. Parikh and S. Boyd, “Proximal algorithms,” Foundations Trends Optim. 1(3), 127–239 (2014). [CrossRef]
42. D. Shin, “Computational 3D and reflectivity imaging with high photon efficiency,” Master’s thesis, Massachusetts Institute of Technology (2014).
43. A. G. Weber, “Usc-sipi image database version 5,” USC-SIPI Rep. 315, 1–24 (1997).
44. J. Ma, J. Zhao, and A. L. Yuille, “Non-rigid point set registration by preserving global and local structures,” IEEE Trans. on Image Process. 25(1), 53–64 (2016). [CrossRef]
45. J. Ma, J. Zhao, J. Jiang, H. Zhou, and X. Guo, “Locality preserving matching,” Int. J. Comput. Vis. 127(5), 512–531 (2019). [CrossRef]
