Open Access
Add to BibSonomy
Abstract
We present a method for multi-depth imaging that uses polarisation modulation to reconstruct multiple depths from photon-counting observations. The echo photon signals of the scene with multiple targets are modelled under the low-flux condition based on a confocal scanning system. We establish a computational method by constructing the relationship between the received photon rate after polarisation modulation and several variables described for multi-echo signals: the detected average number of photons before polarisation modulation and the flight time corresponding to phase shift based on polarisation modulation. The photon rate is directly calculated from the photon-counting value received by a photon-counting detector based on the Poisson negative log-likelihood function. We suggest solving the abovementioned relationship by changing the waveform of the operating voltage applied to the modulator to derive analytic forms of the detected average number of photons and the flight time. We experimentally demonstrate that the proposed method can accurately reconstruct the reflectivity and depth images of two targets in a 3D scene with a significant improvement beyond what the polarisation modulation imaging method for single-depth can achieve. Moreover, we demonstrate the effectiveness of the proposed method by varying the reflection properties of the targets, the number of echo photons (e.g. less than one photon in a pulse) and the background noise. We exhibit a fast imaging capability with the imaging frequency of one pixel at 8 kHz and the root mean-square error of depth smaller than 6 cm.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Time-of-flight (TOF) methods for 3D imaging have attracted considerable research interests in many applications, such as autonomous driving [1], search and rescue, and remote sensing [2,3], to capture the shape and depth of a scene. A single pixel of the 3D image may correlate to multiple reflective light as different depths due to the partially reflective characteristic of objects. Therefore, the problem of reconstructing multiple depths (typically called as multi-depth imaging [4]) plays an important role.
Typical TOF methods for obtaining multiple depths in 3D scenes can be classified into three main categories: amplitude-modulated continuous wave (AMCW) method, time slicing method and time histogram method. The AMCW method obtains distance information by emitting a modulated light signal and measuring the phase shift of the light reflected back onto the sensor [5,6]. To address the multipath interference (MPI) problem in the AMCW method, Qiao et al. [7] exploited a multi-frequency approach for decomposing specular MPI into its components using log-sum sparse regularisation. Marco et al. [8] further formulated the multi-depth imaging process in the AMCW method as a spatially varying convolution problem. However, the AMCW method cannot be applied in the low-light and long-range applications due to limited sensitivity and detection range. In recent years, single-photon avalanche photodiode (SPAD) [9–11], which has excellent sensitivity for single-photon detection, demonstrates a capability of low-flux TOF imaging at long distances (e.g. $45$ km) [12]. Therefore, SPAD has been widely adopted in the time slicing and time histogram method. In the time slicing method, time gate scanning is used to obtain transient images from SPAD camera for multi-depth imaging [13,14]. Morimoto et al. [15] presented a multi-depth detection with a spatial resolution of $700\times 500$ based on a novel photon-counting (quanta) image sensor [16,17]. However, a large number of transient images (e.g. a detection of $1.5$ m needs a maximum data of $25$ Gbps in [15]) are required in the time slicing method to analyse time information, which limits the detection range and time resolution. Different from the AMCW and time slicing methods, the time histogram method directly calculates the distance by obtaining the flight time through a time-to-digital converter (TDC) hardware [18]. In particular, the multiple depths can be identified from the time histogram generated by a pulsed illumination source with high emitting frequency. For example, Shin et al. [4] proposed a full-waveform imaging framework that recovers the multi-depth profile from single-photon observations. Tachella et al. [19] achieved 3D reconstructions of complex scenes with the time-correlated single-photon-counting lidar. Then, they further optimised the method to image outdoor scenes with an unknown number of surfaces [20]. However, the integration of TDC and SPAD is difficult [21,22], which leads to a low spatial resolution of the time histogram method.
Although various TOF methods have been investigated in the past few decades, multi-depth imaging with high spatial resolution under low-light conditions is still challenging due to the following reasons: (1) The AMCW method cannot work well when the echo signals are extremely small. (2) Multi-depth measurements always need a large detection range; therefore, tremendous data flow is generated and hard for processing in the time slicing method. (3) The spatial resolution of the time histogram method is always low due to the difficulty of the hardware integration. (4) Recently, polarisation modulation strategy [23], which can achieve fast 3D imaging with a high spatial resolution in low-light conditions, brings a new insight for depth measurement. However, the modulation strategy in [23] cannot be applied in the multi-depth imaging because the echo signals of the objects at different depths cannot be distinguished.
We propose a multi-depth photon-counting imaging based on polarisation modulation (MDPPM) to solve the abovementioned problem. In particular, we model the echo photon signals of the scene with multiple targets under the low-flux condition based on a confocal scanning system. We introduce a computational method by constructing the relationship between the received photon rate after polarisation modulation and several variables specialised for multi-echo signals: the detected average number of photons before polarisation modulation and the flight time corresponding to phase shift based on polarisation modulation. The photon rate is directly calculated by the photon-counting value received by a photon-counting detector based on the Poisson negative log-likelihood function. We further change the waveform of the operating voltage of the modulator to solve the aforementioned relationship. Consequently, efficient solutions of the detected average number of photons and the flight time can be derived.
The main contributions of this study are summarised as follows:
- - We use the polarisation modulation method to achieve multi-depth 3D imaging. Therefore, high spatial resolution with high ranging accuracy can be achieved under low-light conditions with small amount of received data.
- - We utilise an inhomogeneous Poisson process to describe the photon detection and model the echo photon signal of a multi-target scene based on the K-reflector model. We derive an efficient solution of the detected average number of photons before polarisation modulation and the flight time corresponding to phase shift based on polarisation modulation by solving their relationship with the received photon rate after polarisation modulation, where we mainly change the waveform of the operating voltage applied to the modulator. In particular, a typical example for a scene with two targets is given. This way allows us to achieve high-quality reconstruction in the multi-depth scene with a significant improvement beyond what the polarisation modulation imaging method for single-depth can achieve.
- - We build an experimental system of multi-depth photon-counting imaging to verify the efficiency of the proposed method in real experiments and the effectiveness of the reflection properties of the targets, the number of echo photons and the background noise. Such a system remains unverified to the best of our knowledge.
The rest of the paper is organised as follows. We introduce the proposed method in Section 2. The experimental results are presented in Section 3. Conclusions of this study are drawn in Section 4.
2. Methodology
2.1 Confocal scanning system for multi-depth imaging
The diagram of the proposed confocal scanning system for multi-depth imaging is shown in Fig. 1. A focused pulsed laser periodically illuminates a patch of the target scene with the repetition period $T_G$. We suppose the distance between the farthest target and the detector is $R$. Then, we have the following condition: $T_G>\frac {2R}{c}$, where $c$ is the speed of light to avoid aliasing. In Fig. 1, depths from two targets are selected as an example for demonstration. In our system, we assume that targets 1 and 2 are partially specular. Consequently, target 1 will reflect some light to illuminate a patch of target 2, and a portion of target 1’s diffuse light will return along the original path as an echo signal of target 1. Thereafter, a portion of target 2’s diffuse reflected light will further return along the original path as an echo signal of target 2. Similar to [13,24], we neglect the further inter-reflection between targets 1 and 2. We utilise a QWP to filter the multiple echo signals that have a polarisation state orthogonal to the laser beam. Thus, the passed multiple echo signals will be reflected by a PBS. As the multiple echo signals are usually weak to only a few photons, we use a PCD to receive the multiple echo signals after passing through the polarisation-modulated ranging module. We use a dual-axis galvo to scan a photon-counting image with $100\times 100$ pixels. Each spatial point is illuminated by one measurement with $N$ pulses. Therefore, we can utilise the number of received photons to calculate the intensity and distance of both the targets even when multiple echo signals are extremely weak (e.g. less than one photon). The reflection properties of the target including (1) a full specular reflector-like glass, (2) a partial specular reflector-like ceramic tile and (3) an ideal diffuse surface [25] are important factors in multi-depth imaging. Similar to related multi-depth imaging methods, the proposed method also assumes that the targets are all partially specular.
Fig. 1. Diagram of the proposed confocal scanning system for multi-depth imaging. QWP: quarter-wave plate. PBS: polarisation beam splitter. LP: linear polarizor. PCD: photon-counting detector.
In the low-flux condition, we use an inhomogeneous Poisson process to describe photon detections produced by the PCD. In a single measurement period $T_G$, the time-varying rate of the Poisson process can be expressed as $\eta \Re _{\theta, \alpha } s(t)+d$, where $\eta$ is the quantum efficiency of the PCD and $d$ represents the rate of an independent homogeneous Poisson process generated by detector’s dark counts. $\Re _{\theta, \alpha }$ is the photon passing rate of the polarisation-modulated ranging module, and it will be explained in detail in the Section 2.2. $s(t)$ represents the photon-flux waveform of echo signals measured in counts/seconds. The probability mass function $p(k)$ for the number of photons detected in response to a single measurement period $(0, T_G]$ is expressed as follows:
whereFor the multiple targets, we can utilise the K-reflector model [26,27] to describe $s(t)$ as
where $N_{si}$ and $t_i$ are the detected average number of photons before polarisation modulation in a single pulse and flight time of target $i$, respectively. $K$ is the number of the targets.2.2 Principle of MDPPM
In MDPPM, we use an electro-optic phase modulator (EOM) [28,29] to modulate the echo signals and calculate the detected average number of photons before polarisation modulation and flight time from the received photon-counting values. Under a specific operating voltage, EOM controls the phase shift to ensure that the echo signals have the polarisation-modulated state related to the photon flight time. Therefore, we can calculate the polarisation-modulated state by the calculated photon rates from received photon-counting values based on the Poisson negative log-likelihood function. Subsequently, the detected average number of photons before polarisation modulation and flight time can be obtained by multiple polarisation modulation. The principle of the MDPPM with two targets is shown in Fig. 2 for intuitive explanation. The photons reflected from two targets at different distances pass through LP1, EOM and LP2 in turn, and then, they are received by PCD. The polarisation direction of LP1 ($p$ axis) is $45^{\circ }$ from the $x$ axis and the polarisation direction of LP2 is at an angle $\theta$ to the $p$ axis. We describe the effects of the EOM by introducing phase shift $\alpha$, which is proportional to the operating voltage of EOM. Thus, the photon passing rate of the polarisation-modulated module can be described as follows:
We use $V_t$ to represent the instantaneous voltage of EOM and $V_{\pi }$ to denote the half-wave voltage (the voltage value when the phase shift of the EOM is $\pi$). Then, we have $\alpha =(\pi V_t)/V_{\pi }$. When we apply a voltage that linearly increases with time (a period of $T_G$) to the EOM ($V_t=V_{\pi }/T_G t$), the phase shift will be also linearly increased with time, i.e., $\alpha =\frac {\pi }{T_G} t$. Then, we can rewrite Eq. (2) based on Eq. (3) and Eq. (4) as follows:
Under the low-light condition, the PCD obtains either zero detection with the probability $p(0)$ or one detection with the complementary probability in each period. Thus, received photon-counting value $k$ is binomially distributed as follows:
where $k=0,1,\ldots,N$. $N$ is the number of laser pulses.On the basis of Eq. (6), the negative log-likelihood function $\mathcal {L}$ of $m$ given photon-counting value $k$ is computed as follows [30]:
By maximising the negative log-likelihood function $\mathcal {L}$ in Eq. (7), we can obtain:
From Eq. (5) and Eq. (8), we can calculate the photon rate $m$ from the photon-counting value $k$, which contains the information of $N_{si}$ and $t_i$.
2.3 Implementation of MDPPM
In the following, we utilise a typical two-depth scene as an example to demonstrate the implementation of MDPPM. In particular, we need to solve four variables $N_{s1}$, $N_{s2}$, $t_{1}$ and $t_{2}$ in this application. In this study, we propose a new polarisation modulation strategy that only adjusts the operating voltage of the EOM whilst the direction of LP2 remains constant. Compared with the previous polarisation modulation method [23] (need to adjust the direction of LP2 and the operating voltage of the EOM), this strategy can simplify the operation process and is more conducive to the realisation of multi-step parallel operation. To achieve a analytic solution, we specifically adjust the operating voltage of the EOM to derive six equations for the solution of $N_{s1}$, $N_{s2}$, $t_1$ and $t_2$ in the following steps:
(1) We keep the direction of the LP2 to $45^{\circ }$ (parallel to the $y$ axis) and set the voltage linearly decrease within $[0,V_{\pi })$, i.e., $V_t=\frac {V_{\pi }}{T_G}(T_G-t)$. Then, we can obtain
Based on Eq. (9), Eq. (10), Eq. (11), Eq. (12), Eq. (13) and Eq. (14), $N_{s1}$, $N_{s2}$, $t_1$ and $t_2$ can be calculated as
3. Experiments and results
We construct an experimental system to verify the multi-depth imaging performance of the proposed method in a common scene with two targets. The pulsed laser (Thorlabs NPL64B) demonstrates a wavelength of $640$ nm with a working frequency of $10$ MHz. The EOM (Thorlabs EO-PM-R) exhibits a working frequency of $10$ MHz and a half-wave voltage of $7$ V. The PCD (Thorlabs SPCM20A/M) is used to obtain photon-counting values. This detector has a quantum efficiency of $23\%$ at a wavelength of $640$ nm, an effective area of $180$ $\mathrm{\mu}$m and approximately $25$ dark counts per second. Figure 3 shows the experimental setup of the imaging system and our experimental scene. Target 1 is placed approximately $100$ cm away from the imaging system. With a lateral tilt, target 1 can reflect the laser onto target 2 behind the occlusion. The distance between targets 2 and 1 is approximately $50$ cm. Since the maximum capacity of PCD in a single measurement is limited to $10000$ bins, leading to an imaging resolution of $100\times 100$ in our experiment.
Fig. 3. Experimental setup of the imaging system and experimental scene. (a) confocal scanning system for multi-depth imaging. (b)–(c) positions of targets 1 and 2.
3.1 Comparison experiment of polarisation modulation for single-depth and multi-depth
Traditional polarisation-modulated imaging methods for single-depth [23,31] cannot reconstruct a multi-target scene with high quality because the reflected signals between multiple targets will cause interference. MDPPM can effectively distinguish the information of each target by modulating and calculating the echo signals of multiple targets. This way can reduce the interference between multiple targets in photon-counting imaging for guaranteeing high quality of target reconstruction. To demonstrate the abovementioned viewpoint, we conduct the following two experiments: (1) we select a white plastic cube as target 1 and use two plates with different reflection properties (coating plate with a high specular reflection component and plastic plate with a high diffuse reflection component) in another position as target 2. In this experiment, the $N_{s1}$ calculated by MDPPM method can be directly regarded as the reflectivity of the cube and target 2 is used as an interference object. (2) We change the positions of targets 1 and 2 from the experiment (1). Consequently, the calculated $N_{s2}$ in the experiment (2) is related to the reflectivity of targets 1 and 2. In this experiment, we assume that target 1 has uniform reflectivity; therefore, the calculated $N_{s2}$ is mainly correlated to the reflectivity of target 2. We compare the traditional photon-counting polarisation-modulated imaging method for single-depth (PPSD) [23] and the proposed polarisation-modulated imaging method for multi-depth to image the plastic cube in the two different experimental scenes, respectively. For each experiment, we set the number of pulses per pixel in one measurement to $1000$, i.e., $N=1000$. The reconstruction results are shown in Fig. 4, and the results of quantitative analysis are given in Table 1.
Fig. 4. Comparison of reconstruction results in different methods. (a) RGB image, ground truth of photon-counting image and ground truth of depth images. The unit of the color bar: cm. The red profile on the photon-counting image is analysed quantitatively. (b) Reconstruction result of experiment (1). (c) Reconstruction result of experiment (2).
Table 1. Quantitative indicators of reconstruction results of different methods.
Figure 4(a) shows the RGB image, the ground truth of photon-counting image and the ground truth of depth images. The ground truth of photon-counting image is directly captured by the PCD when the cube is placed $100$ cm away. The ground truth of depth images are generated by the PPSD, where the cube is placed $100$ and $150$ cm away from the PCD without multiple reflections. The ground truth of photon-counting image and the depth images are taken as ground truth of the reflectivity and depths (the ground truth images in the following experiments are all obtained in this way). On the basis of the ground truth, we calculate the contrast of the red profile from the photon-counting image and the root mean square error (RMSE) of the depth image for quantitative analysis. The contrast is calculated by $C=\frac {I_{max}-I_{min}}{I_{max}+I_{min}}$, where $I_{max}$ and $I_{min}$ are the maximum and minimum of the red profile from the photon-counting image. Figure 4(b) is the reconstruction results of experiment (1). The detected average number of photons is about $1.3$. The reconstruction result of the PPSD is poor due to the interference of plate as target 2. In particular, when target 2 is the plastic plate with a large diffuse reflection component, the pattern on the cube is difficult to distinguish and the contrast of the calculated area is low. Therefore, the PPSD cannot effectively reconstruct the depth of the cube. By contrast, the MDPPM method can clearly reconstruct the spatial details of the cube, and the RMSE of the reconstruction in depth is smaller than $2$ cm. Figure 4(c) is the reconstruction results of experiment (2), where the pulsed laser scans the cube behind the occlusion through the plate. The detected average number of photons is about $0.8$. The echo signals include the signal directly reflected from the plate and the signal indirectly reflected from the surface of the cube. The PPSD and MDPPM have clear reconstructed spatial quality of the cube when target 1 is the coating plate with high specular reflection component. However, the depth of PPSD is inaccurate. When the plastic plate with high diffuse reflection component is used as target 1, the reconstruction results of the PPSD have no efficient information, whilst the MDPPM can reconstruct a slight blurry cube. Furthermore, the RMSE of the reconstructed depth image is smaller than $5$ cm. Therefore, we can conclude that the PPSD cannot efficiently reconstruct a single object in the multi-depth scene with high quality, and it may even fail completely in the case of strong multi-reflected light. However, MDPPM can be effectively applied in a multi-depth scene by modelling and modulating multi-reflection echo.
3.2 Imaging experiments with different reflection properties of targets
We use objects with different reflection properties in the following experiment to analyse the influence of the reflection properties of targets. For each experiment, we set $N=1000$ and the detected average number of photons is about $1.2$. The experimental results are demonstrated in Fig. 5. We choose glass plate, coating plate and plastic plate as target 1, as shown in Fig. 5(a), where their specular reflection components are gradually decreased. At the same time, we use the metal triangle plate (Fig. 5(b)) with high specular reflection component and the plastic cube (Fig. 5(c)) with low specular reflection component as target 2. We define $\frac {N_2}{N_1}$ as the parameter of reflection properties to quantitatively evaluate the reflection properties of targets 1 and 2. $N_1$ and $N_2$ are the sum of received average number of photons of all pixel points from targets 1 and 2, respectively. Therefore, an increase in $\frac {N_2}{N_1}$ implies an increase in the specular reflection component of targets 1 and 2. Figures 5(a)–(c) show the ground truths of the plates (as target 1), metal triangle plate and plastic cube. The reconstruction results of metal triangle plate and plastic cube as target 2 are shown in Figs. 5(d) and (e), respectively. Figures 5(d) and (e) show that, when $\frac {N_2}{N_1}$ is within $[0.8, 2.5]$, the reconstruction performance of the MDPPM is superior, and targets 1 and 2 obtain high-quality reconstruction. However, when $\frac {N_2}{N_1}$ is extremely large, the reconstruction of target 1 will be difficult given that the echo signals are weakened. Accordingly, when $\frac {N_2}{N_1}$ is too small, the reconstruction of target 2 will also be challenging. However, the proposed method is still effective in the two cases. For example, when $\frac {N_2}{N_1}$ is equal to $3.1$, the reflectivity image of target 1 is relatively accurate, with only some artefacts, and the measurement error of the depth image is only slightly increased. When $\frac {N_2}{N_1}$ is equal to $0.3$, the reflectivity image of target 2 becomes fuzzy, but the pattern of number $20$ on the object is still distinguishable, and the reconstruction of depth image is also effective. This observation proves the effectiveness of MDPPM for targets with different reflection properties.
Fig. 5. Experimental results of imaging with targets of different reflection properties . (a) Plates of three different reflection properties taken as target 1, and the RGB images and the ground truth of depth image are given. (b)–(c) RGB images, ground truth of photon-counting images and ground truth of depth images of the metal triangle plate and plastic cube. The blue and red profiles on the ground truth of photon-counting images are used to calculate the contrast of the reflectivity images. (d)–(e) Reconstruction results of different targets by the proposed method. (f) Contrast. (g) SNR. (h) RMSE. The unit of the color bars: cm.
A quantitative analysis including contrast of reflectivity image and the RMSE is shown in Table 2. Considering that the reflectivity of the three plates as target 1 is uniform, we do not calculate the contrast of their reflectivity images. The effectiveness of the proposed method can be efficiently demonstrated. For example, the average contrast of the selected section for target 2 in Table 2 is $0.44$. In the meantime, the average RMSEs of depth images of targets 1 and 2 are $1.62$ and $4.4$ cm, respectively.
Table 2. Quantitative indicators of reconstruction results of different reflection properties. Subscript (1) represents the data for target 1, and subscript (2) represents the data for target 2.
3.3 Imaging experiments with extremely weak echo signal and background noise
We use a cup with a curved shape and complex pattern to demonstrate the proposed method of imaging performance at extremely weak echo signals and high background noise. We inject extraneous background light using an incandescent lamp. We utilise an attenuator before the laser to adjust the detected average number of the photons ($N_s$) reflected from the targets to analyse the effects of extremely weak echo signal and background light noise on imaging. We use the coating plate as target 1 and the cup behind the occlusion as target 2. For each experiment, we set $N=100$. The reconstruction results are shown in Fig. 6. Figures 6(a)–(c) are the reconstruction results when the echo signal is weakened from $N_s=1$ to $N_s=0.5$. Similar to the echo signal, the background light is also reflected by the galvo and the polarisation beam splitter, and then passed through the polarization modulation module before received by the PCD. In a single measurement period $T_G$, the average number of photons from background light received by the PCD is about $0.7$, which is comparable to the average number of signal photons. The first line is the reconstruction results without background noise, and the second line is the reconstruction results with background noise. Reconstruction results of the proposed method are effective, and the background noise has small influence on the reconstruction results. When we attenuate the echo signal to $N_s=0.5$, although the reconstruction results of the reflectivity images have noise and the error of the reconstruction results of the depth images also increases, the pattern on the target can still be identified and the reconstruction performance of the proposed method is stable after the addition of background noise. Figure 6(d) shows the RGB image, ground truth of photon-counting image and ground truth of depth image. We calculate the contrast of the region (labelled by the red box) of the photon-counting image and the RMSE of the depth image for quantitative analysis. The results in Figs. 6(e)–(f) show that the influence of background noise increases with the reduction in $N_s$. However, the proposed method can still achieve an effective reconstruction result with contrast above $0.36$ and RMSE smaller than $6$ cm when $N_s=0.5$.
Fig. 6. Experimental results of imaging with extremely weak echo signal and background noise. (a)–(c) Reconstruction results when the echo signal is weakened from $N_s=1$ to $N_s=0.5$. The first line is the reconstruction results without background noise, and the second line is the reconstruction results with background noise. (d) RGB image, ground truth of photon-counting image, and ground truth of depth image of the cup. (e) Contrast of the area in the red box on the photon-counting image. (f) RMSE of the depth images. BL: background light.
3.4 Fast imaging capability
We further demonstrate the fast imaging capability of the proposed method using a cup with letter pattern. The imaging frequency of a single pixel for one measurement is set to $1$, $4$, $8$ and $10$ kHz. Accordingly, $N$ is set to $1000$, $250$, $125$, and $100$, respectively. This condition implies the possibility of fast-imaging speed for an array-based photon-counting imaging system if the proposed technique is adopted. We use a coating plate as target 1 and a ceramic cup as target 2. We set the detected average number of photons is about $0.5$. The reconstruction results are shown in Fig. 7. As shown in Fig. 7(a), the reconstruction results of the reflectivity images are generally clear. For example, when the imaging frequency is increased to $10$ kHz, the shape of the cup and the outline of the letters’ pattern could still be distinguished in the reconstruction result of the reflectivity image. The quality of reconstruction results of the depth image are gradually decreased with the increase in the imaging frequency. Figure 7(b) shows the RGB image, ground truth of photon-counting image and ground truth of depth image. We calculate the contrast of the red profile labelled in the photon-counting image and the RMSE of the depth image for quantitative analysis. The results in Figs. 7(c)–(d) show that the reconstruction outcomes can maintain stable performance when the imaging frequency is smaller than $8$ kHz. When the imaging frequency increases to $10$ kHz, an obvious decrease in contrast and an increase in RMSE are observed, and this condition degrades the performance of the proposed method.
Fig. 7. Experimental results of fast imaging. (a) Reconstruction results when the imaging frequency of a single pixel for one measurement is $1$, $4$, $8$ and $10$ kHz. (b) RGB image, ground truth of photon-counting image and ground truth of depth image of the cup. The unit of the color bars is cm. (c) Contrast of the red profile on the photon-counting image. (d) RMSE of the depth images.
4. Conclusion
In this paper, we propose a multi-depth photon-counting imaging method based on polarisation modulation. To the best of our knowledge, this study is the first to apply polarisation modulation to multi-depth imaging to achieve high spatial resolution reconstruction under low-light conditions. We use an inhomogeneous Poisson process to describe the photon detection and model the echo photon signal of multi-target scene based on K-reflector model. We establish a computational method by calculating the relationship amongst the received photon rate before polarisation modulation, the detected average number of photons after polarisation modulation and flight time corresponding to phase shift based on polarisation modulation. The photon rate is calculated from the photon-counting value by the photon-counting detector based on Poisson negative log-likelihood function. To solve the detected average number of photons and flight time of different targets, we select a typical scene of two targets as an example, where we change the waveform of the operating voltage applied to the modulator to derive six equations for an efficient solution. In the experiment, we first demonstrate that the proposed method can accurately reconstruct the reflectivity and depth images of two targets in a 3D scene with a significant improvement beyond what the polarisation modulation imaging method for single-depth can achieve. Furthermore, we verify the effectiveness of the proposed method under different conditions including targets with different reflection properties, extremely low echo photon number (less than one photon in a pulse) and background noise. We exhibit a fast imaging capability with the imaging frequency of one pixel at $8$ kHz and ranging accuracy of RMSE smaller than $6$ cm. In the future, the following studies will be our next work: (1) We will focus on multi-depth imaging of diffuse objects, where light scattering should be considered. (2) We will design a system that can perform multiple imaging steps in parallel to further improve the imaging speed and simplify the imaging process. (3) We will apply the MDPPM to mechanical-scan-free systems with 2D photon-counting detector arrays to achieve multi-depth imaging with high spatial resolution and high frame rate.
Funding
National Natural Science Foundation of China (61971315); Natural Science Foundation of Hubei Province (2018CFB435).
Acknowledgments
The authors wish to thank the editor and the anonymous reviewers for their valuable suggestions.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. J. Hecht, “Lidar for self-driving cars,” Opt. Photonics News 29(1), 26–33 (2018). [CrossRef]
2. Z.-P. Li, J.-T. Ye, X. Huang, P.-Y. Jiang, Y. Cao, Y. Hong, C. Yu, J. Zhang, Q. Zhang, C.-Z. Peng, F. Xu, and J.-W. Pan, “Single-photon imaging over 200 km,” Optica 8(3), 344–349 (2021). [CrossRef]
3. X. Tian, R. Liu, Z. Wang, and J. Ma, “High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision,” Inf. Fusion 77, 19–28 (2022). [CrossRef]
4. D. Shin, F. Xu, F. N. Wong, J. H. Shapiro, and V. K. Goyal, “Computational multi-depth single-photon imaging,” Opt. Express 24(3), 1873–1888 (2016). [CrossRef]
5. J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Fourier analysis on transient imaging with a multifrequency time-of-flight camera,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2014), pp. 3230–3237.
6. M. L. Simpson, M.-D. Cheng, T. Q. Dam, K. E. Lenox, J. R. Price, J. M. Storey, E. A. Wachter, and W. G. Fisher, “Intensity-modulated, stepped frequency cw lidar for distributed aerosol and hard target measurements,” Appl. Opt. 44(33), 7210–7217 (2005). [CrossRef]
7. H. Qiao, J. Lin, Y. Liu, M. B. Hullin, and Q. Dai, “Resolving transient time profile in ToF imaging via log-sum sparse regularization,” Opt. Lett. 40(6), 918–921 (2015). [CrossRef]
8. J. Marco, Q. Hernandez, A. Munoz, Y. Dong, A. Jarabo, M. H. Kim, X. Tong, and D. Gutierrez, “Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging,” ACM Trans. Graph. 36(6), 1–12 (2017). [CrossRef]
9. W. Chen, S. Li, and X. Tian, “Robust single-photon counting imaging with spatially correlated and total variation constraints,” Opt. Express 28(2), 2625–2639 (2020). [CrossRef]
10. I. Gyongy, S. W. Hutchings, A. Halimi, M. Tyler, S. Chan, F. Zhu, S. McLaughlin, R. K. Henderson, and J. Leach, “High-speed 3D sensing via hybrid-mode imaging and guided upsampling,” Optica 7(10), 1253–1260 (2020). [CrossRef]
11. M. Buttafava, F. Villa, M. Castello, G. Tortarolo, E. Conca, M. Sanzaro, S. Piazza, P. Bianchini, A. Diaspro, F. Zappa, G. Vicidomini, and A. Tosi, “SPAD-based asynchronous-readout array detectors for image-scanning microscopy,” Optica 7(7), 755–765 (2020). [CrossRef]
12. Z.-P. Li, X. Huang, Y. Cao, B. Wang, Y.-H. Li, W. Jin, C. Yu, J. Zhang, Q. Zhang, C.-Z. Peng, F. Xu, and J.-W. Pan, “Single-photon computational 3D imaging at 45 km,” Photonics Res. 8(9), 1532–1540 (2020). [CrossRef]
13. A. Kirmani, T. Hutchison, J. Davis, and R. Raskar, “Looking around the corner using ultrafast transient imaging,” Int. J. Comput. Vis. 95(1), 13–28 (2011). [CrossRef]
14. A. C. Ulku, C. Bruschini, I. M. Antolović, Y. Kuo, R. Ankri, S. Weiss, X. Michalet, and E. Charbon, “A 512 × 512 SPAD image sensor with integrated gating for widefield FLIM,” IEEE J. Sel. Top. Quantum Electron. 25(1), 1–12 (2019). [CrossRef]
15. K. Morimoto, A. Ardelean, M.-L. Wu, A. C. Ulku, I. M. Antolovic, C. Bruschini, and E. Charbon, “Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications,” Optica 7(4), 346–354 (2020). [CrossRef]
16. J. Ma, S. Masoodian, D. A. Starkey, and E. R. Fossum, “Photon-number-resolving megapixel image sensor at room temperature without avalanche gain,” Optica 4(12), 1474–1481 (2017). [CrossRef]
17. A. Gnanasambandam, O. Elgendy, J. Ma, and S. H. Chan, “Megapixel photon-counting color imaging using quanta image sensor,” Opt. Express 27(12), 17298–17310 (2019). [CrossRef]
18. J. Rapp, Y. Ma, R. M. Dawson, and V. K. Goyal, “High-flux single-photon lidar,” Optica 8(1), 30–39 (2021). [CrossRef]
19. J. Tachella, Y. Altmann, X. Ren, A. McCarthy, G. S. Buller, S. Mclaughlin, and J.-Y. Tourneret, “Bayesian 3D reconstruction of complex scenes from single-photon lidar data,” SIAM J. Imaging Sci. 12(1), 521–550 (2019). [CrossRef]
20. J. Tachella, Y. Altmann, N. Mellado, A. McCarthy, R. Tobin, G. S. Buller, J.-Y. Tourneret, and S. McLaughlin, “Real-time 3D reconstruction from single-photon lidar data using plug-and-play point cloud denoisers,” Nat. Commun. 10(1), 4984 (2019). [CrossRef]
21. C. Zhang, S. Lindner, I. M. Antolović, J. M. Pavia, M. Wolf, and E. Charbon, “A 30-frames/s, 252×144 SPAD flash LiDAR with 1728 dual-clock 48.8-ps TDCs, and pixel-wise integrated histogramming,” IEEE J. Solid-State Circuits 54(4), 1137–1151 (2019). [CrossRef]
22. R. K. Henderson, N. Johnston, F. M. Della Rocca, H. Chen, D. D.-U. Li, G. Hungerford, R. Hirsch, D. McLoskey, P. Yip, and D. J. Birch, “A 192 × 128 time correlated SPAD image sensor in 40-nm CMOS technology,” IEEE J. Solid-State Circuits 54(7), 1907–1916 (2019). [CrossRef]
23. R. Liu, X. Tian, and S. Li, “Polarisation-modulated photon-counting 3D imaging based on a negative parabolic pulse model,” Opt. Express 29(13), 20577–20589 (2021). [CrossRef]
24. A. Turpin, V. Kapitany, J. Radford, D. Rovelli, K. Mitchell, A. Lyons, I. Starshynov, and D. Faccio, “3D imaging from multipath temporal echoes,” Phys. Rev. Lett. 126(17), 174301 (2021). [CrossRef]
25. K. Xu, W. Jin, S. Zhao, J. Liu, H. Guo, S. Qiu, and D. Wu, “Image contrast model of non-line-of-sight imaging based on laser range-gated imaging,” Opt. Eng. 53(6), 061610 (2013). [CrossRef]
26. D. Freedman, Y. Smolin, E. Krupka, I. Leichter, and M. Schmidt, “SRA: Fast removal of general multipath for ToF sensors,” in European Conference on Computer Vision, (Springer, 2014), pp. 234–249.
27. A. M. Wallace, J. Ye, N. J. Krichel, A. Mccarthy, R. J. Collins, and G. S. Buller, “Full waveform analysis for long-range 3D imaging laser radar,” EURASIP J. Adv. Signal Process. 2010(1), 896708 (2010). [CrossRef]
28. K. Beha, D. C. Cole, P. Del’Haye, A. Coillet, S. A. Diddams, and S. B. Papp, “Electronic synthesis of light,” Optica 4(4), 406–411 (2017). [CrossRef]
29. M. Zhang, C. Wang, P. Kharel, D. Zhu, and M. Lončar, “Integrated lithium niobate electro-optic modulators: when performance meets scalability,” Optica 8(5), 652–667 (2021). [CrossRef]
30. D. Shin, A. Kirmani, V. K. Goyal, and J. H. Shapiro, “Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors,” IEEE Trans. Comput. Imaging 1(2), 112–125 (2015). [CrossRef]
31. Z. Chen, B. Liu, E. Liu, and Z. Peng, “Electro-optic modulation methods in range-gated active imaging,” Appl. Opt. 55(3), A184–A190 (2016). [CrossRef]
