Open Access
Add to BibSonomy
Abstract
Non-line-of-sight (NLOS) detecting technology is used to locate and track a hidden object through scattered light after three reflections. This technology is expected to be applied in automatic driving, danger rescue, etc. Herein, we improve the real-time performance of NLOS detection in two aspects. Firstly, we introduce the method of dynamic programming to detect extremely weak signals from the hidden object based on current and previous information, which can unmix the target signal from noise at a shorter exposure time. Secondly, we simplify the positioning method by extracting the time difference from the histogram and training a feed-forward network that speeds up the process of spatial positioning. Based on the improvements, we demonstrated fast tracking experiments in a corridor 20 m away, tracking a moving hidden object at framerates of 50 Hz. These improvements make NLOS fast detection a step forward.
© 2021 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Recently, non-line-of-sight (NLOS) imaging has been developed rapidly [1–8]. Gathering information out of sight has important significance, ranging from automatic driving to danger rescue. Furthermore, this technology will have many applications in the future.
For NLOS imaging, in 2012, Velten et al. designed a system to reconstruct the obscured object through three bounce indirect reflections based on the method of backpropagation, using an ultrafast pulsed laser and a streak camera [9]. Subsequently, some improved methods were proposed [10–12]. In 2018, Batarseh et al. measured the spatial coherence of light fields to retrieve the information of the hidden object [13]. In 2018, O’Toole et al. explored a light-cone transform method regarding the process of imaging as a linear inverse problem, which reduced the computational complexity [14–16]. Different from using the particle nature of light, some methods modeled the light transport through wave optics [17–20]. Besides, machine learning technology was introduced in solving the problem of NLOS [6,21–26].
In some of the applications, it is more useful to quickly locate and track hidden objects. For NLOS detection, in 2016, Gariepy et al. located and tracked the hidden object 1 m away, based on the photon time-of-flight, using a single-photon avalanche diode (SPAD) camera [27]. By applying a gauss fitting function, they measured the time-of-flight from the laser to their camera and thus formed an ellipse representing the probability distribution of the hidden object. As a result, they achieved tracking of the moving object by 3 seconds intervals. In addition, they pointed out the extremely low signal-to-noise ratio (SNR) is the limitation of this technique. In 2017, Chan et al. showed a tracking system that extended the distance to a long range of 53 m with 4 seconds to collect data. In their experiments, they used a slanted chalkboard as the scattering surface to simulate an open door [28]. Then, in 2017, Chan et al. used three single-pixel SPAD to collect the returned photons at the same time, and updated the position of the hidden object every 1.5 s [29]. In 2018, O’Toole et al. also conducted experiments of tracking a planar traffic sign in space at framerates of 3 Hz with 0.1s exposure time, in which they measured the time-of-flight at three detection points on the wall and positioned the hidden object by finding the intersection part of the three spheres [14]. As SPAD requires a long-time collection of returned photons, which affects the real-time performance, in 2019, James et al. use an avalanche photodiode to locate the hidden object at the corner of 4.4 m, and the tracking frequency is 10 Hz [30]. In contrast, the single-photon detector is better in detection distance and accuracy. In 2020, a Gauss filtering algorithm was introduced to improve the SNR in NLOS detection [31]. In 2020, Isogawa et al. proposed a method for positioning the hidden object by estimating the parameters in the transient sinograms to position the hidden object [8]. In 2021, Wu et al. tracked a moving object at 1.43 km [4], using three telescopes combined with SPAD to collect the returned photons, and tracked an object at a frame rate of 1 Hz. Due to the limitation of the signals with low SNR, the real-time performance is affected.
Two main reasons affect the real-time performance of NLOS detection. One reason is that the collected histogram with an extremely low SNR requires more acquisition time to collect more returned photons. Reducing the acquisition time will result in a lower SNR. The other reason is that the traditional positioning method has a large amount of computation. The method regards the peak in the collected histogram as the probability distribution of the arrival time for photons and maps the time distribution to the probability density in space. In the process, the spatial probability density is calculated for each pixel, which takes a lot of time.
In this work, we introduce two methods to improve the real-time performance of NLOS detection. Firstly, for the aspect of data utilization, we introduced the method of dynamic programming (DP) to detect weak signals based on current and previous histograms, regarding the detection as a multi-stage optimization problem. Secondly, we extracted the time difference from the histogram and trained a network to learn the relationship between the time difference and the spatial position of the hidden object, which reduced the amount of computation. Based on the improvements, we demonstrated fast tracking experiments in a real scene with a wall as the scattering surface, which results in fewer echo photons and lower SNR. Ultimately, we tracked a moving hidden object 20 m away at a frequency of 50 Hz.
2. Experimental setup
We performed the detecting experiments in a corridor at a distance of 20 m to demonstrate the feasibility of NLOS fast tracking. Firstly, the detection distance of 20 meters causes the attenuation of photons in the process of emission and collection, which will make the signal poor. Secondly, the scenarios of the experiments are shown in Fig. 1 (a), in which the scattering surface is a white wall in the real scene, making the effect of diffuse reflection worse and the number of returned photons less, as shown in Fig. 1 (b). Thirdly, at the receiving end, we used only small imaging lenses to collect photons without the use of telescopes. Under the limitation of SNR, in order to reduce the acquiring time, we used a high-power pulsed laser in the experiments, which can emit more photons of each light pulse. Meanwhile, we adopted the efficient superconducting nanowire single-photon detector (SNSPD) that can detect more returned photons entering the lens. In addition, we used four lenses to collect the returned photons at the same time, which avoids the scanning process and improves the collection efficiency.
Fig. 1. Experimental scenarios. (a) shows that we send a lot of photons to the wall with a pulsed laser and detect the returned photons after multiple reflections with the lenses. The photons entering the lenses are detected by SNSPD, and then the TCSPC output the histograms. The hidden object at the corner is out of our sight. (b) shows a picture of the experimental environment.
The system incorporated a pulsed laser (1550 nm peak wavelength, 40 MHz repetition frequency, ∼100ps pulse width, ∼2W average power), a time-correlated single-photon counting (TCSPC) module (qutools-quTAG, time jitter less than 9.9ps, time resolution of 1ps, maximum counting rate of 33 MHz), an SNSPD (SINGLE-QUANTUM, detection efficiency is between 70% and 80%, time jitter 33-79ps, deadtime less than 30 ns, dark count rate less than 300 Hz), and four imaging lenses (50 mm focal length).
The process of the NLOS detection is as follows. The pulsed laser illuminates a spot on the wall at the end of the corridor. Then, the light pulse emitted by the laser reaches the hidden object after diffuse reflection on the wall and returns to the wall the next time. We measure the photon arrival time at the different positions on the wall with four lenses simultaneously. The collected photons are filtered through a bandpass filter (1550${\pm} $2.4 nm) to reduce the noise. Subsequently, the filtered photons are coupled with fibers and then transferred to the SNSPD. Finally, the TCSPC module output the photon arrival time histogram subsequently.
3. Weak signal detection using dynamic programming
3.1 Principles
Reducing the acquisition time not only requires more photons to be detected on the device, but also requires us to algorithmically separate the weak signals from the noise. Hence, in this section, we introduced the DP method to detect weak signals using the current and previous histograms. This method accumulates the optimal values from the previous frame to the current histogram iteratively at each stage. Using this method, each current histogram contains the information of multiple frames, thus the current histogram is enhanced because of the correlation between frames. The advantages of this method are as follows. Through improving the data utilization, the acquisition time can be compressed further. Especially, the method is suitable for detecting moving objects. Extending the collection time causes the photon arrival time to be mixed for the hidden object in different locations, resulting in a wider third bounce which is difficult to locate. The DP method accumulates the optimal values of the previous frame in the current histogram, which effectively highlights the current arrival time and finds the current object location easily.
As depicted in Fig. 2, the first step is to set the number of iterations m and initialize the value function ${I_{k - m}}$ with ${w_{k - m}}$. The second step is to iteratively calculate the value function of the current stage ${I_k}$ through the optimization function, described by Eq. (1). Then, we can find the maximum value of the current stage value function ${I_k}^{\ast }$ by Eq. (2) and the corresponding photon arrival time ${i_k}^{\ast }$ by Eq. (3). The general procedure for DP is listed in Algorithm 1. We assume that the length of the histogram is N.
Fig. 2. The Schematic of DP method. In the first place, we assume that the number of iterations is m, and initialize the value function ${I_{k - m}}$ with ${w_{k - m}}$. Then we iteratively calculate the current value function ${I_k}$ by accumulating the optimal value of the previous stage to the current histogram. According to the current value function ${I_k}$, the optimal position ${i_k}^{\ast }$ can be derived. ${q_k}$ is the range to find the optimal value in the value function. In addition, the optimization strategy is the maximum of the value function.
3.2 Experimental testing of the DP method
In this section, we tested the effectiveness of the DP method, collecting 0.01s to find the location of the target signal in the environment that is shown in Fig. 1 (a). When the acquisition time is quite short and the number of received photons is very small, it is difficult to identify the location of the target signal and obtain the information about the hidden object, as shown in the first row of Fig. 3. We used the DP method combined with a mean filter to filter the random noise, as described in Eq. (5).
where g(i) represents the filtered value for each time bin, f(i) represents the origin value for each time bin, k is the number in the filtered window, L is the length of the filtered window and N is the total bins in the histogram. Then, we iterated three times (m=3) to obtain the value function of the current histogram. Subsequently, we can see the location of the third bounce, as shown in the second row of Fig. 3. It is beneficial for the subsequent operation. In theory, the acquisition time can be reduced by m+1 times using this method, thereby reducing the acquisition time and increasing the tracking frequency. The number of iterations m should be a trade-off between acquisition time and computation time. Excessive iterations will increase the computation time and thus reduce the tracking frequency. Relying too much on previous information can lead the system to be unresponsive.Fig. 3. Testing results of DP method at a short exposure. The four images in the first row are background-subtracted histograms after shifting the first bounce to the origin. The four images in the second row are the result histograms that easily identify the target signal.
The DP method applies not only to stationary objects but also to moving objects. For stationary objects, we can accumulate the previous frames on the current histogram directly, which is equivalent to collecting multiple frames. However, for moving objects, a direct accumulation of the histograms of the previous frames may introduce errors. We experimentally collect the histograms for faster moving object. And the speeds of the moving object were about 2 cm/s, 5 cm/s, 10 cm/s, and 20 cm/s, respectively. To compare the effectiveness of the two methods, we regard the position of the third bounce in current histogram as the reference position. Therefore, we choose the obvious target signals that can be seen easily.
In Fig. 4, we can see that a direct accumulation of the previous signal leads to bifurcation and inaccuracy compared to the reference target signal. The faster the object moves and the more frames are accumulated, the larger errors will be introduced in the direct accumulated signal. For the DP method, although the processed signal becomes wider because of the search range, the processed signal after the DP method is still consistent with the reference position. In summary, we consider that the DP method provides a reasonable way for the moving object, to accumulate the signal from the previous frames on the current one.
Fig. 4. Comparison of the effectiveness of two different methods. Vertically, the first to fourth rows indicate the different velocities of the hidden object, respectively. Horizontally, the first column shows the result of accumulating the previous 5 frames using the two methods. Similarly, the second column shows the result of accumulating the previous 20 frames using both methods. The blue line indicates the results after processing of the DP method, the orange line indicates the results after directly accumulating, and the sky-blue dashed line indicates the current signal. All signals have been normalized.
4. Fast positioning method using the network
4.1 Reviews of the conventional method
The schematic of NLOS positioning is shown in Fig. 5. Firstly, the pulsed laser illuminates a spot “L” on the scattering surface. Some of the scattered photons directly return to the detector, forming the first bounce. At the same time, another part of the scattered photons reaches the hidden object and returns to the scattering surface the next time. This part of the photons then enters the detector and forms the third bounce. The general method fits a Gaussian to the third bounce and regarding it as a probability distribution of the arrival time for photons with a center t and a standard deviation $\sigma $ [27]. According to the light paths in Fig. 5, subtracting the distances in view gives the summation of the distances ${\parallel} {{\vec{r}_2}} \parallel $ and ${\parallel} {{\vec{r}_3}} \parallel $, as described by Eq. (6).
Fig. 5. Schematic of NLOS positioning. The schematic shows the light paths in the detecting process. The pulse laser illuminates the point “L” on the scattering surface and the detector collects the returned photons from “A”, “B”, “C”, and “D” on the scattering surface. ${\parallel} {{\vec{r}_1}} \parallel $ represents the distance between the laser and the point “L”. ${\parallel} {{\vec{r}_2}} \parallel $ represents the distance from the point “L” to the hidden object. ${\parallel} {{\vec{r}_3}} \parallel $ represents the distance from the hidden object to the detection point “B”. ${\parallel} {{\vec{r}_4}} \parallel $ represents the distance from the detection “B” to the detector lens. Some of the scattering photons at point “L” will return to the detector lens directly, corresponding to ${\parallel} {{\vec{r}_1}^{\prime}} \parallel $.
4.2 Network-based positioning
In the traditional method, the spatial probability density is calculated for each pixel, which takes a lot of time. In this work, we extract the arrival time corresponding to the first bounce peak and third bounce peak without uncertainty, such as ${t_1}$, ${t_2}$ in Fig. 6(a), and model the relationship between the time difference and the spatial position of the hidden object through a feed-forward neural network with sigmoid hidden neurons and linear output neurons, as shown in Fig. 6(b).
Fig. 6. Schematic of the time difference and the network. (a) shows the extracted time difference is the distance between the first and third bounce peaks in a histogram. Ideally, the collected histogram will have two peaks, which represent the first bounce from the surface directly and the third bounce from the hidden object indirectly. ${t_1}$ and ${t_2}$ are the arrival time of the first bounce and third bounce peaks respectively. (b) shows that the network gets the time differences of four channels and outputs the coordinates of the hidden object.
Initially, to establish the coordinate system, we take the point “L” as the origin of the coordinate system and assume that the coordinates of detection points “A”, “B”, “C”, “D” on the surface are (${x_1},{y_1},{z_1}$), (${x_2},{y_2},{z_2}$), (${x_3},{y_3},{z_3}$), (${x_4},{y_4},{z_4}$) respectively. The positive direction of the x-axis, y-axis, and z-axis is shown in Fig. 5. In this way, we assume that the coordinates of the hidden object are (x, y, z). The input data X is a vector consisting of the time difference (${d_1}$, ${d_2}$, ${d_3}$, ${d_4}$) extracted from the four-channel histograms respectively. If the distance from the detector to the laser is neglected, the time difference can be approximated as $({\parallel {{\vec{r}_2}} \parallel{+} \parallel {{\vec{r}_3}} \parallel } )/c$, written as Eq. (7).
As an example, the time difference for the detection point “A” is calculated as shown in Eq. (8). The target vector Y is the spatial coordinates of the hidden object. The input vector X and the target vector Y are displayed as Eq. (9).
The feed-forward network has 4 input units, 10 hidden units, and 3 output units. The function is written as Eq. (10).
To verify the feasibility of the positioning method, we conducted simulation experiments and error analysis in this section. In the simulation, we assume that the hidden object is a particle and the photon arrival time has a Gaussian noise ${\sim}{{\cal N}}({0,10} )$. The coordinates of the detection points “A”, “B”, “C” and “D” are (0, 0, 0.3 m), (-0.37 m, 0, 0), (0, 0, -0.2 m), (0.227 m, 0, 0) respectively. We know that 3 detection points can accomplish spatial positioning, and the reason why we choose 4 detection points on the wall is that the extra detection point can improve the accuracy. There are three main reasons for choosing the location of the detection point on the wall: (1) If the distance is too close, the accuracy of the positioning will be reduced. (2) If the distance is too far the number of returned photons is reduced. (3) Spatial positioning of the object requires the detection points at different heights. Then, based on the positioning method above, we trained the network to position the hidden object.
As shown in Fig. 7(a), we calculated the spatial position of the hidden object. Additionally, we also simulated the hidden moving object, as shown in Fig. 7(b-c). The speed of the moving object in Fig. 6(b) is about 1.1 cm/s and the speed in Fig. 7(c) is about 1.4 cm/s.
Fig. 7. The simulation results of positioning and tracking. The cross “L” represents the illuminated point by the laser and the crosses “A”, “B”, “C”, and “D” represent the four detection points on the scattering surface. The red cross indicates the predicted position, and the black square indicates the real position of the hidden object. (a) shows the simulation results of positioning. (b) We set the trajectory of the moving object from (0.1 m, 1.42 m, 0.2 m) to (0.9 m, 1.02 m, 0.1 m). The black dotted line represents the real trajectory. (c) The trajectory of the moving object is set from (0.1 m, 0.32 m, 0.2 m) to (0.9 m, 1.12 m, 0.092 m).
Furthermore, we analyzed the root-mean-square error (RMSE) of positioning in different positions, as shown in Fig. 8(a-i). In this situation, the horizontal error is between 0.004-0.024 m, the longitudinal error is between 0.02-0.14 m, and the vertical error is between 0.004-0.025 m. In addition, we note that the accuracy decreases as the detection distance increases. This is mainly because the long-distance makes the ellipses quite close. The simulation results show that the positioning method is feasible.
Fig. 8. The RMSE of NLOS positioning. (a-c) are the error of positioning in the x, y, and z directions when the vertical coordinate of space is 50 cm. (d-f) are the error in the x, y, and z directions when the vertical coordinate of space is zero. Figure 3(g-i) are the error in the x, y, and z directions when the vertical coordinate of space is -50 cm.
4.3 Experimental testing of network-based positioning
For the verification of the positioning method, we performed the positioning experiments in a corridor 20 m away, as shown in Fig. 1(a). We placed the hidden object at five different positions in the corner and position the stationary object respectively, in which the object is a foam board with a width of 18 cm and a height of 30 cm. The size of the object has an impact on the calculation of the time difference. When the echo signals returned from a large object are mixed and bifurcated, the positioning method will introduce errors because the assumption of the hidden object is a particle. The positioning results are shown in Fig. 9. Based on the network-based positioning method, inputting the four-channel time difference into the trained network, we get the spatial coordinates of the hidden object. After measuring, the horizontal error is between 0.02-0.07 m, the longitudinal error is between 0.02-0.06 m, and the vertical error is between 0.06-0.15 m. The vertical coordinates are all within the height range of the object. This result shows that the method meets the positioning requirements.
Fig. 9. Experimental results of NLOS positioning. (a-e) shows positioning results for the stationary object at five different positions. The red cross “L” is where the laser illuminates the wall. The black crosses “A”, “B”, “C”, “D” are the detection points on the wall. The blue cross represents the position of the hidden object. The rectangle represents the hidden object. (f) shows the top view of the NLOS area.
To test the computing time of the positioning method, we compare it with the positioning method, as shown in Fig. 10. In the calculation, the general method, as mentioned in section 4.1, calculates the joint probability density of each pixel in the space, in which the calculating process has been optimized by matrixing. The computing time may limit the updating frequency of NLOS positioning. We turn to the computing time of the positioning with the network at the same position space, which is less than 0.01 s. The time complexity of the proposed method is O ($\textrm{n}\log n$), mainly determined by the sorting algorithm in which n represents the length of the histogram. The short computing time allows us to update the position in quite a short exposure time.
Fig. 10. The computing time of two methods. With the same input histograms, we tested the computation time of the two methods for 10 frames. The red line shows the computing time of the general method, while the blue line shows the computing time of positioning with the network.
5. Tracking experiments
To verify the feasibility of fast tracking, we performed the tracking experiments in a corridor 20 m away, as shown in Fig. 1(a). The process of positioning the hidden object is depicted in Fig. 11. Initially, we extracted the arrival time of the first bounce peak in the background histogram and set it to zero to facilitate finding the third bounce peak. Then, we shift the first bounce to the origin of the coordinates system and subtract the background to separate the signal. Alternatively, the background can be estimated by the median of the histograms [27,32,33] or subtracted by previous histograms to extract the moving object [28,34,35]. We discretized the collected histograms into 2500 bins, and each time bin was 10 ps. Next, the four-channel histograms were passed through a mean filter, as described as Eq. (5), to reduce the random noise. Through the DP method, we iterated three times to calculate the value function for each frame. Subsequently, we extracted the time difference in the value function, in which the first arrival time is on the origin. Inputting the four-channel time difference to the trained network, we figured out the spatial position of the hidden object. Due to the existence of observation error, we use a Kalman filter to estimate the position of the hidden object.
Fig. 11. Flow diagram of positioning the hidden object for each frame. The four steps in the first row are the processing of the histogram. The histograms at the top of the box are the results of the steps. The second row is the process of positioning the hidden object. Extracting the time difference from histograms, we put the time differences into the network and then get the spatial coordinates of the hidden object. Because of the existence of observation error, we utilize a Kalman filter to estimate the position.
As shown in Fig. 12 (a-e), we placed the slide in five different positions and tracked the moving objects at a frequency of 50 Hz, in which the process of collecting and positioning are parallel. In the experiments, we increased the tracking frequency of NLOS detection to 50 Hz based on the improvements in hardware and algorithm. Additionally, the tracking frequency is limited by the processing time. We are able to track hidden objects with much shorter acquisition times, but the calculating time of each frame takes 0.02s on a laptop (Intel Core i7-10750H, 2.6 GHz), resulting in a tracking frequency of 50 Hz. In other words, increasing the speed of computation can improve the tracking frequency and has the potential for higher tracking frequency. Considering the high-power laser is not safe for eyes. We reduced the laser power to 400 mW with a repetition frequency of 40 MHz and repeated the experiment to track the hidden target at a frequency of 10 Hz. The results are placed in Supplement 1.
Fig. 12. Experimental results of tracking a moving object. (a-e) show the tracking results of five different paths (see Visualization 1). The red cross “L” is the position that the laser illuminates the wall. The black crosses “A”, “B”, “C”, “D” are the detection points on the wall. The blue cross represents the position of the hidden object of each frame. The speed of the moving object is 2.133 cm/s.
6. Conclusion
In this paper, we introduced the DP method to detect the weak signals returned from the hidden object, regarding the detection of weak signals as a multi-stage optimization problem, which can unmix the target signal from noise at a shorter exposure time. Secondly, we simplified the positioning method by extracting the time difference from the histogram and training a feed-forward network to output the spatial position of the hidden object directly. These methods are beneficial to improve the real-time performance of NLOS detection. Ultimately, we conducted fast tracking experiments in a corridor 20 m away and achieved tracking a hidden object at a frame rate of 50 Hz. Among them, the frame rate was limited by the processing time of the procedure. Through program optimization and parallel computing in hardware, the update frequency may be increased further. The increased tracking frequency indicates the possibility for NLOS fast application, like tracking an unmanned aerial vehicle (UAV) in NLOS space.
Funding
National Natural Science Foundation of China (62005289); Youth Innovation Promotion Association (2017428, 2018411, 2020372); Excellent Youth Foundation of Sichuan Scientific Committee (2019JDJQ0012); West Light Foundation of the Chinese Academy of Sciences; 1000 Talents Sichuan Program.
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
Supplemental document
See Supplement 1 for supporting content.
References
1. K. Tanaka, Y. Mukaigawa, and A. Kadambi, “Polarized non-line-of-sight imaging,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, June 14, 2020 - June 19, 2020, Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (IEEE Computer Society, 2020), 2133–2142.
2. B. Wang, M. Y. Zheng, J. J. Han, X. Huang, X. P. Xie, F. Xu, Q. Zhang, and J. W. Pan, “Non-line-of-sight imaging with picosecond temporal resolution,” Phys. Rev. Lett. 127, 127 (2021). [CrossRef]
3. T. Maeda, Y. Wang, R. Raskar, A. Kadambi, and Ieee, “Thermal Non-Line-of-Sight Imaging,” in 2019 Ieee International Conference on Computational Photography (2019).
4. C. Wu, J. J. Liu, X. Huang, Z. P. Li, C. Yu, J. T. Ye, J. Zhang, Q. Zhang, X. K. Dou, V. K. Goyal, F. H. Xu, and J. W. Pan, “Non-line-of-sight imaging over 1.43 km,” Proc. Natl. Acad. Sci. U. S. A. 118, e2024468118 (2021). [CrossRef]
5. S. W. Seidel, J. Murray-Bruce, Y. Ma, C. Yu, W. T. Freeman, and V. K. Goyal, “Two-Dimensional Non-Line-of-Sight Scene Estimation from a Single Edge Occluder,” IEEE Trans. Comp. Imag. 7, 58–72 (2020). [CrossRef]
6. M. Isogawa, Y. Yuan, M. O’Toole, and K. Kitani, “Optical Non-Line-of-Sight Physics-Based 3D Human Pose Estimation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2020), pp. 7011–7020.
7. D. Faccio, A. Velten, and G. Wetzstein, “Non-line-of-sight imaging,” Nat. Rev. Phys. 2(6), 318–327 (2020). [CrossRef]
8. M. Isogawa, D. Chan, Y. Yuan, K. Kitani, and M. O’Toole, “Efficient Non-Line-of-Sight Imaging from Transient Sinograms,” in Computer Vision – ECCV 2020 (Springer International Publishing, 2020), 193-208.
9. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3(1), 745 (2012). [CrossRef]
10. C. Jin, J. Xie, S. Zhang, Z. Zhang, and Y. Zhao, “Reconstruction of multiple non-line-of-sight objects using back projection based on ellipsoid mode decomposition,” Opt. Express 26(16), 20089–20101 (2018). [CrossRef]
11. V. Arellano, D. Gutierrez, and A. Jarabo, “Fast back-projection for non-line of sight reconstruction,” Opt. Express 25(10), 11574–11583 (2017). [CrossRef]
12. M. La Manna, F. Kine, E. Breitbach, J. Jackson, T. Sultan, and A. Velten, “Error Backprojection Algorithms for Non-Line-of-Sight Imaging,” IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1615–1626 (2019). [CrossRef]
13. M. Batarseh, S. Sukhov, Z. Shen, H. Gemar, R. Rezvani, and A. Dogariu, “Passive sensing around the corner using spatial coherence,” Nat. Commun. 9(1), 3629 (2018). [CrossRef]
14. M. O’Toole, D. B. Lindell, and G. Wetzstein, “Confocal non-line-of-sight imaging based on the light-cone transform,” Nature 555(7696), 338–341 (2018). [CrossRef]
15. F. Heide, M. O’Toole, K. Zang, D. Lindell, S. Diamond, and G. Wetzstein, “Non-line-of-sight Imaging with Partial Occluders and Surface Normals,” ACM Trans. Graph. 38(3), 1–10 (2019). [CrossRef]
16. M. O’Toole, D. B. Lindell, G. Wetzstein, and M. Assoc Comp, “Real-time Non-line-of-sight Imaging,” in Special-Interest-Group-on-Computer-Graphics-and-Interactive-Techniques (SIGGRAPH) Conference, 2018).
17. X. Liu, I. Guillén, M. La Manna, J. H. Nam, S. A. Reza, T. Huu Le, A. Jarabo, D. Gutierrez, and A. Velten, “Non-line-of-sight imaging using phasor-field virtual wave optics,” Nature 572(7771), 620–623 (2019). [CrossRef]
18. D. B. Lindell, G. Wetzstein, and M. O’Toole, “Wave-Based Non-Line-of-Sight Imaging using Fast f-k Migration,” ACM Trans. Graph. 38(4), 1–13 (2019). [CrossRef]
19. X. C. Liu, S. Bauer, and A. Velten, “Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems,” Nat. Commun. 11(1), 1645 (2020). [CrossRef]
20. S. A. Reza, M. La Manna, S. Bauer, and A. Velten, “Phasor field waves: A Huygens-like light transport model for non-line-of-sight imaging applications,” Opt. Express 27, 29379–29399 (2019). [CrossRef]
21. P. Caramazza, A. Boccolini, D. Buschek, M. Hullin, C. F. Higham, R. Henderson, R. Murray-Smith, and D. Faccio, “Neural network identification of people hidden from view with a single-pixel, single-photon detector,” Sci. Rep. 8, 11945 (2018). [CrossRef]
22. M. Lyu, H. Wang, G. Li, and G. Situ, “Exploit imaging through opaque wall via deep learning,” ArXiv Preprint arXiv:1708.07881 [cs.NE] (2017).
23. G. Musarra, P. Caramazza, A. Turpin, A. Lyons, C. F. Higham, R. Murray-Smith, and D. Faccio, “Detection, identification, and tracking of objects hidden from view with neural networks,” Proc. SPIE 10978, 1097803 (2019). [CrossRef]
24. T. Yu, M. Qiao, H. Liu, and S. Han, “Non-Line-of-Sight Imaging Through Deep Learning,” Acta Optica Sinica 39, 1–12 (2019).
25. G. Satat, M. Tancik, O. Gupta, B. Heshmat, and R. Raskar, “Object classification through scattering media with deep learning on time resolved measurement,” Opt. Express 25(15), 17466–17479 (2017). [CrossRef]
26. W. Chen, S. Daneau, F. Mannan, F. Heide, and I. C. Soc, “Steady-state Non-Line-of-Sight Imaging,” in 2019 Ieee/Cvf Conference on Computer Vision and Pattern Recognition (2019), pp. 3783–6792.
27. G. Gariepy, F. Tonolini, R. Henderson, J. Leach, and D. Faccio, “Detection and tracking of moving objects hidden from view,” Nat. Photonics 10(1), 23–26 (2016). [CrossRef]
28. S. Chan, R. E. Warburton, G. Gariepy, J. Leach, and D. Faccio, “Non-line-of-sight tracking of people at long range,” Opt. Express 25(9), 10109–10117 (2017). [CrossRef]
29. S. Chan, R. E. Warburton, G. Gariepy, Y. Altmann, S. McLaughlin, J. Leach, and D. Faccio, “Fast tracking of hidden objects with single-pixel detectors,” Electron. Lett. 53(15), 1005–1008 (2017). [CrossRef]
30. J. Brooks and D. Faccio, “A Single-Shot Non-Line-of-Sight Range-Finder,” Sensors 19(21), 4820 (2019). [CrossRef]
31. Y. Ren, Z. Xie, Y. Luo, S. Xu, H. Ma, and Y. Tan, “Non-Line-of-Sight Location With Gauss Filtering Algorithm Based on a Model of Photon Flight,” IEEE Photonics J. 12, 99 (2020). [CrossRef]
32. R. Cutler and L. Davis, “View-based detection and analysis of periodic motion,” in Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170), 1998), 495–500 vol.491.
33. R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, “Detecting moving objects, ghosts, and shadows in video streams,” IEEE Trans. Pattern Anal. Machine Intell. 25(10), 1337–1342 (2003). [CrossRef]
34. R. Pandharkar, A. Velten, A. Bardagjy, E. Lawson, M. Bawendi, and R. Raskar, and Ieee, “Estimating Motion and Size of Moving Non-Line-of-Sight Objects in Cluttered Environments,” in2011 Ieee Conference on Computer Vision and Pattern Recognition (2011), pp. 265–272.
35. T. S. Ralston, G. L. Charvat, and J. E. Peabody, “Real-time through-wall imaging using an ultrawideband multiple-input multiple-output (MIMO) phased array radar system,” in IEEE International Symposium on Phased Array Systems & Technology, 2010).
