1. Introduction

Conventional imaging systems, such as the built-in cameras available in every smartphone, intrinsically assume that the objects of interest are located in the imager’s direct line-of-sight (LoS), as depicted in Fig. 1(a). Standard optical, microwave and ultrasound imaging systems fall in this category. In a non-line-of-sight (NLOS) scenario, the goal is to recover the image of a target that is not directly visible from the imager, as shown in Fig. 1(b). Being able to create NLOS images has potential impacts in different applications ranging from law enforcement, to medical applications and space exploration.

 

Fig. 1. Shown in (a) is a typical LOS scene where the imaging system is collecting data of the object under investigation. Conversely shown in (b) is an example of a NLOS scenario, where an occluder wall blocks the imaging systems’ field of view. The optical imaging system discussed in this paper is capable of collecting data that has been scattered from the relay surface and create an image of the hidden object around the corner.

Download Full Size | PDF

For a solid understanding of the theoretical background on light scattering and how photons can be used to create an image of a target hidden around a corner, we refer to [1–12] (and references therein). In this manuscript, we focus our attention on the experimental aspects of NLOS imaging.

Experimental efforts published in prior art can be classified into different categories depending on the equipment and reconstruction algorithms employed. For example, some imaging systems comprise inexpensive, off-the-shelf diode lasers and CMOS time-of-flight (ToF) cameras, as light sources and detectors, respectively (e.g.: [13–15]). In a second approach, a laser pointer and inexpensive 2D cameras are used to track occluded target by simulating the light scattering in the scene and then fitting the experimental data to the simulated one [16,17]. Other imaging systems comprise more expensive equipment, which typically includes an ultra-fast laser and ToF detectors that are able to sense even a single photon [18–22]. Backprojection-based algorithms are typically employed to reconstruct the image of the targets (cf. [20,23–26]) and there exist fast implementation of such algorithms (e.g.: [27]). Moreover, recent publications [26,28–31] have shown that the NLOS imaging problem can be treated as a diffractive wave propagation problem and therefore modeled as a LoS problem. In the end, other techniques (cf. [32]) exploit the presence of occluding objects in the light path.

For the remainder of the manuscript, we consider the arrangement in Fig. 1(b), where we collect data an active light source that illuminates a relay surface, on a point $x_{l, m}$. The photons scatter in all directions and a fraction of them arrive at the hidden object. The photons scatter from the object once again in all directions and some return to the relay wall. Assuming that the detector is focused on a point of the relay surface, $x_{c, n}$, we collect the ToF (or time-of-arrival, TOA) of these photons, with respect to an initial origin time. We repeat this data acquisition for a specific number of laser positions, $m=1, \dots , M$ and camera positions $n=1, \dots , N$.

Previous publications (e.g. [23,26]) acquire data by uniformly scanning a pre-defined area on a flat relay wall. This grid is computed offline and stored in memory. Any changes to the relay surface or positioning of the imaging system with respect to the relay wall requires adjusting the offline laser grid, which may be difficult, if not tedious. Other published works, such as [33,34], consider non-planar relay surfaces and moving targets, but still rely on a pre-computed grid, which requires a slow measurement. Moreover, the LOS geometry has to be calibrated as well. In this work we address the problem of non-planar, dynamic relay surfaces. To keep track of a moving relay surface, we equip the imaging system with a second detector that locates the laser position ($x_{l,m}$ on the relay surface at any given time as depicted in Fig. 1(b)).

2. System description

2.1 Hardware setup

A more detailed schematic of a NLOS imaging system is shown in Fig. 2. As a light source, we utilize a Onefive Katana 10 HP fiber laser, that is able to output $35 \pm 15~\mbox {ps}$ long pulses at an operating wavelength of $512~\textrm {nm}$ and with a $10~\textrm {MHz}$ repetition rate. To illuminate an area of the relay surface, we employ a Thorlabs GVS012 2D large beam diameter galvo system with silver coated mirrors. Similarly to the previous section, we define $x_{l,m}$ as the $m$-th laser position. To detect the returning photons, we employ two silicon single photon avalanche diodes (SPADs) [35–37]. To decrease the background noise, both of these SPADs are equipped with a bandpass filter with a full width at half maximum (FWHM) bandwidth of $10~\mbox {nm}$. When a photon arrives at the SPAD’s detector head - namely where the active chip is located - it triggers an internal electron avalanche. This has two effects on the SPAD: 1) it becomes ‘blind’ to subsequent photons for a specific amount of time, called the hold-off time, and 2) after an internal delay, an electrical signal is generated to the output. An eight channel PicoQuant HydraHarp 400 time-correlated single photon counting (TCSPC) unit is used to calculate the photons ToA with a a picosecond resolution. The two SPADs in Fig. 2 are positioned in two different locations of the imaging system, as we exploit them to recover the information of the visible and hidden scene, as described below.

 

Fig. 2. Capture system described in Sec. 2.

Download Full Size | PDF

2.2 Data acquisition

The underlying assumption in traditional acquisition methods is that the relay surface does not move or alter during the entire course of data acquisition. When this is true, the position $x_{l,m}$ does not change over time and the collected time response corresponds to the photons scattering from this considered point. When the relay surface moves, the collected time response corresponds to the photons arriving around $x_{l,m}$. If we were to use this laser position value in the reconstruction algorithm, it would cause substantial image reconstruction degradation. The use of a second detector as described above and the TCSPC time-tagged time-resolved (TTTR) mode can be used to address this issue. In TTTR, the TCSPC only records the photon time events, but does not parse them into histograms.

To acquire TTTR data, the laser beam is first positioned at its starting position, for example the top-left corner shown in Fig. 3(a). The laser beam then moves in a raster scan (cf. green line), at a constant speed, $v$, until the ending position, bottom-right corner in Fig. 3(a), is reached. As the laser beam moves along the surface, the TCSPC collects the ToA data arriving from the free-running and gated SPADs, which arrive at two different TCSPC inputs. The detected and stored events for each channel (SPAD) are shown in Fig. 3(a).

 

Fig. 3. Data acquisition method described in Sec. 2.2. The TCSPC TTTR mode registers only the time events. Afterwards, we need to discretize the scanned area into laser positions and associate the time events to laser positions.

Download Full Size | PDF

The next step is to generate the histograms for both SPADs. We discretize the relay surface into $M$ points, say $x_{l,m}$, $m=\{1, \dots , M\}$, which correspond to the laser positions on the relay surface. Note that, at this time, we do not know the 3D coordinates of $x_{l,m}$, they will be calculated using the information from the first bounce (free-running SPAD information). Considering the data coming from the free-running SPAD, the events associated with the generic position $x_{l,m}$ are the ones that satisfy the following condition

The 3D coordinates of the laser positions, $x_{l,m}$, with respect to an origin in our imaging system, can be determined using the knowledge of the galvo mirror voltages (there is a direct relationship between voltage applied and mirror angle) and the location of the first bounce peak (which translate into distance), contained in $h_{f,m}(t)$.

3. Results and discussion

In Fig. 4, we depict the relay surface employed for our experiments. It comprises a rectangular frame holding two white curtains, divided by a mount with a statue on top of it. The gated SPAD is focused on the statue, whereas the laser scans the area inside the rectangle shown in black. The imaging system is approximately $2.20~\textrm {m}$ away from the relay surface and the hidden objects are placed at approximately $1~\textrm {m}$ from the relay surface. Our post-processing algorithm discretizes the relay surface into approximately $47$ thousand laser positions and, for each, creates a histogram with an equivalent exposure time of $5~\textrm {ms}$. Afterwards, we apply the phasor field reconstruction algorithm, as described in [26,28–31,38]. For the purpose of this manuscript, we have selected two different hidden scenes that we want to reconstruct and these are shown in Fig. 5(a) and Fig. 6(a). The former is a simple, almost binary scene that contains well separated objects (a ’4’ and two poster tubes in a box). The latter scene is more complex and roughly resembles the one in [26].

 

Fig. 4. The relay wall for this experiment comprises two curtains mounted on a rectangular frame. The black lines represent the laser scanning area. Moreover, we assume that the gated SPAD is focused on the statue between the two curtains. The red arrow indicates where the hidden scene is located w.r.t. to the relay surface. To create motion, we use an air fan, which is located on the right (but not visible here).Visualization 1.

Download Full Size | PDF

 

Fig. 5. Experiment 1: (a) depicts the true hidden scene, whereas (b) shows the reconstructions with the fan off, (c) with the fan set to ‘high’.

Download Full Size | PDF

 

Fig. 6. Experiment 2: (a) depicts the true hidden scene, whereas (b) shows the reconstructions with the fan off, (c) with the fan set to ‘high’, and (d) when the fan is set to ‘high’, but we reconstruct using the laser positions retrieved from (b).

Download Full Size | PDF

Figures 5(b) and 6(b) show the reconstruction results when the relay wall is static (motionless). The results are comparable to the ones obtained in [26],-in the case of which- the relay surface was Lambertian and flat. More specifically, Fig. 5(b) shows that the ‘4’ and the tubes are completely visible and distinguishable. The reconstruction of the more complex scene is shown in Fig. 6(b). Even in this case the reconstructions closely resembles the hidden scene: we have recovered the shelves, the books, the statue and the letter ‘T’, the poster tubes (right) and part of the wall behind the shelf (lower-mid). It should be noted that the top-left shelf is partly reconstructed. Moreover, the bottom-right shelf (containing the mannequin) is missing from the reconstruction. However, this shelf is directly in front of the statue, where there is a gap between the two curtains. It is highly likely that the ToF information regarding this shelf has been lost because of this gap. Alternatively, the wall behind the shelf provides a stronger return w.r.t. the shelf.

In our second set of data acquisition, the curtains have been set in motion by an air fan. A video showing the acquisition while the laser is scanning and the relay wall is moving has been provided as a supplemental material as Visualization 1 (the video has been sped up by 10x). The reconstruction results for the scene in Fig. 5(a) are shown in Fig. 5(c). Once again, it can be concluded that the number ’4’ and the tubes have been reconstructed, since they are clearly visible. Because of the motion, it seems that the poster tubes have a lower intensity in this image w.r.t. Fig. 5(b). This could be explained by the fact that, as the curtains were moving, the laser was impinging on areas that were not directly facing the targets. Figure 6(c) depicts the reconstruction for the complex scene shown in Fig. 6(a). Once again, it is clear that the bookshelf, the books, the statue and the letter ‘T’ have been successfully recovered. It is also interesting to note that, although the bottom-right shelf is still missing, we are still able to recover the wall behind the shelf. The tubes have a lower return, but are still visible.

It is important to note that the reconstructions we have shown in Fig. 5(c) and Fig. 6(c) are possible because we have used the free-running SPAD to locate the laser positions on the relay surface as it was moving. To show the importance of this information, we have run a reconstruction using the multi-bounce information gathered from the moving curtains scene and the laser position information gathered from the static curtains scene. The result is depicted in Fig. 6(d), where it is easy to see that the reconstruction does not provide any information about the hidden scene. In other words, this results shows that it is of paramount necessity - especially in a moving relay surface scenario - to be able to correctly associate first bounce data with multi-bounce data, as we proposed here.

4. Conclusion

Considering an optical NLOS imaging scenario, we have shown how it is possible to acquire meaningful data when the relay surface is considered rugged, discontinuous and also in motion. This can essentially be achieved using a secondary detector (a SPAD, in our case) that collects the first bounce ToF information, together with the TCSPC TTTR mode. Similar to only a few other methods in prior art, our acquisition time is in the order of minutes. Furthermore, we compare the reconstruction results with a dynamic relay surface to ones where the relay surface is static using the standard phasor field reconstruction approach in both cases. The former results show minimal deterioration w.r.t. the latter ones. In future work, we foresee employing SPAD arrays and faster reconstruction algorithms which would allow to reduce the acquisition time (at the expense of the data processing) and a quicker rendering of the hidden scene respectively.