1. Introduction

Non-line-of-sight (NLOS) imaging aims at visualizing elements that are out of direct line-of-sight in a scene. Many of the imaging methods are based on combining pulsed laser sources and time-resolved sensors [1–6]. Time-resolved sensors, such as Single-Photon Avalanche Detectors (SPADs), measure the temporal profile of light transport [7–11]. Broadly speaking, such techniques reconstruct geometry that is hidden from the direct view by analyzing the temporal footprint of light transport of such geometry after a bounce on a diffuse relay wall [12–15], as shown in Fig. 1. The NLOS imaging landscape has seen rapid progression in recent years, as experimental setups have become more reliable, faster, and more accurate, which has generated an effervescence in reconstruction methods with varying accuracy and reconstruction time. We refer the reader to any of the existing surveys on the topic [3–5,16,17]. However, this body of reconstruction methods has been formulated independently, with ad-hoc assumptions and usually simplified light transport models adapted to particular cases. Two of the major formulations are back-propagation and wave-based propagation [18,19]. Back-propagation formulations model the transport of light within the hidden scene and aim at recovering the geometry inverting the transport model [1,9,20–22]. Wave-based propagation formulations aim at modeling the time-of-flight of light within the phase of a complex function, applying well-known Fourier optics to image the hidden scene [18,19,23–25].

 

Fig. 1. A classic NLOS scenario looking around a corner. The displayed room has a relay wall that is illuminated by a laser source. The light diffused by the relay wall illuminates the hidden T-shaped geometry. The indirect illumination reflected from the hidden geometry is reflected by the relay wall back to the time-resolved sensor. Figure adapted from [26].

Download Full Size | PDF

A key part of this paper’s contribution is the formulation of a cohesive theoretical framework that not only encloses several reconstruction methods but can also explain their limitations and virtues in a cohesive way. One of the key insights that has inspired our work is the realization that the magnitudes propagated in both major formulations (time-resolved real numbers in back-propagation and phasors in wave-based propagation) are, indeed, elements in a Hilbert space $\mathcal {H}$ connected through a linear operator.

We build on top of the idea that reconstruction methods are unable to accurately predict the hidden geometry due to insufficient information and the inherent ambiguities of light transport within the hidden scene. Therefore, the output of such methods can be interpreted as a probability distribution, for each point in the volume, of being part of the hidden geometry. This leads to another key insight inspiring our work: thinking about the measurement of a hidden scene as the measurement of an operator in quantum mechanics. This in turn links with quantum mechanics theory and the uncertainty principle, in which it is impossible to predict particular outcomes of events, but it is possible to estimate probabilities of such events.

A fundamental piece in the formulation of quantum mechanics is the use of Dirac notation to describe quantum systems [27,28]. As we show, the use of Dirac notation in an NLOS context provides a cohesive theoretical framework that will explain why the methods proposed so far work the way they do, as well as to connect the reconstruction methods with the fundamental physics of light propagation.

In this work, we will demonstrate the following theoretical contributions i) the derivation of back-propagation, phasor fields, and f-k migration formulations for NLOS imaging from a single equation (Section 4, Section 5, and Section 6), ii) the demonstration that the Rayleigh-Sommerfeld diffraction operator is the propagation operator for wave-based imaging methods (Section 5), and iii) the demonstration that back-propagation and wave-based imaging methods are equivalent point-to-point (Section 7), which is supported by a prior demonstration that propagation operators are unitary (Annex). The paper also brings other contributions, such as modeling general temporal profiles for light source emission (most methods assume delta pulses), as well as novel insights on the general properties of propagation operators, including the Rayleigh-Sommerfeld diffraction operator commonly used in phasor fields. Other NLOS imaging formulations [29–35] may also be derived from our proposed framework, as we discuss at the end of this manuscript.

Based on this work, we expect to enable future research including, but not limited to i) bringing the mathematical toolset of quantum mechanics to NLOS imaging, which can result in more optimal and higher performing computational methods; ii) implement other imaging and reconstruction methods that may model the interaction of light with the hidden scene more accurately by finding an appropriate Hilbert space representation and specific propagation operator; iii) define optimal scene sampling and reconstruction strategies based on the expected properties of the operators, for example, their rank, dimensionality, and symmetry; and iv) use the generality of our proposed framework to devise algorithms that will work with all NLOS imaging methods.

2. Background

Non-line-of-sight (NLOS) imaging aims at visualizing elements that are out of direct line-of-sight. Broadly speaking, images are obtained by analyzing the temporal footprint of light transport in a hidden scene after bouncing on a diffuse relay wall. NLOS imaging techniques also leverage time-of-flight (ToF) data to estimate the range and location of elements. ToF data may be captured in other parts of the electromagnetic spectrum outside the visible part.

To image a hidden scene, a set of impulse response signals of the scene is first captured. This is achieved by illuminating points on the relay wall with short laser pulses, which play the role of the delta excitation of the system. The points on the relay wall illuminated by the laser reflect, in a diffuse manner, some of the light to the hidden scene. This is considered as the first light bounce. The second light bounce is the diffuse reflection of the hidden scene back to the relay wall. The light coming back from the hidden scene bounces a third time on the relay wall, being captured at different points by time-resolved sensors synchronized with the emission of the laser pulses. Two of the major formulations to realize NLOS imaging are back-propagation and wave-based propagation.

The back-propagation formulation creates a voxelization of the NLOS scene and estimates the presence of geometry on a particular voxel based on the cumulative contributions of light intensities captured by the time-resolved sensor at specific timestamps [1,20,21]. The timestamps are computed based on the estimated time-of-flight of the light according to the relative position of the voxel with respect to the illumination and sensor points on the relay wall. Back-propagation uses the computed timestamps to look up the intensity of light captured at the corresponding time index on the time-resolved signal. A more mathematical background on back-propagation is provided at the beginning of Section 4.

Wave-based propagation formulations aim at modeling the time-of-flight of light within the phase of a complex function so that virtual light sources and cameras can be modeled and well-known Fourier wave optics methods can be used to image the hidden scene. Two of the most popular wave-based formulations are phasor fields [18], and f-k migration [19].

In the rest of this section, our goal is to familiarize the reader with the terminology and some fundamental concepts of the Dirac notation and quantum mechanics. More mathematical backgrounds on phasor fields and f-k migration are provided at the beginning of Section 5 and Section 6 respectively. Table 1 describes the symbols in this paper.

Table 1. Table describing the symbols used throughout this paper. $^{(1)}$ We will use $\mathbf {x}_{n}$ and $n$ indistinctly to refer to an illumination point in $\mathcal {L}$. $^{(2)}$ We will use $\mathbf {x}_{m}$ and $m$ indistinctly to refer to a sensing point in $\mathcal {S}$.

View Table

2.1 Dirac notation and quantum mechanics

Quantum mechanics is a fundamental theory in physics developed in the first half of the 20th century that aims at describing the physical properties of matter at the scale of atoms and subatomic particles. Although being controversial and often conflicting with our common-sense intuition derived from our experience in the world, quantum mechanics has been successful in providing correct predictions and results in every situation to which it has been applied.

A key component in quantum mechanics is the use of Dirac notation to represent the different states of a quantum system and its dynamic evolution. Dirac created this notation to simplify the formulation of quantum mechanics in mathematical terms [27,28] and quickly became as de-facto standard notation in the field. The scope of this section is to provide enough background in terms of terminology an intuition of the key mathematical elements in Dirac notation. In particular, we aim at describing the state of a pulse of light and how this pulse propagates and interacts with a hidden scene. We refer the interested reader to other sources for more in-depth explanations [36–40].

2.1.1 Kets and bras

Dirac notation is also known as bra-ket notation. A ket represents the state of a quantum system and it is expressed as $|u\rangle$ (any letter other than $u$ may be used to identify the ket). This ket is a vector, or element, in a Hilbert space $\mathcal {H}$. In general, the term Hilbert space refers to an infinite-dimensional space where the inner product is complete. This definition also includes finite-dimensional spaces, which automatically satisfy the condition of completeness. Without loss of generality, we can think of a ket as a column vector with a dimension equal to $K$ in which the components are complex:

A bra is the adjoint of a ket. Therefore, the bra is the transposed and conjugated version of the ket. Without loss of generality, the bra is expressed as a row vector with dimension $K$ in which the components are the conjugated components of the ket:

Notice that the transposed conjugated $|\cdot \rangle ^{\dagger }$ is denoted as $\langle \cdot |$ in Dirac notation. Note that the dimension $K$ of these vectors can be infinite, provided, as stated before, that their inner product is complete.

2.1.2 Inner product

The inner product defines the contraction of a bra with a ket, which is expressed as:

In practical terms, the inner product is the scalar product of two vectors in which the components are complex. The inner product of a ket with its bra is the square of the norm of the ket:

Note that the inner product is expressed as a bracket of $\langle v |$ and $|u\rangle$ and that both $\langle v |$ and $|u\rangle$ have the same dimension. Hence, the name of braket notation for this formalism [27]. The generalization of Eq. (3) to an infinite dimensional space is the integral:

Kets play a prominent role in representing the states of quantum systems. Kets are the states upon which operators act; for instance, $A |u\rangle$ means that operator $A$ is acting on $|u\rangle$, because in standard algebra operators are placed on the left side of the element they act on. Bras are considered the dual representation of the Kets and, thus, adjoint operators act on the element placed on their left side. For instance, $\langle u | A^{\dagger }$ means that the adjoint operator of $A$ is acting on $\langle u |$. Considering matrices and vectors as particular cases, we can consider $A |u\rangle$ as the product of matrix $A$ with column vector $\mathbf {u}$, whereas we can consider $\langle u | A^{\dagger }$ as the product of row vector $\left [ \mathbf {u}^{*} \right ]^{T}$ with matrix $A^{\dagger } = \left [ A^{*} \right ]^{T}$.

2.1.3 Outer product

The outer product is defined as the cartesian product of a ket with a bra:

3. Fundamental equations for NLOS imaging

In this section we propose two simple equations based on quantum mechanics and Dirac notation that will allow us deriving back-propagation and wave-based imaging methods such as phasor fields and f-k migration.

An illumination point $n$ is where light hits the relay wall, and thus undergoes a first diffuse bounce towards the hidden scene. This diffused light illuminates the hidden scene, which in turn reflects some light back to the relay wall after the second bounce. A sensing point $m$ is where the light reflected by the hidden scene hits the relay wall, which, in turn reflects part of the light back to the sensor. This final reflection to the sensor is considered to be the third bounce. For more details about the light path in a NLOS scene, refer to the beginning of Section 2.

Let us start by defining the kets that represent the state of light at both the illumination and sensing points. Let us define $|l_{n}\rangle$ as the state of light at an arbitrary illumination point $n$ located on the virtual light source surface $\mathcal {L}$, and $|s_{m}\rangle$ as the state of light at an arbitrary sensing point $m$ located in the virtual camera surface $\mathcal {S}$. We can then define an operator $\hat {T}_{m,n}$ as the outer product between the kets representing light measured at the sensing points on $\mathcal {S}$ and the bras representing light at the illumination points on $\mathcal {L}$ as:

Equation (9) defines a simple, comprehensive operator to model the interaction of light with a hidden scene in a NLOS context; we call $\hat {T}$ the scene operator. In the following, we will use individual elements of the scene operator $\hat {T}_{m,n}$ instead of its matrix form $\hat {T} = \left [ \hat {T}_{m,n} \right ]$.

Equation (9) can be expressed more explicitly by introducing propagation operators, which actually model the propagation of light from point-to-point, whereas $\hat {T}_{m,n}$ models a convenient mathematical relationship between the state of light at the illumination point and the state of light at the sensing point.

Let’s define a propagation operator $\hat {U}_{m,n}$ that models the interaction of light with the hidden scene by propagating light from an illumination point $n$ on $\mathcal {L}$ to a sensing point $m$ on $\mathcal {S}$. The contributions from all illumination points to an arbitrary sensing point $m$ are given by:

Plugging Eq. (10) into Eq. (9) yields:

Equation (9) will allow us to demonstrate a series of general properties of the different NLOS reconstruction methods and it is the starting point to i) derive the back-propagation, phasor fields, and f-k migration formulations from the same Eq. (9) in Section 4, Section 5, and Section 6, respectively, thus setting evidence that our proposed theoretical framework represents a cohesive formulation for the NLOS problem; ii) demonstrate that the Rayleigh-Sommerfeld diffraction operator is the propagation operator for wave-based imaging methods, such as phasor fields and f-k migration in Section 5; and iii) demonstrate that back-propagation and wave-based imaging methods are equivalent in Section 7.

Both Eq. (9) and Eq. (11) are general, and be seen as sheet equations under which several NLOS reconstruction methods can be formally described. To realize this, we need to i) identify the form that kets take in the particular Hilbert space that describes light under a given method. For example, wave-based imaging methods are conveniently described in frequency space (phasors), whereas time-domain methods (back-propagation) are described in a real time-resolved space; and ii) find the adequate propagation operator $\hat {U}_{m,n}$ that acts on the kets in a particular Hilbert space based on the specific restrictions of the reconstruction method. Other NLOS reconstruction methods may be derived from our proposed framework as well, following this methodology.

4. Derivation of the back-propagation formulation

The back-propagation formulation and reconstruction was one of the first methods proposed to reconstruct a hidden scene from an arbitrary number of illumination and sensing points [1]. In practical terms, illumination is realized using laser pulses whereas sensing is realized using time-resolved detectors. In the back-propagation formulation, for every voxel $\mathbf {x}_{v}$ of a hidden scene inside the imaging volume $\mathcal {V}$, the imaged signal corresponding to that voxel $G(\mathbf {x}_{v})$ from a set of time-resolved measurements $H_{m,n}(t)$ on the relay wall is defined as:

 

Fig. 2. Schematic of the back-propagation formulation. A mesh of voxels partitions the imaging space containing the hidden geometry. The signal for each voxel $G(\mathbf {x}_{v})$ is computed as the contributions from all time-resolved signals $H_{m,n} (t)$ corresponding to time $t_{v}$, which corresponds to the total time-of-flight $d_{1}$ of light from the illumination source to illumination point $n$ on the relay wall, time-of-flight $d_{2}$ from illumination point $n$ to a voxel $\mathbf {x}_{v} \in \mathcal {V}$, time-of-flight $d_{3}$ from voxel $\mathbf {x}_{v}$ to sensing point $m$ on the relay wall, and time-of-flight $d_{4}$ back to the sensor from sensing point $m$. $\mathbf {x}_{n}$ and $\mathbf {x}_{m}$ describe the position coordinates of illumination point $n$ and sensing point $m$ respectively.

Download Full Size | PDF

In quantum mechanics, an observable is an experimental measurement of an operator. We will show that the back-propagation formulation described in Velten et al. [1] can be derived by inverting the observable corresponding to the operator defined in Eq. (11). One of the fundamental postulates of quantum mechanics is that it is not possible to measure all possible outcomes of an operator simultaneously; instead, we can only measure the probability of all the possible outcomes of such operator. The Born rule [40] defines how to measure the observable $O$ corresponding to an operator $\hat {O}$, which defines the observable $O$ as the expected value of operator $\hat {O}$ using suitable basis functions $\{|b_{a}\rangle \}$ representing the system’s state:

The way that the expected value of an operator is defined by the Born rule determines that the result is a Real number or function. Although our scene operator is not based on an actual physical quantum mechanics system, it is formally defined as one and it is a Hermitian operator and, thus, it benefits from the mathematical body applicable to quantum mechanics. The reason to use the observable of operator Eq. (11) in deriving the back-propagation formulation is that the observable is a Real magnitude and back-propagation uses the intensity of the captured light, which is also a Real magnitude.

The extension of Eq. (12) to the continuous domain leads to Equation S2 in the supplemental material of Velten et al. [1], which states that:

We can evaluate the observable corresponding to the operator in Eq. (11) for a single illumination-sensor pair using the Born rule as:

The propagation term $\langle b | \hat {U}_{m,n}|l_{n}\rangle$ is:

Plugging Eq. (16) into Eq. (17) yields:

5. Derivation of the phasor fields formulation

In this section we show how the phasor fields formulation can be derived from Eq. (11). Prior to demonstrating the derivation, we will introduce a brief background of such formulation.

5.1 Phasor fields formulation background

The phasor fields formulation is a wave-based computational framework used in NLOS imaging that transforms the relay wall into a computational virtual light emitter and virtual camera. The immediate consequence is that it transforms NLOS scenarios into virtual line-of-sight ones, in which we can use well-known tools from Fourier wave optics to model sophisticated imaging systems.

The way to transform the relay wall into a virtual emitter, or virtual camera, is by modeling the wavefront of a virtual monochromatic light at different points and instants on the relay wall as a phasor. A phasor $\mathcal {P}_{\omega }(\mathbf {x}, t)$ at point $\mathbf {x}$, time $t$, and frequency $\omega$ is defined as:

The propagation of a phasor at point $\mathbf {x}_{1}$ lying on a surface $S_{1}$ to a point $\mathbf {x}_{2}$ lying on a surface $S_{2}$ is expressed as:

To image a hidden scene using phasor fields, we first must capture a set of impulse response signals of the scene. A single impulse response signal of the scene captured by an illumination-sensor pair $\{ m, n \}$ is expressed as $H_{m,n}$, where $n$, $m$ are the illuminated and sensed points on the relay wall. The virtual light source surface $\mathcal {L}$ comprises all points $\mathbf {x}_{n}$ on the relay wall acting as virtual emitters, whereas the virtual camera $\mathcal {S}$ comprises all points $\mathbf {x}_{m}$ on the relay wall acting as virtual sensors. Phasor fields allows to model the illumination on $\mathcal {L}$ as phasors $\mathcal {P}(\mathbf {x}_{n}, t)$, and the light captured at $\mathcal {S}$ as another phasor $\mathcal {P}(\mathbf {x}_{n}, t)$. Given the linearity and time-invariance of light transport, the phasor at the virtual camera can be expressed as the convolution between the phasor at the virtual emitter with the impulse response signal:

Finally, to generate the image of the hidden scene $I(\mathbf {x})$ as seen from the virtual camera, an image formation model $\Phi [\cdot ]$ is applied over $\mathcal {P}(\mathbf {x}_{m}, t)$ such that:

Phasor fields enables great flexibility in modeling different virtual emitters, virtual sensors, and image formation models based on different particular characteristics of the NLOS geometry and experimental hardware.

5.2 Derivation

ToF data is captured in the time domain. However, any signal captured in the time domain can be expressed as the superposition of its frequency components. To derive phasor fields from Eq. (9) we thus need to express the signals captured in the time-domain as the inverse Fourier transform of the corresponding signals in the frequency-domain. Moreover, a convenient Hilbert space to describe wave-based propagation methods is the frequency space, or Fourier space. Other Hilbert spaces may be used as well and, indeed, our proposed framework enables that possibility.

The ket representing the captured signal in the time-domain can be expressed as the inverse Fourier transform of the signal in the frequency-domain as:

The time component of the bra representing the illumination has to be expressed as the conjugate of the inverse Fourier transform as:

Using these expressions of the ket and bra in Eq. (9) results in:

Now, assuming that the illumination is a delta in time with a Gaussian distribution of energy in space results in:

Note that Eq. (27) is equivalent to the original phasor fields formulation in Eq. (21), as follows:

 

Fig. 3. (a) A laser source illuminates point $n$ on the relay wall with a delta light pulse $\delta _{n}(t)$. (b) An ultra-fast sensor captures time-resolved illumination $H_{m,n}(t)$ on multiple visible points $m$ on the relay wall. By convolving the impulse response with an illumination function, the phasor field formulation transforms the relay wall into a virtual light source with aperture $\mathcal {L}$ (c), and also into virtual camera with aperture $\mathcal {S}$ (d). These transformations allow modeling a computational lens that focuses the response from all points at each point in the bounding volume $\mathcal {V}$ of the hidden scene, effectively transforming the NLOS geometry into a virtual line-of-sight one. Figure adapted from [26].

Download Full Size | PDF

5.3 Rayleigh-Sommerfeld diffraction operator is the propagation operator in wave-based imaging methods

The Schrödinger equation describes the temporal evolution of a quantum system, which is expressed as:

The solution of the Schrödinger equation at an arbitrary position $\mathbf {x}$ and time $t$ for a free particle, in our case a photon, is described by a wavefunction $|\psi (\mathbf {x}, t)\rangle$ [37,40] such that:

Defining $|\psi (\mathbf {x}_{n})\rangle \equiv |\psi (\mathbf {x}_{0}, t_{0})\rangle$ as the wavefunction describing the state of light at the virtual light source surface $\mathcal {L}$ at $t_{0} = 0$ and $|\psi (\mathbf {x}_{m}, t)\rangle$ as the wavefunction describing the state of the captured light at the virtual camera $\mathcal {S}$ at a later time $t$, we obtain:

The contribution from all the point sources lying on the virtual light surface $\mathcal {L}$ results in:

6. Derivation of the f-k migration formulation

The f-k migration formulation is applied to confocal measurements in which the measurement of each illumination point happens at the same position of the sensing point, i.e. $\mathbf {x}_{m} = \mathbf {x}_{n} = \mathbf {x}$. Considering a single illumination-sensing pair located at the same position and writing the dependency on position and time explicitly, turns Eq. (11) into:

The propagation of light from $t$ to $t'$ can be expressed as:

Multiplying Eq. (37) by $|l_{n}(\mathbf {x}, t)\rangle$ on both sides we obtain:

The propagation equation that is used as the starting point for the f-k migration formulation is Eq. (1) in Lindell et al. [19], which states that:

Now, using the following identification:

The solution for Eq. (42) relies on the boundary condition:

This results shows how the f-k migration formulation can also be derived from our framework within the context of confocal measurements and the boundary conditions described by Lindell et al [19]. The physical interpretation of Eq. (43) is the virtual wavefront that complies with the boundary condition set by the f-k model.

7. Back-propagation and wave-based imaging methods are equivalent point-to-point

As we explained in Section 2, back-propagation and wave-based propagation are the two most used formulations to image hidden scenes in NLOS scenarios. While back-propagation voxelizes the imaging space and computes the signal contributions corresponding to the time of flight between the illumination and sensing points and a particular voxel, wave-based propagation models the propagation defining virtual light sources and virtual cameras converting the NLOS geometry into a line-of-sight geometry (hence, wave-based propagation formulations are also referred as forward-propagation formulations).

At first sight, it may look as if back-propagation and wave-based methods are quite different; however, our proposed formulation allows us to demonstrate formally that both are actually equivalent. The demonstration of this equivalence is based on the fact that propagation operators are unitary, which is demonstrated in Annex A. A unitary operator is an operator such that $\hat {U} \hat {U}^{\dagger } = \hat {U} \hat {U}^{-1} = \mathbb {I}$, the identity operator.

Fig. 4 depicts a diagram showing the operators and their propagation directions for both back-propagation and forward propagation, which will help visualizing the algebra developed in this section. The state of light at point $m$ is represented by $|s_{m}\rangle$, the state of light at point $\mathbf {x}_{v}$ is represented by $|\mathbf {x}_{v}\rangle$, the state of light at point $n$ is represented by $|l_{n}\rangle$, and the state of light at the virtual point $\mathbf {x}_{v}'$ is represented by $|\mathbf {x}_{v}'\rangle$.

 

Fig. 4. Diagram showing the operators and their propagation directions for both back-propagation and forward propagation to help visualizing the algebra developed in Section 7. $|l_{n}\rangle$ represents the illumination al point $n$, $|s_{m}\rangle$ represents the signal at sensing point $m$, $|\mathbf {x}_{v}\rangle$ represents the signal at the voxel $\mathbf {x}_{v}$ in the back-propagation model, and $|\mathbf {x}_{v}'\rangle$ represents the signal at the virtual voxel $\mathbf {x}_{v}'$ in the forward propagation model. The result expressed by Eq. (52) demonstrates that back-propagation and forward propagation are equivalent point-to-point.

Download Full Size | PDF

The propagation from $n$ to $m$ goes through $\mathbf {x}_{v}$, therefore:

In general, we will consider that $\hat {U}_{b,a}$ is the propagation operator of light from a generic point $a$ to a generic point $b$.

Now, the propagation from point $m$ to point $\mathbf {x}_{v}'$ is given by:

Plugging Eq. (46) into Eq. (48) yields:

We can define the propagation operator from $\mathbf {x}_{v}$ to $\mathbf {x}_{v}'$ as:

Now, using the property that $\hat {U}_{\mathbf {x}_{v}', \mathbf {x}_{v}}$ must be unitary:

This result demonstrates that propagating the state of light at point $m$ using the inverse of the forward operator from point $m$ to point $\mathbf {x}_{v}'$ in Eq. (48) is the same as inverting the propagation operator from point $\mathbf {x}_{v}$ to point $m$ defined in Eq. (46). In other words, we have demonstrated quite naturally that both back-propagation and wave-based imaging methods are equivalent point-to-point. This equivalence may open the possibility to develop hybrid formulations to optimize imaging scenes based on certain features of the scene and the experimental setup.

This results imposes certain conditions on the properties of the transport matrix $\left [ \hat {U}_{m,n} \right ]$, which considers the propagation among all illumination and sensing points. For instance, Eq. (8) describes the propagation of light from an illumination point $\mathbf {x}_{n}$ to a sensor point $\mathbf {x}_{m}$. However, the signal collected at the sensor point will be the contribution of all the illumination points and, hence:

Similarly, the inverse of Eq. (53) is:

Equation (54) is not the direct inverse of Eq. (53). Equation (53) expresses the contributions of all illumination points on a sensing point $m$, whereas Eq. (54) represents the contribution of all sensing points to an illumination point $n$ if we were to invert the propagation from all sensing points to illumination point $n$.

Using the fact that each component $(m,n)$ of transport matrix $\hat {U}_{m,n}$ is unitary, i.e. $\hat {U}_{m,n}^{-1} = \hat {U}_{m,n}^{\dagger } = \hat {U}_{n,m}^{*}$, Eq. (55) can be further expressed as:

Equation (57) implies that the trace of the matrix resulting from the product in Eq. (57) is equal to one (or a Real number if $\left [ \hat {U}_{m,n} \right ]$ is not normalized):

Although the transport matrix $\left [ \hat {U}_{m,n} \right ]$ as a whole might not be invertible, it must comply with Eq. (58) and, therefore, it can be used to invert it by using diagonalization or matrix decomposition methods.

8. Application: NLOS with non-delta illumination

Equation (9) and Eq. (11) provide the fundamental description of light interaction and propagation in a NLOS context using Dirac notation. This description is general enough to represent a varied range of NLOS reconstruction methods, such as back-propagation, phasor fields and f-k migration. The use of this notation not only serves as a general theoretical framework, but it enables derivations that are independent of the reconstruction algorithm as well. In this section we illustrate this with one of such derivations, in which we lift one of the main assumptions of current NLOS methods: the use of delta pulses of light as illumination source.

As stated in Eq. (10), the propagation operator $U_{m, n}$ relates the state of light on capture $|s_{m}\rangle$ and the state of the illumination $|l_{n}\rangle$. The information about the hidden scene lies within this propagation operator $U_{m, n}$ and NLOS reconstruction algorithms assume the illumination $|l_{n}\rangle$ to be a delta in time. However, real-world illumination hardware does not emit delta but near Gaussian pulses, accounting for excitation and decay. The width (standard deviation) of such Gaussian pulses is extremely small for sophisticated, expensive lasers, being larger (and therefore impractical for NLOS reconstructions) for lower-cost emitters. Using our framework, we can disentangle the effect of the illumination from the capture, enabling the use of Gaussian distributed temporal models for emitters (cheaper lasers), instead of delta models, which in turn enables the use of cheaper time-resolved sensors as well.

Similar to other techniques [18,25], we assume a single illumination point in ${n}$ with known temporal emission profile, which transforms Eq. (11) into:

Because phasor fields operates in the frequency space, we can express $\left\|{l_{1}}\right\|^{2}$ as the summation of the square amplitude of each frequency component of the signal as:

A delta pulse can be normalized such that:

Because of the superposition principle, we can decompose Eq. (61) for each frequency component $\nu$ as well. Then, dividing Eq. (61) on sides by $\left\|{l_{1, \nu }}\right\|^{2}$, we actually obtain a delta-coupled scene operator for each frequency component $\nu$:

Now, we can add all frequency components of the scene operator as:

We then use this delta-decoupled operator for the NLOS reconstructions instead of the original signal. In other words, we can achieve the same effect of a delta-like illumination from an arbitrary time-resolved illumination by applying Eq. (65) and using $\hat {T'}_{m, 1}$ as input for the NLOS reconstructions.

To validate our results, we have synthesized a hidden scene consisting of a Stanford dragon with a transient renderer optimized for NLOS imaging [41]. The dragon is at 1 meter of distance to a relay wall of $2\times 2$ meters, with a sensing grid of $128\times 128$ points $\mathcal {S}$, and a single illumination point $\mathcal {L}$ at the center of the relay wall. In Fig. 5 we show the results of reconstructing the hidden scene with the phasor fields framework [18], with the temporal model of a Gaussian pulse with different widths: 250, 500, 750, and 1000 picoseconds. As the top row shows, as the width of the pulse increases, the reconstruction becomes blurrier until no figure can be recognized. Using our framework, however, we obtain reasonable reconstructions (where the figure of the dragon is clearly recognizable) even in the presence of very wide Gaussian pulses.

 

Fig. 5. Results of the reconstruction of a dragon with phasor fields using a Gaussian light pulse of increasing width: 250, 500, 750, and 1000 picoseconds. Top: reconstruction of the raw signal with Gaussian illumination. As the width of the pulse increases the reconstructions become blurrier, to the point where the dragon cannot be recognized. Bottom: reconstruction using our framework. Even in the presence of very wide pulses, we can still obtain reasonable reconstructions.

Download Full Size | PDF

Aspects of the signal such as the effect of noise and decreasing temporal resolution can be explored under our proposed framework following on the steps of the example described in this section. However, such exploration lies outside the scope of this paper, which aims to establish a foundation for such exploratory works.

9. Discussion and conclusion

This work draws inspiration from Dirac himself, who said: "if one is working from the point of view of getting beauty in one’s equations, and if one has really a sound insight, one is on a sure line of progress", and: "a theory with mathematical beauty is more likely to be correct than an ugly one that fits some experimental data". We have focused on laying down the theoretical groundwork to provide a comprehensive formulation of the different mechanisms and imaging methods in NLOS imaging. In particular, formulating the NLOS problem in terms of the Dirac notation and defining a novel scene operator (Eq. (9)) has allowed us i) to derive the back-propagation, the phasor fields, and the f-k migration formulations from a unified framework; ii) to demonstrate that the Rayleigh-Sommerfeld diffraction operator is the propagation operator for wave-based propagation imaging formulations; and iii) to demonstrate that back-propagation and wave-based imaging formulations are equivalent. We have demonstrated that the unitary property of propagation operators is necessary so that the experimental measurements are real-valued magnitudes, reversible in time, and consistent with Helmholtz reciprocity principle. The development of this work has been based on the key realization that phasors defined in the wave-based formulation can be considered as projections of kets on different representations and operators in a Hilbert space that model the interaction of light between a relay wall and a hidden scene. The different projections of the kets yields the different formulations, including back-propagation, phasor fields, and f-k migration. The conclusions of this paper are consistent with traditional wave optics framework. However, we provide a different perspective that is more formal and general in the context of formulating NLOS imaging. We actually express the concept of time-resolved NLOS imaging in a way that is agnostic to the reconstruction algorithm and, therefore, our conclusions are not tied to particular algorithms.

Other NLOS methods could also be derived from our framework. Although a full formal derivation falls out of the scope of this paper, we provide here some meaningful insights towards this goal. For instance, the technique proposed in Saunders et al. [29] is based on measuring the shifts of the shadows caused by an occluder that is placed between the NLOS scene and the relay wall. Using our framework the shift of the measurement points on the relay wall could be modeled as an affine transformation of the propagation operator $\hat {U}$ connecting the illumination and the sensed points. Such affine transformation would be directly related to the relative position of the occluder and the hidden scene. The technique proposed in Pei et al. [30] is based on modeling arbitrary illumination patterns represented by different point-spread-function patterns, and performing an optimization process to minimize noise and select the most representative sampling points. One of the strengths of our framework is that it allows arbitrary illumination functions, therefore Pei et al. fits naturally in it. The technique proposed in Cao et al. [31] is based on actively changing the focus of the illumination using wavefront shaping. Being a wave-based method it could be derived similarly to the derivation of phasor-fields and f-k migration, with operator $\hat {U}$ being mapped to the operator in charge of wavefront shaping and focusing.

Finally, inspired by another Dirac’s quote stating that "I understand an equation when I can predict the properties of its solutions, without actually solving it", we have laid down the foundation of a cohesive mathematical framework to formulate NLOS challenges at large and we expect it will provide a deeper understanding of the aspects that are most important in imaging NLOS scenes as well as to devise more optimal and fast scene sampling and computational methods. This effort may include the formulation of a generalized light transport operator to model properties such as polarization, scattering, dispersion, or birefringence. The key process for incorporating those properties would be to represent them in a suitable Hilbert space with the corresponding propagation operators. This work is the first stepping stone for future research directions involving both theoretical and experimental results.