1. Introduction

Over the last decade, considerable progress has been made on the daunting task of seeing around corners by collecting and processing light reflected from a hidden scene into a visible space. This field of research, as well as the analogous one of seeing through diffusive transmission media like ground glass or fog, is called non-line-of-sight (NLoS) imaging [1–6]. One of the more promising developments in this line of work is phasor-field (${{\mathcal {P}}}$-field) imaging. Initially proposed by Reza et al. [7] and demonstrated by Liu et al. [8,9], the key idea behind ${{\mathcal {P}}}$-field imaging is to amplitude modulate a visible light source at radio or microwave frequencies and to manipulate, detect, and process the modulation envelope, i.e., the short-time-average (STA) irradiance, as if it were an optical wave using techniques from traditional line-of-sight (LoS) imaging. The difficulty in NLoS imaging arises from ordinary walls and diffusive transmission media being rough at the optical-wavelength scale, resulting in diffusely-scattered, spatially-incoherent light. However, at the much larger scale of radio and microwave wavelengths, i.e., the modulation wavelength of the ${{\mathcal {P}}}$ field, these walls and transmission media are smooth, and so the ${{\mathcal {P}}}$ field retains modulation-frequency directionality information in a manner similar to that for coherent light at the optical frequency in LoS scenarios.

Although the ${{\mathcal {P}}}$-field concept proved intuitively pleasing and experimentally impressive, we felt there was room for a sounder understanding of its theoretical underpinnings. So, contemporaneously with Teichman [10], we began a program of research dedicated to pursuing such understanding. Thus far the fruit of that labor is a theoretical framework for ${{\mathcal {P}}}$-field propagation through transmissive geometries which serve as proxies for more typical reflective NLoS scenarios [11]. Our framework is capable of characterizing both computational [11] and physical-optics ${{\mathcal {P}}}$-field imaging [12], efficiently accounting for hidden-space specularities and occlusions by means of the two-frequency spatial Wigner distribution (TFSWD) [11], as well as analyzing the effects of laser speckle [13]. Our framework, however, is limited to the paraxial, i.e., small-angle, propagation regime as it heavily relies on the Fresnel diffraction integral, whereas other formulations started from the more general Rayleigh–Sommerfeld diffraction integral [7,10].

The paraxial approximation is valid for propagation of an optical wave at wavelength $\lambda _0$ across a distance $L$ when the transverse diameters of interest at the input and output planes, $d_0$ and $d_L$ respectively, satisfy $(d_0+d_L)^{4}/128\lambda _0 L^{3} \ll 1$. However, typical parameter values for many NLoS imaging scenarios violate this condition, sometimes greatly [8]. An alternative sufficient condition for the validity of Fresnel diffraction is that the input field’s angular spectrum be comprised solely of components that lie close to the axis of propagation. Unfortunately, insofar as this alternative is concerned, the defining characteristic of NLoS scenes is their involving rough surfaces that scatter light over a broad range of angles. Accordingly, it seems that the paraxial approximation is likely to be violated in many, if not most, NLoS scenarios. Thus our paraxial theory of ${{\mathcal {P}}}$-field propagation should be extended to a nonparaxial theory by replacing our assumption of Fresnel diffraction with the more broadly valid Rayleigh–Sommerfeld diffraction. This is especially true as we hope to adapt our transmissive framework to the reflective geometry of real NLoS problems. In this paper, we take initial, key steps towards that goal.

First, in Sec. 2, we review our paraxial ${{\mathcal {P}}}$-field propagation framework both for its own merit and to remotivate our introduction of the TFSWD to handle hidden-space specularities and occlusions. Then, in Sec. 3, we use the Rayleigh–Sommerfeld diffraction integral to derive the appropriate ${{\mathcal {P}}}$-field propagation integral for free-space propagation following a diffuser and show the equivalent implication for the propagation of the diffuser-averaged STA irradiance. In Sec. 4 we call attention to the paraxial free-space propagation primitive for the TFSWD and show its equivalent formulation in terms of the 6D light field, viz., the TFSWD’s temporal Fourier transform. Using geometric intuition, we propose a nonparaxial propagation primitive for the 6D light field by replacing the paraxial terms in our previous result with their nonparaxial equivalents. We verify this proposal implies the correct behavior for the diffuser-averaged STA irradiance, and exhibit its equivalent formulation in terms of the TFSWD. We then provide a more formal derivation of that TFSWD equivalent which is valid under specific partial-coherence conditions. Finally, in Sec. 5, we provide a pair of differential equations that characterize free-space TFSWD propagation without restriction.

2. Paraxial ${{\mathcal {P}}}$-field and TFSWD propagation

To begin, we review the paraxial phasor-field propagation framework that we developed in Ref. [11]. We use paraxial, scalar-wave optics in a transmissive geometry that serves as an unfolded proxy for three-bounce NLoS imaging, as depicted in Fig. 1. Our framework characterizes propagation through such geometries in a plane-by-plane fashion, i.e., with a set of primitives that describe interactions with the variety of planar elements displayed in Fig. 1 and propagation from one non-empty plane to the next. Our paraxial assumption only enters into the propagation primitives, so it is only these primitives that need modification to extend our framework to the nonparaxial regime.

 

Fig. 1. Transmissive geometry that serves as an unfolded proxy for a three-bounce NLoS scenario whose hidden space contains a specular component and an occluder. The thin blue rectangles represent idealized thin diffusers that serve as analogs for the first and final bounces off the visible wall. The black line in the central plane represents a target in the hidden space with both specular and diffuse components. The small black rectangle between this target and the final plane represents an occluder in the hidden space that partially obstructs the light returning to the visible wall.

Download Full Size | PDF

The fundamental quantity we concern ourselves with is the optical field’s baseband, $\textrm {W}^{1/2}/\textrm {m}$-units, complex field envelope, $E_z(\boldsymbol \rho _z,t)$, which modulates a $\omega _0$-angular-frequency optical carrier to produce the optical field, $U_z(\boldsymbol \rho _z,t)=\textrm {Re}[E_z(\boldsymbol \rho _z,t)e^{-i\omega _0 t}]$, where $\boldsymbol \rho _z$ is the two-dimensional transverse spatial coordinate in the plane denoted by $z$ [14]. We take the bandwidth of this envelope to satisfy ${{\Delta \omega }} \ll \omega _0$ and its frequency-domain representation to be ${{\mathcal {E}}}_z(\boldsymbol \rho _z,\omega ) = \int \!\textrm {d}{t}\, E_z(\boldsymbol \rho _z,t) e^{i\omega t}$. In the frequency domain, propagation of this envelope through our transmissive geometry is characterized by the following primitives:

Propagation through diffusers: The frequency-domain field envelopes $\{{{\mathcal {E}}}'_k(\boldsymbol \rho _k,\omega ) : k = 0,2\}$ emerging from the $z=0$ and $z=L_1+L_2$ plane diffusers are given by

Propagation through transmissivity masks: The frequency-domain field envelope ${{\mathcal {E}}}'_d(\boldsymbol \rho _d,\omega )$ emerging from the deterministic occluder is given by

Fresnel diffraction: Fresnel diffraction governs free-space propagation from $z=0$ to $z=L_1$, viz.,

The STA irradiance is the squared magnitude of the complex field envelope, $I_z(\boldsymbol \rho _z,t)=|E_z(\boldsymbol \rho _z,t)|^{2}$, and we define the ${{\mathcal {P}}}$ field to be its diffuser-averaged Fourier transform:

Using the ${{\mathcal {P}}}$ field alone, propagation past the target plane in Fig. 1 can only be accomplished if the target is purely diffuse, i.e., if $\langle {F(\boldsymbol \rho _1)} \rangle =0$, in which case we have

All of the elements encountered by light in a typical NLoS scenario will effect linear transformations upon the complex field envelope. Moreover, phasor-field imaging is primarily concerned with second moments of the complex field envelope [15]. Consequently, for plane-by-plane propagation through NLoS scenarios for the sake of recovering ${{\mathcal {P}}}$-field or diffuser-averaged-STA-irradiance input-output behavior it suffices to follow the propagation of the complex field envelope’s space-time autocorrelation function,

In effect, the TFSWD and 6D light field serve as augmented versions of the ${{\mathcal {P}}}$ field and diffuser-averaged STA irradiance, respectively. Both contain equivalent information to the space-time autocorrelation, and we will work with each as convenient. Thus, they provide a means for developing the sort of modular primitives we seek, for any NLoS scenario of interest, while making the process of recovering a cumulative ${{\mathcal {P}}}$-field input-output from such primitives as painless as possible and, in particular, much easier than doing so from the autocorrelation function or, worse, the complex field envelope directly. That is, in our opinion, they are the simplest and most intuitive quantities, in terms of their relation to the ${{\mathcal {P}}}$ field, that suffice to handle the broadest class of NLoS scenarios.

In Ref. [11] we derived the following propagation primitives for the TFSWD:

Propagation through diffusers:

Propagation through deterministic occluders:

Propagation through specular-plus-diffuser masks:

Although not shown in Ref. [11], and not exploited thus far in our work, it is not hard to extend that paper’s derivation of Eq. (15) to show

Propagation through deterministic, planar LTI filters:

Most of these propagation primitives, namely those for propagation of the complex field envelope and TFSWD through diffusers and transmissivity masks, apply in general, without any assumption about the propagation angles, as they describe transformations at a single transverse plane. So, these elements already apply to nonparaxial operation, and accordingly we need only extend our Fresnel-diffraction results for the ${{\mathcal {P}}}$ field and TFSWD to the nonparaxial regime to have a complete set of nonparaxial primitives for Fig. 1-like scenarios.

3. Rayleigh–Sommerfeld ${{\mathcal {P}}}$-field propagation

Our development of a nonparaxial propagation primitive for the ${{\mathcal {P}}}$ field parallels that for Fresnel propagation as in Ref. [11]. We consider propagation first through a diffuser and then through a free-space distance $L_1$, as depicted in the first portion of Fig. 1. The key difference in the derivation is to replace Eq. (3) of that paper (Eq. (5) here)—the Fresnel diffraction integral for the complex field envelope—with the Rayleigh–Sommerfeld diffraction integral [23]

4. 6D light field and TFSWD

Next we turn our attention to the TFSWD. Here it turns out to be convenient to first seek the 6D light field’s primitive for nonparaxial propagation starting from its paraxial-propagation predecessor,

The preceding interpretation immediately exposes the imprecision of the Fresnel approximation. Neither $L_1$, as in the denominator of our interpretation for ${{\mathbf {s}}}$, nor $L_1+\left | {\boldsymbol {\rho }_+-\boldsymbol \rho _0} \right |^{2}/2L_1$, as in the time delay, are really the propagation distance of such a hypothetical ray. That distance would be $\sqrt {L_1^{2}+\left | {\boldsymbol {\rho }_+-\boldsymbol \rho _0} \right |^{2}}$. Say then we take ${{\mathbf {s}}} = \left ( {\boldsymbol {\rho }_+-\boldsymbol \rho _0} \right )/\sqrt {L_1^{2}+\left | {\boldsymbol {\rho }_+-\boldsymbol \rho _0} \right |^{2}}$ and solve for $\boldsymbol \rho _0$. We find that

 

Fig. 2. Trigonometry for Rayleigh–Sommerfeld propagation of the 6D light field. The red arrow represents a unit vector pointing in the propagation direction of a hypothetical ray propagating from $(\boldsymbol \rho _0,0)$ to $(\boldsymbol \rho _+,L_1)$.

Download Full Size | PDF

Taking the appropriately corrected values for the transverse offset and the total time delay, this suggests that the correct free-space propagation primitive for the 6D light field is

Inspired by this success, we propose the equivalent nonparaxial free-space propagation primitive for the TFSWD,

We begin that demonstration by shifting our attention from the output plane’s complex field envelope ${{\mathcal {E}}}_z(\boldsymbol \rho _z,\omega )$ to its angular spectrum $\tilde {\mathcal {E}}_z({{\mathbf {s}}},\omega )$, in which the field envelope is expressed as a superposition of propagating plane waves and all evanescent components are ignored, i.e.,

Before moving on, let us restate and more carefully consider our two key assumptions: (1) that $\langle {\tilde {\mathcal {E}}'_0({{\mathbf {s}}},\omega )\tilde {\mathcal {E}}_0^{\prime *}(\boldsymbol {\tilde {\mathrm {s}}},\tilde \omega )} \rangle$ differs appreciably from 0 only when $\left | {{{\mathbf {s}}}-\boldsymbol {\tilde {\mathrm {s}}}} \right | \ll 1$; and (2) that $\langle {{{\mathcal {E}}}'_0(\boldsymbol \rho _0,\omega ){{\mathcal {E}}}_0^{\prime *}(\boldsymbol {\tilde {\rho }}_0,\tilde \omega )} \rangle$ differs significantly from 0 only for $\left | {\boldsymbol \rho _0-\boldsymbol {\tilde {\rho }}_0} \right |$ small enough that $\left | {(\boldsymbol \rho _0+\boldsymbol {\tilde {\rho }}_0)/2-z{{\mathbf {k}}}/\sqrt {{k_0}^{2}-{{{\mathbf {k}}}}^{2}}} \right | \gg (\omega -\tilde {\omega }) \left | {\boldsymbol \rho _0-\boldsymbol {\tilde {\rho }}_0} \right | / 4[\omega _0+(\omega + \tilde {\omega })/2]$. We will make that consideration when $E_0(\boldsymbol \rho _0,t)$ is space-time factorable, i.e., when $E_0(\boldsymbol \rho _0,t) = E_0(\boldsymbol \rho _0)S(t)$, so that the complex field envelope emerging from the 0 plane and its temporal Fourier transform are

Unfortunately, the preceding argument does not suffice for validating Eq. (34) as the nonparaxial replacement for Eq. (17) because the latter applies to all the free-space propagation paths in Fig. 1. In particular, Eq. (17)’s derivation in Ref. [11] does not require the input TFSWD to be that for a field emerging from a diffuser. Hence our paraxial-propagation primitive for the TFSWD includes free-space propagation from the $z=L_1$ and $z=L_1+L_d$ planes in Fig. 1, whereas what we have proven so far for its nonparaxial replacement does not. To show that Eq. (34) can be applied to those non-diffuser situations we must work harder by allowing $E_0'(\boldsymbol \rho _0)$’s correlation function to be more general than the $\delta$-function form implied by Eq. (2). In that case—using $\boldsymbol \rho _+ = (\boldsymbol \rho _0+\boldsymbol {\tilde {\rho }}_0)/2$, $\boldsymbol \rho _- = \boldsymbol \rho _0-\boldsymbol {\tilde {\rho }}_0$, ${{\mathbf {s}}}_+ = ({{\mathbf {s}}} + \boldsymbol {\tilde {\mathrm {s}}})/2$, ${{\mathbf {s}}}_-= {{\mathbf {s}}}-\boldsymbol {\tilde {\mathrm {s}}}$, $\omega = \omega _++\omega _-/2$ and $\tilde \omega = \omega _+ - \omega _-/2$—we have that

To gain further insight into the preceding partial-coherence validity conditions for Eq. (17), suppose that

To make the preceding analysis applicable to interaction with an occluder, suppose that the frequency-domain envelope ${{\mathcal {E}}}_0'(\boldsymbol \rho _0,\omega )$ illuminates an occluder $P(\boldsymbol \rho _0)$ in the 0 plane resulting in a frequency-domain envelope ${{\mathcal {E}}}_0''(\boldsymbol \rho _0,\omega ) = {{\mathcal {E}}}_0'(\boldsymbol \rho _0,\omega )P(\boldsymbol \rho _0)$, where, e.g., $P(\boldsymbol \rho _0)$ could be the Gaussian pinspeck considered in Ref. [11]’s analysis of occlusion-aided ${{\mathcal {P}}}$-field imaging. This new frequency-domain envelope’s correlation function is then

5. Helmholtz equation

The last section required considerable work to obtain and validate—under some restrictions—a nonparaxial free-space propagation primitive for the TFSWD. So, it is worth seeking an alternative, differential-equation characterization that might provide a better route to such a primitive. In terms of the underlying positive-frequency optical field—which, in the time domain we denote $U_z^{(+)}(\boldsymbol \rho _z,t) \equiv E_z(\boldsymbol \rho _z,t) e^{-i\omega _0 t}$—the most general description we can provide is the wave equation,

Despite its not being clear what the unrestricted TFSWD free-space propagation primitive would be, we can characterize the TFSWD’s behavior quite generally. First we note that the TFSWD is a Fourier transform of the complex-field-envelope correlation $\langle {{{\mathcal {E}}}_z(\boldsymbol \rho _z,\omega ){{\mathcal {E}}}_z^{*}(\boldsymbol {\tilde {\rho }}_z,\tilde \omega )} \rangle$. It is not hard to see that this correlation must obey two Helmholtz equations, one for the first transverse spatial coordinate and frequency, and another for the second pair of coordinates. From the chain rule and the Fourier transform’s linearity, we can easily derive a pair of differential equations for the TFSWD which can be summarized by

Not surprisingly, Eq. (34) does not satisfy these differential equations. After all, it only suffices to reproduce the approximate Rayleigh–Sommerfeld behavior of the ${{\mathcal {P}}}$ field and not the full Rayleigh-diffraction behavior. However, it seems unlikely that any clean propagation primitive will satisfy these equations exactly, as all of our propagation derivations thus far have made liberal use of the quasimonochromatic assumption, which is of course not captured by the Helmholtz equation. Still, it is intellectually satisfying to have a general description in terms of such differential equations, and they may prove useful in some presently unforeseen circumstances.

6. Summary and discussion

Motivated by initial proposals and experimental demonstrations that established the utility of the ${{\mathcal {P}}}$-field concept for NLoS imaging, we embarked on a program to flesh out its theoretical understanding, beginning with study of paraxial, transmissive geometries. In this paper, mindful of the limitations of our earlier work’s paraxial assumption, we have derived nonparaxial post-diffuser propagation primitives for the ${{\mathcal {P}}}$ field and, equivalently, the diffuser-averaged STA irradiance using the Rayleigh–Sommerfeld diffraction integral for the complex field envelope. We have also proposed nonparaxial free-space propagation primitives for the 6D light field and, equivalently, the TFSWD, which enable handling a wider variety of NLoS scenarios than is possible using the ${{\mathcal {P}}}$ field alone. Our proposal was inspired by applying geometric intuition to our result for the Fresnel limit, and its validity is suggested by the fact that it reproduces our derived propagation result for the diffuser-averaged STA irradiance. We then provided a more formal derivation of this primitive using the angular-spectrum representation of the complex field envelope under certain reasonable constraints on the relevant correlation functions. An example in which our TFSWD propagation primitive will fail is as follows. Consider the spatial-Wigner-distribution limit of our Rayleigh–Sommerfeld diffraction primitive, viz.,

Despite this considerable progress, which successfully extends our previously developed framework to the nonparaxial regime, much work remains to be done, particularly in applying these results. In developing this extension, we have not reanalyzed the ${{\mathcal {P}}}$ field’s computational and physical imaging scenarios, nor have we reconsidered the effects of speckle. Our previous investigation of these topics benefited from the ability to obtain closed-form results from Gaussian integrals, a feat which will likely not be possible when using our new nonparaxial results. Consequently, it may be necessary to abandon the pursuit of closed-form insights in favor of numerical analysis and simulation for the nonparaxial case. However, in doing so, we expect to find that the essence of our results for imaging and speckle carry over nicely to the nonparaxial regime.

Even beyond these application-oriented opportunities, there is still room for further understanding of the pure theory. While our derivation of the Rayleigh–Sommerfeld propagation primitive for the TFSWD under partially-coherent circumstances is promising evidence for its validity, as is the reproduction by that primitive of the correct STA-irradiance behavior, it would be preferable to have a formal derivation of it for propagation from arbitrary planes, without restriction. A step towards this goal might be to find the Rayleigh–Sommerfeld ${{\mathcal {P}}}$-field input-output relation for occluder-interrupted, post-diffuser propagation in the geometric-optics limit, as we earlier showed to be intuitive in the paraxial regime [26]. Finally, the pinnacle of understanding nonparaxial propagation would be to find a Rayleigh-diffraction equivalent for the TFSWD or some other propagation primitive that satisfies the Helmholtz equation or, equivalently, our new differential equations for the TFSWD. More generally, progress in this area would include any practical developments in the understanding of the TFSWD’s differential equations that lead to a better, theoretically-sound understanding of the general propagation behavior of phasor-field quantities. In steering towards such broadly-applicable results, we hope that the results of this paper will serve as the foundation for transitioning our planar, transmissive framework to one that directly addresses standard, reflective NLoS geometries and the full complexities of 3D reconstruction.