1. Introduction

The plenoptic function provides a complete description of light under the geometrical optics approximation, where both the location and angle of each light ray are known [1]. The light field is a subset of the plenoptic function and is expressed as a four-dimensional (4D) radiance function having two angular and two spatial variables [2]. Our imaging scenario is illustrated in Fig. 1(a), where scattered light from non-line-of-sight (NOLS) objects is detected reflecting from a rough surface, such as a wall.

 

Fig. 1. (a) Assumed imaging scenario consisting of the hidden side and the accessible side. (b) and (c) Photographs of the real-scale experimental setup implementing the scenario illustrated in (a).

Download Full Size | PDF

In previous work, light-field reconstruction methods and related analyses have been proposed [3–17], and object location information has been extracted from light field data [15–25]. Moreover, previous authors [26–36] have developed passive methods to estimate NLOS information from shadows cast on the wall from occulders. However, the majority of these works assumed visible radiation and were compromised by the presence of ambient light.

Other previous work [37–39] has reported that infrared light facilitates NLOS imaging because it reduces scatter and heat radiating objects are seen as light emitters. A method assuming mirror-like scatterers has been proposed using LWIR radiation [37]; however, most construction walls have significant scattering even in the LWIR wavelength range. Other authors [38] have proposed a method for NLOS localization from information obtained by separating the specular and the diffuse reflections from the scatterer using LWIR light. However, its applicability has not been examined in practical cases such as observing objects with temperatures only slightly above the ambient background and scenes involving multiple objects. Here we investigate light-field techniques that substantially remove the intrinsically large scattering present in more realistic scenarios.

Authors in the biomedical field [16,17] have derived a forward model to describe volumetric scattering and have proposed an iterative method to localize point sources from 4D light field data. However, their formulation involves a large-dimensional matrix with a corresponding solution procedure that is computationally expensive, particularly in the 3D localization case. Moreover, it needs 2D (horizontal and vertical) camera scanning for our assumed scenario because it uses the full 4D light field. In contrast, we report on full 3D localization of the hidden objects that only requires a single horizontal scan of the IR camera, making it more amenable to real-time operation. Further, our method only requires solutions to independent small convolutions and forward filtering computations, which is more efficient and can be accelerated by an FFT algorithm and parallel processing.

Finally, methods based on stereo vision [40] have been developed to estimate objects’ depths from two or a low number of camera images. These methods can be efficient; however, they encounter a fundamental difficulty caused by the light scattering from the wall. Principally, the scattering effect (i.e., spatial blur) needs to be removed from each camera image of the wall by using spatial deconvolution for a better location estimation. However, the size of the blur kernel needed for the deconvolution changes depending on the unknown object’s depth location. Therefore, the image of the wall is generally a result of a mixture of convolutions having multiple blur kernels of different unknown sizes, resulting in a difficult-to-solve inverse problem. Here, our newly proposed procedure is free of this issue in the transverse and depth localization of the hidden objects and is more robust against this issue in the vertical localization of the objects by fully utilizing the light field information.

In our previous work, we have provided a fundamental analysis of the light field reconstruction process to remove the scattering effect [3,4] and proposed novel methods for determining the location of NLOS objects from the scattered light field [15]. In this current paper, by assuming prior knowledge of the wall’s Bidirectional Reflectance Distribution Function (BRDF) such as by remotely measuring the BRDF [41], we utilize the light field from scattered Long-Wavelength InfraRed (LWIR) radiation to realize full 3D localization of NLOS objects. A variety of applications are expected in the areas of security, safety, and rescue. Our measurement strategy only requires a single horizontal scan by a portable consumer LWIR camera. This quasi-real-time data collection captures the scattered light in the form of a light field cube, which is a function of horizontal spatial and angular variables (i.e., ${x_m}$ and ${\theta _m}$) and a vertical spatial variable (i.e., ${y_m}$). Using information from the ${x_m}$ and ${\theta _m}$ dimensions of the cube, we show in section 3 the recovery of transverse ${x_m}$ and depth ${z_m}$ locations of both human subjects and human-like objects. In section 4, we derive a method to extract the NLOS object’s vertical information from the remaining ${y_m}$ dimension of the cube, and propose a procedure to substantially remove the scattering effect from the entire cube. Finally, in section 5, we demonstrate the full 3D location estimation of the NLOS objects by utilizing all the information in the data cube. Throughout this paper, all experiments were carried out using NLOS human subjects and human-like objects in a real-scale hallway containing practical diffusive construction walls (e.g. painted drywall) as the scattering surface.

2. Quasi-real-time measurement strategy for light field cube acquisition

2.1 Single horizontal camera scanning

As illustrated in Fig. 1(a), we assume a scene consisting of hidden and accessible sides. The unknown NLOS objects (human subjects or thermally-elevated objects) are assumed to be present at arbitrary locations on the hidden side, whereas a single person equipped with a portable IR camera stays in the accessible area which is out of the direct view from the unknown objects. Our measurement method consists of scanning a single camera in a horizontal plane. A person equipped with an IR camera walks through the accessible area to capture many 2D images of a part of the scattering wall [illustrated as the dashed rectangle in Figs. 1(a) and (c)] from many different angles.

Figures 1(b) and (c) show photographs of our full-scale setup which implements the aforementioned scenario. In the first half of our experiments, we used a painted drywall and a Masonite board coated with a diffusive paint as examples of practical walls [Figs. 3(g) and (h) show their BRDFs]. The LWIR light was measured with a small portable micro-bolometer-based infrared camera with a spectral sensitivity ranging from 8∼14 μm. The camera was connected to a motorized stage, and a single horizontal angular scan was carried out to implement the aforementioned strategy. We note that the measurement by this automated system required only a few seconds (limited by the camera’s frame-rate), and we expect that quasi real-time measurements are possible.

2.2 Definitions, light field cube, and the separable BRDF approximation

Figure 2(a) shows an unfolded optical system corresponding to Fig. 1(a). The light field is generally expressed as a 4D function having two spatial and two angular variables, where ${x_m}$ and ${y_m}$ represent the horizontal and vertical spatial dimensions and ${\theta _m}$ and ${\phi _m}$ represent the horizontal and vertical angular dimensions, respectively. These spatial and angular measurements are defined on the scatterer, and the angles are defined with respect to the scatterer’s normal. Using these variables, we define the following three light fields: 1) ${L_{meas}}({{x_m},{y_m},{\theta_m},{\phi_m}} )$ represents the scattered light field measured by the IR camera; 2) $L_{obj}^0({{x_m},{y_m},{\theta_m},{\phi_m}} )$ represents the light field from the hidden objects before the scattering; and 3) $L_{obj}^{{z_m}}({{x_o},{y_o},{\theta_o},{\phi_o}} )$ represents the NLOS object’s light field on a virtual object plane at a particular ${z_m}$ location, where ${x_o}$, ${y_o}$, ${\theta _o}$, and ${\phi _o}$ are variables on the virtual object plane. In addition, $[{{a_m},{b_m}} ]$, $[{{A_m},{B_m}} ]$, and $[{{{\tilde{A}}_m},{{\tilde{B}}_m}} ]$ represent measurement ranges in ${\theta _m}$, ${x_m}$, and ${y_m}$ dimensions, respectively.

 

Fig. 2. (a) Unfolded optical system equivalent to the scenario in Fig. 1(a). (b) Light field cube measured by the horizontal camera scanning.

Download Full Size | PDF

In Fig. 2(b), the plane B illustrates the 2D spatial information (represented by the ${x_m}$ and ${y_m}$ axes) captured in a single camera image of the wall from a specific camera angle. By changing the camera angle in the horizontal plane, the remaining angular information (represented by the ${\theta _m}$ axis) is measured. Therefore, all information measured by our strategy is expressed as a 3D slice of a 4D light field represented as a light field cube, $L({{x_m},{y_m},{\theta_m}} )$ as illustrated in Fig. 2(b).

As illustrated in Fig. 2(a), the wall’s scattering property is described by the BRDF denoted as ${f_{BRDF}}({{\theta_o},{\theta_m},{\phi_o},{\phi_m}} )$, where ${\theta _o}$ and ${\phi _o}$ correspond to incident angles, and ${\theta _m}$ and ${\phi _m}$ are outgoing angles, respectively. In this current paper, the BRDF is assumed to be homogeneous and is approximated as a separable function, which is expressed as a product of a function of two horizontal angles, denoted $f_{BRDF}^H$, and that of two vertical angles, denoted $f_{BRDF}^V$, as:

3. Transverse ${{\boldsymbol x}_{\boldsymbol m}}$ and depth ${{\boldsymbol z}_{\boldsymbol m}}$ localization of the NLOS objects enhanced by the LWIR light field

In this section, we first describe the ambient light suppression and the process to enhance the weakly scattered signal using the LWIR light field. Secondly, we experimentally demonstrate localization of the NLOS human subject or human-like objects [39]. We restrict these initial calculations to use only the 2D horizontal dimensions of the light field cube [i.e., ${x_m}$-${\theta _m}$ data evaluated at a central vertical ${y_m}$ location, corresponding to the plane A in Fig. 2(b)] for estimating the transverse ${x_m}$ and depth ${z_m}$ locations of the hidden objects. The vertical variables ${y_m}$ and ${\phi _m}$ are omitted for simplicity.

3.1 Visible ambient light suppression

We first compare conventional visible and LWIR photographs in the situation where a single human subject exists in the hidden side. Figures 3(a) and (d) show the direct views of the hidden side. Although it is very difficult to discriminate between the human subject and the background in Fig. 3(a), the subject is clearly distinguishable from the background in the LWIR image shown in Fig. 3(d). Note that most of the clutter in the background tends to have similar temperatures and emissivity in the LWIR wavelength range, leading to a highly uniform background. This greatly helps to discriminate between objects with elevated temperatures and a uniform temperature background. Figures 3(b) and (e) are visible and LWIR photographs of the painted drywall scatterer, respectively. In Fig. 3(b), it is impossible to recognize the scattered signal from the human subject because the drywall is brightly illuminated from external sources. In Fig. 3(e), however, the radiance of the ambient light from the clutter and wall is relatively uniform and a faint signal of the human subject on top of this bias is visible, albeit barely, within the dashed ellipse. Clearly, additional signal processing is needed to enhance the scattered NLOS signal. Figures 3(c) and (f) show an equivalent case for the painted Masonite board.

 

Fig. 3. Comparison of hallway photographs at visible and IR wavelengths, and the scatterers’ BRDFs. Conditions for (a) ∼ (f): A human subject was placed at 2.3 m from the scatterer; ambient temperature = 23 deg C; Optris PI-400 IR camera (NETD 40mK, H382${\times} $V288 pixels). (a) and (d) Direct views of the hidden side. (b) and (e) Photographs of the painted drywall. (c) and (f) Photographs of the painted Masonite. (g) and (h) The painted-drywall’s and the painted-Masonite’s specular component of LWIR BRDF $f_{BRDF}^H$, respectively, after removal of the diffuse component. (Note: The BRDFs measurements were made using the same method as used for the light field measurement using a vertical narrow IR heater element as a NLOS object.)

Download Full Size | PDF

3.2 Signal enhancement by utilizing the LWIR light field information

Here, we explain the signal enhancement process realized by fully utilizing light field information. For the current experimental demonstration, we used a painted drywall scatterer and two human-temperature objects located at 1.7 m and 2.9 m from the scatterer (for details and other conditions, see the caption of Fig. 4).

 

Fig. 4. Experimental results of the signal enhancement process. Conditions: Two human-temperature objects (constructed from plastic water pipes maintained at a constant temperature of 37 deg C) were placed at 1.7 and 2.9 m from the painted drywall scatterer [whose BRDF is shown in Fig. 3(g)]. The light fields are expressed as a function of horizontal pixel location of the camera sensor and the camera angle taken from sequential camera frames. (a) Raw LWIR light field. (b) Light field after applying pixel averaging (H15${\times} $V100 window) obtained from (a). (c) Enhanced light field ${L_{meas}}$ obtained from (b) by the SVD first-rank subtraction. Note: One camera count corresponds to one NETD (40mK).

Download Full Size | PDF

As seen in Figs. 3(e) and (f), the IR camera image contains a large unwanted signal term with a very weak NLOS information-bearing signal term superimposed on top. This information-bearing signal originates from the specular component of the BRDF and is usually several orders of magnitude smaller than the unwanted signal. For our commercially available microbolometer camera, its magnitude is on the order of the camera’s Noise Equivalent Temperature Difference (NETD). Figure 4(a) shows a raw IR light field expressed as a function of the horizontal pixel location of the camera sensor and the camera angle (corresponding to camera frames taken while scanning horizontally). Note that, by assuming the camera’s keystoning effect is small, the horizontal pixel location and the camera angle can be roughly approximated as the horizontal spatial dimension on the scatterer (i.e., the ${x_m}$ axis) and the horizontal angular dimension on the scatterer (i.e., the ${\theta _m}$ axis), respectively.

As can be seen in Fig. 4(b), the quantization noise can be reduced by a spatial pixel averaging of the camera sensor. The distribution shown in this figure is thought to be a mixture of the faint NLOS information-bearing signal and the significant unwanted signal. This unwanted signal mainly originates from the following two sources: 1) the diffuse component of the BRDF and 2) the (possibly non-uniform) radiation from the scatterer. In our previous work [4], the diffuse BRDF term was approximated by a separable function expressed as a product between an incident-angle function and an outgoing-angle function on the scatterer. Assuming the BRDF is homogeneous, the scattered light originating from this term is expressed as a product between two terms, one which is a function of ${x_m}$ and the other which is a function of ${\theta _m}$. The second source can be represented as a collection of IR emitters on the scattering wall; therefore, it can be expressed as a product of two terms, one which is a function of ${x_m}$ and a second constant term with no ${\theta _m}$ dependence. In addition, microbolometer calibration errors distort the entire signal [43]. These errors consist of fixed pixel calibration errors (such as a non-uniform offset in pixels) affecting the spatial ${x_m}$ dimension of the light field and dynamic changes in sensor calibration that alter the bias of each frame, affecting the angular ${\theta _m}$ dimension of the light field in our measurement procedure.

For the aforementioned reasons, the unwanted signal and the calibration error can be roughly approximated as a separable function expressed as a product of a function of the horizontal pixel location (${\sim} {x_m}$ dimension) and that of the camera angle (${\sim} {\theta _m}$ dimension). This type of unwanted signal can be largely eliminated by subtracting the first-rank component (i.e., the most significant separable component) of the singular value decomposition (SVD) [44,45] computed from the distribution in Fig. 4(b). Figure 4(c) shows the result of this subtraction, where we can clearly see the NLOS information-bearing signal as the tilted line spread function. This enhanced light field is referred to as ${L_{meas}}$ in the following development. Note that the tilt angle of the NLOS signal represent the parallax effect inherent in light fields, and this feature differentiates the NLOS signal from the separable unwanted signal and error. Moreover, in Appendix A, we experimentally demonstrate the current enhancement process in the presence of a more significant nonuniform temperature distribution on the wall.

3.3 Estimation of transverse ${{\boldsymbol x}_{\boldsymbol m}}$ and depth ${{\boldsymbol z}_{\boldsymbol m}}$ locations of the NLOS objects

In this section, we describe the process of extracting the hidden objects’ information from the enhanced light field (i.e., ${L_{meas}}$ from the previous section). We further demonstrate the NLOS localization by using methods developed based on a refocusing algorithm.

3.3.1 Horizontal angular deblurring process

From the previous section, the NLOS information-bearing signal still has significant angular blur corresponding to the angular spread of the specular component of BRDF. Here, we start by removing this blur from ${L_{meas}}$ to extract the NLOS object’s information. Related research was reported by authors in the computer vision field [5–13]. From our previous work [3,4], we can apply a light-field reconstruction process which we refer to as horizontal angular deburring. To make this inverse problem tractable, the system operator connecting $L_{obj}^0$ and ${L_{meas}}$ needs to be formulated as a system of many independent Fredholm integral equations of the 1^st kind, expressed using angular variables (i.e., ${\theta _o}$ and ${\theta _m}$) and having the scatterer’s BRDF $f_{BRDF}^H$ as an integration kernel. Each equation is formulated at a single spatial point on the scatterer [represented by ${x_m} = {c_r}$ for $r = 0,1,2 \ldots $ illustrated in-between Figs. 5(a) and (b)] and includes data on the 1D ${\theta _m}$-line in ${L_{meas}}$ and that in $L_{obj}^0$ [illustrated as the pairs of two dashed lines connected by arrows in Figs. 5(a) and (b)]:

 

Fig. 5. Experimental demonstration of the light field reconstruction to remove the blurring effect. Conditions: Measurement spatial range (i.e., the baseline) ${x_m} \in [{ - 18\textrm{cm},\,18\textrm{cm}} ]$ and measurement angular range ${\theta _m} \in [{1\, \textrm{ deg},\,58\, \textrm{ deg}} ]$ (other conditions were the same as those in Fig. 4). (a) ${L_{meas}}({{x_m},\; {\theta_m}} )$ enhanced by H15${\times} $V100 pixel averaging and the SVD first-rank subtraction from section 3.2. (b) Object light field $L_{obj}^0({{x_m},{\theta_m}} )$ reconstructed from (a) by a Wiener filter from [4]. (c) Angular profiles of ${L_{meas}}$ and $L_{obj}^0$.

Download Full Size | PDF

By solving Eq. (2) at many spatial points under the prior knowledge of the BRDF, the entire 2D object radiance $L_{obj}^0({{x_m},{\theta_m}} )$ in Fig. 5(b) can be reconstructed from the 2D measured and enhanced radiance ${L_{meas}}({{x_m},\; {\theta_m}} )$ in Fig. 5(a). As seen in Fig. 5(b), this process clearly reveals the two line-spread functions corresponding to the two NLOS human-like objects, where the angular width ${\Delta _\theta }$ in Fig. 5(b) represents an uncertainty caused by the scattering event and the noise remaining in ${L_{meas}}$. Finally, we note that this equation can be efficiently solved in the frequency domain of ${\theta _m}$ (i.e., ${f_\theta }$ domain), because the effect of the diffuse component of the BRDF has been largely eliminated from ${L_{meas}}$ by the SVD first-rank subtraction (section 3.2) and Eq. (2) can be well approximated as a convolution using the specular component of the BRDF as a shift-invariant kernel.

A brief review of our analysis on the retrievable NLOS information from scattered light is as follows. From sections 4.A and 5.D in our previous work [4], we have showed that, by modeling the aforementioned reconstruction process as a Fourier truncation regularization, the Mean-Squared Error (MSE) caused in the reconstruction process can be expressed as an analytical function of ${f_\theta }$ (where ${f_\theta }$ is the frequency variable associate with ${\theta _m}$). This angular frequency value ${f_\theta }$ corresponds to the highest frequency component passed by the regularization procedure. From this analytical function, a particular ${f_\theta }$ minimizing the MSE can be theoretically predicted under the prior knowledge of the BRDF (such as the degree of specularity $\gamma $ from [4]) and statistical properties of the noise remaining in ${L_{meas}}$. This MSE minimizer, denoted ${f_{\theta \; max}}$, can be understood as the highest angular frequency which is retrievable from the scattered light. As we can expect, the uncertainty ${\Delta _\theta }$ illustrated in Fig. 5(b) is approximated by an inverse value of ${f_{\theta \; max}}$ and limits the NLOS localization accuracy.

3.3.2 NLOS localization based on the refocusing algorithm

After suppressing the deleterious scattered light, we now localize the hidden objects from the reconstructed $L_{obj}^0$ using our refocusing-based methods. The refocusing algorithm was developed in the computer vision field [46–48] and can be used to produce a spatial ${x_m}$ - ${z_m}$ distribution (commonly known as focal-stack) from light fields. Peak locations in this distribution correspond to estimated object locations; however, the distribution has an excessive depth blur inherent in the refocusing algorithm. This blur degrades the localization accuracy and needs to be improved. (Note: Other major methods used for localization, such as the inverse Radon transform and the Hough transform, share the same depth blur issue.)

From our previous work [15], we have developed two localization techniques based on the refocusing algorithm. The first technique, the radiance-preserving refocusing (RP-refocusing), is expressed as spatial integration by Eq (3):

 

Fig. 6. Experimental demonstration of the NLOS localization obtained from the $L_{obj}^0$ in Fig. 5(b). (a) Spatial distribution computed by the standard refocusing method from $L_{obj}^0$ in Fig. 5(b). (b) Spatial distribution computed by the RP-refocusing method in Eq. (3) from $L_{obj}^0$ in Fig. 5(b). (c) Spatial frequency spectrum obtained from Fig. 5(b) by using the relation from Eq. (4). (d) Final localization result obtained by fusing the RP-refocusing and the ODF results (note: the stars in the figure indicate actual object locations, and the peak values of two bright areas have been normalized).

Download Full Size | PDF

In the following, we review the ODF method. Our previous work [15] showed that the standard refocusing operation (consisting of the light-field’s propagation and the projection along $\theta $ dimension) serves as a mapping between the light field and the full spatial frequency spectrum. This can be seen as an alternative expression of the commonly known projection-slice theorem. From section 4.A in [15], the refocusing operation maps a point in a 1D spatially Fourier-transformed light-field [i.e., a point of $({{f_x},{\theta_m}} )$ where ${f_x}$ is the frequency variable associate with ${x_m}$] to a corresponding point in the full spatial frequency spectrum [i.e., a point of $({{f_x},{f_z}} )$ where ${f_z}$ is the frequency variable associate with ${z_m}$] by the following relation:

In Fig. 6(c), we can see the triangle-shaped window corresponding to the angular measurement range $[{{a_m},{b_m}} ]$, where the inversely Fourier transformed data of this triangle-shaped distribution produces the standard refocusing result shown in Fig. 6(a). From Fig. 6(c), we can expect that a wider ${f_z}$ opening of the triangular window at a higher ${f_x}$ values will lead to a smaller depth blur, whereas the narrower ${f_z}$ opening at lower ${f_x}$ values produces a larger depth blur. (Note that the ${f_z}$ opening is indicated by the horizontal arrow in the figure.) Based on this observation, we have developed a spatial frequency filter to select the best spectral information to improve the depth uncertainty. By assuming an object depth location ${d_{obj}}$ (i.e., object’s depth distance from the scatterer) has been roughly estimated, the highest spatial frequency ${f_{x\; max}}$ retainable by this particular object can be approximated from the light-field’s propagation as:

3.4 Statistical analysis of the localization accuracy

In this section, after briefly reviewing the analysis on the localization accuracy from our previous work [15], we show experimentally obtained statistics on the localization accuracy at LWIR wavelengths. The theoretical limit of localization accuracy has been derived in [15] in consideration of the scattering effect. According to this analysis, the transverse ${x_m}$ localization uncertainty, denoted ${\Delta _x}$, is approximated by an inverse value of ${f_{x\; max}}$ as in Eq. (6). Also, the depth ${z_m}$ localization uncertainty, denoted ${\Delta _z}$, can be expressed as Eq. (7) based on the parallax effect seen in light fields:

For the current experiments, we used an actual human subject as the NLOS object and painted drywall as the scattering surface. Figures 7(a) and (b) show the mean values of estimated depth and transverse locations, respectively. We can see reasonable agreements between the estimated and the actual locations. Figure 7(c) shows the Standard Deviation (SD) value of the estimated locations. As can be seen, the depth SD value rapidly becomes larger as the distance increases, whereas the transverse SD value shows a gradual increase. This observation agrees with the theoretical prediction from Eqs. (6) and (7) showing that the depth resolution limit is a quadratic function of the absolute depth whereas the transverse resolution limit is linearly proportional to the depth. Also, the transverse and depth sizes of the bright areas in Figs. 7(d) ∼ (f) [and Fig. 6(d)] are consistent with the theoretical predictions. Additional statistical analysis on the influence of the object’s temperature is shown in Appendix B.

 

Fig. 7. Experimentally obtained localization statistics. (a) and (b) Mean values of estimated depth ${z_m}$ and transverse ${x_m}$ locations, respectively. (c) Standard deviation values of estimated depth and transverse locations, respectively. (d)∼(f) Spatial distributions obtained by the RP-refocusing and the ODF method. The brightest yellow peak corresponds to the estimated object location whereas the star indicates the actual location. Conditions: A single human subject was placed at four different depth locations (1.7∼3.5 m) in front of a painted drywall scatterer. The experiment was run 30 times to produce the statistical values.

Download Full Size | PDF

4. Scattering effect removal from the entire light-field cube

For realization of the full 3D location estimation, we can straightforwardly extend the methods used in section 3 by considering the 4D light field; however, this requires scanning a camera in both the horizontal and vertical directions. Here, without increasing the measurement complexity, we utilize the remaining vertical spatial information contained in the light field cube [i.e., ${y_m}$ axis in Fig. 2(b)] for the 3D localization. In this section, we first propose a method to correct for the blurring effect contained in the 1D ${y_m}$ data in the cube and then propose a procedure to remove the scattering effects from the entire cube as a preparation for 3D localization.

4.1 Vertical spatial deblurring process

For removing the blurring effect in the vertical dimension, we need to solve an angular integral equation, expressed using vertical angles (i.e., ${\phi _o}$ and ${\phi _m}$) and using the vertical BRDF $f_{BRDF}^V$ as an integration kernel. However, information regarding the vertical angle is not available in the light field cube. For this reason, we reformulate the angular equation into a spatial integral equation using the vertical spatial variable ${y_m}$ by assuming that the hidden objects’ ${x_m}$ and ${z_m}$ locations are previously estimated by the methods described in section 3. In the following, we consider the 2D vertical dimensions of the light field (i.e., ${y_m}$ and ${\phi _m}$ dimensions). The horizontal variables ${x_m}$ and ${\theta _m}$ are omitted for simplicity.

Figure 8(a) shows the unfolded optical system of our scenario corresponding to Fig. 2, where the camera captures a wall image from a particular camera position. We focus on the virtual vertical plane illustrated in Fig. 8(a), which contains the camera position and the vertical dotted line on the wall located at a particular ${x_m}$ location. $\tilde{z}$ is an axis defined by the camera position and the particular ${x_m}$ location of the vertical dotted line, and the horizontal angle ${\theta _m}$ is determined by $\tilde{z}$. Also, the vertical angle ${\phi _m}$ is defined on the vertical plane as illustrated.

 

Fig. 8. (a) Unfolded optical system corresponding to Fig. 2(a). (b) Blurred light field ${L_{meas}}({{y_m},\; {\phi_m}} )$ observed on the vertical dotted line on the wall in (a).

Download Full Size | PDF

Here, the aforementioned vertical dotted line on the wall is of interest for the blurring effect removal. [This vertical dotted line is also illustrated in Figs. 2(a) and (b).] Let us assume a single point-source exists on the vertical plane in the hidden scene as illustrated in Fig. 8(a). The distance between the camera position and the vertical dotted line is denoted L and the distance between the vertical dotted line and the point source is assumed to be ${\tilde{d}_{obj}}$ (corresponding to ${z_m} \cong{-} {\tilde{d}_{obj}}\cos {\theta _m}$). In this situation, as illustrated in Fig. 8(b), the blurred light field of the point source, i.e., ${L_{meas}}({{y_m},\; {\phi_m}} )$ observed on the vertical dotted line in Fig. 8(a), is expressed as the line spread function whose slope is approximately equal to ${\tilde{d}_{obj}}$ and whose angular profile is the same as the wall’s vertical BRDF $f_{BRDF}^V({{\phi_m}} )$. Note that $f_{BRDF}^V$ is defined similarly to Eq. (2) using the vertical variables and is approximated to be a shift-invariant specular component because the diffuse component has been eliminated in section 3.2.

For our mathematical derivation, we consider a particular 1D spatial distribution along the ${y_m}$ axis (i.e., ${\phi _m} = 0$) in Fig. 8(b). From the geometrical relation seen in Fig. 8(b) and by assuming the paraxial approximation, the spatial distribution of the line spread function on the ${y_m}$ axis can be expressed as a spatially scaled BRDF by:

On the other hand, the camera captures the light field data along the tilted dotted line in Fig. 8(b), whose slope is approximately equal to $- L$. Therefore, the vertical spatial profile in the camera image, denoted ${u_{cam}}({{y_m}} )$, can be seen as a result of the projection applied to the data on the tilted dotted line, as illustrated in Fig. 8(b). Both Eq. (8) and ${u_{cam}}({{y_m}} )$ in Fig. 8(b) show the vertical distribution from the same point source; however, there is a difference in spatial scaling. From the geometrical relation seen in Fig. 8(b), the scaling factor can be derived and the adjusted camera image, i.e., ${L_{meas}}({{y_m},\; 0} )$, can be expressed on the ${y_m}$ axis as:

We now move to a more general situation, where some NLOS objects, denoted $L_{obj}^{{z_m}}({{y_o},\; {\phi_o}} )$ and having a uniform angular ${\phi _o}$ distribution (i.e., IR light emitters), are assumed to be present at a single depth distance ${\tilde{d}_{obj}}$ [i.e. at the same depth location of the point source in Fig. 8(a)]. By considering the fact that propagating $L_{obj}^{{z_m}}({{y_o},\; 0} )$ to the vertical dotted line on the scatterer in Fig. 8(a) does not change the spatial y distribution (because ${\phi _o} = 0$), we can see that $L_{obj}^{{z_m}}({{y_o},\; 0} )$ is equal to $L_{obj}^0({{y_m},\; 0} )$. In this situation, it can be seen that Eq. (8) is the convolution kernel representing the blurring effect, Eq. (9) is the blurred distribution as a result of the convolution, and $L_{obj}^0({{y_m},\; 0} )$ is the unknown object distribution before the scattering. We finally obtain the spatial system equation expressed on the ${y_m}$ vertical dotted line on the scatterer as:

Because we can estimate the depth ${\tilde{d}_{obj}}$ by the method described in section 3.3, Eq. (10) can then be solved for the unknown object’s vertical spatial distribution $L_{obj}^0({{y_m},\; 0} )$. Obviously, Eq. (10) can be efficiently solved in the ${f_y}$ spatial frequency domain. This new process of vertical spatial deblurring is the complement to the horizontal angular deblurring developed in section 3.3.1. Note that the data along the particular vertical dotted line in Fig. 8(a) corresponds to that along the vertical-dotted line in the light field cube illustrated in Fig. 2(b). The same vertical deblurring can be applied to any vertical ${y_m}$ data in the light field cube. A more general case, where multiple NLOS objects have different depth locations, will be discussed in section 4.3.3.

4.2 Combined procedure to remove the scattering effect from the entire light field cube

By combining the horizontal angular deblurring used in section 3.3.1 and the vertical spatial deblurring in the last section, we can substantially remove the blurring effect from the entire light field cube by the following four steps:

4.3 Experimental demonstrations

To demonstrate the procedure described in the last section, we used a board covered with wallpaper as the scatterer and three warm water-pots located at 1.7∼2.3 m from the wall as the NLOS objects. Figure 9(a) shows the horizontal and vertical BRDFs of the wall. For other conditions, see the caption of Fig. 9. Note that each pot has an information-bearing signal level that is 30% lower than that of a human subject (due to its smaller size), indicating the pots are more difficult to detect than actual human subjects.

 

Fig. 9. Experimental demonstration of the procedure from section 4.2 in the case of a single camera image. Conditions: A scattering surface consisting of a Masonite board covered with a wallpaper, three 45-degC water pots used as NLOS objects [one pot was placed at 1.7 m from the scatterer, and other two were vertically stacked (with 38 cm vertical spacing) and placed at 2.3 m from the scatterer], ICI 8640S IR camera [H366${\times} $V512 pixels were used, NETD 20mK], measurement ranges: ${\theta _m} \in [{0\textrm{deg},\,44\textrm{deg}} ]$, ${x_m} \in [{ - 15\textrm{cm},\,15\textrm{cm}} ]$ and ${y_m} \in [{ - 20\textrm{cm},\,20\textrm{cm}} ].$ (a) The wallpaper’s specular component of the LWIR BRDF; $f_{BRDF}^H$ and $f_{BRDF}^V$, after removal of the diffuse component. (b) and (c) IR camera images before and after the pixel averaging (21${\times} $21-pixel averaging window), respectively. (d) ${L_{meas}}({{x_m},{y_m}} )$ after performing the 1^st step from section 4.2. (e) Horizontally deburred data obtained by the 2^nd step. (f) Vertically deburred data, i.e., $L_{obj}^0({{x_m},{y_m}} )$, obtained from (e) by the 4^th step (Tikhonov regularization) with the blurring effect horizontally and vertically removed.

Download Full Size | PDF

4.3.1 Single camera image case

Let us first consider a vertical 2D slice of the cube illustrated as the plane B in Fig. 2(b). This slice corresponds to a single camera image taken from a specific camera angle. Figure 9(b) and (c) show the IR camera images before and after the pixel averaging, respectively. Because of the unwanted signal and the camera calibration error (section 3.2), the information-bearing signal from the NLOS objects is barely recognizable. Figure 9(d) results from implementing the 1^st step of the procedure described in section 4.2 and shows the enhanced ${L_{meas}}({{x_m},{y_m}} )$, where we can clearly see the information-bearing signal. However, the signals from the three objects are connected due to the significant blurring. Figure 9(e) shows the results of the 2^nd step. The objects have been clearly resolved in the horizontal direction; however, two of them are still connected in the vertical direction. Figure 9(f) shows the final result recovering the object light field $L_{obj}^0({{x_m},{y_m}} )$ after applying the 4^th step. The three objects have been clearly resolved in both the horizontal and vertical directions.

4.3.2 Entire light field cube case

All of the experimental conditions were the same as those in Fig. 9. Figure 10(a) shows the enhanced light field cube ${L_{meas}}({{x_m},{y_m},{\theta_m}} )$ in the 1^st step of the procedure from section 4.2, where the information-bearing signals from the three NLOS objects are connected due to the significant blurring. Figure 10(b) shows the result of the 2^nd step, where two horizontally separated distributions can be seen. Finally, Fig. 10(c) shows the recovered light field cube $L_{obj}^0({{x_m},{y_m},{\theta_m}} )$ with the blurring effect horizontally and vertically removed by the 4^th step. We can clearly identify the three isolated light field distributions corresponding to the three NLOS objects, which is a requirement for full 3D localization.

 

Fig. 10. Experimental demonstration of the procedure from section 4.2, applied to the entire light field cube. All of the conditions were the same as those in Fig. 9. (a) enhanced ${L_{meas}}({{x_m},{y_m},{\theta_m}} )$ after applying the 1^st step from section 4.2. (b) Horizontally deblurred cube obtained by the 2^nd step. (c) Vertically deblurred cube, i.e., $L_{obj}^0({{x_m},{y_m},{\theta_m}} )$, obtained from (b) by the 4^th step with the blurring horizontally and vertically removed.

Download Full Size | PDF

4.3.3 Considering vertical deblurring in a multiple depth case

In section 4.1, we assumed for Eq. (10) that the NLOS objects are present at a single depth location ${\tilde{d}_{obj}}$ (i.e., single depth case). In a more general case where multiple NLOS objects exist having different depth locations, an average value of their depths can be used as ${\tilde{d}_{obj}}$ to approximately solve Eq. (10). This assumption compromises the solution. However, in many cases, this compromise can be avoided utilizing the procedure described in section 4.2 for the following reasons: (1) The horizontal angular deblurring (2^nd step from section 4.2) is inherently free of this assumption and the blurring effect can always be accurately removed in the horizontal dimensions. (2) Prior to the vertical spatial deblurring (4^th step), we apply the horizontal angular deburring (2^nd step). This gives us an opportunity to horizontally separate the multiple objects, which converts the multiple depth case to the single depth case. For example, Fig. 9(d) shows a single bright area where the blurred signals from the multiple objects overlap. This appears to be a multiple depth case. However, after applying the horizontal angular deblurring, Fig. 9(e) shows two horizontally separated bright areas (at the point indicated by the white arrow in the figure), where Eq. (10) can then be accurately solved as a single-depth case in each separated section. If the horizontal separation is difficult to perform, we can choose a more suitable 2D slice of the cube by considering the fact that the parallax effect can improve the horizontal separation.

5. Full 3D location estimation of the NLOS objects from the light field cube

In this section, we will estimate the full 3D locations of the NLOS objects from the cube information with the horizontal and vertical blurring effects removed. Figure 11(a) illustrates the light field cube, i.e., $L_{obj}^0({{x_m},{y_m},{\theta_m}} )$, and each horizontal plane in the figure represents an ${x_m}$-${\theta _m}$ slice at a particular height ${y_m}$. Our 3D localization method consists of applying the horizontal 2D ${x_m}$-${z_m}$ localization method from section 3.3.2 to many ${x_m}$-${\theta _m}$ slices [i.e., $L_{obj}^0({{x_m},{\theta_m}} )$] at many different height ${y_m}$ locations of the cube. An example of the 2D localization result obtained at the central ${y_m}$ location is shown in Fig. 6(d). Once many 2D localization results have been obtained at many ${y_m}$ heights, a 3D spatial distribution can be constructed by vertically stacking the 2D results, as illustrated in Fig. 11(b). Note that each 2D result is placed on a tilted plane whose tilt angle corresponds to the vertical ${\phi _m}$ angle.

 

Fig. 11. Full 3D localization method using the horizontally and vertically deblurred light field cube information. (a) $L_{obj}^0({{x_m},{y_m},{\theta_m}} )$ corresponding to Fig. 10(c) expressed as a collection of ${x_m}$ - ${\theta _m}$ slices at many ${y_m}$ locations. (b) Vertically stacked 2D ${x_m}$-${z_m}$ localization results computed at many ${y_m}$ locations to construct a 3D spatial distribution.

Download Full Size | PDF

Figure 12(a) shows the 3D spatial distribution obtained by the aforementioned method as the final 3D localization result. Conditions were the same as those in section 4.3. The three isolated shapes in the figure correspond to the three hidden water-pots. The shapes locations are the estimated NLOS locations, and their sizes are related to the localization uncertainty reflecting the BRDF and the noise characteristics (see section 3.4). Figures 12(b) and (c) show the top and side views of the 3D localization results, respectively. In both figures, the estimated locations have relatively good agreements with the actual hidden locations. Recall that the raw IR camera image shown in Fig. 9(b) contains only the slowly varying distribution because of the unwanted signals and the significant scattering; however, the three NLOS objects have been clearly separated and their locations have been estimated with a relatively good accuracy by our newly proposed methods.

 

Fig. 12. Final 3D localization results of the NLOS objects. All conditions are the same as those in Fig. 9. (a) An oblique view of the 3D distribution obtained by the 3D localization method from section 5. The three distinct shapes correspond to the three water pots in the hidden scene. (b) and (c) Top and side views of the 3D distribution seen in (a).

Download Full Size | PDF

In this new 3D localization method, the horizontal (i.e., transverse ${x_m}$ and depth ${z_m}$) locations are estimated at the theoretically highest accuracy as described in section 3, whereas the vertical ${y_m}$ location is approximately estimated under the assumptions explained in section 4. Considering the fact that horizontal location is often more important than vertical location in many scenes, this enhanced horizontal information is likely to be more useful. The reduced accuracy in the vertical direction is offset by the ease and speed of gathering the data in a single angular plane, which makes it amenable to a quasi-real-time measurement.

6. Conclusion

We have developed and experimentally demonstrated a novel 3D NLOS localization procedure from a scattered LWIR light field. Although it is difficult to see the NLOS information in the initial camera images of the wall, our procedure allowes us to accurately assess the 2D or 3D positions of the hidden objects by making full use of the IR light field cube information. We note that our procedure only requires a single horizontal camera scanning for the measurement, and the major computations in the procedure are independent small deconvolutions which can be effectively accelerated by the FFT algorithm and parallelized processing. Because of these features, the locations of the NLOS objects can potentially be estimated in quasi real-time. We also note that the experimental results shown in this report were produced using a portable consumer IR camera; therefore, our method’s performance is expected to be greatly extended by using a camera having a larger lens aperture and/or a lower NETD. Further, the assumption of the prior knowledge of the BRDF may potentially be eliminated by a blind-deconvolution process, which is left for future work.

Appendix A: Removal of unwanted signal from non-uniform temperature distribution on the scattering wall (Corresponding to section 3.2)

Here, we show additional experiments to consider the non-uniform temperature distribution on the scatterer. In this experiment, we initially placed a hot water pipe close to the scattering surface to produce an artificial temperature distribution. The light field was subsequently measured immediately upon removing the hot water pipe.

Figures 13(a)∼(d) show the results obtained with a relatively constant temperature distribution across the scatterer, whereas Figs. 13(e)∼(h) show results with a large artificial temperature distribution. By comparing Figs. 13(b) and (f), we can see that the unwanted signal from the temperature distribution on the wall is significantly larger than the NLOS information-bearing signal. However, after subtracting the first-rank component of the SVD (section 3.2), Fig. 13(g) clearly reveals the NLOS information, although some shape distortion can be seen in the distribution. (Note: This distortion comes from the fact that the specular component is not necessarily orthogonal to the SVD first rank component.) Fig. 13(h) illustrates the reconstructed $L_{obj}^0$ obtained from (g), which has comparable quality to that in Fig. 13(d). Particularly, the tilt-angle of the line function in Fig. 13(h) appears to be almost the same as that in Fig. 13(d), meaning we can expect a reasonable localization result from it. We note that the current demonstration assumed the relatively slowly varying temperature distribution which allows for neglecting the keystone effect. For more challenging temperature distribution cases, such as fast varying and/or larger temperature distributions, additional efforts may be necessary for compensation.

 

Fig. 13. Experimental demonstration of the removal of the extraneous signal presented by a variation in wall temperature. Figures (a)∼(d) show results without the artificial temperature distribution on the scattering wall, whereas Figs. (e)∼(h) show results with the temperature distribution. Conditions: A single 37 deg C water pipe was placed at 2.3 m from the scatterer as the NLOS object. A Masonite board coated with a diffusive paint was used as the scatterer [whose BRDF is shown in Fig. 3(h)]. (a) and (e) Raw IR camera images of the scatterer. (b) and (f) Light fields after applying the pixel averaging. (c) and (g) Enhanced light fields ${L_{meas}}$ after subtracting the first-rank component of the SVD computed from (b) or (f). (d) and (h) Reconstructed light field $L_{obj}^0$ obtained from (c) or (g) by the horizontal angular deburring (section 3.3.1).

Download Full Size | PDF

Appendix B: Localization accuracy as a function of object’s temperature (Corresponding to section 3.4)

Figure 14 shows the localization results using a NLOS water pipe having five different temperatures between 31∼69 deg C. As shown in Figs. 14(a) and (b), the mean values of the estimated locations reasonably predict the NLOS locations. Note that the absolute locations shown in Figs. 14(a) and (b) have been calibrated to produce accurate results at 37 deg C. The calibration becomes less accurate as the temperature increases. Recall that both Eqs. (6) and (7) are inversely proportional to ${f_{\theta max}}$, indicating that both the transverse ${x_m}$ and the depth ${z_m}$ uncertainties are expected to similarly increase as ${f_{\theta max}}$ decreases. The standard deviation tendencies seen in Fig. 14(c) agree well with this theoretical prediction, since lower temperatures lead to smaller ${f_{\theta max}}$ value as shown in Fig. 14(c).

 

Fig. 14. Localization accuracy as a function of the temperature of the NLOS object. Conditions: A water pipe was located at 2.3 m from a Masonite board coated with a diffusive paint [whose BRDF is shown in Fig. 3(h)]. The pipe had five different temperatures; 31, 40, 52 and 69 deg C. The experiment was run 30 times to produce the statistical values. (a) and (b) Mean values of the estimated depth ${z_m}$ and transverse ${x_m}$ locations, respectively. (c) Standard Deviation (SD) values of the estimated locations. Numbers in the figure are ${f_{\theta \; max}}$ values [cycles/radian]. (d) ∼ (f) Localization results obtained by the methods from section 3.3 under three different object temperatures; 69, 52 and 31 deg C (note: the star indicates the actual location).

Download Full Size | PDF

Funding

Defense Advanced Research Projects Agency (DARPA), REVEAL Program, HR0011-16-C-0024.

Acknowledgement

Sasaki designed and conducted the research on the NLOS localization procedure and the analysis on the localization accuracy, performed the experiments, and drafted the paper. Hashemi developed the automated IR light field measurement setup including a pixel-tracking system. Leger provided the fundamental consept of this study, proposed use of longer wavelength light for extending localization performance, and suggested the quasi-real-time measurement strategy. All of authors read and approved the final manuscript. We would like to thank Dr. Di Lin for technical discussions and Prof. Jarvis Haupt for meaningful advice on the signal enhancement process. Portions of this work were presented at the OSA Imaging and Applied Optics Congress (COSI) in 2021 [39].

Disclosures

The authors declear no conflict of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. G. Wetzstein, I. Ihrke, D. Lanman, and W. Heidrich, “State of the art in computational plenoptic imaging,” in Eurographics (2010).

2. M. Levoy, “Light fields and computational imaging,” Computer 39(8), 46–55 (2006). [CrossRef]  

3. T. Sasaki and J. Leger, “Light-field reconstruction from scattered light using plenoptic data,” in Proc. SPIE10772, 1077203 (2018) [CrossRef]  .

4. T. Sasaki and J. Leger, “Light field reconstruction from scattered light using plenoptic data,” J. Opt. Soc. Am. A 37(4), 653–670 (2020). [CrossRef]  

5. R. Ramamoorthi and P. Hanrahan, “A signal-processing framework for inverse rendering,” in 28th Annual Conference on Computer Graphics and Interactive Techniques (2001), 117–128.

6. D. Mahajan, R. Ramamoorthi, and B. Curless, “A theory of frequency domain invariants: spherical harmonic identities for BRDF/lighting transfer and image consistency,” IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 197–213 (2008). [CrossRef]  

7. R. Ng, R. Ramamoorthi, and P. Hanrahan, “Triple product wavelet integrals for all-frequency relighting,” in SIGGRAPH, ACM Transactions on Graphics (ACM, 2004), 477–487. [CrossRef]  

8. O. Aldrian and W. A. P. Smith, “Inverse rendering of faces with a 3D morphable model,” IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1080–1093 (2013). [CrossRef]  

9. T. Haber, C. Fuchs, P. Bekaert, H. P. Seidel, M. Goesele, and H. P. A. Lensch, “Relighting objects from image collections,” in IEEE Conference on Computer Vision and Pattern Recognition (2009), 627–634.

10. S. R. Marschner and D. P. Greenberg, “Inverse lighting for photography,” in Fifth Color Imaging Conference (1997), 262–265.

11. T. Aoto, T. Sato, Y. Mukaigawa, and N. Yokoya, “Linear estimation of 4-D illumination light field from diffuse reflections,” in Asian Conference on Pattern Recognition (ACPR) (2013), 495–500.

12. T. Aoto, T. Sato, Y. Mukaigawa, and N. Yokoya, “4-D light field reconstruction by irradiance decomposition,” IPSJ T Comput Vis Appl 9(1), 13 (2017). [CrossRef]  

13. R. Wang, J. Tran, and D. Luebke, “All-frequency relighting of non-diffuse objects using separable BRDF approximation,” in Eurographics Symposium on Rendering (2004), 345–354.

14. A. V. Sambasivan, R. G. Paxman, and J. D. Haupt, “Computer Graphics meets Estimation Theory: Parameter Estimation Lower Bounds for Plenoptic Imaging Systems,” 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 1021-1025 (2019)

15. T. Sasaki and J. Leger, “Non-line-of-sight object location estimation from scattered light using plenoptic data,” J. Opt. Soc. Am. A 38(2), 211–228 (2021). [CrossRef]  

16. H.-Y. Liu, E. Jonas, L. Tian, J. Zhong, B. Recht, and L. Waller, “3D imaging in volumetric scattering media using phase-space measurements,” Opt. Express 23(11), 14461–14471 (2015). [CrossRef]  

17. N. C. Pégard, H.-Y. Liu, N. Antipa, M. Gerlock, H. Adesnik, and L. Waller, “Compressive light-field microscopy for 3D neural activity recording,” Optica 3(5), 517–524 (2016). [CrossRef]  

18. M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, “Light field microscopy,” ACM Trans. Graph. 25(3), 924–934 (2006). [CrossRef]  

19. M. Broxton, L. Grosenick, S. Yang, N. Cohen, A. Andalman, K. Deisseroth, and M. Levoy, “Wave optics theory and 3-D deconvolution for the light field microscope,” Opt. Express 21(21), 25418–25439 (2013). [CrossRef]  

20. J.-X. Chai, X. Tong, S.-C. Chan, and H.-Y. Shum, “Plenoptic sampling,” in SIGGRAPH (2000), 307–318.

21. S.-C. Chan and H.-Y. Shum, “A spectral analysis for light field rendering,” in IEEE International Conference on Image Processing (2000), Vol. 2, 25–28.

22. C. Zhang and T. Chen, “Spectral analysis for sampling image-based rendering data,” IEEE Trans. Circuits Syst. Video Technol.13, 1038–1050 (2003).

23. D. Dansereau and L. T. Bruton, “A 4-D dual-fan filter bank for depth filtering in light fields,” IEEE Trans. Signal Process. 55(2), 542–549 (2007). [CrossRef]  

24. D. Dansereau and L. Bruton, “A 4D frequency-planar IIR filter and its application to light field processing,” in IEEE International Symposium on Circuits and Systems (ISCAS) (2003), Vol. 4, pp. IV-476–IV-479.

25. C. U. S. Edussooriya, D. G. Dansereau, L. T. Bruton, and P. Agathoklis, “Five-dimensional depth-velocity filtering for enhancing moving objects in light field videos,” IEEE Trans. Signal Process. 63(8), 2151–2163 (2015). [CrossRef]  

26. T. Maeda, G. Satat, T. Swedish, L. Sinha, and R. Raskar, “Recent advances in imaging around corners,” arXiv:1910.05613 (2019).

27. D. Faccio, A. Velten, and G. Wetzstein, “Non-line-of-sight imaging,” Nat. Rev. Phys. 2(6), 318–327 (2020). [CrossRef]  

28. A. Torralba and W. T. Freeman, “Accidental pinhole and pinspeck cameras: Revealing the scene outside the picture,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition, 374-381 (2012).

29. K. L. Bouman, V. Ye, A. B. Yedidia, F. Durand, G. W. Wornell, A. Torralba, and W. T. Freeman, “Turning corners into cameras: principles and methods,” in IEEE International Conference on Computer Vision (ICV) (2017), 2289–2297.

30. S. W. Seidel, Y. Ma, J. Murray-Bruce, C. Saunders, W. T. Freeman, C. C. Yu, and V. K. Goyal, “Corner occluder computational periscopy: estimating a hidden scene from a single photograph,” in International Conference on Computational Photography (ICCP) (2019), 1–9.

31. C. Saunders, R. Bose, J. Murray-Bruce, and V. K. Goyal, “Multidepth computational periscopy with an ordinary camera,” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain (2020), 9299–9305.

32. D. Lin, C. Hashemi, and J. R. Leger, “Passive non-line-of-sight imaging using plenoptic information,” J. Opt. Soc. Am. A 37(4), 540–551 (2020). [CrossRef]  

33. M. Baradad, V. Ye, A. B. Yedidia, F. Durand, G.W. Wornell, A. Torralba, and W. T. Freeman, “Inferring light fields from shadows,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 6267–6275.

34. A. B. Yedidia, M. Baradad, C. Thrampoulidis, W. T. Freeman, and G. W. Wornell, “Using unknown occluders to recover hidden scenes,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019).

35. Di. Lin and J. Leger, “Locating non-line-of-sight features and recovering imagery conveyed through parallax in passive plenoptic imaging,” J. Opt. Soc. Am. A 38(8), 1104–1119 (2021). [CrossRef]  

36. C. Hashemi and J. Leger, “Exploiting the Visible Spectrum to Look Around Corners,” in Imaging and Applied Optics Congress, OSA Technical Digest (Optical Society of America, 2020), paper CTh5C.3.

37. T. Maeda, Y. Wang, R. Raskar, and A. Kadambi, “Thermal non-line-of-sight imaging,” Proc. IEEE Int. Conf. Comput. Photogr. 2019, 1–11 (2019).

38. M. Kaga, T. Kushida, T. Takatani, K. Tanaka, T. Funatomi, and Y. Mukaigawa, “Thermal non-line-of-sight imaging from specular and diffuse reflections,” IPSJ T Comput Vis Appl 11(1), 8 (2019). [CrossRef]  

39. T. Sasaki, C. Hashemi, and J. Leger, “Depth Estimation of Non-line-of-sight Objects from Scattered Light Enhanced by the Infrared Light Field,” in OSA Imaging and Applied Optics Congress 2021 (COSI), OSA Technical Digest, paper CW5B.5.

40. N. Lazaros, G. C. Sirakoulis, and A. Gasteratos, “Review of Stereo Vision Algorithms: From Software to Hardware,” International Journal of Optomechatronics 2(4), 435–462 (2008). [CrossRef]  

41. T. Luo, D. Lin, M. Mah, C. Hashemi, J. Leger, and J. J. Talghader, “Photography-based real-time long-wave infrared scattering estimation technique,” J. Opt. Soc. Am. A 38(7), 1041–1050 (2021). [CrossRef]  

42. R. Montes and C. Ureña, "An overview of BRDF models," University of Granda, Technical Report LSI-2012-001 (2012).

43. J. M. López-Alonso, J. Alda, and E. Bernabéu, “Principal-component characterization of noise for infrared images,” Appl. Opt. 41(2), 320–331 (2002). [CrossRef]  

44. M. Piccardi, “Background subtraction techniques: a review,” Proc. IEEE Int. Conf. Systems, Man, Cybernetics, 3099–3104 (2004).

45. P. Sharma, B. Kumar, D. Singh, and S.P. Gaba, “Critical analysis of background subtraction techniques on real GPR data,” Def. Sci. J. 67(5), 559–571 (2017). [CrossRef]  

46. A. Isaksen, L. McMillan, and S. Gortler, “Dynamically reparameterized light fields,” in Annual Conference on Computer Graphics (2000), 297–306.

47. R. Ng, “Fourier slice photography,” in SIGGRAPH’05 (2005), 735–744.

48. R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Stanford Technical Report CTSR 2005-02 (2005).