Open Access
Add to BibSonomy
Abstract
We report camera-free three-dimensional (3D) dual photography. Inspired by the linkage between fringe projection profilometry (FPP) and dual photography, we propose to implement coordinate mapping to simultaneously sense the direct component of the light transport matrix and the surface profiles of 3D objects. By exploiting Helmholtz reciprocity, dual photography and scene relighting can thus be performed on 3D images. To verify the proposed imaging method, we have developed a single-pixel imaging system based on two digital micromirror devices (DMDs). Binary cyclic S-matrix patterns and binary sinusoidal fringe patterns are loaded on each DMD for scene encoding and virtual fringe projection, respectively. Using this system, we have demonstrated viewing and relighting 3D images at user-selectable perspectives. Our work extends the conceptual scope and the imaging capability of dual photography.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Dual photography is a computational imaging technique that measures the light transport matrix (LTM) to synthesize images of scenes from the perspectives of non-imaging devices [1]. This technique is built upon the principle of Helmholtz reciprocity [2], which indicates that the directions of rays are irrelevant for measuring the transport of light energy between two points. Thus, it is possible to establish a radiometric equivalence of light transport under the interchange of light sources and detectors [3]. As an embodiment of Helmholtz reciprocity in optical imaging, dual photography uses images taken from the camera’s perspective (defined as primal images) to synthesize images as if viewed from the projector’s perspective (defined as dual images). As an imaging-based paradigm, dual photography can be performed without the knowledge of scene geometry and without calibration to model the projective behavior of the camera and projector. Because of these advantages, it has contributed to various areas of study, including computer graphics [4], lensless imaging [5], non-line-of-sight imaging [6], and ghost imaging [7]. As a universal platform, the concept of dual photography can be readily integrated with many imaging modalities, including spectroscopy [8,9], profilometry [10–14], and plenoptic imaging [15].
Among existing techniques, dual photography implemented with single-pixel imaging (SPI) has received considerable attention in recent years [1,8,16,17]. In SPI systems, spatial light modulators are used to either actively encode the illumination to a scene or passively modulate the image of the scene with time-varying patterns. Synchronized with spatial light modulators, a detector without spatial resolution measures the integrated light intensity received from the pattern-encoded scene [18–23]. SPI offers a competitive edge over conventional photography in system cost, detection sensitivity, and responsive spectrum [24,25]. Most importantly, the capability of patterned illumination enables SPI to flexibly implement various encoding schemes—including raster scanning and compressed sensing—to fast and accurately measure the LTM. Despite these advances, many aspects of SPI-based dual photography are less explored. For example, although routinely performed on three-dimensional (3D) objects, dual photography has not been performed on 3D images. In addition, scene relighting is provided merely from primal and dual perspectives. Finally, previous dual photography work has been dominated by the use of commercial projectors, which have to produce grayscale patterns via weighted temporal integration [26]. The resulting dark noise and digitization error [27] limit the performance of SPI-based dual photography systems.
2. Methods
2.1 Linkage of light transport matrix (LTM) to fringe projection profilometry (FPP)
To overcome these limitations, in this paper, we report camera-free 3D dual photography implemented in an SPI platform. Our method is inspired by observing the analogy between the functionality of the LTM and the operating principle of fringe projection profilometry (FPP) [28]. An LTM associates the illumination from each projector’s pixel to corresponding camera pixels. Each LTM’s element, determined by the properties of the scene geometry, is determined by using imaging-based methods, such as raster scanning of a single illuminating pixel while measuring the intensity of responsive camera pixels [29]. The LTM is comprised of columns that represent vectorized images of the impulse scattering response of the scene. The information recorded by the LTM can be divided into a direct component (which accounts for light paths that experience only one interaction with the scene) and a global component (which accounts for all other paths reaching the camera) [30]. Since the former considers light that reflects/scatters from a single point in the scene, the direct component of the LTM obeys epipolar geometry and can serve as the basis for building a two-dimensional (2D) pixel-to-pixel mapping between the projector and camera.
A 2D pixel-to-pixel mapping can also be recorded by structured light methods in a projector-camera system. For example, in the context of projector calibration used in FPP [31], sinusoidal fringe patterns are used to associate phase-encoded projector coordinates with camera pixels of an image of a flat calibration grid. After the measurement of phase values ${\varphi _h}({u,v} )$ and ${\varphi _v}({u,v} )$ by the projection of horizontal and vertical fringe patterns, each camera pixel $({u,\, v} )$ is linked with their corresponding projector pixel $({u^{\prime},\, v^{\prime}} )$ without the knowledge of calibration parameters of either device (Fig. 1).
Fig. 1. Coordinate settings of 3D dual photography using fringe projection profilometry (FPP)-based coordinate mapping.
These results indicate that the 2D pixel correspondence established by the FPP method can equivalently represent the direct component of the LTM. Thus, a dual image with sub-pixel precision can be synthesized via pixel-wise transformation and subsequent grid interpolation of camera pixels in the coordinate space of the projector. Moreover, hypothetical illumination conditions from perspectives of either the camera or the projector can be synthesized by adjusting transform-associated camera pixel values based on their coordinates before interpolation.
Besides elucidating the 2D pixel-to-pixel relationship between the camera and the projector, the FPP-based method also describes the projective relationship of a 3D point $({x,\, y,\, z} )$ onto the camera pixel $({u,\, v} )$ and the projector pixel $({u^{\prime},\, v^{\prime}} )$ by using the projective model of a pinhole camera [32], expressed by
2.2 System
To verify the proposed imaging method, we have developed a single-pixel three-dimensional dual photography (STDP) system consisting of an illumination module and a collection module (Fig. 2). In the illumination module, a continuous-wave laser (CNI Lasers MRL-III-671) is the light source. Following beam expansion and collimation (via lenses L1 through L4), mirrors M1 and M2 steer the beam onto a 0.45” digital micromirror device (DMD 1, Ajile Light Industries AJD-4500), from which the diffraction order with the maximum efficiency is selected by an iris placed on the Fourier plane of a 4f imaging system consisting of lenses L5 and L6. The image of DMD 1, formed at the intermediate image plane, is projected by a projection lens PL (Nikon AF-P DX NIKKOR 18-55 mm f/3.5-5.6) onto a 3D object.
Fig. 2. Schematic of the single-pixel three-dimensional dual photography (STDP) system. CL, Collection lens; DMD 1 and DMD 2, Digital micromirror device; IIP, Intermediate image plane; L1–L9, Lens; M1–M3, Mirror; PD, Photodiode; PL, Projection lens.
The light scattered from the 3D object enters the collection module. The light is collected by a collection lens CL (Opteka 85 mm f/1.8). The formed image is relayed by lenses L7 and L8 onto another 0.45” digital micromirror device (DMD 2, Ajile Light Industries AJD-4500). A mirror M3 reflects the light modulated by DMD 2 to a condenser lens L9 that focuses the light onto a photodiode (Thorlabs DET100A2). Photocurrent signals, digitized by a data acquisition card (National Instruments PCIe-6321), are transferred to a computer for image processing.
2.3 Data acquisition
Calibration of the STDP system is first carried out using single-pixel imaging to determine the matrices of ${{\boldsymbol A}_0}$ and ${{\boldsymbol A}_1}$ in Eq. (1). For the illumination module, calibration imaging is performed by displaying, on DMD 1, a sequence of masking patterns the same as the ones used in data acquisition (to be explained below), together with a fixed all-“on” pattern on DMD 2. Several single-pixel images of a planar calibration object featuring a checkerboard pattern are obtained in this way across various poses. These images, combined with knowledge of the checkerboard pattern dimensions, allows for estimating ${{\boldsymbol A}_0}$ with appropriate software [34]. The same method is applied to the calibration of the collection module by swapping the displayed patterns on DMD 1 and DMD 2. This procedure produces single-pixel images of the calibration object centered on DMD 2, from which ${{\boldsymbol A}_1}$ is estimated.
For data acquisition, DMD 1 displays complete sets of binary masking patterns generated by a cyclic S-matrix [35,36], which is known for enhancing the signal-to-noise ratio in SPI reconstruction [37,38]. Generated by the twin-prime construction [39–41], the cyclic S-matrix of size $MN \times MN$ can be obtained for each pair of twin primes M and $N = M + 2$. Denoting the elements of such a matrix by ${\boldsymbol S} = [{{s_{ji}}} ]$, where $j,i = 0, \ldots ,\; MN - 1$, the first row of this matrix is defined by
In the collection module, a total of six sinusoidal fringe patterns are displayed on DMD 2 for virtual fringe projection [42,43]. They are divided into two groups. The first three patterns have horizontal fringes with a period of ${\lambda _\textrm{h}} = 864\;\mathrm{\mu }\textrm{m}$. The second group have vertical fringes with an equal period ${\lambda _\textrm{v}} = 864\;\mathrm{\mu }\textrm{m}$. These grayscale sinusoidal fringes are converted into binary patterns for DMD 2 using an error diffusion algorithm [44] (see the inset in Fig. 2). Because each encoding pixel in the masking patterns, when imaged onto DMD 2, occupies an array of micromirrors, the high-spatial-frequency noise carried by these binary sinusoidal patterns is filtered. Therefore, the original grayscale sinusoidal fringes are virtually projected to the 3D object. As a result, a representative signal ${c_{jk}}$ measured by the photodiode during display of masking pattern j and fringe pattern k can be expressed by
Fig. 3. Imaging a low-poly sculpture of the Stanford bunny using the STDP system. (a) Signal acquired by the photodiode for a fringe image. (b) A reconstructed fringe image of the object. (c) World coordinate system containing point cloud data of the object. (d) A recovered 3D image.
Besides these fringe patterns, two additional projector patterns consisting of a single narrow stripe with either vertical or horizontal orientation are used to illuminate subsets of camera pixels, denoted by ${\textrm{P}_\textrm{v}}$ and $\mathrm{{P_h}}$. These patterns also associate the datum coordinates $u_0^{\prime}$ and $v_0^{\prime}$, chosen as the center of vertical and horizontal coordinates of the projector.
2.4. Image reconstruction
The image reconstruction starts by recovering the vectorized fringe image ${{\boldsymbol p}_{\boldsymbol k}}$ by
We characterized the spatial resolution and the depth resolution of the STDP system. For the quantification of spatial resolution, we analyzed the images of the checkerboard target used in system calibration. The line profiles of both vertical and horizontal edges of a selected square were averaged to calculate the edge spread functions (ESFs). Taking the derivative of the ESFs yielded the line spread functions (LSFs). The spatial resolutions of the STDP system, defined by the averaged full-width at half-maximum of the LSFs in the horizontal and vertical directions, were determined to be 2.98 mm. To calculate the depth resolution, we analyzed the variation in the point-cloud geometry observed for a tilted white planar target with no pattern. The standard deviation of measured depth over an area of 14 cm × 17 cm was calculated as the system’s noise level. The depth resolution, defined as twice the system’s noise level [26], was quantified to be $\textrm{2}\textrm{.59}\;\textrm{mm}$.
2.5. Dual photography and scene relighting
Because the cyclic-S-matrix-based masking patterns actively illuminate the 3D object, the reconstructed primal images take the perspective of the illumination module. Accordingly, the dual view is defined from the direction of the collection module. The STDP system allows two methods for dual photography and scene relighting. First, the obtained 2D-to-2D [i.e., $({u,v} )$-to-$({u^{\prime},v^{\prime}} )$] coordinate mapping allows generating 2D dual images with scene relighting akin to previous dual photography methods. In particular, from the primal image pixels $({u,v} )$ with intensities $I({u,v} )$ and an optional relighting function $r({u,v} )$, an intensity ${I_\mathrm{r}}({u^{\prime},v^{\prime}} )= I({u,v} )r({u,v} )$ can be defined on the points $({u^{\prime},v^{\prime}} )$, from which the 2D dual image and 2D relit dual image can be obtained via interpolation. Moreover, using additional 3D-to-2D [i.e., $({x,y,z} )$-to-$({u,v} )$ and $({x,y,z} )$-to-$({u^{\prime},v^{\prime}} )$] coordinate mappings with known calibrations, 3D dual images with scene relighting can be generated by projecting recovered point data $({x,y,z} )$ and associated intensity values $I({x,y,z} )$. For any choice of suitable matrices ${{\boldsymbol A}_\mathrm{{vp}}}$ (“vp” stands for “virtual projector”) and ${{\boldsymbol A}_\mathrm{{vc}}}$ (“vc” stands for “virtual camera”), the projective relationship, ${\rho _\mathrm{{vp}}}{[{u^{\prime},v^{\prime},1} ]^T} = {{\boldsymbol A}_\mathrm{{vp}}}{[{x,y,z,1} ]^T}\; \textrm{and}\; {\rho _\mathrm{{vc}}}{[{u,v,1} ]^T} = {{\boldsymbol A}_\mathrm{{vc}}}{[{x,y,z,1} ]^T},$ can be used to synthesize images centered on the matrix ${{\boldsymbol A}_\mathrm{{vc}}}$ with optional relighting $r({u,v} )$ from the projective matrix ${{\boldsymbol A}_\mathrm{{vp}}}$ achieved from interpolation of the scattered data, i.e., ${I_\mathrm{r}}({u^{\prime},v^{\prime}} )\; \, = \; \,I({x,y,z} )r({u,v} )$. By setting ${{\boldsymbol A}_\mathrm{{vp}}} = {{\boldsymbol A}_0}$ and ${{\boldsymbol A}_\mathrm{{vc}}} = {{\boldsymbol A}_1}$ from the experimental calibration, the 3D-to-2D coordinate mapping performs equivalent dual image synthesis as to the 2D-to-2D coordinate mapping. With the relighting choice $r({u,v} )\, = \,1$, both methods yield ${I_\mathrm{r}}({u^{\prime},v^{\prime}} )= {I_r}({u,v} )$, which is equivalent to the statement of Helmholtz reciprocity. For other choices of ${{\boldsymbol A}_\mathrm{{vc}}}$ and ${{\boldsymbol A}_\mathrm{{vp}}}$, in general, the reciprocity principle no longer assures the radiometric accuracy of output images. However, the synthesis of realistic dual imaging and scene relighting can still be achieved from user-selectable perspectives by the 3D-to-2D coordinate mapping.
3. Results
3.1 3D dual photography
To verify the performance of STDP, we imaged two 3D objects: a tilted plane with a laser hazard symbol and a low-poly sculpture of the Stanford bunny. Figure 4(a) shows the primal images. Synthetic 2D dual images using the 2D-to-2D coordinate mapping are shown in Fig. 4(b). We also recovered 3D dual images of each object. As a comparison, the results obtained by using the 3D-to-2D coordinate mapping are shown in Figs. 4(c) and (d). Figure 4(c) was produced by projecting the 3D information to 2D datasets, whose results show good resemblance to Fig. 4(b). Finally, Fig. 4(d) shows dual photography retaining 3D information.
Fig. 4. 3D dual photography. (a) Primal images. (b) 2D dual images synthesized by using the 2D coordinate mapping. (c)–(d) 2D dual images (c) and depth-encoded 3D dual images (d) synthesized by using the FPP-based 3D-to-2D coordinate mapping.
3.2 3D scene relighting at the primal and dual perspectives
To demonstrate scene relighting using STDP, we digitally illuminated the same objects in Fig. 4 with a relighting pattern consisting of bright circles in a hexagonal arrangement, from both the primal view and the dual view. Figures 5(a) and (b) show results generated by using the 2D-to-2D coordinate mapping. As a comparison, 3D dual images, produced by using the 3D-to-2D coordinate mapping, are shown in Fig. 5(c). We also verified these results experimentally by displaying the relighting pattern on DMD 2 [Fig. 5(d)]. A detailed comparison of a local feature [insets in Fig. 5(d)] demonstrates the feasibility of our method.
Fig. 5. 3D scene relighting. (a) 2D relit primal and dual images of the laser hazard symbol object and the bunny object synthesized by using the 2D-to-2D coordinate mapping. (b) 3D relit primal and dual images synthesized by using the 3D-to-2D coordinate mapping. (c) As (b), but with depth information displayed. (d) Experimental verification of scene relighting at the primal view. Comparison of a local feature (marked by the yellow dashed boxes) in (a)–(d) are shown in the zoomed views on the top left, top right, bottom left, and bottom right, respectively.
3.3 3D dual photography and scene relighting beyond the primal and dual perspectives
We further explored the imaging capability of STDP to synthesize photography from user-selectable perspectives between the primal and dual perspectives. This capability is demonstrated with a 3D object of a tilted plane with a maple leaf symbol for various choices of ${{\boldsymbol A}_\mathrm{{vc}}}$ and ${{\boldsymbol A}_\mathrm{{vp}}}$ interpolated between the primal and dual view projection matrices (Figs. 6 and 7). In particular, to generate 3D dual imaging with user-selectable perspectives, ${{\boldsymbol A}_\mathrm{{vc}}}$ was interpolated by ${{\boldsymbol A}_\mathrm{{vc}}} = ({1 - t} ){{\boldsymbol A}_0} + t{{\boldsymbol A}_1}$ for t varying from 0 to 1 (see Visualization 1). Images at five representative views (for $t = 0,\; 0.25,\; 0.50,\; 0.75,\; \textrm{and}\; 1$) are shown in Fig. 6. Synthetic images in Fig. 6(a) correspond to uniform relighting [i.e., $r({u,v} )\, = \,1$]. In Figs. 6(b) and (c), the same relighting pattern used in Section 3.2 was implemented, with ${{\boldsymbol A}_\mathrm{{vp}}}$ set equal to ${{\boldsymbol A}_0}$ and ${{\boldsymbol A}_1}$, respectively. The images corresponding to values of $t = 0$ in Fig. 6(b) and $t = 1$ in Fig. 6(c) indicate ${{\boldsymbol A}_\mathrm{{vp}}} = {{\boldsymbol A}_\mathrm{{vc}}}$, in which case the alignment of views causes the relighting pattern to appear undistorted by scene geometry. We also demonstrated synthetic relit images for which the perspectives of the relighting pattern were varied as ${{\boldsymbol A}_\mathrm{{vp}}}$=$({1 - t} ){{\boldsymbol A}_0} + t{{\boldsymbol A}_1}$ for t ranging from 0 to 1 (see Visualization 2). Figures 7(a) and 7(b) show five representative images with ${{\boldsymbol A}_\mathrm{{vc}}}$ fixed at ${{\boldsymbol A}_0}$ and ${{\boldsymbol A}_1}$, respectively. As before, it was noted that the situation ${{\boldsymbol A}_\mathrm{{vc}}} = {{\boldsymbol A}_\mathrm{{vp}}}$ produced an undistorted relighting pattern for the values of $t = 0$ and $t = 1$ in Fig. 7(a) and 7(b), respectively.
Fig. 6. 3D dual photography (a) and scene relighting (b)-(c) with user-selectable camera views. Each column labels a value of t for which ${{\boldsymbol A}_\mathrm{{vc}}} = ({1 - t} ){{\boldsymbol A}_0} + t{{\boldsymbol A}_1}$ was chosen to interpolate views between the experimental primal ($t = 0$) and dual ($t = 1$) views. In (a), $r({u,v} )\, = \,1$. In (b) and (c), $r({u,v} )$ was chosen to be a pattern featuring a hexagonal arrangement of circles, while ${{\boldsymbol A}_\mathrm{{vp}}}$ was set equal to ${{\boldsymbol A}_0}$ and ${{\boldsymbol A}_1}$, respectively.
Fig. 7. 3D scene relighting with five user-selectable perspectives of illumination. Each column labels a value of t for which ${{\boldsymbol A}_{vp}} = ({1 - t} ){{\boldsymbol A}_0} + t{{\boldsymbol A}_1}$ was chosen to interpolate relighting perspectives between the primal ($t = 0$) and dual ($t = 1$) views. The same relighting pattern used in Fig. 6(b) and (c) was applied to (a) and (b), for which ${{\boldsymbol A}_{vc}}$ was set equal to ${{\boldsymbol A}_0}$ and ${{\boldsymbol A}_1}$, respectively.
4. Discussion and conclusions
We have demonstrated camera-free 3D dual photography. FPP-based 3D-to-2D coordinate matching is used both to sense the direct component of the LTM and to associate 3D coordinates with 2D pixels of the camera and the projector. We have developed the STDP system employing two DMDs for both active illumination and virtual fringe projection on an SPI platform. The STDP system has enabled dual photography and scene relighting on 3D images. It has also extended dual photography and scene relighting to user-selectable perspectives between the primal and dual views.
The STDP system possesses several advantages. First, akin to conventional dual photography techniques, the synthesis of 2D dual images in STDP can be achieved without the knowledge of scene geometry, camera/projector parameters, and surface properties. Moreover, STDP extends the performance of dual photography on 3D images. Finally, binary patterns are displayed on both DMDs employed in the STDP system, and grayscale fringe patterns are virtually projected to the 3D objects. These features assure the accuracy in 3D surface profilometry with high pattern stability by avoiding dark noise and digitization error.
The spatial resolution of the STDP system is mainly limited by two factors. First, the number of encoding pixels in each masking pattern sets the upper bound of the spatial resolution. Nonetheless, these masking patterns must be projected with high fidelity. Thus, the number of encoding pixels is ultimately limited by the pixel count of DMD and the optical bandwidth of the illumination module. Second, the photodiode needs to resolve a varying signal across the used masking patterns. As the size of the encoding pixels is decreased, the magnitude of the difference in photodiode signals for displayed masking patterns will diminish. Thus, the signal-to-noise ratio of the photodiode practically limits the spatial resolution of the STDP system.
The future work will focus on further enhancing the imaging capability of the STDP system for new applications. The use of a laser and a photodiode with each DMD, together with Helmholtz reciprocity, makes the STDP system functionally symmetric. This structure allows flexibly implementing many widely adapted algorithms, including ones used in compressed sensing [47–50] and non-line-of-sight imaging [21,51,52], to improve the imaging speed and to measure the global components of the LTM. Meanwhile, by exploiting the advantages of photodiodes in their high sensitivities and broad responsive spectra [53,54], the STDP system may open new opportunities in few-photon imaging and infrared 3D dual photography, which could facilitate multi-spectral photorealism in computer graphics [1,8]. Furthermore, the use of multiple collection modules could also generalize our experiment while reducing 3D occlusion and enhancing resolutions. STDP is an attractive technique when side-view imaging is exclusively allowed. The possible scenarios include in-situ and undisturbed imaging of animals in their natural habitats (e.g., in cracks at a corner of a stone) [55]. It will also create new FPP-based applications for computer graphics and entertainment. All of these directions are promising research topics in the future.
Funding
Natural Sciences and Engineering Research Council of Canada (ALLRP 551076-20, ALLRP-549833-2020, CRDPJ-532304-18, RGPAS-507845-2017, RGPIN-2017-05959); Canada Foundation for Innovation (37146); Fonds de recherche du Québec – Nature et technologies (2019-NC-252960); Fonds de Recherche du Québec - Santé (267406, 280229).
Acknowledgments
The authors thank Jeremy Gribben and Alan Boate from Ajile Light Industries for their assistance in DMD equipment and software.
Disclosures
The authors declare no conflicts of interest.
References
1. P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, M. Levoy, and H. P. Lensch, “Dual photography,” ACM Trans. Graph. 24(3), 745–755 (2005). [CrossRef]
2. M. Born, E. Wolf, and A. B. Bhatia, Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light (Cambridge University, 1999).
3. E. Veach, Robust monte carlo methods for light transport simulation (Stanford University, 1998).
4. S. J. Koppal and S. G. Narasimhan, “Beyond perspective dual photography with illumination masks,” IEEE Trans. Image Process. 24(7), 1 (2015). [CrossRef]
5. A. Koppelhuber and O. Bimber, “Computational imaging, relighting and depth sensing using flexible thin-film sensors,” Opt. Express 25(3), 2694–2702 (2017). [CrossRef]
6. M. O’Toole, D. B. Lindell, and G. Wetzstein, “Confocal non-line-of-sight imaging based on the light-cone transform,” Nature 555(7696), 338–341 (2018). [CrossRef]
7. A. M. Paniagua-Diaz, I. Starshynov, N. Fayard, A. Goetschy, R. Pierrat, R. Carminati, and J. Bertolotti, “Blind ghost imaging,” Optica 6(4), 460–464 (2019). [CrossRef]
8. P. Sen and S. Darabi, “Compressive dual photography,” Computer Graphics Forum 28(2), 609–618 (2009). [CrossRef]
9. Z. Zhang, S. Liu, J. Peng, M. Yao, G. Zheng, and J. Zhong, “Simultaneous spatial, spectral, and 3D compressive imaging via efficient Fourier single-pixel measurements,” Optica 5(3), 315–319 (2018). [CrossRef]
10. I. Ralasic, M. Donlic, and D. Sersic, “Dual Imaging–Can Virtual Be Better Than Real?” IEEE Access 8(1), 40246–40260 (2020). [CrossRef]
11. N. Chiba and K. Hashimoto, “Ultra-Fast Multi-Scale Shape Estimation of Light Transport Matrix for Complex Light Reflection Objects,” in 2018 IEEE International Conference on Robotics and Automation (IEEE, 2018), pp. 6147–6152.
12. J. Wang, A. C. Sankaranarayanan, M. Gupta, and S. G. Narasimhan, “Dual structured light 3d using a 1d sensor,” in Computer Vision – ECCV 2016, (Springer, 2016), pp. 383–398.
13. M. O’Toole, J. Mather, and K. N. Kutulakos, “3d shape and indirect appearance by structured light transport,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2014), pp. 3246–3253.
14. Y.-L. Xiao, Y. Wen, S. Li, Q. Zhang, and J. Zhong, “Large-scale structured light 3D shape measurement with reverse photography,” Opt. Laser Eng. 130, 106086 (2020). [CrossRef]
15. C.-K. Liang and R. Ramamoorthi, “A light transport framework for lenslet light field cameras,” ACM Trans. Graph. 34(2), 1–19 (2015). [CrossRef]
16. P. Sen and S. Darabi, “A novel framework for imaging using compressed sensing,” in 2009 16th IEEE International Conference on Image Processing (IEEE, 2009), pp. 2133–2136.
17. P. Sen, “On the relationship between dual photography and classical ghost imaging,” https://arxiv.org/abs/1309.3007, (2013).
18. M.-J. Sun and J.-M. Zhang, “Single-pixel imaging and its application in three-dimensional reconstruction: a brief review,” Sensors 19(3), 732–745 (2019). [CrossRef]
19. J. Liang, L. Gao, C. Li, and L. V. Wang, “Spatially Fourier-encoded photoacoustic microscopy using a digital micromirror device,” Opt. Lett. 39(3), 430–433 (2014). [CrossRef]
20. B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3D Computational Imaging with Single-Pixel Detectors,” Science 340(6134), 844–847 (2013). [CrossRef]
21. G. Musarra, A. Lyons, E. Conca, Y. Altmann, F. Villa, F. Zappa, M. J. Padgett, and D. Faccio, “Non-Line-of-Sight Three-Dimensional Imaging with a Single-Pixel Camera,” Phys. Rev. Appl. 12(1), 011002 (2019). [CrossRef]
22. Y. Zhang, M. P. Edgar, B. Sun, N. Radwell, G. M. Gibson, and M. J. Padgett, “3D single-pixel video,” J. Opt. 18(3), 035203 (2016). [CrossRef]
23. E. Salvador-Balaguer, P. Latorre-Carmona, C. Chabert, F. Pla, J. Lancis, and E. Tajahuerce, “Low-cost single-pixel 3D imaging by using an LED array,” Opt. Express 26(12), 15623–15631 (2018). [CrossRef]
24. M. P. Edgar, G. M. Gibson, and M. J. Padgett, “Principles and prospects for single-pixel imaging,” Nat. Photonics 13(1), 13–20 (2019). [CrossRef]
25. G. M. Gibson, B. Sun, M. P. Edgar, D. B. Phillips, N. Hempler, G. T. Maker, G. P. Malcolm, and M. J. Padgett, “Real-time imaging of methane gas leaks using a single-pixel camera,” Opt. Express 25(4), 2998–3005 (2017). [CrossRef]
26. C. Jiang, P. Kilcullen, X. Liu, A. Boate, J. Gribben, T. Ozaki, and J. Liang, “Real-time High-speed Three-dimensional Surface Imaging using Band-Limited Illumination Profilometry with a CoaXpress Interface,” Opt. Lett. 45(4), 964–967 (2020). [CrossRef]
27. L. J. Hornbeck, “Digital light processing for high-brightness high-resolution applications,” Proc. SPIE 3013, 27–40 (1997). [CrossRef]
28. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Laser Eng. 48(2), 133–140 (2010). [CrossRef]
29. S. Seitz, Y. Matsushita, and K. Kutulakos, “A theory of inverse light transport,” in Proc. IEEE Int. Conf. Comput. Vis. (IEEE, 2005), pp. 1440–1447.
30. N. Chiba and K. Hashimoto, “3D Measurement by Estimating Homogeneous Light Transport (HLT) Matrix,” in 2017 IEEE International Conference on Mechatronics and Automation (IEEE, 2017), pp. 1763–1768.
31. Z. Zhang, “Flexible camera calibration by viewing a plane from unknown orientations,” in Proceedings of the Seventh IEEE International Conference on Computer Vision (IEEE, 1999), pp. 666–673.
32. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. 2nd Edition ed. (Cambridge University, 2004).
33. S. Zhang, D. Royer, and S.-T. Yau, “GPU-assisted high-resolution, real-time 3-D shape measurement,” Opt. Express 14(20), 9120–9129 (2006). [CrossRef]
34. J.-Y. Bouguet, “Camera Calibration Toolbox for Matlab,” http://www.vision.caltech.edu/bouguetj/calib_doc/.
35. M. Harwit and N. J. A. Sloane, Hadamard transform optics (Academic, 1979).
36. S. W. Golomb, “Cyclic Hadamard Difference Sets — Constructions and Applications,” in Sequences and their Applications, C. Ding, T. Helleseth, and H. Niederreiter, eds. (Springer-Verlag, 1999), pp. 39–48.
37. N. Gopalsami, S. Liao, T. W. Elmer, E. R. Koehl, A. Heifetz, A. C. Raptis, L. Spinoulas, and A. K. Katsaggelos, “Passive millimeter-wave imaging with compressive sensing,” Opt. Eng. 51(9), 091614 (2012). [CrossRef]
38. EŞ Ermeydan and İ Çankaya, “Compressed sensing with cyclic-S Hadamard matrix for terahertz imaging applications,” AIP Conf. Proc. 1926, 020040 (2018). [CrossRef]
39. R. G. Stanton and D. A. Sprott, “A Family of Difference Sets,” Can. J. Math. 10, 73–77 (1958). [CrossRef]
40. N. J. A. Sloane and M. Harwit, “Masks for Hadamard transform optics, and weighing designs,” Appl. Opt. 15(1), 107–114 (1976). [CrossRef]
41. N. J. A. Sloane, T. Fine, P. G. Phillips, and M. Harwit, “Codes for Multislit Spectrometry,” Appl. Opt. 8(10), 2103–2106 (1969). [CrossRef]
42. Z. Zhang and J. Zhong, “Three-dimensional single-pixel imaging with far fewer measurements than effective image pixels,” Opt. Lett. 41(11), 2497–2500 (2016). [CrossRef]
43. J. Liang, R. N. Kohn, M. F. Becker, and D. J. Heinzen, “Homogeneous one-dimensional optical lattice generation using a digital micromirror device-based high-precision beam shaper,” J Micro. Nanolithogr. MEMS MOEMS 11(2), 023002 (2012). [CrossRef]
44. J. Liang, S.-Y. Wu, R. N. Kohn, M. F. Becker, and D. J. Heinzen, “Grayscale laser image formation using a programmable binary mask,” Opt. Eng. 51(10), 108201 (2012). [CrossRef]
45. D. C. Ghiglia and L. A. Romero, “Robust two-dimensional weighted and unweighted phase unwrapping that uses fast transforms and iterative methods,” J. Opt. Soc. Am. A 11(1), 107–117 (1994). [CrossRef]
46. D. C. Ghiglia and M. D. Pritt, Two-Dimensional Phase Unwrapping: Theory, Algorithms, and Software. (Wiley-Interscience, 1998).
47. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). [CrossRef]
48. D. Takhar, J. N. Laska, M. B. Wakin, M. F. Duarte, D. Baron, S. Sarvotham, K. F. Kelly, and R. G. Baraniuk, “A new compressive imaging camera architecture using optical-domain compression,” Proc. SPIE 6065, 606509 (2006). [CrossRef]
49. M.-J. Sun, M. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nat. Commun. 7(1), 12010 (2016). [CrossRef]
50. J. Liang, “Punching holes in light: Recent progress in single-shot coded-aperture optical imaging,” Rep. Prog. Phys., in press (2020).
51. D. Faccio, A. Velten, and G. Wetzstein, “Non-line-of-sight imaging,” Nat. Rev. Phys. 2(6), 318–327 (2020). [CrossRef]
52. X. Liu, S. Bauer, and A. Velten, “Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems,” Nat. Commun. 11(1), 1645 (2020). [CrossRef]
53. A. Kirmani, D. Venkatraman, D. Shin, A. Colaço, F. N. Wong, J. H. Shapiro, and V. K. Goyal, “First-photon imaging,” Science 343(6166), 58–61 (2014). [CrossRef]
54. X. Peng, X. Y. Zhao, L. J. Li, and M. J. Sun, “First-photon imaging via a hybrid penalty,” Photonics Res. 8(3), 325–330 (2020). [CrossRef]
55. J. S. Jaffe, K. D. Moore, J. McLean, and M. P. Strand, “Underwater optical imaging: status and prospects,” Oceanography 14(3), 64–75 (2001). [CrossRef]
