1. Introduction

Resolving a scene’s texture and depth is one of the classical problems in imaging. Perhaps, its most challenging incarnation occurs in the context of a lensless camera, where the traditional lens in front of a sensor is replaced with a modulating element—an amplitude [1,2], phase [3,4] or diffractive [5] mask. A scene point casts a “shadow” or point spread function (PSF) on the sensor that encodes its angular location as well as depth, and the sensor observes a linear sum of such PSFs arising from all scene points. Numerous prior work in this space [2,3,6–9] have successfully demonstrated 3D reconstructions by solving an inverse problem.

From a theoretical perspective, 3D imaging with lensless cameras is poorly understood. While prior works have produced ample empirical observations for specific prototypes, they are difficult to generalize and interpret. In particular, the measurement operators associated with lensless imagers are hard to analyze for scenes with extended depth. As a consequence, there has been little work in analyzing the fundamental limits in achievable spatial and axial resolutions with a lensless camera; nor is there a clear understanding of how the various parameters of the imager—including the mask pattern and the scene/sensor to mask distances—affect its ability to recover texture and depth.

One of our primary observation is that the analysis of performance of lensless cameras is complicated by the dimensionality gap between the scene, which is three-dimensional (3D), and the lensless measurements, which are invariably two-dimensional (2D) or, at best, a collection of such 2D images. To alleviate this gap, we draw inspiration from z-stacking or focus stacking and consider, as a thought experiment, the 3D space of measurements formed by moving the image sensor axially, i.e., by changing the mask to sensor distance. We show that, under assumptions that are commonplace in prior work, the z-stacked measurements for an amplitude mask-based lensless camera are the result of a 3D convolution of the scene, represented as a volumetric albedo function, with a 3D kernel that is dependent of the mask. This convolutional structure of the measurement operator is immensely consequential, since it provides the foundation for characterizing the lensless camera’s spatial and axial resolving power by simply computing the modulation transfer function (MTF) of the associated PSF. This result further enables us to compare and evaluate various masks and parameters of the lensless camera, thereby answering the previously elusive questions. Finally, since z-stacked measurements encompass those made by a traditional system with fixed sensor-to-mask distance, our results provide an upper bound on their performance.

Contributions. We make the following contributions.

The analysis in this paper is meaningful for typical lensless cameras with static sensors. Since the static sensor measurement is a subset of the z-stack measurement, the 3D resolution analysis derived in this paper serves as an upper bound to the performance of a typical static sensor lensless cameras.

Limitations. From a theoretical perspective, the derivation of the main results require a number of assumptions on the scene and the image formation that, in principle, reduce their applicability. These include the use of a volumetric model that ignores occlusion, shading, and specularities and the use of a ray tracing approach that ignores diffraction caused by the small features in the mask. Volumetric modeling is a commonly-made assumption in this literature (for example, see FlatCam [1], FlatScope [2] or SweepCam [7]); hence, these results characterize their performance with the caveat that there is an additional mismatch between the actual measurements and the assumed measurement model. Lastly, the MTF analysis only characterizes the effectiveness of the measurement operator (or the imaging system), but does not consider the use of sophisticated computational techniques for solving the inverse problem; here, the use of scene priors can potentially offer reconstruction that is better than what our analysis predicts.

2. Related work

The theory and analysis proposed here builds upon two largely independent topics: lensless imaging, and focus stack photography.

2.1 Lensless measurement operators

For a lensless camera with an amplitude mask [1,10–12], its measurement operator is convolutional when all scene points are restricted to lie on a single front-parallel plane. Hence, for a scene not restricted to a single depth, the measurement operator can be described as a sum of 2D convolutions; see Fig. 1(a). To stabilize the reconstruction process, previous work use priors in the form of sparsity [2] and data-driven models [13–15].

 

Fig. 1. Structure of 3D lensless imaging operators for a scene with two depth planes with albedo $t_{z_1}, t_{z_2}$ and lensless measurements $i$. (a) Single measurement; (b) multiple measurements; (c) Hua et al. [7] computationally focus measurements on specific depths to obtain systems that approximate single 2D convolution with low-frequency residual. (d) Zheng et al. [8] represents the Fourier transform of multiple measurement matrix as a block-diagonal matrix. (e) Z-stack measurements, after re-parameterization, can be obtained by a 3D convolution between the 3D scene volume and a 3D kernel defined by the mask pattern.

Download Full Size | PDF

One approach to improve the conditioning of the operator is to capture multiple measurements with different mask patterns. The measurements corresponding to the different masks can simply be concatenated, as in Fig. 1(b), to obtain a joint system with improved conditioning and invertibility. However, the computational burden in implementing this operator can be quite formidable, especially for high-resolution sensors. Hua et al. [7] capture multiple measurements with a translating mask to facilitate computational refocusing to different depths in the measurement space; the resulting operation approximates imaging points only from focused depth with the rest in severe defocus, resulting in the measurement operator shown in Fig. 1(c). Ignoring boundary effects of the convolution, [8] formulate the measurement operator in the frequency domain of the measurement; here, the sum of convolution operator reduces to a block diagonal structure, as seen in Fig. 1(d), which can be implemented very efficiently.

In this paper, we show that the z-stacked measurements, under an appropriate re-parameterization, are convolutional in the scene’s volumetric albedo. Antipa et al. [3] make a similar observation, implementing the sum of 2D convolution as a 3D convolution. However, there are notable differences including their use of a phase mask, which has a different image formation from ours. Further, our use of z-stacked measurements introduces important re-parameterizations of the scene and measurements that are critical to the derivation of the convolutional model. Finally, we detail a number of important consequences of the convolution model, which goes significantly beyond prior work.

2.2 Focus stack photography

Focus stack photography acquires multiple images of a scene by sweeping the focus plane of the imaging system [16,17]; typically, this is achieved via axial movement of the sensor with respect to the imaging lens or by using focus-tunable optics [18]. Focus stacks have been studied for 3D scene estimation, using focus [19] and defocus [20] cues, as well as obtaining extended depth-of-field images [21]—the latter being useful in microscopy [22] where the extremely shallow depth of field is often a challenge.

Sundaram and Nayar [23] study recoverability of depth of a textureless scene from a focus stack, and present a theoretical analysis of MTFs that is similar to this work. Specifically, they show that the volumetric measurement formed by a scene point in a tele-centric system is shift-invariant. They derive the optical transfer function associated with this system and establish conditions when the depth of a textureless scene is resolvable. The theory presented in this paper can be interpreted as the lensless counterpart to their work; in spite of the conceptual similarity between the two results, the differences in image formation between a lens-based and lens-less imager lead to distinct results and consequences.

3. Z-stacking and the convolutional model

We begin with a recap of the image formation model for imaging a 3D scene with a lensless camera, followed by derivation for obtaining the z-stacked measurements with a 3D convolutional model, and finally address its implication with static sensor prototypes.

3.1 Measurement model for a 3D scene

Consider an image sensor placed behind an amplitude mask defined with an attenuation function $m({\bf x})$ with ${\bf x} = (x, y)$; we assume that the mask is aligned with the $z=0$ plane (see Fig. 2). Following prior work [7], we model the scene as a volumetric albedo function $t({\bf x}, z)$. When the sensor is placed on the plane $z=d < 0$ (i.e., a distance $d$ behind the mask), the intensity observed at a point ${\bf p} = (p, q)$ on the sensor is given as

 

Fig. 2. Z-stacked lensless measurements. The sensor is translated axially, along $z$, to different sensor-to-mask distances $d$ to obtain a z-stack. Figure adapted from [7].

Download Full Size | PDF

3.2 3D convolution model for a z-stack

We now consider a lensless camera where the sensor captures multiple measurements while moving axially (i.e., with varying sensor-to-mask distances). An important distinction is that these z-stacked measurements are volumetric; specifically, we can extend Eq. (1) to write the measurements, $i({\bf p}, d)$, to be explicitly dependent on the spatial locations ${\bf p}$ as well as axial location $d$.

Change of variables. We now show that the z-stack $i({\bf p}, d)$ is related to a re-parameterized volumetric scene albedo $t({\bf x}, z)$ via a convolution operator. Starting from Eq. (1), we can rewrite $i({\bf p}, d)$ by rearranging the terms in $m(\cdot )$ as follows:

The equation above is significantly simplified if we change the variables from Cartesian coordinates to the following choice:

This new parameterization changes $(x, y)$ to the tangent of the angle at the origin, and depth $z$ to its reciprocal. Supplement 1 contains an illustration of this parameterization. Finally, we use ${\widetilde {i}}({\widetilde {\bf p}}, {\widetilde {d}})$ and ${\widetilde {t}}({\widetilde {\bf x}}, {\widetilde {z}})$ to denote the z-stack measurements and the scene albedo, respectively, in their new variables. With this, we can rewrite Eq. (2) as

Here, the $1/{\widetilde {z}}^4$ is the modulus of determinant of the Jacobian underlying the change of variables. Defining the 3D kernel $k({\widetilde {\bf x}}, {\widetilde {z}})$ and depth-normalized texture ${\widetilde {t}}'({\widetilde {\bf x}}, {\widetilde {z}})$ as

To proceed further and obtain the desired convolutional model, we need to make an additional assumption pertaining to the limits of integration. Specifically, the integral in Eq. (6) is physically meaningful—i.e., consistent with what a sensor would measure—only when the sensor and the scene are on opposite sides of the mask. After all, the mask plays no role when the sensor and scene are on its same side. We can implicitly enforce the sensor-mask-scene configuration with the following two conditions: first, the scene albedo ${\widetilde {t}}'({\widetilde {\bf x}}, {\widetilde {z}}) = 0$ when ${\widetilde {z}} \le 0$; and second, the output ${\widetilde {i}}({\widetilde {\bf p}}, {\widetilde {d}})$ is evaluated only for ${\widetilde {d}} < 0$. We can now change the limits of integration to get

Hence, the z-stack of measurements is related to the scene’s volumetric albedo, both under their respective re-parameterizations, via a 3D convolution operator whose kernel is dependent only on the mask pattern.

The convolutional model presented in Eq. (7) is the centerpiece of our contributions. As we show in the next section, it can be leveraged to characterize basic properties of the imager, including the modulation transfer function (MTF) that is defined as the modulus of the Fourier transform of the convolutional kernel. Before we delve into this analysis, a few observations are in order.

Connection to non-line-of-sight imaging. The interested reader is referred to a recent work in non-line-of-sight imaging that formed the inspiration for the derivation above. The measurement models in non-line-of-sight imaging are similar to lensless imaging in their lack of structures that facilitate fast implementation or analysis. O’Toole et al. [24] show that an appropriate re-parameterization of the variables describing the scene and measurements lead to a convolution operator. A notable difference is that non-line-of-sight operator is 5D and [24] select a 3D subset to match the dimensionality of the scene and measurements; in contrast, our work matches the dimensionality by enhancing the space of the measurements by z-stacking.

4. Analysis

We now derive an expression for the Fourier transform of the PSF, and use it to characterize the lateral and axial resolution of the device in terms of its mask.

4.1 Derivation of the MTF

A key feature of an imaging system whose measurement operator is convolutional is that we can easily characterize the fundamental limits of achievable resolution. This is often done by computing the MTF of the system, which is the magnitude of the Fourier transform of the PSF. Note the resolution is derived for the scene with parameterization as described in Section 3.2, but the resolution limits can be mapped back to the original space. We show an example in Section 4.2.0.1.1.

In our case, the PSF is a 3D function $k({\widetilde {\bf x}}, {\widetilde {z}})$ that is defined in Eq. (5). Let $K(f_x, f_y, f_z)$ be the 3D Fourier transform of 3D kernel $k({\widetilde {x}}, {\widetilde {y}}, {\widetilde {z}})$.

Plugging in definition of $k(\cdot )$ from Eq. (5), we can simplify $K(\cdot )$ as

Here, $r_\ell (\cdot, \cdot )$ is the Radon transform of $\ell (x, y)$, the Laplacian of the mask, defined as

A detailed derivation is provided in Supplement 1.

Subsequent analysis is simplified if we define $\rho = \sqrt {f_x^2+ f_y^2}$ as the magnitude of angular frequency and $\psi = \tan ^{-1}(f_y/f_x)$ as its directionality. The Fourier transform of the 3D PSF in polar coordinates for $(f_x, f_y)$, denoted as $K^P$, can now be written as

Numerical verification. We observe from Eq. (10) that if $\rho$ is fixed to some value, say $\rho = \rho _0$, then

Hence, the 2D slice of $K^P$ for a fixed $\rho$ is a scaled copy of $r_\ell$, the Radon transform of the Laplacian of the mask pattern. Figure 3 provides a numerical verification of this; for a collection of masks commonly used in lensless imaging, we compute the Laplacian of the mask, its Radon transform, and a $\psi$-$f_z$ slice of the magnitude spectra $|K^P(\rho, \psi, f_z)|$, for a fixed value of $\rho$. We observe that the Laplacian matches well with the 2D slice of the magnitude spectra with two notable sets of artifacts: aliasing of the DFT which results in repeating copies, and the sinc-like decay, especially along the $f_z$ axis, which we attribute to the windowing of the kernel; we elaborate on this in the next section.

 

Fig. 3. Comparison of mask patterns and their MTFs. Multiple mask patterns (a, inset) are shown with their Laplacian (a) and the Radon transform of Laplacian (b). (c) shows $|K^P(\rho =0.4, \psi, f_z)|$, a $\psi$-$f_z$ slice, obtained numerically; the similarity between row (b) and (c) verifies our analytical expression of $K$. (d) shows $\log \int _{\psi }|K^P(\rho, \psi, f_z)|d\psi$ obtained numerically. The red lines marked in (d) correspond to the butterfly structure defined in Eq. (12); this is a consequence of the compact spread of the mask, indicated by the red dotted circles in (a). The structures of the MTF in (d) are constrained within the butterfly structure, except for the leakage due to the windowing of the PSF in the spatial domain.

Download Full Size | PDF

4.2 Dependence of the MTF on the mask

The expressions for the Fourier transform of the PSF also provide a direct way to understand how the mask affects the MTF of the resulting system. We study this next.

Spatial extent of the mask. Many of the masks that we use have a finite spatial extent. The MTFs of such masks exhibit an important symmetry. Specifically, suppose that the mask $m(x, y)$ is zero outside of a disc of radius $R_m$, i.e., $m(x, y) = 0, \ \forall (x, y) \textrm { s.t. } x^2 + y^2 \ge R_m^2$, then the Radon transform of Laplacian satisfies

This implies that the Fourier transform can be non-zero only when $|f_z/\rho | \le R_m$, or

Hence, if we visualize a different 2D cross-section of $K^P$, one corresponding to a fixed $\psi$, then we expect to see non-zero values only with in the “butterfly” shape defined by the set defined above. Figure 3(d) visualizes this for a number of masks.

The butterfly structure places an important constraint on achievable spatio-axial resolution. Similar structures also arise in the analysis of spatio-temporal resolutions in videos [25]; as in our analysis, such structures serve to strongly couple achievable resolutions across the two domains. Given measurements that resolve in tangent of angle subtended with a resolution of $\delta _p$ (the ratio of sensor pixel pitch to sensor-to-mask distance) , we can only resolve frequencies corresponding to $\rho \in [-\frac {1}{2\delta _p}, \frac {1}{2\delta _p} ]$. Hence, for a mask with support restricted within a disc of radius $R_m$, the maximum resolvable axial resolution is

This naturally explains the lack of depth resolution in a pinhole mask and the improvement in performance with multiple pinholes, as well as larger-sized masks based on M-sequences or random constructions, since they have a larger $R_m$ than pinholes. It is worth noting that the bound discussed above is expected to be loose, since it only considers the diameter of the mask and ignores the specific pattern within.

Example. This analysis allows us to compute the upper bound of 3D resolution for lensless camera prototypes. For example, in the FlatScope prototype [2], the spatial extent of mask is contained within a disc of radius $R_m={1.84}\;\textrm{mm}$. The spatial resolution is bounded by $\delta _p = \frac {\Delta p}{d} = {1.12E-2}$, where $\Delta p = {2.24}\;\mathrm{\mu}\textrm{m}$ is the effective pixel pitch and $d={0.2}\;\textrm{mm}$ is the mask-to-sensor distance. FlatScope reports lateral resolution of less than ${2}\;\mathrm{\mu}\textrm{m}$. After converting the sensor angular resolution ($\delta _p$) to the scene spatial resolution ($\delta _p \times z$), our analysis predicts the ${2}\;\mathrm{\mu}\textrm{m}$ spatial resolution holds true for scenes that are farther than $z = {178}\;\mathrm{\mu}\textrm{m}$ from the mask. This is the depth range in which FlatScope have shown experimental results ($z > {170}\;\mathrm{\mu}\textrm{m}$). The axial resolution of the prototype has an upper bound from Eq. (13): $f_{zmax} = \frac {R_m}{2 \delta _p} = {82.1}\textrm{mm}$ in frequency of diopters. FlatScope shows an axial resolution of less than $ {15}\;\mathrm{\mu}\textrm{m}$ at depth $z= {270}\;\mathrm{\mu}\textrm{m}$ from experiments. Our analysis predicts that diopter resolution is bounded by $\Delta \tilde z = \frac {1}{2 f_{zmax}} = \frac {\delta _p}{R_m}$. The resulting depth resolution at $z={270}\;\mathrm{\mu}\textrm{m}$ can be computed as $\Delta z = \vert \frac {\partial z}{ \partial {\widetilde {z}}} \Delta {\widetilde {z}} \vert = \frac {\Delta {\widetilde {z}}}{{\widetilde {z}}^2} = z^2 \frac {\delta _p}{R_m} = {0.44}\;\mathrm{\mu}\textrm{m}$. The predicted theoretical upper bound of axial resolution is better than the reported empirical axial resolution, as it does not account of the limitation further imposed by the subsampling operator discussed in Section 4.4 or sensor non-idealities. This suggest further improvements to the axial resolution may be possible with obtaining z-stack and better handling of measurement noise and quantization.

A counter-intuitive consequence of the butterfly structure is that the depth resolving power of a pinhole, modeled as a disc, seems to change when we increase its radius. We discuss this next.

Pitch of the mask. Given a mask pattern—for example, a pinhole—changing its pitch scales the mask pattern, and consequently, the Radon transform of its Laplacian only along its shift axis. This suggests that we can get better depth resolution simply by scaling a mask pattern. Intuitively, this makes sense as the defocus blur is enlarged and hence, given a sensor pitch, we can distinguish smaller changes in depth better.

Sparsity of the Laplacian. Another significant factor that detrimentally affects performance of reconstruction is the presence of nulls in the MTF. We observe such nulls in the simple masks consisting of one or a few pinholes; the Laplacian of such masks have positive and negative intensities which cancel out during the line-integrals, and so their Radon transform has sparse structures as well (as seen in Fig. 3(c)). As a consequence, while increasing the size of the pinhole enlarges the butterfly structure, the sparsity of Radon transform results in the same amount of frequency measurements, albeit those that can reach higher axial resolutions.

DC term and the effect of windowing. The expressions in Eq. (9) and Eq. (10) also indicate that the Fourier transform tends to infinity when we approach the origin (i.e., the DC term). This is not surprising since the DC term is equal to the integral area under the curve (AUC) of the PSF. Since the PSF $k({\widetilde {\bf x}}, {\widetilde {z}})$ is simply a projection of the mask $m$ along the depth dimension, its AUC is infinite. Practical considerations, however, force us to work with a windowed kernel when we do the simulations of Fig. 3. In particular, when we seek to recover a scene and sensor measurements that both have finite extents in tangent of the angle and reciprocal of the depth, then the effective kernel that we see is a windowed version of the theoretical kernel.

Windowing has two distinct effects on $K$. The term at the origin is now finite; $K$ approaches the AUC of the windowed PSF as we approach the origin. Next, windowing in spatial domain results in convolution with a sinc function in the frequency domain; this results in the vertical and horizontal spread of the MTF, especially in regions outside the butterfly structure in Fig. 3(d).

Effect of axial sampling density and range. When the z-stack is uniformly sampled within some range in the diopter space, the effect of sampling density follows the traditional trade-offs characterized by the Nyquist–Shannon sampling theorem [26]; specifically, to avoid aliasing, this model would only be able to handle scenes which are band-limited along depth as determined by the sampling rate. The effect of the sampling range is to be interpreted as windowing of the ideal sampling of the z-stack, which has been discussed in the previous paragraph.

4.3 Lateral and axial resolution

We are interested in expressions for the MTF as functions of $\rho$ and $f_z$, respectively, corresponding to characterizing the lateral and axial resolution. To get such expressions, we start with the modulus of $K^P(\rho, \psi, f_z)$ and integrate/marginalize the variables that we want to exclude. Such a marginalization assumes that all frequency components are equally important, and can be interpreted as an average over an ensemble of 2D signals, without any priors.

Lateral resolution. For lateral resolution, we are interested in characterizing the MTF, purely as a function of $\rho$, which we can obtain by summing over the modulus of $K$ over $f_z$ and $\psi$.

This suggests that the lateral resolution for different masks is similar, except for a constant. Further, it also suggest that the tail of $\textrm {MTF}(\rho )$ decays as $1/\rho ^2$; therefore, visualizing $\textrm {MTF}(\rho )$ in a log-log plot should produce linear profiles with a slope of -2. We verify this in Fig. 4 for the masks shown earlier in Fig. 3.

Axial resolution. To obtain MTF as a function of $f_z$, we can marginalize the modulus of $K(f_x, f_y, f_z)$ over $f_x$ and $f_y$, or equivalently, $\rho$ and $\psi$.

Suppose we define $h(f_z)$ as

This suggests that the axial MTF has a tail decay of $1/f_z$, or linear with a slope of $-1$ in a log-log plot. We confirm this using simulations in Fig. 4.

 

Fig. 4. Lateral and axial MTF of different masks. The lateral and axial MTF of different masks are numerically obtained via FFT of the 3D kernel. The slopes of the lines agree with that of $\rho ^{-2}$ and $f_z^{-1}$ respectively, as shown in black dotted lines. This validates the analytical expression in Eq. (14) and Eq. (16).

Download Full Size | PDF

Remarks. The expressions on the slices of the MTF in Eq. (14) and Eq. (16) are obtained by integrating out the variables that are not of immediate interest. Implicitly, this assumes that all frequencies are equally important; in reality, natural scenes have their energy concentrating on low-frequencies and hence, the plots have to be interpreted with this distinction in mind. This also explains why the open aperture appears at an higher value in the $\textrm {MTF}(\rho )$ plot; Fig. 3(d) shows that the open aperture samples higher depth frequencies compared to the pinhole so de-emphasizing the high frequencies – to match their distribution in natural scenes – will likely result in an aperture MTF curve that is worse off as compared to the pinhole.

4.4 Reduction to the static sensor scenario

The analysis of the MTF of the z-stacked measurements have immediate consequences for traditional lensless cameras, such as FlatCam [1], FlatScope [2], and SweepCam [7], that rely on a single (or multiple) image measurements without sensor movement; after all, we can simply choose to retain a single image in the z-stack, which would correspond to a static sensor scenario. The measurement operator associated with a static sensor can be written as a sub-sampling operator applied to that of the z-stacked measurements; specifically, if we denote the measurement operators associated with the z-stacked sensor and the static sensor as $\mathcal {A}_{\textrm {zcam}}$ and $\mathcal {A}_{\textrm {static}}$, respectively, then they are related as follows:

Remarks. The predictions we make are in the reparameterized space, where the spatial coordinates are represented as tangent of angles subtended and the axial coordinates are in reciprocal of depth. These transformed coordinates have natural intepretability in the context of imaging. In traditional lens-based camera, spatial coordinates on the sensor are linear in the tangent of the angle subtended by scene points. Similarly, the size of the defocus blur, with or without a lens, is linear in the reciprocal of depth. Thus, an analysis in the transformed space is meaningful once we evaluated it through this perspective.

5. Simulation results

Setup. Since the discussion above was heavily based on a volumetric representation of the scene, we aim to test the proposed method in realistic situations that involve light fall-off, occlusion, foreshortening, and sensor angular response; such effects can be significant for scenes very close to the mask. Specifically, we render the simulated measurements by ray-tracing a mesh-based scene so that it captures the above mentioned effects. However, the renderer does not model diffraction and other wave effects. Details of the renderer and its comparison to the volumetric modeling can be found in Supplement 1. Additionally, we add Gaussian noise based on the dynamic range of a typical machine vision sensor (71.95 dB). For z-stacking, we translate a 13.13mm$\times$8.75mm sensor from 5mm to 10mm to the mask, in 64 steps linear in $\frac {1}{d}$, and render a measurement at each location. The dimension of our measurements is given by $114\times 171\times 64$. The dimension of reconstructed volume is the same. Each of the reconstructions were finished within 5 minutes. For single measurement results, we use the measurement captured at the furthest location in the z-stack.

Performance of different masks. We experimentally validate our resolution analysis of mask patterns in Fig. 5. We image a scene with different mask patterns under the proposed z-stacking. The scene contains five points of diameter $ {200}\;\mathrm{\mu}\textrm{m}$ evenly spaced on a line between point (-2mm, 2mm, 5mm) and (4mm, -4mm, 10mm). Figure 5 visualizes the 3D kernels for a set of masks, and reconstructions using regularized least squares ($\ell _2$) and a sparse $\ell _1$ prior (with fista [27] ). The results reflect the observations made in Sec. 4.3 about the spatial and axial resolution of the masks. As expected, pinhole has almost no depth resolution, and the result reflects this by producing long streaks instead of points. Stereo masks have better depth resolution, and results in shorter streaks in the $\ell _1$ reconstructions. Note the stereo mask $\ell _2$ regularization results show long streaks; this suggests typical stereo systems observe depth with strong dependence on sparsity of the scene. The longer M-sequence mask has the best depth resolution, and results in sparse points. It is also instructive to note that using a sparse prior always produces isolated points in the volume; however, for the masks with poor depth resolution, the location of those points are incorrectly reconstructed. This highlights the importance of theoretical analysis, as the use of strong priors makes it difficult to analyze the performance of lensless cameras.

 

Fig. 5. Different masks and their reconstructions. The 3D kernel and reconstructed volumes are all plotted in re-parameterized space as given in 3.2. The scene consists of five points aligned on a diagonal line; the ground truth location of points are overlayed with green asterisks in the reconstructions. Mask pattern with larger spread have higher depth resolution – mseq-2,8 reconstruct points as points instead of line streaks in the other masks under $\ell _2$ regularization. While a sparse prior ($\ell _1$) results in sparse points in the reconstruction, they can be located at the wrong depth for masks with poor depth resolution (pinhole); thus it is essential to characterize the cameras’ resolution theoretically in addition to empirical observations.

Download Full Size | PDF

6. Discussions

This paper provides a theoretical characterization of the performance of lensless images, in terms of spatio-axial resolution, and the central role played by the mask. Our primary result relies on the construction of a measurement operator that is convolutional; this involves two steps: using z-stacked measurements to obtain a 3D space of measurements, and a reparameterization of the space. This construction connects the MTF of the system to a simple transformation of the mask. More importantly, it makes a concrete set of predictions on the achievable spatial and axial resolutions: a butterfly structure that limits the depth resolution based on sensor pitch and the mask extend, and tail decays on the marginal MTFs over spatial and axial frequencies. Finally, we verify these predictions on a set of commonly-used masks. We envision the impact of this work and its relevance to the imaging community to be two-fold. First, it analyzes spatial and axial resolutions for prior art in lensless imaging [2,7–9,28]. Second, it provides a pathway for the design of lensless cameras, in mask design and in acquiring z-stacked measurements, which we detail in the following paragraphs.

Design of masks. The theory developed in this work, especially Eq. (9) and Eq. (10), provide a crisp connection between the mask used and the MTF of the resulting system; specifically, that the MTF is a resampling of the Radon transform of the Laplacian of the mask used. Radon transforms are invertible, and basic Fourier analysis suggests that the Laplacian of the mask specifies the mask completely, except for the DC offset and slope. This suggests that mask design can be formulated as an optimization problem in $r_\ell$ under some desired cost function.

Acquiring z-stacked measurements. The analysis in this paper also raises the intriguing possibility of building lensless cameras with axial sensor motion, so as to provide a richer space of measurements. Perhaps, the most significant among these stems from the challenges in acquiring z-stack images. Axial motion of a sensor invariably results in lateral motion as well, which needs to be accounted for, via careful calibration. Axial motion can potentially be in conflict with the primary motivation of lensless cameras, namely the need for imaging with a compact footprint. Yet, such mechanisms for translation are routinely used in cellphone lenses for autofocusing, and so there is the possibility that such a feature can be implemented for the sensor as well. Finally, the resampling of sensor measurements required to enable the convolutional model is non-uniform and, hence, results in a loss in sensor resolution.