1. Introduction

Light field microscopy, first presented by Levoy et. al in 2006 and 2009 [1, 2] and further improved upon by Broxton et. al in 2013 [3], is a method for single-snapshot volumetric imaging that employs a microlens array in the optical path of a fluorescence microscope. Unlike the widefield microscope, which records the spatial distribution of light collected by the microscope objective, the light field microscope (LFM) records both the spatial and angular distribution of the light. Captured light field images can be processed to reconstruct a full three dimensional (weakly scattering) volume or a single two dimensional z plane in the volume. In [3] we presented a reconstruction method that uses an inverse-problem framework with a wave-optics optical model of the LFM and solved it using 3D deconvolution. We showed that unlike traditional microscopy, in which high resolution can only be achieved for a small range of z depths around the native focus plane of the microscope (its depth of field), the volume reconstructed from the captured light field image of the LFM preserves significant portion of lateral spatial resolution at each z plane; even over a hundred microns away from the native object plane.

However, as we showed in [3], whether reconstructing a full volume or only a two dimensional z plane in that volume from a light field image, the achievable lateral resolution of the LFM is not uniform across z depths. Figure 1(b) shows qualitatively the resolution that can be achieved with a 20× 0.5NA water-dipping objective and a 125μm-pitch f/20 microlens array, and a more detailed analysis of the resolution is given in section 4.1 and Fig. 5 of [3]. Our analysis showed that at the center z planes of the imaged volume, over a z range spanning 10 μm around the native object plane of the LFM, the achievable lateral resolution is significantly degraded compared with that of a widefield fluorescence microscope at its native object plane (Fig. 1(a), left-most column). Farther away from the native object plane the achievable spatial resolution degrades gradually. However, the LFM working range, over which the lateral resolution is high, remains much larger compared with a widefield fluorescence microscope. This non-uniformity of resolution hinders practical use of the LFM for some applications that require localization and measurement of fine 3D structures in a sample that is spread over a large range of z depths. In [4, 5], the authors demonstrate the use of LFM to image neural activity in a transparent larval zebrafish. In supplemental video 6 of [5], reconstructed neurons located at the native object plane of the LFM show up as large uniform square patches compared to neurons at other z depths, which are better resolved.

 

Fig. 1 US Air Force (USAF) 1951 resolution test target translated to depths below the native object plane (z = 0 μm) and imaged using a light field microscope with a 20× 0.5NA water-dipping objective. (a) Images taken with a conventional widefield microscope as the target is translated to the z-heights denoted below each image. (b) Light field deconvolution using the method developed in [3] while the microscope was defocused to the same heights as in (a). The resolution is poor at the native plane (red frame in leftmost column), peaks at z = −20 μm and gradually decreases with depth. (c) wavefront coded LFM, which in this example consists of a single cubic phase mask, placed in the back focal plane of the objective. The low-resolution at the native object plane is significantly improved (green frame in leftmost column), and the resolution at z = −100 μm is also slightly improved compared with (b) (rightmost column, red and green frames). This comes at the expense of reduced peak resolution at z = −20 μm.

Download Full Size | PDF

In this paper we propose a novel extension to the LFM that aims to mitigate the nonuniformity of lateral resolution across depth by placing phase masks in the optical path of the LFM, in order to produce a more uniform performance across depth. This technique, called wavefront coding, shapes the point spread function of the microscope and is a natural extension to traditional light field microscopy. It can be easily incorporated into the optical model and light field deconvolution framework we proposed in [3]. We propose two types of designs: (1) placing a single phase mask at the back focal plane of the microscope’s objective (we call this the objective mask), and (2) combining this objective mask with an array of smaller phase masks positioned at the native image plane of the microscope, as an integral part of the microlens array (microlenses masks). We also show that properly-designed masks can improve the LFM resolution profile for reconstructing both 2D planes (off the object native plane) and 3D volumes and create either a more uniform resolution profile across z depths. Our design solves the low-resolution problem around the native object plane, and can even create a resolution profile that favors the native object plane over other depths. Our data suggests a general design principle for our wavefront coded LFM - using an objective phase mask to control the size of the point spread function (PSF) at the native image plane that is sampled by the microlenses; and using microlenses phase masks to shape the PSF at the detector plane so that high frequencies can be resolved over a wide range of depths.

Since wavefront coding techniques were first proposed by Cathey and Dowski [6] phase masks have been designed and employed by researchers in various imaging modalities. In microscopy, phase-only elements have been placed in the back focal plane of a microscope’s objective, mainly for the purpose of extending the depth of field of the microscope when imaging samples that are spread over a wide range of depths [7–10], but also for the purpose of reducing the depth of field, allowing for better optical sectioning [11].

In the context of integral imaging, Castro et al. proposed to add an array of quartic phase masks to an integral imaging acquisition system made of a microlens array and a detector, in order to improve each microlens depth of field [12]. The three-dimensional reconstruction method requires to convolve each microlens image with a single inverse filter to remove the mask blur, and then the volume is reconstructed using a pinhole ray-optics model. However, the proposed method does not aim to reconstruct above the resolution of the microlens array, nor does it consider a complex optical system such as a camera or a microscope, where diffraction takes place.

A light field image could also be captured together with a widefield image, the latter providing high-resolution information about the native object plane as proposed in [13]. This method requires splitting the incoming light between two cameras or capturing the light field and wide-field images sequentially, which is less suitable for imaging dynamic phenomena. Another approach to improve the low resolution at the native object plane is to divide the light captured by the objective into two optical paths with a known path difference between them, and place two microlens arrays (potentially with different focal lengths) side by side in front of the detector [14]. However this method has the drawback that it sacrifices half the sensor area.

It is worth noting that there are some degrees of freedom in the design and use of the LFM that can help mitigate the problem of non-uniform resolution at the expense of other properties. For example, by varying the microlens array pitch a tradeoff between the peak resolution of the LFM and the resolution farther away from the native focal plane can be achieved - smaller pitch would result in higher peak resolution in the vicinity of the native object plane and worse lateral resolution farther away from it. It is also possible to avoid the low resolution at the native object plane by focusing the LFM outside the volume of interest in the sample. However, this sacrifices half of the microscope’s working range.

The paper is organized as follows: in Section 2 we present an extension to the optical model we presented in [3] for the wavefront coded fluorescence LFM that includes phase masks. This model is then used to generate simulated results and to deconvolve experimental data. In Section 3 we derive a performance metric for the LFM based on Fisher information. The metric is independent of the type of reconstruction algorithm used and is well correlated with standard resolution measurement methods. We demonstrate that it can be a useful tool for designing phase masks and optimizing LFM performance. Finally, in Section 4 we present both experimental results using a fabricated glass objective phase mask, as well as simulated results for other proposed microscope configurations, including microlenses phase masks.

2. Optical model

The reconstruction of a 2D plane or a 3D volume from a light field image requires solving an inverse problem of the form

In [3] we also presented a physical-optics model of light propagation through the LFM, based on wave optics theory. The model assumes a fluorescence microscope in which the objective and tube lens are set up as a doubly-telecentric 4-f system where the back focal plane of the objective coincides with the tube lens’s front focal plane. The wavefront at the native image plane of the microscope, generated by a point source in the volume, is calculated using scalar diffraction theory for a circular aperture objective [15] (which is accurate for low-NA objectives with an aperture diameter much larger than the wavelength of incoming light, for which polarization effects can be neglected). This wavefront is then multiplied by the transmission function of the microlens array and the result is propagated a distance equal to one microlens focal length to the detector plane where an intensity image is recorded. In order to solve the inverse problem, the optical model is discretized and measurement noise is modeled to have a Poisson distribution (i.e. mostly shot noise).

In this work we extend this optical model to include a phase mask in the back focal plane of the microscope’s objective, and phase masks at the aperture of the microlenses of the microlens array. Our model takes advantage of the explicit Fourier transform relationship between parallel planes in the microscope, and is fast to compute numerically. This is especially important, as we later define a performance metric, based on the forward optical model, to search for suitable phase masks that improve the performance of the LFM.

We define our coordinate system so that its origin is located on the optical axis at the front focal plane of the objective, which we also refer to as the native object plane. Positive z values are defined towards the objective. x and y are spatial coordinates on planes perpendicular to the optical axis. A point source with electric field amplitude A, located at p = [x_p, y_p, z_p]^T in the volume (marked by the red dot in Fig. 2) emits a spherical wavefront that propagates to the front focal plane of the objective. According to the Huygens-Fresnel integral [16] and assuming r ≫ λ, at the front focal plane of the objective we have

 

Fig. 2 Schematic diagram of the light propagation through our proposed wavefront coded light field microscope. Phase masks (in green) are placed in the back focal plane of the microscope’s objective, where the telecentric stop is placed in a doubly-telecentric microscope configuration, and also in the aperture of each microlens in the microlens array.

Download Full Size | PDF

The objective and tube-lens in a well-corrected, doubly-telecentric microscope can be modeled as 4-f system, as shown in Fig. 2. The objective’s focal length, denoted f_obj can be calculated from the known magnification M and the tube lens focal length f_tl

Under the Fresnel approximation, the wavefront at the objective’s front focal plane U₀ (x, y, p), multiplied by the objective’s pupil function and the wavefront at the back focal plane of the objective $U_1^{-}(x,y,\mathbf{p})$ form a Fourier transform pair (up to a complex scaling factor)

In a 4-f telecentric microscope configuration, the back focal plane of the objective coincides with the front focal plane of the tube lens, and therefore the wavefront at the back focal plane of the tube lens $U_2^{-}(x,y,\mathbf{p},{\mathrm{\Theta }}_{\mathit{obj}})$, is a scaled Fourier transform of $U_1^{+}(x,y,\mathbf{p},{\mathrm{\Theta }}_{\mathit{obj}})$

Our original design of the LFM features a microlens array at the native image plane of the microscope (where the detector is positioned in a widefield fluorescence microscope). As we described at length in [3], the microlens array acts as a sensing element that enables the recording of both spatial and angular information about the light distribution in the imaged sample. We extend the optical model to include an arbitrary phase function in each microlens aperture. Denoting T_se (x, y, Θ_μlens) the general transmission function of the sensing element, P_se (x, y) the aperture function and Q_se (x, y, Θ_μlens) the phase function, the transmission function takes on the form of a rectangular grid, with added phase component that multiplies the quadratic phase of each microlens

T_se (x, y, Θ_μlens) multiplies the incident wavefront $U_2^{-}(x,y,\mathbf{p},{\mathrm{\Theta }}_{\mathit{obj}})$. For convenience, we define Θ = [Θ_obj, Θ_μlens]^T and we have

Finally, the resulting incoherent light field PSF at the detector plane is given by the squared modulus of the coherent PSF

3. Analysis of the performance of the LFM based on Fisher information

Characterizing the performance of an LFM is not a trivial task. Common methods such as measuring the full-width-half-max of the optical system’s PSF or using a resolution target (such as a US Air Force 1951 test target) to measure the achievable resolution cannot be applied to the light field image directly. Rather, these methods must be applied to the image or volume reconstructed from the light field image (see [3]) and are therefore become sensitive to the choice of reconstruction algorithm and any priors the algorithm applies. In addition, reconstructions are computation intensive and thus time consuming. For these reasons, it is inconvenient to rely on them when designing objective or microlenses phase masks for enhancing the LFM performance. It is therefore beneficial to develop a performance metric that is independent of the reconstruction but correlates well with resolution measurement methods.

We propose to characterize the performance of our wavefront coded LFM by calculating the Fisher information of the optical system. This is similar to the approach used in [6], in which the Fisher information was used to analyze the sensitivity of a cubic phase mask to misfocus. Subsequently, the Fisher information was also used in design and analysis of other phase masks for enhancing the depth of field of imaging systems [18–20]. In these works, the phase mask performance is analyzed and optimized by calculating the Fisher information of the optical system’s PSF with respect to the Zernike defocus coefficient W₂₀. The Fisher information was used to measure how the PSF changes with defocus. A phase mask that minimizes the Fisher information results in a PSF that is insensitive to defocus, thus granting the optical system extended depth of field. In contrast, in [21] the authors designed a phase-amplitude mask that results in a rotating PSF that yields a high Fisher information with respect to defocus. This allowed them to estimate the depth of a two-dimensional thin object from the rotation angle of the PSF via a deconvolution process. In our setting, we will see that the Fisher information can measure the sensitivity of the light field PSF to the position of a point source in the volume and serve as the basis for a metric that the calculates the performance of the microscope.

We adapt the Fisher information-based approach to our needs by defining a measure of how the light field PSF changes when the position of a point source translates slightly in the volume. Formally, consider the light field PSF h (x, y, p, Θ) observed at the detector plane. Normalized properly, h (x, y, p, Θ) can be regarded as a two-dimensional probability density function, de-fined over the detector area, which depends on the unknown parameters vector p. Therefore, we can calculate the Fisher information matrix ℐ (p, Θ) with respect to the unknown point source position p, from the observed light field PSF

High Fisher information values suggest that the light field PSF is sensitive to the exact position of the point source in the volume - i.e. slight shifts in the point source position in the volume result in a significant, detectable change in the light field PSF recorded on the detector. And vice versa - low Fisher information values mean that the recorded light field PSF appears the same on the detector, regardless of small shifts in the point source position in the volume. In this case, the imaging system suffers from inherent ambiguity with respect to the point source position that will hinder reconstruction methods that try to localize the point source in the volume.

The proposed Fisher information matrix depends only on the optical model of the system, irrespective of which reconstruction algorithm is employed. It is also relatively fast to compute numerically - it requires calculating three Fourier transforms and four matrix multiplications per object point (this depends on the exact elements in the optical path), and therefore can be used for both analysis and synthesis purposes.

The proposed performance metric was used to design the combination of phase masks proposed in section 4.4 - the parameters of the phase masks Θ were chosen to maximize a performance metric that is based on the Fisher information matrix in Eq. (2). The performance metric is a weighted sum of the Fisher information matrix elements

In Fig. 8(e) and Fig. 8(f) we show a comparison between the lateral resolution limit measured on reconstructed planes of a simulated USAF 1951 resolution target and the proposed performance metric for different configurations of phase masks in our wavefront coded LFM, respectively. To generate the performance metric plot we set ω_ij to consider only the elements of the Fisher information matrix that correspond to lateral resolution (in the directions of the axes x_px_p, y_py_p and the diagonals x_py_p and x_py_p). The values ℐ_ij (p, Θ) were calculated for a point source on the optical axis of the microscope for different z planes. The derivative was calculated numerically using a finite difference approximation with a step of 0.5 μm (slightly below the smallest feature on the USAF 1951 target). The comparison confirms that the performance metric we propose correlates well with actual resolution measurement on reconstructed USAF 1951 data.

4. Experimental and simulation results

In this section we present simulated and experimental results that demonstrate the performance of our wavefront coded LFM. We begin with experimental results using an objective phase mask that we fabricated. In section 4.1 we discuss our experimental setup and in section 4.2 we show experimental results with only a cubic phase mask placed at the objective’s back focal plane, and using a standard microlens array with no phase masks. We compare the lateral resolution of the standard LFM to our wavefront coded LFM and discuss the reasons for the difference in performance. We also demonstrate experimentally the improvement in resolution when reconstructing full volumes in section 4.3 - both for a synthetic sample and for a biological application. In section 4.4 we show simulation results of additional configurations of our wavefront coded LFM, including phase masks in the microlenses apertures. We discuss the merits of different configurations and compare their respective resolution performance across depth. Finally, in section 4.5 we give a ray-space analysis of our wavefront coded LFM with two phase masks configurations. The analysis provides more insight on how the use of phase masks affects sampling of spatial and angular information of the LFM.

4.1. The experimental setup

In order to test the performance of our wavefront coded LFM we altered our light field microscope setup described in [3] to include a physical phase mask in the objective back focal plane. It is not straightforward to access the objective’s actual back focal plane in our setup so we added relay optics to create a 1:1 conjugate back focal plane where we can easily place a phase mask. Figure 3 shows an image of our setup and a schematic diagram of the optical path. A sample may be trans-illuminated from below by a diffused halogen light source (that emits incoherent light) and through a 0.9NA condenser lens (Nikon) arranged in Köhler-illumination, or epi-illuminated through the objective using a xenon arc lamp at 488 nm wavelength. Note that the condenser NA has to be equal or larger than the objective’s NA so that light completely fills the objective’s back aperture and the resolution is therefore not limited by the illumination (in this case, the number of the pixels under a microlens that collect useful data about the sample is maximized). The sample is imaged by a 20× 0.5NA water immersion objective (Nikon) through a 535 nm fluorescence emission filter (Semrock 535/50) and through a tube lens, L1 (Nikon 200mm). The physical phase mask is placed between a pair of relay lenses, L2 and L3 (3” Edmund Optics achromatic doublets, f=200 mm) which together form a 1:1 4-f system. We chose to match their focal length to that of the tube lens to create a 1:1 image of the back focal plane of the objective. The exact position of the glass phase mask (PM) is determined according to the location of the conjugate objective’s back focal plane (note that if the microscope is not exactly telecentric, as is the case in our setup, this plane may be different than the back focal plane of L2). The mask is mounted on a stage that allows for XY translation and rotation about the optical axis for accurate alignment. L3 is placed one focal length after the phase mask. The remainder of the light path is the standard path used in [3]: a microlens array (MLA, RPC 125 μm pitch, f/20) is positioned at the native image plane of the microscope, with lenses L4 and L5 (Nikon 50mm f/1.4 and Nikon 35 mm f/2, arranged in back-to-back configuration) used to relay the detector (DET, Retiga 4000R) plane to precisely one microlens focal length behind the microlens array. This optical setup effectively implements the optical path shown in Fig. 2 (without the microlens phase masks). Note that the pair of relay optics L2, L3 and L4, L5 are not modeled of our optical model since we assume they are high quality optics and since we use only the center part of these large lenses to minimize aberrations. Therefore, the relay optics have very little effect on the light field PSF. We note however, that the choice of focal lengths for L4 and L5 does affect the number of detector pixels under each microlens, as explained in [3]. In our configuration we chose to have 25 × 25 pixels under each microlens.

 

Fig. 3 Wavefront coded light field microscope - image and schematic diagram. The setup is composed of a fluorescence microscope (objective OBJ and tube lens L1), relay optics (L2 and L3) that create a conjugate back focal plane of the objective, a phase mask (PM) that is placed at that conjugate plane on an translation stage, a microlens array (MLA) and a detector (DET), relayed one microlens focal length behind the microlens array using relay optics (L4 and L5).

Download Full Size | PDF

We chose to manufacture and test the well known cubic phase mask since it is relatively easy to fabricate and due to its promising performance in simulations. The cubic mask, made by RPC Photonics, was fabricated by depositing a 20 mm × 20 mm polymer (with refractive index n = 1.52 for emission wavelength of λ = 535 nm) on BK7 glass substrate and etching the cubic shape onto the polymer. The phase function it implements is p(x, y) = 117 (x³ + y³), where the spatial coordinates x and y are normalized to the square mask size. The free parameter α = 117 was chosen to take advantage of the entire depth range achievable by the fabrication process over the 20 mm × 20 mm mask area (the depth limitation of the process was 75 μm). Note that the mask area is larger than the diameter of the back aperture of the 20× 0.5NA objective we use (the back aperture diameter d_obj = 2 f_objNA [15] is equal to 10 mm in the case of our 20× 0.5NA objective). Since our phase mask has width and height twice that size, it can also accommodate other objectives - particularly ones with relatively low magnification and high NA that have larger back aperture diameter.

To test the accuracy of the fabrication we measured the profile of the mask using a coherence correlation interferometer (CCI HD, Taylor Hobson). Figure 4(a) shows the measured mask sag and the back aperture diameter of the microscope objective. The maximal surface error compared to the design was measured to be 500 nm over the entire mask area and 300 nm inside the objective’s back aperture diameter.

 

Fig. 4 The cubic mask placed at the conjugate back focal plane of the light field microscope objective and its effect on the LFM PSFs for a point on the native object plane. (a) The measured sag of the cubic phase mask. The white circle indicates the size of the back aperture of the 20× 0.5NA objective. (b) With the phase mask, a point source at the native object plane generates a cubic PSF at the native image plane of the LFM (left column, top row) and a complicated diffraction pattern that spans 4 microlenses on the detector (left column, bottom row). Translating the point source by 2 μm in x and y (right column) changes the light field PSF intensity profile significantly. (c) Without the cubic phase mask. By contrast, the light field PSF after a 2 μm translation in x and y still resembles a disk. The greater change in (b) than (c) reflects the higher sensitivity of our wavefront coded LFM to small changes in the specimen at the native object plane, hence its improved spatial resolution at that plane.

Download Full Size | PDF

4.2. Experimental results with a single phase mask - resolution analysis

Before presenting the resolution measurements with the objective cubic phase mask, it is beneficial to study light field PSFs for a point source at the native object plane to gain insight into the increase in resolution at that plane, attributed to the phase mask. Figures 4(b) and 4(c) show the effect of a small lateral translation on the PSFs at the native image plane and on the light field PSFs at the detector plane, with and without the cubic phase mask, respectively. The cubic PSF shape at the native image plane spans an area of about 150 μm×150 μm and is much larger than the diffraction spot of the standard LFM. It is spread by the microlens array and forms a complicated diffraction pattern on the detector that spans four microlenses. This light field PSF is more sensitive to the exact position of the point source on the native focus plane than the light field PSF without the phase mask. When the point source is translated 2 μm laterally, the diffraction pattern changes significantly. The diffraction spot of the standard LFM on the other hand, is spread by a single microlens to a circular disk on the detector plane. Shifting the point source position laterally causes only a slight change in the light field PSF - since the shifted small diffraction spot on the native image plane still lies within the numerical aperture of a single microlens, it produces in a very similar disk pattern on the detector plane. Therefore, the cubic phase mask reduces the ambiguity in localizing a point source on the native object plane.

To confirm the improvement in lateral resolution in actual experiments we used the same procedure as in [3] where we imaged a high resolution USAF 1951 test target (Max Levy DA052) at each of 201 z depths spanning 200 μm, and reconstructed the image of each target, restricting the light emission to voxels at the known z depth of the target in each image. We chose to reconstruct each image with a spatial sampling period 16 times smaller than the native sampling period of the LFM, as determined by the microlens array pitch (or microlenses spacing) divided by the objective magnification (its reciprocal is referred to herein as the native resolution of the LFM). The resulting reconstructions of the resolution target (Figs. 5(a) and 5(c)) were then analyzed using a similar contrast metric as in [3] to produce the resulting modulus transfer function (MTF) plot shown in Figs. 5(b) and 5(d). Each z plane containing a USAF 1951 pattern was registered to a ground truth image and the contrast was calculated for each of the target’s regions of interest according to

 

Fig. 5 Experimental resolution measurements of the standard LFM and our wavefront coded LFM, with a cubic mask placed at the objective back focal plane. (a) Light field deconvolution of experimental data captured with no phase mask. At the native object plane (z = 0 μm) the resolution is low, but at other planes, much higher spatial resolution can be reconstructed. The resolution gradually decreases as we move farther away from the native object plane. (b) Experimental MTF measurements for the standard LFM, based on contrast calculation of different spatial frequencies in the USAF 1951 target. High contrast values MTF are shown as hot (white-yellow), while lower values are cold (blue). Peak resolution of more than 500 lp/mm is achieved at z = 15 μm, but the resolution fluctuates in a region of about 50 μm around the native object plane. (c) The same reconstructed planes, now with the cubic phase mask. The resolution at the native object plane is significantly improved, is similar for z=±50 μm and is also slightly improve at z=±100 μm. (d) The corresponding MTF heat maps show that with the cubic phase mask, the resolution is more uniform around the native object plane, but peak resolution drops to about 350 lp/mm.

Download Full Size | PDF

Using the LFM with no phase masks, we see several limitations. Over a large range of depths around the native object plane of the LFM the achievable spatial resolution fluctuates significantly, with certain z depths (−15 μm off the native object plane) reaching high spatial resolution, while other z depths (around −25 μm off the native object plane) only reaching lower spatial resolution. In particular, the lateral spatial resolution of the LFM at z depths very close to the native object plane (the z-depth the microscope is focused at ±5 μm around it) is significantly degraded compared with that of a traditional fluorescence microscope. Here, resolution is limited by the fact that the detector pixels collect aliased, redundant information about the spatial position of an object in the volume, as explained in [3]. In the example shown in Fig. 5, a LFM with a 20× objective and 125 μm pitch microlens array yields a maximal attainable frequency of 80lp/mm at the native object plane. At these z depths, objects with fine details cannot be reconstructed with high fidelity. Moreover, even at z depths where high spatial resolution is achieved, lower spatial frequencies are degraded, resulting in inconsistent performance. For example, note the lower contrast at frequencies around 200 lp/mm, between −10 μm to −20 μm off the native object plane. Farther away from the native object plane, the achievable spatial resolution degrades gradually, limiting the effective working range of the microscope (even though it is still much larger compared with a widefield fluorescence microscope).

With the cubic phase mask however, many of these limitations are overcome. The resolution is significantly more uniform around the native object plane and is also slightly improved farther away from it. Moreover, the MTF frequency response at each depth is more monotonic decreasing with frequency. This more uniform resolution profile comes at the expense of peak resolution performance - the peak resolution when using a cubic phase mask reaches only about 64% of the peak resolution of the LFM when no phase mask is used.

4.3. Volumetric reconstructions with the cubic phase mask

We conclude the experimental results by comparing three-dimensional volumes reconstructed from light fields that were captured with and without our experimental objective cubic phase mask. Unlike the planar reconstruction, in volumetric reconstructions, the volume is rendered without any prior knowledge about the z planes where the is specimen is located. Moreover, unlike a USAF 1951 target, in the examples we give the specimen is not planar but is spread over a large range of depths. We imaged with our 20× 0.5NA objective and reconstructed the volumes with a sampling period 4 times smaller than the native sampling period. Figures 6(a) and 6(b) show max projections of a 500 μm × 500 μm × 300 μm volume of 2 μm-size fluorescence beads embedded in agarose, with no phase mask and with the objective cubic phase mask, respectively. Beads were epi-illuminated at 488 nm. Figure 6(c) shows XY and XZ slices of a bead on the native object plane, along with the corresponding one-dimensional cross sections of these slices. The blue curve shows the lateral profile and the red curve shows the axial profile of the bead. Using the phase mask, both lateral and axial resolution is significantly improved, as can be noted from the full width half maximum measurements. In Fig. 6(c) we show the same plots for a bead 70 μm above the native object plane. In this case, the phase mask improves the lateral resolution (blue curves) slightly, but the axial resolution remains very similar.

 

Fig. 6 Volumetric imaging of 2 μm fluorescent beads embedded in agarose, with and without an objective cubic phase mask. (a) Max projections through the volume with no phase mask. (b) Max projections through the volume with the cubic phase mask. (c) Reconstructed bead on the native object plane (red boxes in (a) and (b)) and the corresponding X and Z intensity profiles showing the FWHM for each case. The cubic mask significantly improves both lateral resolution and axial resolution. (d) Reconstructed bead +70 μm off the native object plane (yellow boxes in (a) and (b)). The phase mask improves the lateral resolution slightly but the axial resolution remains the same as the no phase mask case.

Download Full Size | PDF

To illustrate an application that benefits from the improved resolution around the native object plane we also applied our new technique toward in vivo calcium imaging in larval zebrafish, a task which requires volumetric imaging at video frame rates and ability to resolve neurons about 5–10 μm in size across all z depths. We used our 20× 0.5NA objective to image a volume of the same size as before at 3 Hz (the limiting frame rate of our camera). Epi-illumination at 488 nm was used to excite 535 nm fluorescence emission from active neurons in 8 dpf zebrafish expressing calcium indicators pan-neuronally, Tg (elavl3:GCaMP5G) in nacre background.

Two time series were acquired, one with and without the previously described cubic phase mask, and min-subtracted reconstructed volumes from a single time point in each time series are shown in Fig. 7. The improvement in resolution at the native plane, shown in Figs. 7(a) and 7(b), makes it possible to better resolve fluorescent sources located there, which are indistinguishable without the phase mask. This improvement with the cubic phase mask comes without any loss in z-resolution, as illustrated by the XZ maximum projections in Figs. 7(c) and 7(d).

 

Fig. 7 In vivo volumetric calcium imaging of a larval zebrafish at camera-limited frame rate. (a) The native object plane out of a reconstructed volume at a single point in time using conventional light field deconvolution. The poor spatial resolution makes it hard to distinguish individual fluorescent sources. (b) With the use of a single cubic phase mask at the objective’s back focal plane, the resolution is improved. The insets in (a) and (b) show magnification of the telencephalon, an olfactory and learning center in the forebrain. c,d) XZ maximum projections of the same volumes in (a) and (b) illustrate that no z resolution is traded away to obtain the improvement shown in (b). “Banding” artifacts shown in (c), caused by the low resolution planes (black arrow), are gone in (d).

Download Full Size | PDF

4.4. Simulation results of additional configurations

The experimental results with the cubic phase mask raise an interesting question: whether designing a different objective phase mask, or using a combination of an objective phase mask and microlenses phase masks may result in better overall performance of the wavefront coded LFM. A question that might follow is whether there is an optimal configuration for the wavefront coded LFM that is preferable to all others. Answering these two questions in full is not straightforward, but we take a step in this direction by simulating and analyzing two additional configurations of phase masks: a configuration with a spiral mask at the back focal plane of the objective and a configuration the cubic phase mask at the back focal plane of the objective, coupled with an additional array of cubic phase masks located at the microlenses apertures.

Our spiral phase mask is the continuous version of the phase function proposed in [22]. Instead of dividing the aperture into several annular Fresnel zones with decreasing width, we define the mask in polar coordinates to be

The second configuration we propose includes a cubic masks at the back focal plane of the objective as well as cubic masks at the aperture of the microlenses. Adding phase masks to the microlenses offers additional degrees of freedom and control over the shape of the light field PSF. With a cubic phase mask at its aperture the point spread function of each microlens becomes a cubic PSF. We chose a value of α = 5 for the free parameter of the mask. As we will see, the addition of microlenses phase mask on top of an objective mask can significantly improve the resolution of the LFM around the native object plane. We note that unlike the objective phase mask, which requires adding at least one additional optical element to the optical setup (and may require more if there is a need to create a conjugate plane as is the case in our setup), the microlenses phase masks can be implemented as part of the microlens array by altering the microlenses sag.

Our simulations were conducted in the following way: we created an empty volume and placed the ground truth image of a USAF 1951 target in it, at a certain z-depth. We then applied the forward optical model on the volume g according to Eq. (1) to create a light field image f from it. This synthetic light field image was deconvolved and processed in the same way as the experimental light fields shown in section 4.2. We repeated this procedure, placing the USAF 1951 image at different z depths in our simulated volume for a range of 200 μm around the native object plane of the microscope.

Figure 8 shows the simulation results for the two configurations we propose (see Figs. 8(c) and 8(d)) as well as the standard LFM configuration with no phase masks and our experimental objective cubic phase mask for comparison (see Figs. 8(a) and 8(b) respectively). The simulated results in Figs. 8(a) and 8(b) are in agreement with our experimental results for both these cases, although the experimental results in Fig. 5 reach a lower limiting resolution and exhibit stronger reconstruction artifacts. In Figs. 8(c) and 8(d) we show the reconstructed USAF 1951 target for the combination of cubic masks and for the spiral mask, respectively and in Fig. 8(e) we show the 10% contrast resolution limit, calculated from the reconstructed USAF 1951 stacks for each z-depth. A comparison to the Fisher information performance metric is given in Fig. 8(f).

 

Fig. 8 Simulation results of USAF 1951 resolution target at different z planes for several LFM configurations. (a) Standard LFM with no phase masks. (b) An objective cubic phase mask. (c) A combination of cubic phase masks: an objective mask as in (b) and adding a cubic phase mask at each microlens aperture. The resolution around the native object plane is significantly improved compared with (a) and (b), but degrades faster farther away from it. (d) A spiral objective phase mask. The resolution is uniform across a range of 200 μm, but peak performance is reduced compared with (a)–(c). (e) 10% contrast resolution limit across depth for the configurations (a)–(d). At the native object plane, the standard LFM (cyan curve) shows severe aliasing artifacts which result in inaccurate measurement of the resolution. (f) The proposed Fisher information-based performance metric. The metric correlates well with the limiting resolution in (e). Existing differences are due to different the Fisher information metric does not calculate the maximal frequency for a certain contrast threshold, but rather the derivative at a certain fixed step size in the spatial domain.

Download Full Size | PDF

The performance of the spiral mask (Fig. 8(d)) is the most uniform over the depth range - the resulting limiting resolution is slightly above 250 lp/mm, more than threefold improvement over the native resolution of the LFM (80 lp/mm in our configuration). Even at a distance of ±100 μm away from the native plane the resolution hardly degrades. This uniform performance is expected since the spiral PSF keeps its size and general shape and only rotates around a center point with depth. However, peak resolution is sacrificed even more compared with the cubic phase mask. Even with the lower peak resolution, the spiral phase mask is a good candidate for applications that require uniform resolution performance over a large range of depths.

In contrast, the combination of cubic masks at the objective back focal plane and the microlenses apertures shown in Fig. 8(c) results in a non-uniform resolution profile across depth, with high resolution at the vicinity of the native object plane that decreases farther away from it more rapidly compared with the other configurations. To understand why the addition of these phase mask trades the resolution of the LFM near the native object plane and farther away from it in this manner, we analyze the PSF under a single microlens as a result of point source in the volume located at two different z depths. Figures 9(a) and 9(b) show a single microlens diffraction spot for an objective cubic phase mask and for a combination of objective and microlenses cubic masks respectively, both for a point source 25 μm away from the native object plane. When using only an objective cubic phase mask, the diffraction pattern under one microlens is a large spot and consequently its 1D cross-section MTF (the absolute value of the Fourier transform of the central horizontal row of the 2D diffraction spot), shown in red in Fig. 9(c), cannot support high frequencies. Adding the cubic phase mask at the microlens aperture shapes this spot into a cubic PSF that has an improved frequency response (blue curve in Fig. 9(c)) and can therefore support higher spatial frequencies, resulting in higher spatial resolution in the reconstruction. Farther away from the native object plane at z = 80 μm the opposite is true: the resulting spot when using standard microlenses (Fig. 9(d)) is more compact than the cubic PSF (Fig. 9(e)) and can therefore support higher spatial frequencies, as shown in Fig. 9(f).

 

Fig. 9 Analysis of the diffraction spot under a single microlens with and without a cubic phase mask at its aperture. In both cases, a cubic phase mask is used in the objective back focal plane. (a) The diffraction spot for a point source at z = 25 μm with no microlens phase mask. (b) the diffraction spot for the same point source position as in (a), with a cubic phase mask at the microlens aperture. (c) Cross-section MTFs of the spots in (a) - in red and in (b) - in blue. The PSF of the microlens with the cubic phase mask shows better frequency response. (c) The diffraction spot for a point source at z = 80 μm with no microlens phase mask. (d) The diffraction spot for the same point source position as in (c), with a cubic phase mask at the microlens aperture. (e) Cross-section MTFs of the spots in (c) - in red and in (d) - in blue. At this depth, the MTF of the microlens with the cubic phase mask is worse compared with not using a microlens phase mask. In this example, adding cubic masks to the microlenses is advantageous only for a certain range of depths of about 80μm around the native object plane.

Download Full Size | PDF

4.5. Ray-space analysis of the wavefront coded light field microscope

Further understanding of the effects of using different phase masks can be gained by observing the Cartesian 2D ray-space diagrams for two wavefront coded LFM configurations: (1) an objective cubic phase mask, and (2) microlenses cubic phase masks. Ray-space diagrams show how the detector pixels integrate light rays from different spatial positions on the native object plane, denoted the x plane, and different angles over the objective’s back focal plane, denoted the u plane (note that for the sake of clarity, the diagrams are two dimensional, having only one spatial coordinate and one angular coordinate). The size of the x plane is limited by the objective’s field of view and the size of the u plane is defined as the size of the telecentric stop of the microscope. Although they are based on ray optics and do not consider diffraction effects, we found that these diagrams do provide insight into the benefit of using phase masks to improve the low resolution at the LFM’s native object plane. Figure 10 shows one-dimensional ray-space diagrams along with backward ray-tracing (from the detector plane to the volume) illustrations for an LFM with no phase masks and for the two phase mask configurations mentioned above. The ray tracing illustrations are based on tracing rays through a paraxial model of the LFM in optical design software Zemax.

 

Fig. 10 Backward ray tracing diagrams and corresponding ray-space diagrams for four LFM configurations, assuming paraxial optics. Scale is exaggerated to highlight differences. (a)–(d) Back-traced rays from a detector pixel for no phase masks, microlenses cubic phase masks, objective cubic phase masks and both microlenses and objective masks configurations, respectively. Ray colors distinguish different points inside a pixel’s integration area. The objective and microlenses phase masks spread the rays differently: Ray bundles from each point on the detector are refracted by the microlenses masks so that their intersections with the u plane form a parabolic profile. The objective phase mask on the other hand, introduces different phases to ray bundles that intersect at different positions over the back focal plane, so that their positions on the x plane form a parabolic profile. (e) Ray-space diagram for (a). When no phase masks are used, a thin object on the native object plane (the yellow vertical stripe) is sampled only by pixels under a single microlens (denoted by same-color areas), which collect light over the same area on the x plane. The lack of diversity in position measurement leads to low spatial resolution in the reconstruction. (f) Using only microlenses phase masks the sampling pattern of the x plane does not change. Therefore, the low spatial resolution at the native object cannot be improved. (g) With an objective mask, pixels that sample the object now cover partially overlapping areas on x. The added spatial information leads to higher spatial resolution in the reconstruction. (h) When using objective and microlenses phase masks together, the object is sampled by even more pixels, resulting in further improvement in spatial resolution in the reconstruction.

Download Full Size | PDF

When no phase masks are used (Fig. 10(a)), rays back-traced from a single pixel (blue rectangle on the detector) are spread to parallel ray bundles by the microlens which are then focused by the tube lens on the u plane. They are expanded again by the objective to parallel bundles that intersect with the x plane. The ray bundles from one pixel cover an area on the u plane equal to the objective’s back aperture size d_obj divided by the number of pixels under a microlens; and on the x plane the rays spread over an area of size d_μlens/M_obj. Hence, in u − x space (Fig. 10(e)) every detector pixel samples a rectangular area. The ray-space diagram Fig. 10(e) illustrates the low resolution problem at the native object plane of the LFM: a thin isotropically-emitting object on the x plane, with width smaller than d_μlens/M_obj as illustrated by the vertical yellow stripe in Fig. 10(e), is only sampled by pixels under a single microlens (same-color rectangles in Fig. 10(e)). In this case, the resulting light field PSF looks like a full disk as shown in Fig. 4(c). Since all the pixels collect light from the same area on the x plane (but from different angles of view), the measurements are redundant and it is not possible to reconstruct the object with higher spatial resolution than the width of that area (i.e. the native sampling period of the LFM), without using prior knowledge about the object. This redundancy is also illustrated in Fig. 3 in [3] which illustrates the spatial sampling of the volume by rays back-traced from the detector - at z = 0 μm (the native object plane) the ray intersections are sparse, and the plane is therefore sampled only at intervals corresponding to d_μlens/M_obj (6.25 μm in this case).

By adding microlens cubic phase masks, we see that rays traced-back from each point on the detector (Fig. 10(b)) are spread by the phase mask and intersect the u plane at positions that form a parabolic profile. The rays intersect the x plane over an area with the same size as in Fig. 10(a). In u − x space (Fig. 10(f)) each pixel covers the same area d_μlens/M_obj on the x plane as in Fig. 10(e), but has a parabolic profile in u (this profile is the sum of many shifted parabolic profiles, each produced by a different point inside the pixel’s integration area on the detector). The parabolic curvature depends on the power of the cubic phase masks. It is evident that the pixels that sample the object on the x plane (vertical yellow stripe) still contain redundant information about its position. Therefore, the microlenses phase masks do not solve the native object plane resolution problem.

Placing a cubic phase mask in the objective’s back focal plane (with no microlenses phase masks) however, changes the intersection of rays with the x plane. Backward-traced rays from a every point on the detector (Fig. 10(c)) intersect the u plane at a single point, and are spread on the x plane over the same area size, as is the case in Fig. 10(a). But since groups of rays coming from different points within a pixel area intersect the u plane at different positions, they are refracted differently by the phase mask and are therefore spaced according to a parabolic profile on the x plane. Summing the contributions from all points within a pixel, results in a parabolic profile on the x plane as shown in Fig. 10(g). Our ray-space analysis for this case matches results presented in [23] and [24] for a camera system with a cubic phase mask in its main aperture. When using an objective cubic mask, the resulting light field PSF spreads over several microlenses as shown in Fig. 4(b). An object on the native object plane is therefore sampled by pixels under several microlenses, which collect light from different, partially overlapping spatial positions on x. The additional spatial information can be used to reconstruct the object’s position with higher spatial accuracy than the native resolution of the LFM. When using objective and microlenses masks together (Fig. 10(d)), rays are refracted by both masks and result in parabolic profiles on both u and x planes. The resulting sampling pattern of the detector pixels (Fig. 10(h)) shows that even more pixels sample the object on the native object plane compared with just using an objective mask. The greater diversity of positions and angles of rays in the measurement of the native object plane, and the fact that with the cubic phase mask each microlens PSF can support higher spatial frequencies for that plane result in even higher spatial resolution in the reconstruction, as we show in section 4.4.

5. Discussion

In this paper, we have presented an extension to the light field microscope that adds phase masks in the back focal plane of its objective and in the microlenses apertures. We have shown how these phase masks can control the shape of the light field PSF and change the microscope’s resolution profile to be much more uniform across depth. We analyzed the resolution of several configurations of our wavefront coded LFM theoretically, experimentally, and in simulations. We have also proposed a method to characterize and optimize the LFM’s performance based on Fisher information that is independent of the deconvolution algorithm used.

Our proposed extension to the LFM allows more control over its resolution profile and suggest that the profile can be tailored to the desired application. In particular, we note that the spiral objective phase mask is especially well-suited for volumetric imaging, where one can obtain very uniform resolution performance over the entire volume while still enjoying the benefits of light field microscopy (namely the ability to capture high speed three-dimensional data with sub-second time dynamics).

There are several future research directions that are worth mentioning. First, we plan to validate our simulated results experimentally, either by fabricating additional objective phase masks and microlens arrays with phase masks or by using a phase spatial light modulator (SLM). The SLM would allow us to implement and test various mask designs quickly. Second, we plan to incorporate the Fisher information metric in a general optimization scheme to search for phase masks that give even better resolution than those we proposed herein. We believe however, that such an optimization scheme should be constrained and directed in order for it to converge to useful mask designs. The comparisons in Fig. 8 and Fig. 9 and the ray-space analysis we presented suggest two constraints for such an optimization scheme in order to obtain high resolution over a wide range of z depths. First, for the entire range of z depths, the extent of the PSF at the native image plane of the LFM has to be large enough so that it sampled by several microlenses, but not too large so that it remains sensitive to the exact position of the point source in the volume. In addition, the PSF should be sufficiently different in shape for different object z depths to be able to support volumetric reconstructions well (as an example, the spiral mask we presented fulfills these requirements). Second, the diffraction pattern created by the microlenses and phase masks should support high spatial frequencies over the entire depth range. It seems that the design principle for optimizing our wavefront coded light field microscope should address the first requirement by optimizing the objective phase mask, since only this mask can affect the extent and shape of the PSF at the native image plane where the microlens array is located. The second requirement can be satisfied by optimizing the microlenses phase masks to control the diffraction pattern at different depths.

Finally, we would also like to use our extended optical model and Fisher information to explore and compare the performance of more general extensions to the microlens array. These might include using different types of phase masks at neighboring microlenses, or replacing the microlens array with an alternative sensing element, such as the coded attenuation mask proposed in [25] for capturing compressive light fields in a camera setup.