1. Introduction

Computational imaging has emerged as a technique that can overcome some of the limitations of traditional imaging methods. The approach involves indirectly imaging objects and integrating computational techniques in order to provide novel functionality or improve imaging performance [1,2]. Some computational imaging approaches use principles from the field of compressed sensing (CS), which provides a theoretical framework for reconstruction of signals from fewer samples than dictated by traditional Nyquist sampling, provided the signal is sparse in some known domain [3,4]. In the context of imaging, where signals are often sparse in wavelet domains, CS principles enable images to be recovered with more resolvable points than pixels in the physical sensor. This capability can allow for high quality imaging in applications where sensor size is constrained or where high pixel count CCD/CMOS arrays are expensive or unavailable [5].

Research in computational imaging based on the CS framework, or compressive imaging, has focused heavily on the formulation of mathematical CS models, and on development of efficient CS image reconstruction algorithms [6]. This focus is continually pushing the limits of CS theory, but practical limitations can easily be overlooked if theoretical concepts are not tested in hardware with consideration of end applications [7]. In this report, we focus on some challenges involved in practical implementation of CS theory in benchtop imaging hardware platforms, including challenges with alignment of optical components, determining the optimal mask element size relative to the sensor pixel size, dealing with optical aberrations, light coherence, light throughput, and dispersive behavior of spatial light modulators. We previously showed that the introduction of CS methods using spatially-coded intensity modulating masks at a conjugate image plane leads to a highly parallel version of the single pixel camera [8]. This earlier work used digitally-generated synthetic objects to examine challenges such as optical aberrations and alignment factors that arise in an experimental platform. We showed that by measuring and incorporating these real-world system imperfections into the mathematical model, the quality of reconstructed images can be improved. We extend this work here by reporting results from an experimental study imaging real objects in a benchtop platform. Our setup permits a direct comparison of two fundamental compressive imaging architectures, one with masks positioned at a conjugate image plane and the other with masks positioned at a Fourier (or aperture) plane. We extend our description of how to deal with practical issues and we examine the consequences of imaging under spatially coherent and incoherent illumination. We compare the two architectures under relaxed optical conditions in order to determine the advantages and disadvantages of each so we can further develop one candidate architecture in a later study.

Our long term goal is to apply compressive imaging methods to biomedical imaging applications where sensor size is restricted and we limit the scope of this study to architectures with potential for use in this area. In particular, we believe applications in biomedical optics would benefit from compressive imaging approaches. For example, compressive imaging could have a particularly positive impact on the development of minimally invasive imaging devices that improve in vivo visualization of tissue in clinical settings. Conventional optical techniques for in vivo imaging are unable to produce images with the required histopathology-level detail necessary for diagnosis because miniaturizing imaging devices places size constraints on imaging sensors. Compressive imaging approaches have shown to improve the resolution capabilities of an optical imaging system, including some biomedical applications [1], and could permit higher definition imaging for minimally invasive devices. Additionally, with the growing focus on portable, low-cost biomedical technologies for point of care diagnostics, computational imaging approaches can shift the focus from expensive hardware upgrades to computational advances, which improves device performance without dramatically increasing costs [9]. While our future plans involve translation to biomedical optics, we believe the challenges identified in this report will be of interest to the broader computational imaging community, as we present specific limitations in CS modeling and algorithms that should be considered for any hardware implementation.

2. Compressive imaging architectures

Many mathematical models have been developed for compressive imaging systems since the emergence of the single pixel camera, alongside new advancements in CS theory [10]. The implementation of CS-based imagers in hardware has also expanded to include several novel architectures, as reported in previously published reviews on compressive imaging [11–14]. To narrow down potential compressive imaging architectures for biomedical applications where sensor size is limited, we focus the following discussion on some example architectures from papers that have reported hardware platforms capable of improving spatial resolution of sensor arrays. Typically, these architectures use physical masks in the system to perform spatial multiplexing, a process by which coded low-resolution images are captured and processed to generate high-resolution images. Hardware implementation for this type of compressive imaging generally differs based on the action of the coded mask (modulating light amplitude and/or phase) and its location in the optical path (at a conjugate image plane, Fourier plane, or elsewhere). In each of these situations, coded masks are commonly implemented using an SLM rapidly switching between different patterns. The system may operate with spatially coherent or spatially incoherent light (we will refer to these simply as coherent and incoherent), with consequences for the mathematical forward model describing the image forming process (see Sec. 3.1). Under coherent illumination, such as laser light spatially filtered by a single-mode fiber or pinhole, the system is linear in complex amplitude, whereas under incoherent illumination, such as that provided by a diffuse or extended source, the system is linear in optical intensity [15]. Compressive imaging models that assume phase modulation is possible in the system are more appropriate for applications that image with coherent light [5]. While some biomedical optics applications do use wide field coherent illumination (such as coherent microscopy techniques, or laser speckle imaging), many image light that is inherently incoherent (such as light microscopy, fluorescence, or diffuse reflectance imaging) and therefore imposing phase modulation is more challenging [5].

Early CS-based imagers used amplitude-only SLMs placed at a conjugate-image plane within the imaging path to modulate an object’s intensity point-by-point [16]. In these image plane coding (IPC) architectures, the modulated light is integrated at a single photodetector or a focal plane array (Fig. 1(a)). An extensively studied variation of this architecture is the CASSI system and its offshoots [17]. This architecture uses CS for hyperspectral and temporal imaging along with super-resolution applications [18], typically using dispersive elements for spectral coding. Some CASSI developments are able to produce CS reconstructions in a single snapshot, using a single mask [19]. CASSI snapshot spectral imagers are efficient at improving spatial resolution, but multiple low-resolution measurements are required to attain this resolution improvement, which sacrifices imaging speed [19]. Other IPC systems have been developed to address this issue and achieve high-speed imaging [20]. Other architectures use programmable apertures for Fourier plane coding (FPC), in which the SLM is placed at a conjugate Fourier plane (Fig. 1(b)). Here, the SLM modulates the spatial frequencies present in the object [21]. This configuration allows for amplitude and/or phase modulation and provides the opportunity for superresolution from only a single measurement, therefore maintaining the possibility of high-speed imaging [22]. A common architecture for implementing phase modulation makes use of two masks for double random phase encoding (DRPE) [23]. DRPE, which was originally developed for optical security applications [24], has been modified for superresolution compressive imaging [25] (Fig. 1(c)). This technique also permits superresoultion with a single exposure and is suitable for diffraction-limited systems, or systems where pixel size limits image resolution [26].

 

Fig. 1 Computational imaging architectures for compressive optical imaging. (a) Image-plane coding (IPC) with the mask at a conjugate image plane. (b) Fourier-plane coding (FPC) with the mask at a Fourier plane in the imaging path. (c) Double random phase encoding uses separate random phase masks placed at object and Fourier planes. (d) Structured illumination architectures use image-plane coding with the mask at a conjugate image plane in the illumination path. (e) Fourier ptychography architectures use masks at a Fourier plane in the illumination path. (f) Lensless CS architectures use only a mask between the sensor and the object to modulate light emanating from the object. (g) Mask-based light field imaging architecture with the mask slightly in front of the sensor.

Download Full Size | PDF

Variations of these architectures can also implement CS imaging with the SLM in the illumination path. Structured illumination architectures (Fig. 1(d)) use an SLM at a conjugate-image plane in the illumination path and follow the same generalized mathematical model as the IPC architecture [27]. Structured illumination microscopy techniques can even reach video rate imaging with better than diffraction limited resolution [28, 29]. Placing the SLM, or more commonly an LED array, at a Fourier plane in the illumination path (Fig. 1(e)) creates an architecture similar to Fourier Ptychography [30]. Compressive Fourier ptychography can achieve approximately an order of magnitude reduction in acquisition time over traditional sequential techniques [30]. Researchers are also investigating lensless CS systems [31–34] (Fig. 1(f)). In general, this architecture use the SLM as a coded aperture in the imaging path to modify the system’s point spread function, the impulse response of the imaging system, which is then exploited during the CS reconstruction process. A computational lensless microscopy platform has been developed for high resolution imaging of various biological samples, but the technique is limited to brightfield transmission imaging [34]. Compressed sensing has also theorized for use in light field imaging [35, 36] where, a microlens array is typically used to capture the light field emanating from a scene. Mask-based designs for light field imaging have shown to achieve better image quality than microlens designs, when using an FPC architecture or coded mask placed just off the sensor plane [37] (Fig. 1(g)).

Before considering a specific CS imaging architecture, it is important to establish the imaging restrictions imposed by the target application. With translation to in vivo biomedical imaging in mind, we can narrow down which architecture is best suited. Imaging fluorescent dyes or proteins is often useful in biomedicine, but since fluorescent emissions are incoherent, architectures that use phase modulation in the illumination path are not good candidates [38]. Willett et al. have previously noted that phase modulation in the imaging path, as is done in DRPE architectures, is more appropriate for coherent imaging [5]. While in theory phase modulation in the imaging path with incoherent light is feasible [39], and hardware implementation has been demonstrated [21], masks with low light throughput are required in order to achieve sufficient diversity of observations for compressive imaging [21]. Therefore, we have chosen to investigate amplitude-only SLM since phase modulation is impractical in many biomedical applications, especially those in low light settings and/or requiring short sensor integration times.

In addition to the amplitude or phase modulating nature of the mask element, the placement of this element within the imaging system can be restricted in some biomedical applications. Amplitude modulation at a conjugate image plane is independent of light coherence state and the same forward model can be used whether the modulation is introduced in the illumination or imaging path [15]. However, in an in vivo setting where trans-illumination is impractical, patterned illumination architectures can be more challenging to implement than using a transparent SLM in the imaging path [27]. With the SLM in the illumination path, the angle of incidence must be considered and stricter requirements are placed on the optics, including restricted F-number and the degree of telecentricity at the object plane [27]. Having the SLM in the imaging path at either a conjugate image or Fourier plane eliminates concerns about angle of incidence by maintaining co-axial alignment. While a lensless CS architecture is appealing for its simplicity, it is not well suited for miniaturization, which is often useful for in vivo imaging. Miniaturization requires reducing the distance between the mask and the sensor which in turn decreases the achievable resolution [40]. Furthermore, noise is amplified at higher spatial frequencies in lensless architectures [33]. Finally, the mask-based architectures used in compressive light field imaging have shown to be capable of slight resolution improvements [37], but this method is most commonly used to allow digital refocusing [41], rather than achieving better resolution than the imaging sensor.

Based on the considerations discussed in this section, we chose to compare compressive imaging with image-plane coding (Fig. 1(a)) to Fourier-plane coding (Fig. 1(b)) to establish which architecture is the stronger candidate for use in biomedical imaging applications where sensor size is limited. Both of these architectures are theoretically capable of using magnitude only modulation and are not ruled out by the miniaturization restrictions discussed earlier. As previously mentioned, coherent and incoherent light is used in various biomedical applications, so we present results under both coherent and incoherent illumination.

3. Methods

In this section, we describe the mathematical forward models for IPC CS imaging and FPC CS imaging under coherent and incoherent illumination. We identify some practical hardware limitations and discuss how we compensate for them in our models. We describe the design and characterization of our experimental platform, which consists of separate IPC and FPC arms and we provide detailed discussion of SLM alignment methods for each arm. We conclude this section by showing that each arm exhibits the same baseline image quality, enabling a direct comparison of IPC and FPC CS imaging performance to be made.

3.1 Mathematical models

In this section, we derive separate forward models for the IPC and FPC arms. For each architecture, we condense the corresponding model to the format of a traditional CS problem (y = Ax) [42], which we solve for a vector representation of the object (x), using the known measurement matrix (A) and vectorized measured observations (y). The components of the measurement matrix differ depending on the architecture and illumination used. We conclude this section with a figure that illustrates the key components for each of our models.

3.1.1 Image plane coding

The model for CS imaging with an image plane mask is effectively the same for coherent and incoherent illumination. The object intensity X is modified by mask $M_i$ to produce a pointwise modulated version M_i◉X of the object intensity, where ◉ denotes elementwise multiplication of entries. This modulated version of the object is then imaged onto the low-resolution sensor plane to provide an observation $Y_i$ with resolution lower than the original object. This is because multiple mask elements are mapped onto each sensor pixel in the sensor array. Under perfect system alignment, $Y_i$ depends only on the object, the mask pattern, and the mask element to sensor pixel ratio d, which represents the number of mask elements that are mapped onto a single pixel at the sensor plane. In mathematical terms, if the object intensity is represented by the matrix X, the mask by the $\sqrt{n}\times \sqrt{n}$ matrix $M_i$, and the observation by the $\sqrt{m}\times \sqrt{m}$ matrix $Y_i$, then the forward model for an ideal IPC system with an element-to-pixel ratio $d=n/m$ can be written as:

In practice, optical aberrations, physical limitations in hardware, and misalignments between the mask and sensor all affect the measured observations. We previously showed that accounting for these non-idealities within the system model improves the quality of CS reconstructed images [8]. This is done by replacing the matrix H with a system specific convolution kernel in the forward model (Eq. (1)). More complex models for incorporating real-world imperfections can be formulated, but the model put forth here is fairly simple and can provide substantial improvements in the final reconstructions [8].

Under ideal conditions, H is a $\sqrt{d}\times \sqrt{d}$ matrix, as noted before. In practice, however, H should be selected as a larger square matrix to account for contributions from any number of mask elements to individual sensor pixels. Our earlier report also outlined a method to calculate H for an IPC system by using training data collected with the imaging platform and objects of known intensity [8]. In the study reported here, H is estimated as a non-negative Gaussian low-pass filter whose parameters (size and variance) are selected empirically. Specifically, various parameter values are used to obtain reconstructions from real image data using a few observations and then the parameters that produce the best results upon visual inspection are selected. This method of selecting the H matrix produces results of similar quality to an H calculated using the method outlined in [8] for a system with well-aligned optics.

The model in Eq. (1) can be written in terms of vector entities to arrive at a form that is more recognizable for CS reconstruction methods [8]:

3.1.2 Fourier Plane Coding With Coherent Light

The preceding discussion for IPC also applies to the FPC arm, with the major difference being the mask placement at a conjugate Fourier plane. Mathematically, the model for FPC with coherent light can be written as:

With the forward model put forth in Eq. (4), we are outside the domain of traditional CS imaging because the observations made ($y_i$’s) provide only magnitude information, with phase information lost due to the magnitude operator. Thus, phase information must be recovered in addition to solving for the unknown object intensity. Hence, CS reconstruction for FPC imaging with coherent illumination requires phase retrieval coupled with the reconstruction in order to recover the unknown object intensity.

3.1.3 Fourier Plane Coding With Incoherent Light

In an incoherent Fourier optics setting where light is linear in intensity, the point spread function of the system is determined by the mask and can be expressed as ${\left|{\mathcal{F}}^{-1}\left(M_i\right)\right|}^2$ [5]. Therefore, the forward model for the FPC arm under incoherent light can be expressed as a convolution of the magnitude-squared of the inverse Fourier transform of the mask with the object:

Figure 2 provides a visual representation for the main mathematical components of each forward model discussed above. The positions of lenses in each architecture correspond to the diagrams in Fig. 1 where lens L₁ converts the object (X) to the spectral domain and lens L₂ converts back to the image domain. The relay lenses image the modulated scene onto the sensor that captures a downsampled observation (Y). Note that the type of mask (M) used in the FPC model with incoherent light is not a random binary mask. The reasons for this choice are outlined in Sec. 3.2.

 

Fig. 2 Illustration of CS forward models for a single observation. (Top row) Illustration of our IPC model as defined in Eq. (1). (Middle row) Illustration of our FPC model with coherent light as defined in Eq. (3). (Bottom row) Illustration of our FPC model with incoherent light as defined in Eq. (5).

Download Full Size | PDF

3.2 Mask design

We performed software simulations of each architecture with different types of mask patterns to determine which to explore in hardware. For this purpose we chose the commonly used “Cameraman”, “Lena”, and USAF 1951 images as our test images. We perform all the simulations in Matlab by using the mathematical models of the imaging architectures described earlier. We pass the test images through these mathematical models, with ideal and non-ideal H matrices, and in the presence and absence of noise. The masks that produce better quality reconstructions, as dictated by having the lowest mean squared error, are the ones chosen for our experiments with the hardware setup.

Random binary masks are commonly used for CS imaging because they provide sufficient observational diversity, meaning that multiple observations of the same object differ significantly for different masks. Our simulations corroborated this behavior with the IPC model and therefore binary masks were used for all IPC results in this study.

For FPC with incoherent light, random binary masks are poorly suited because the model requires taking the magnitude of the mask’s Fourier transform (see Sec. 3.1.3). For random binary mask, the magnitude of their Fourier transform is effectively a two-dimensional delta function because the central DC component is several orders-of-magnitude larger than all other components. This, combined with the non-negativity constraint intrinsic to an incoherent setting causes observational diversity to decrease as the number of non-zero elements in the mask increases [21]. However, in practice, a minimum number of non-zero elements must be included in the mask to achieve sufficient light throughput and signal-to-noise ratio at the sensor [21]. This dictates that the masks suitable for FPC with incoherent light should have some notion of sparsity in order to achieve the observational diversity necessary for CS, but not so much that the light throughput is insufficient to produce enough signal at the detector. Therefore, we performed simulated experiments in Matlab using random masks (Fig. 3, top row), sparse random binary masks, masks with gratings of different frequencies and at different angles, masks with rings of varying widths and varying radii, and finally masks resembling Gabor filters (Fig. 3, middle and bottom rows). For random binary masks (Fig. 3, top row) the image domain equivalent of the mask that is convolved with the scene is close to an impulse, so there is low observational diversity from mask to mask. Gabor-like masks with few small impulses (Fig. 3, middle row) have high sparsity and maintained sufficient observational diversity. However, pixel intensity of observations is low due to the low light throughput of the masks. Increasing the throughput of Gabor masks (Fig. 3, bottom row) renders the image domain equivalent closer to an impulse and reduces the observational diversity across different masks, but is necessary to produce higher intensity observations It is important to balance this tradeoff when selecting masks for incoherent FPC imaging. From a practical standpoint, the masks that provided ample light through with some observational diversity were the masks resembling Gabor masks.

 

Fig. 3 Masks for FPC with incoherent light. (Top row) Random binary masks displayed at the Fourier plane have corresponding image domain equivalents with very large DC components, which reduces observational diversity across different masks. (Middle row) Low-throughput Gabor masks contain few non-zero elements, which reduces the DC component in relation to other components and increases diversity of observations. (Bottom row) High-throughput Gabor masks behave closer to random binary masks and result in lower observational diversity. Images are scaled for visualization.

Download Full Size | PDF

For FPC with coherent light, phase masks are traditionally implemented using liquid crystal devices. However, as noted in Sec. 2, our design constraints lead us to focus only on amplitude modulation. These masks modulate the light amplitude in the Fourier domain and our simulations suggest they can provide ample observational diversity for exploiting CS principles. In summary, the results presented throughout this paper use random binary {0,1} masks with a 50/50 pass/block ratio for all IPC results and for FPC with coherent light. For FPC with incoherent light, we use Gabor filter masks.

3.3 Experimental platform

We assembled a benchtop platform consisting of a common illumination arm and object stage, separate IPC and FPC arms and a third “witness camera” arm for ground truth imaging (Fig. 4). We use two identical digital micromirror devices (DMDs) (Texas Instruments, DLP® LightCrafter 6500 Evaluation Module) as SLMs that display two-dimensional 8-bit mask patterns. The DMDs are positioned at a conjugate image plane and a conjugate Fourier plane in the IPC and FPC arms, respectively. Two flip mirrors are used to direct light from the object through one of the three arms. Both CS imaging arms consist of identical lenses aligned in a 4f system where the focal length of L₁ and L₂ is 150 mm (Thorlabs, AC254-150-A). Infinity-corrected microscope objectives L₃ (Olympus, UPLFLN 10X2) image each modulated projection of the object onto a 14-bit CCD array with 1384 × 1036 pixels, each 6.45 μm square (Point Grey Research, GRAS-14S5M-C). The witness arm contains a CMOS sensor with 3856 × 2764 pixels, each 1.67 μm square (The Imaging Source, DMK 24UJ003).

 

Fig. 4 Photo of the benchtop compressive imaging platform displaying beam paths for the IPC arm (solid line) and the FPC arm (dashed line). The high-resolution “witness” camera takes ground truth images of the object for comparison to computationally reconstructed images (dotted line).

Download Full Size | PDF

We use three separate light settings for trans-illumination of the object: incoherent broadband, incoherent narrowband, and coherent. To generate an incoherent broadband light setting we use a cold white LED (Thorlabs MCWHL5). This produces negligible chromatic aberrations in the IPC arm, but the discrete, tilting mirrors of the DMD act as a blazed diffraction grating when placed at a conjugate Fourier plane [43]. This behavior results in a laterally blurred image at the camera under broadband illumination as different wavelengths exit the DMD at different angles, making FPC CS difficult under broadband incoherent illumination. However, FPC can still be evaluated under spatially incoherent light by reducing the bandwidth of the source, minimizing the dispersive behavior of the DMD. For a region of 960 × 960 mirrors, the DMD acts as a blazed grating of N = 1920 lines with the highest power image appearing in the m = 6th diffraction order. A light source with a center wavelength of λ₀ = 633 nm and bandwidth of Δλ = λ₀/mN = 0.055 nm or greater would lead to significant dispersion, resulting in image features blurred along the direction of mirror tilt [44]. Therefore, to generate an incoherent narrowband light setting we illuminate objects with collimated light from a single-mode fiber-coupled HeNe laser (λ₀ = 633 nm, Δλ = 0.0006 nm) and transmit the beam through a spinning ground glass diffuser (Thorlabs, DG10-600-MD). This diffuser is simply removed to generate a coherent light setting.

3.4 System alignment

In a compressive imaging system, accurate alignment of the coded mask is necessary for peak performance of CS reconstruction algorithms. This can be difficult to attain in practice because mask elements in the SLM are demagnified at the sensor plane, making them smaller than the camera pixels. As a result, the low resolution image sensor is unable to resolve the high resolution mask elements, eliminating the ability to directly visualize mask elements on the camera for feedback. To address this challenge, we developed separate alignment methods for the IPC and FPC arms to ensure that the mask displayed on the DMD is laterally centered with the CCD (Fig. 5). Each DMD is mounted on a 3-axis translation stage to permit fine positioning. To align the IPC arm we first turn on only the central 4x4 mirrors in the DMD. We then acquire an image with the IPC arm to capture the CCD response. Under perfect alignment (Fig. 5(b)) with an element-to-pixel ratio of d = 4, the 4x4 mirrors should image exactly onto 2x2 CCD pixels. Thus, when the DMD is well aligned with the sensor array, the CCD response shows the center 4 pixels white and all others mostly black (Fig. 5(b)). If the DMD and CCD are misaligned (Fig. 5(a)) then the CCD response exhibits asymmetry for the center pixels, which can be reduced by adjusting the 3-axis positioning stage. Note that in the CCD response of the “well-aligned” system (Fig. 5(b)) the pixels adjacent to the center 4 pixels are not perfectly black. This is an example of the “leakage” due to system specific imperfections discussed in Sec. 3.1 and which we account for with the H term in our model.

 

Fig. 5 Alignment techniques for IPC and FPC imaging. (a,b) For IPC alignment, a central group of 16 mask elements is set in the “on” position and the DMD is translated until light is primarily confined to the central 4 CCD pixels (b). (c,d) For FPC alignment, a central group of mask elements are set in the “off” position and the DMD is translated until the mask produces a high-pass filter effect in the acquired image (d).

Download Full Size | PDF

The same alignment method cannot be used for the FPC arm because mask elements are not directly imaged onto the CCD. Rather, the mask acts as a spatial filter, passing and blocking certain Fourier components. Under coherent illumination, the Fourier transform of the object is visible at the Fourier plane and the DMD can be used as a high-pass filter by blocking low spatial frequencies at the center of the Fourier transform. As a result, the image captured by the CCD will be dark with bright edges. We used coherent illumination and a transparent chrome-on-glass USAF 1951 resolution target to align the FPC arm of our system. We turn on all except the center 4 mirrors of the DMD, which are then aligned to block the DC component, resulting in a high-pass filtered image captured by the CCD in our FPC arm (Fig. 5(d)). If the DMD is misaligned (Fig. 5(c)), the image shows edges that are not highlighted as strongly and a background shifted to higher values. The system maintains alignment for incoherent illumination when the diffuser is removed.

3.5 System characterization

In order to make a direct comparison of IPC to FPC for CS imaging architectures, we used the same hardware components in each arm. In this section, we present images taken without using coded masks or application of CS algorithms to show that each system arm exhibits the same baseline resolution for a given illumination. These “no mask” images represent imaging with a conventional, non-CS imaging system and are taken with each DMD set to reflect light from all mask elements (essentially acting as a plane mirror). For all “no mask” images, we used a region of 120 × 120 CCD pixels to image a portion of the 1951 USAF resolution target with a field of view (FOV) of 7.3 mm × 7.3 mm. The ground truth images taken with the witness camera show the same field of view with a higher pixel density to visualize the true scene for comparison. Witness camera images provide evidence that the additional details resolved using our CS method do in fact match the true details of the tissue sample. Therefore, the witness camera is used solely for evaluation purposes and does not represent a “gold-standard” image for a specific application.

The top row in Fig. 6 shows images taken under coherent illumination. Vertical line profiles of measured pixel intensities across the group 3 elements (depicted by arrows) are provided in the right-hand column. Figure 6(a) is the high-resolution (2880 × 2880 pixel) ground-truth image from the witness camera. Note that the corresponding line profile (black) clearly resolves the line pairs within each element of group 3. When no CS is used, the raw images (120 × 120 pixels) taken with the FPC arm (Fig. 6(b)) and the IPC arm (Fig. 6(c)) are qualitatively similar, but with lower resolution than the ground truth image (Fig. 6(a)). Their corresponding line profiles (blue for FPC and red for IPC) show that the resolution is limited by the CCD pixel size in each arm. The resolution limit of both CS arms is similar under coherent illumination, each resolving three peaks for element 2 in group 3, which corresponds to a feature size of 55.68 μm. Our FPC arm resolved these features with 54.7% contrast and the IPC arm resolved the same features with 74.9% contrast. This experiment established that the two CS imaging arms achieve the same baseline imaging performance under coherent illumination. The middle row of Fig. 6 provides the corresponding images for incoherent narrowband illumination. Again, we see that images from the IPC and FPC arms are similar to each other but of lower quality than the ground truth image, with line profiles resolving three peaks for the 55.68 μm features in group 3 element 2. With incoherent narrowband illumination, our FPC arm and IPC arm resolve these features with 20.4% and 23.4% contrast respectively. Under incoherent broadband illumination (Figs. 6(g)–6(i)), the dispersive behavior of the DMD (see Sec. 3.3) results in blurring along the direction perpendicular to the tilt axis of the DMD in the FPC configuration (Fig. 6(h)). Due to this specific limitation arising from the use of a DMD as the masking device, we elected not to compare IPC to FPC for CS under incoherent broadband illumination. We will continue using incoherent broadband illumination in our IPC system arm for the following experiments as an additional example of IPC compressive imaging. The line profile for this configuration shows the 55.68 μm features in group 3 element 2 are resolved with 47.8% contrast.

 

Fig. 6 “No mask” images of a 1951 USAF resolution target with the witness camera, FPC, and IPC arms under coherent and incoherent illumination. The arrows in panels (a-c) indicate the origin of the intensity line profiles extending vertically across the group 3 elements for each image. Scale bar in (a) represents 1 mm for all images.

Download Full Size | PDF

Modulation transfer function (MTF) curves were also generated for each arm of the system by calculating the contrast for each line pair element in the images shown in Fig. 6. Third order polynomial fits were generated for each MTF data set (Fig. 7). For the witness arm (black curves), the image contrast is worse for incoherent narrowband illumination than the other two light settings because the spinning diffuser does not completely remove the laser speckle pattern [45]. Nevertheless, The MTF curves in Fig. 7 show that under coherent and narrowband incoherent illumination the IPC and FPC arms exhibit comparable imaging performance, allowing a direct comparison of computational imaging in these two architectures. The incoherent broadband curves (dot-dashed lines) are included to demonstrate the “best case” scenario for IPC with incoherent illumination.

 

Fig. 7 Plots of the modulation transfer function (MTF) for each system arm; witness arm (black), IPC arm (red), FPC arm (blue). Line style indicates the illumination setting; coherent (solid), incoherent narrowband (dashed), or incoherent broadband (dot-dashed).

Download Full Size | PDF

4. Results

We compared our two CS imaging architectures to each other and to the common post-processing image enhancement technique of bicubic interpolation. For both CS imaging arms, masks with m = 480 × 480 elements were used with a region of n = 120 × 120 CCD pixels used for acquiring observations with a 7.3mm × 7.3 mm FOV. CS reconstruction algorithms using 100 masked observations then generated images with N = m = 480 × 480 pixels, maximizing the resolution of reconstructed images. For the IPC arm, we used a reconstruction algorithm based on Nesterov's Proximal-Gradient (NPG) method [46]. This algorithm was chosen because it includes the non-negativity constraint required for our CS reconstruction problem, and it results in high-quality image reconstruction [8]. The FPC arm under coherent light requires phase retrieval to be coupled with the CS reconstruction. For this reason, we used the Reshaped Wirtinger Flow Algorithm [47]. Finally, for the FPC arm under incoherent illumination, we used the TwIST method [48], because it is less prone to error than the NPG method when working with low signal-to-noise observations (see discussion in Sec. 3.2). As for the computational time, for a maximum of 20 iterations NPG and TwIST both take around 4 minutes on average for reconstruction when using 100 observations. RWF for coherent FPC takes around 8.5 minutes on average when run for 20 iterations with 100 observations. While these reconstruction times do not enable real-time processing of images, it should be noted here that optimizing the reconstruction algorithms for these architectures is not something we have focused on in this report. The purpose here is to use an appropriate off-the-shelf algorithms and run them in Matlab on a standard PC (Windows 10 64-bit, Core i7-3770 @ 3.40 GHz, 16GB RAM). It has been documented that implementing these algorithms on GPUs can speed up computational time [49], but our main objective for this report is to test each CS architecture in hardware rather than optimize the reconstruction process.

Figure 8 presents results from both CS arms under incoherent illumination. The bicubic interpolation images in the top row (Figs. 8(a), 8(d), and 8(g)) are generated by post-processing the “no mask” images in Fig. 6. Examining the CS reconstructions in the middle row (Figs. 8(b), 8(e), and 8(h)), it is clear that the IPC architecture under both narrowband (Fig. 8(e)) and broadband illumination (Fig. 8(h)) recovers more detail than the respective images post-processed with bicubic interpolation (Figs. 8(d), and 8(g)). Specifically, the zoomed-in region around group 3 on the USAF target shows that the smaller line pairs and numbers that are not discernable in the “no-mask” or bicubic interpolation images are much clearer using CS reconstruction. Furthermore, the associated line profiles (Fig. 8(c), 8(f), and 8(i)) show quantitatively that the IPC CS reconstructions (red lines) can resolve all elements in group 3, including the 35.08 μm group 3 element 6 with 51.4% contrast for narrowband and 56.7% contrast for broadband incoherent illumination. At its limit, the IPC arm resolves up to the 19.69 μm features in group 4 element 5 with 18.8% contrast for narrowband and 29.8% contrast for broadband incoherent illumination. In comparison, we found the same system using bicubic interpolation (blue lines) instead of CS can only resolve up to the 55.68 μm features in group 3 element 2 with 23.4% contrast narrowband and 47.8% contrast for broadband incoherent illumination. Therefore, our IPC compressive imaging architecture is able to resolve features that are nearly one-third the size of those that the same hardware without CS is able to resolve. Additionally, the lower baseline contrast that our incoherent narrowband illumination displayed in Figs. 6 and 7 is also seen in CS reconstructions when comparing IPC under narrowband (Fig. 8(f)) versus broadband (Fig. 8(i)) incoherent illumination. Reconstructions with broadband illumination have slightly higher contrast and more defined features than narrowband illumination.

 

Fig. 8 CS reconstructions vs. bicubic interpolation under incoherent illumination. Top row: Bicubic interpolation. Middle row: CS reconstructions. Bottom row: Line profiles across the group 3 elements. Outlined in red are digitally zoomed regions with arrows indicating the origin of vertical line profiles across the group 3 elements; blue corresponds to bicubic interpolation and red to CS reconstruction. Scale bar in (a) represents 1 mm for all images.

Download Full Size | PDF

Based on both visual and quantitative assessment, there appears to be no clear benefit to using FPC CS imaging over bicubic interpolation (Fig. 8(a)–8(c)). CS imaging results from the FPC arm did not resolve any USAF target line pairs beyond those resolved by bicubic interpolation, with CS reconstructed images no closer to the ground truth image in Fig. 6(d). These CS reconstructions resolve the 55.68 μm group 3 element 2 features with 16.5% contrast whereas the bicubic interpolation resolves the same features with 20.4% contrast. These data suggest that using a coded amplitude mask at a conjugate image plane improves CS image quality more than using the same masks at a Fourier plane, when incoherent illumination is used. The FPC architecture is less effective in part because we are limited to using amplitude-only masks. Specifically, achieving sufficient light throughput requires using amplitude masks with many non-zero elements, which, as noted in Sec. 3.2, makes the CS inverse problem more difficult to solve due to insufficient diversity between observations. Furthermore, we believe that any slight mismatch between how we mathematically model this system and the actual physical setup can lead to poor reconstruction quality. Specifically, we have noted that even minor misalignments cause image quality to degrade quickly.

Results are shown in Fig. 9 for CS reconstructions under coherent illumination. As discussed in Sec. 3.1, the mathematical model for IPC is the same for coherent and incoherent illumination. Consequently, with IPC we see a similar result to the incoherent case shown in Fig. 8, where the CS reconstruction resolves more detail than bicubic interpolation (Fig. 9(d)–9(f)). The CS reconstructions resolve the 35.08 μm features in group 3 element 6 with 23.6% contrast. While the line profile for the IPC CS reconstruction (Fig. 9(f), red) show improved resolution with all elements in group 3 resolved, the results provide just above half the contrast that IPC CS reconstruction with incoherent light provide for the same line pair elements (Fig. 8(f), and 8(i), red). This is potentially related to edge ringing with coherent light. Because the lenses, mirrors, and the DMD itself each have finite extent, the light passing through the system is band limited, meaning that the highest spatial frequency components of the object are cut off. Under coherent illumination, this causes the intensity near sharp edges to appear as a sinc function, with ringing artifacts around edges. Additionally, reflections from curved and flat lens surfaces can lead to interference patterns under coherent illumination (see for example Fig. 6(a)), which may also contribute to lower contrast in the IPC CS reconstructions.

 

Fig. 9 CS reconstructions vs. bicubic interpolation under coherent illumination. Outlined in red are digitally zoomed regions of the group 3 elements with arrows indicating the origin of vertical line profiles; blue for bicubic interpolation and red for CS reconstructions. Scale bar in (a) represents 1 mm for all images.

Download Full Size | PDF

In contrast to IPC, the FPC forward model differs for coherent and incoherent light. Using the model for FPC with coherent light outlined in Sec. 3.1.2, we see that CS reconstructions with our FPC arm are, again, no better than bicubic interpolation. In fact, CS reconstructions appear noisy with substantial artifacts. The line profiles show these CS reconstructions only resolve up to the 55.68 μm group 3 element 2 features with 30.2% contrast. We believe the poor quality of these reconstructions is primarily due to limitations in the phase retrieval algorithm because our numerical simulations of this architecture also failed to generate substantially better images than bicubic interpolation.

To further evaluate the IPC and FPC compressive imaging architectures, we examined whether the number of coded mask elements mapped to each camera pixel (d) influenced the quality of CS reconstructions. For this, we used a region of 960 × 960 DMD mirrors to generate three different size masks: (i) 960 × 960, (ii) 480 × 480, and (iii) 240 × 240. For (i), each individual DMD mirror served as a single mask element. For (ii), groups of 2 × 2 DMD mirrors were binned (assigned the same value) and acted as a single mask element. For (iii), groups of 4 × 4 mirrors were binned. For all mask sizes, the same 120 × 120 pixel region of the CCD was used to acquire images, resulting in element-to-pixel ratios of d = 64, d = 16, and d = 4 for configurations (i), (ii), and (iii), respectively. CS reconstruction images were generated with the same dimensions as the mask size. The data presented in Fig. 10 are for incoherent narrowband illumination only, but similar trends were seen for incoherent broadband and coherent illumination. CS reconstructions with the FPC arm (Fig. 10, top row) appear to become less pixelated as d increases, but image quality and detail does not improve. For all three mask sizes evaluated, the FPC architecture did not produce images of higher quality than bicubic interpolation.

 

Fig. 10 CS reconstructions with different numbers of coded mask elements mapped to each camera pixel, for the FPC and IPC arms under incoherent narrowband illumination. Mask sizes of 240 × 240, 480 × 480, and 960 × 960 mapped to a 120 × 120 pixel region on the camera correspond to mask element to pixel ratios of 4, 16, and 64, respectively.

Download Full Size | PDF

For the IPC arm, more detail was recovered, but the average image intensity became lower as d increased (Fig. 10, bottom row, left to right). The digitally zoomed view of group 3 elements shows that the numbers and line pairs appear sharper as mask size increases. This is because the individual mask elements are smaller, offering higher resolution modulation of the object and recovery of smaller details. However, this improved resolution comes at the cost of image brightness, as demonstrated by the darker image produced from IPC CS reconstruction with 960 × 960 masks. All masks used have the same light throughput regardless of mask size, but individual observations through masks with larger numbers of elements appeared to exhibit reduced contrast. Large masks have more individual elements mapped onto each individual camera pixel, so the resulting average of random black or white elements trends toward a more homogeneous grey. This in turn reduces the diversity between observations, making the CS inverse problem more difficult to solve. Low diversity makes it more difficult for CS reconstruction algorithms to converge, which manifests as an overall darkening in reconstructed images. While the images could be digitally brightened in post-processing for visualization purposes, this would not yield additional detail and would also make artifacts and noise more pronounced. Masks designed to avoid this “greying” could result in brighter reconstructed images in future studies.

Finally, we demonstrated the feasibility of applying these CS imaging principles to a biomedical problem by imaging an H&E stained histopathology section of human colon tissue (Fig. 11). We used the IPC arm of the system with incoherent broadband illumination because that combination was found to produce the best CS reconstructions for the USAF target object (Fig. 8). The same 120 × 120 region of sensor pixels was used with d = 16. A color camera (Point Grey Research, GRAS-14S5C-C) attached to a benchtop microscope was used to capture a ground truth image of the tissue section. We stitched together several smaller FOV images of the sample to construct the color image displayed in Fig. 11(a). This image provides visualization of the small tissue details for comparison. Figure 11(d) shows a digitally zoomed region of this image with individual crypts visible (the round features with purple stained borders). Imaging the same area with the low resolution (monochrome) camera in the IPC arm of our system without CS showed a loss of detail even when bicubic interpolation is used (Fig. 11(b)). Furthermore, individual crypts could no longer be distinguished (Fig. 11(e)). Using the IPC CS architecture with the same sensor resolution, we recovered much of the lost tissue detail (Fig. 11(c)), with individual crypts resolved (Fig. 11(c), and 11(f)).

 

Fig. 11 Application of IPC CS to imaging biological tissue. (a) H&E stained histopathology section of the colon with digitally zoomed region (d). Scale bar in (a) represents 1 mm. (b) Image of the same tissue section from the IPC arm after bicubic interpolation, and corresponding digital zoom (e). 480 x 480 pixel IPC CS reconstruction from 120 x 120 pixel observations (d = 16) with corresponding zoomed region (f). All images were captured with incoherent broadband illumination. Scale bar in (a) represents 1 mm for images (a) – (c).

Download Full Size | PDF

5. Conclusions

Two imaging architectures based on the CS framework were evaluated experimentally in a direct head-to-head comparison. Results indicated that the image plane coding architecture is more effective for reconstructing high resolution images than Fourier plane coding, under both incoherent (Fig. 8) and coherent illumination (Fig. 9). There are multiple potential explanations for these findings. Examination of our forward model for FPC under coherent illumination shows that the mask and object field are convolved prior to the magnitude operation. Solving the inverse problem thus requires knowledge of the phase. Adding a phase retrieval operation on top of solving the CS inverse problem makes the system even more difficult to solve accurately and can lead to lower quality reconstructions. Using a focal plane array sensor that can physically measure both amplitude and phase information, or employing better CS-phase-retrieval algorithms could potentially improve reconstruction accuracy of FPC under coherent illumination. In fact, our simulations indicated that with the availability of accurate phase information, the CS reconstruction can converge to a higher quality image.

When implementing FPC in an incoherent light setting (which is linear in intensity rather than complex amplitude), phase retrieval is no longer necessary. However, with the incoherent FPC model, there is a tradeoff between light throughput and observational diversity, as discussed in Sec. 3.2. Because the object intensity is convolved with the mask at the Fourier plane, the mask must have relatively few non-zero elements in order to generate observations with sufficient diversity [21]. The use of coded masks with a small number of non-zero elements results in good diversity between observations, but poor light throughput, which can degrade the quality of reconstructed images. One reason that the FPC architecture with incoherent illumination was unable to substantially improve image quality over bicubic interpolation could be that there was not sufficient observational diversity. Since our simulations of this architecture showed achieving higher observational diversity is possible if masks have very high sparsity, future studies could evaluate such masks experimentally. However, we found that the majority of light was blocked due to the low throughput of high sparsity masks. This limitation is especially concerning since we intend to further develop a CS architecture for biomedical applications where low light settings are common.

We found that the IPC CS architecture was able to resolve 19.69 μm features, whereas a conventional imaging method without CS using the same hardware was limited by the size of CCD sensors and could only resolve up to 55.68 μm features (Fig. 8). Noticeable artifacts present in reconstructed images using our IPC CS architecture are (1) that CS reconstructions are generally darker than bicubic interpolation images and (2) that CS reconstructions appear to have a textured noise pattern throughout. Both of these artifacts are due to limitations of the reconstruction algorithm. Improved CS algorithms that reduce these artifacts would improve final image quality for this IPC CS architecture.

With the IPC architecture, we found that increasing the number of elements in the modulating mask allows CS algorithms to reconstruct images with higher resolution, but at the expense of overall image brightness (Fig. 10). Beyond imaging the standard binary USAF test target, using IPC with 16 mask elements mapped to each sensor pixel, we showed that 480 × 480 pixel images of biological tissue can be reconstructed from a 120 × 120 pixel sensor, resolving features that were indistinguishable when the same hardware is used for conventional imaging (Fig. 11).

Future investigations should involve evaluating different optimization-based CS algorithms and modified models to better match the experimental system in order to reduce reconstruction artifacts. Optimizing mask design could minimize the number of masks needed to reconstruct images with the desired level of detail, which will potentially assist toward moving these CS-based systems to video rate imaging.