1. Introduction

Fourier ptychography (FP) is a computational imaging technique that offers an effective way to improve the resolution of an imaging system. After acquiring multiple unique images of an object of interest, FP then computationally synthesizes a high-resolution image reconstruction in the Fourier (i.e., spatial frequency) domain. To effectively improve image resolution, FP must convert the multiple intensity images into an estimate of the complex-valued sample, which includes its absorption and phase properties. Unlike holographic approaches, which typically rely on interference with a known reference beam, FP instead solves for the missing phase with an iterative reconstruction algorithm. Accordingly, the technique does not require the reference beam or much additional optical hardware, which makes it particularly well-suited for applications within standard imaging systems.

Over the past 6 years, FP has evolved from its first demonstration within a microscope [1] to become a general technique that is now applied in a wide variety of setups and scenarios. In this review, some historical context of FP is first provided, along with a short table of benefits to highlight its importance. Section 2 contains a simple explanation of FP, along with a mathematical description using Fourier optics. The resolution and the space-bandwidth product (SBP) in an FP system are then discussed, along with some practical considerations about experimental implementation. Section 3 defines the basic properties of an FP system and then draws parallels to other computational imaging techniques such as ptychography, structured illumination microscopy, and synthetic aperture imaging. Section 4 presents a summary of advances in various aspects of FP such as its illumination, phase measurements, aberrations recovery, speed improvements, tomographic capabilities and connections to machine learning. Finally, Section 5 offers a perspective on future directions, along with links to some FP resources.

1.1 Historical context of Fourier ptychography

Before delving into its details, it is helpful to place FP within a historical context. As its name suggests, FP has close roots with a technique termed ptychography (pronounced with a silent ’p’), which is a computational imaging method initially proposed in 1969 by Hegerl and Hoppe for solving the phase problem in electron diffraction measurements [2,3]. It was further developed by Rodenburg and colleagues several decades later, who extended its capabilities to image continuous objects and also applied it to X-ray and optical imaging setups [4–8]. Its unique advantages for X-ray microscopy were first clearly demonstrated in 2007 [9] and modern computing hardware has provided a huge boost to performance, spreading the field of ptychography to also include illumination wavefront sensing [10], tomography [11], partially coherent [12,13] and reflection modalities [14,15], for example, along with the development of a large suite of robust algorithms [10,16–20].

While the concept of FP can trace its roots back to a reciprocity relationship established in 1994 by Landauer and Rodenburg for electron microscopy [8], the technique did not find a successful experimental implementation until approximately two decades later. Unaware of this prior work, in 2013, Zheng et al. uncovered the benefits of ptychography’s mathematics for enabling whole slide microscopic imaging without any moving parts and demonstrated its experimental importance [1]. The simplicity of Zheng et al.’s first FP experimental setup helped the technique gain scientific and commercial interest from a wide variety of research areas. Its relation with ptychography has likewise helped in knowledge transfer and quick acceleration of related research.

1.2 Summary of benefits

The recent rapid development of FP has uncovered a large number of benefits for high-resolution imaging. As noted above and detailed in Section 3, FP shares a close relationship with ptychography, so the two techniques thus offer multiple shared benefits - the ability to create gigapixel scale images, and to measure phase without any interferometric measurements, for example. The experimental implementation of FP, however, is quite different from ptychography leading to a number of unique benefits for FP as compared to standard optical microscopy. As we will describe in detail in later sections, FP’s ability to transfer the experimental complexity to computation using low numerical aperture (NA) optics and angularly varying illumination, often provided by an LED array, lead to the multiple advantages summarized in Table 1.

Table 1. Summary of FP benefits.

View Table

2. Fourier ptychography basics

The natural question that arises when one comes across literature about Fourier ptychography for the first time is, "well, how exactly does it work?" While there are now a number of possible ways to experimentally implement FP, we consider here the first demonstration that was carried out within a standard optical microscope [1] to explain the basic principles of operation, which remain similar across different experimental implementations.

2.1 Image capture setup

In the first experimental demonstration of FP [1], the only new hardware required was a relatively inexpensive array of light emitting diodes (LEDs), which was placed several centimeters beneath a thin, semi-transparent sample. This sample was then imaged by a standard microscope using a low-NA objective lens. We note here, up front, that this standard microscope was not outfitted to image fluorescence, but instead to form an image with the same optical radiation that was used to illuminate the object of interest, from the LEDs. To begin, we will assume that each LED acts as a small point source, and that we turn on just one individual LED in the array to illuminate the sample for the subsequent image formation. Given that the sample of interest contains many small sub-micrometer features at the scale of the wavelength of light, the sample will diffract the incoming light into a large cone (i.e., a distribution of wavevectors, see Fig. 1–2).

 

Fig. 1. Fourier ptychography explanation: (a) Diagram of a standard microscope, where a Köhler-type illumination, consisting of many mutually incoherent emitters, is provided by an extended light source. Only light emanating from a single such emitter is shown. Some of the light is absorbed and the remaining light is scattered (i.e., diffracted) by the object and the imaging system records only a part of this scattered light, resulting in a blurry image. Cameras can only detect intensity, so the phase information of the object is also lost. (b) In Fourier ptychography, a movable point source generates a plane wave of light to illuminate the object. FP uses an array of these point sources to generate a wide array of varying plane wave illumination angles. These point sources are turned on time-sequentially and one image is captured per illumination angle. When the object is illuminated with a plane wave at a higher angle, the scattered light that is normally missed by the imaging system now passes through the objective. Images that capture only the scattered light have a dark background (darkfield) and the images that capture both scattered and unscattered light have a bright background (brightfield). FP combines multiple images of both types using a reconstruction algorithm to form a high-resolution complex-valued result.

Download Full Size | PDF

 

Fig. 2. Spatial-frequency sampling in FP: (a) In a standard imaging setup, the limited acceptance angle ($\theta$) of the lens defines a bandpass filter in the Fourier domain that limits the detected image resolution. (b) Tilting the illumination plane wave by an angle $\phi$ shifts the sample’s Fourier spectrum, which is equivalently described by a shifting lens bandpass filter. (c) Fourier ptychography uses an LED array to provide multi-angle illumination to capture information from many segments of the Fourier spectrum via a shifting bandpass filter.

Download Full Size | PDF

This large cone will almost always exceed the maximum angle of acceptance of the imaging lens, whose half-angle we denote as $\theta$. As one might recall, the resolution of an imaging system is specified by the angle of acceptance of the imaging lens. Specifically, the minimum resolvable distance between two points within a coherent imaging system is proportional to $\lambda /n\sin {\theta }$, where $n\sin {\theta }$ is the lens NA and $n$ is the index of refraction of the medium, which we assume here is air ($n=1$). It should be noted that in an incoherent system (such as one with Köhler illumination or in fluorescence microscopy) the resolution is half of this value. If the lens can detect all of the light scattered by a sample of interest, then it will be able to resolve features down to one optical wavelength in coherent systems, whereas if the lens can only accept a smaller cone of light scattered from the sample, its resolution will be proportionally lower. While one could improve the resolution of the resulting image by selecting a lens that can accept a larger cone of diffracted light (i.e., with a larger NA), the field of view (FOV), depth of field (DOF), and space-bandwidth product of high-NA lenses is typically smaller than lower-NA lenses [34,57].

2.2 Angularly-varying illumination

To maintain a large FOV and DOF, FP uses a low-NA lens for image acquisition, but sequentially captures different segments of the large cone of diffracted light by acquiring multiple images. Each image is captured with the sample under illumination from a unique angle, $\phi$, (i.e., with an angled plane-wave) as diagrammed in Fig. 2(b). If the sample is thin (i.e, obeys the projection approximation [58]), then illuminating the sample from an angle $\phi$ will cause the cone of diffracted light emerging from the sample surface to rotate across the aperture by $\phi$ and a new cone segment will pass through the fixed imaging lens and then onto the image sensor. By shifting the point illumination source to many different positions, it is possible to ensure that a large fraction of the diffracted cone of light passes through the lens and onto the image sensor, albeit at different points in time (i.e., within different images). To effectively capture images of the sample under illumination from a large number of angles, Fourier ptychography sequentially illuminates the sample from different LEDs within the inserted array, and then computationally merges this data into a single image that appears to have passed through a "synthetic" lens, whose effective size can extend across the entire cone of diffracted light to offer a much higher resolution.

2.3 Fourier optics formulation

It is relatively straightforward to model this process mathematically using Fourier optics. Neglecting various proportionality constants for simplicity, we can describe the thin sample’s transmission and phase shifting properties via a complex function $s(x, y)$, which is the unknown quantity that we would like to reconstruct. Here $(x, y)$ denote 2D sample plane coordinates. The sample’s cone of diffracted light at the lens back focal plane is proportional to the Fourier transform of this function, $\hat {s}(f_x, f_y)$, where $(f_x, f_y)$ denote 2D spatial frequencies. Under normal illumination, the light transmitted through the lens and focused to the image plane is proportional to $\mathcal {F}[\hat {s}(f_x, f_y)a(f_x, f_y)]$, where $a(f_x, f_y)$ denotes the amplitude transfer function of finite radius, which effectively acts as a bandpass filter and $\mathcal {F}(\cdot )$ is the Fourier transform operation. If we tilt the incident illumination by an angle $(\phi _x^j,\phi _y^j)$, then the diffracted cone of light at the back focal plane, termed the spectrum, will shift across the lens aperture by a proportional amount to create $\hat {s}\left (f_x-\frac {sin(\phi _x^j)}{\lambda }, f_y-\frac {sin(\phi _y^j)}{\lambda }\right )$, where $\lambda$ is the wavelength of the light and $j$ is the illumination sequence number. Here, $(\phi _x^j,\phi _y^j)$ represent the angle of the LED subtended at the sample plane. By inverse Fourier transforming this quantity to the image plane, we can thus approximately model the $j$th detected image as

2.4 Computational reconstruction

If we measured the complex field of these angular illuminations at the image plane with an interference setup [59,60], then the process of high-resolution image reconstruction from the captured data would be relatively straightforward. One would simply inverse Fourier transform each detected field, place the resulting spatial frequencies in their resulting shifted bandpass filter location in Fourier space, tile all of the bandpass filter locations together into a large composite, and then inverse Fourier transform the tiled composite into the high-resolution result. Unfortunately, standard image sensors can only detect optical intensity, not phase, which necessitates another strategy.

Luckily, there are computational techniques to solve for the unknown phase of a signal from multiple intensity measurements. The process of phase retrieval, which dates back many decades [61–63], has been applied to a wide variety of scenarios where "phase diversity" is created between each measurement [64]. Phase diversity refers to the availability of various forms of information that jointly constrain the phase of a complex signal. In FP, phase diversity is created by forming and capturing the multiple images of the same object under angularly varying illuminations. For successful image reconstruction, there must be a certain amount of redundancy between the measurements. As diagrammed in Fig. 1 this redundancy may be designed into the experiment by ensuring that the sample spectrum shifts less than the lens aperture diameter (i.e., a fraction of the width of the bandpass filter between each image capture). In practice, the shift distance is set at slightly less than one half of the aperture diameter, which implies that a final high-resolution image with $N$ pixels will require the overhead of capturing at least $2N$ measurements with the image sensor, which is often set to $3N$ or more [27,65,66].

Once one has captured redundant FP image data, there are a wide variety of standard alternating projection-type phase retrieval reconstruction algorithms available to produce the high-resolution result [61,67–69]. Many of the insights into FP’s first algorithms were extended directly from those created by the ptychography community [10,16,70]. An overview of the FP reconstruction process is shown in Fig. 3, which is quite similar to the ptychography reconstruction process, as detailed in Section 3.

 

Fig. 3. FP reconstruction algorithm follows an alternate projection type scheme. A $j^{th}$ low-resolution image estimate is generated from a high-resolution image guess. True amplitude of this low-resolution image is known, hence corrected and the phase is left unchanged. This updated low-resolution image is then used to update the high-resolution image guess. This process is repeated for all the LEDs several times to achieve a good reconstruction. There is an overlap between two adjacent illumination angles, which encourages the convergence of this algorithm.

Download Full Size | PDF

 

Fig. 4. FP increases the space-bandwidth product (SBP) of microscopes. (a, b) FP can achieve the same resolution as a 20X 0.5-NA objective lens just by using a 2X 0.08-NA objective lens, but with multi-angle illumination extending to $NA_{\textrm {ill}}=0.42$. FP system has a wider FOV, therefore larger SBP, and extended DOF with longer working distance. (c) Example gigapixel-scale FP reconstruction using a low-NA objective. Similar high-resolution images can be obtained with higher-NA objectives, but over a much smaller FOV (marked as circles).

Download Full Size | PDF

Since its first demonstration, a number of alternative reconstruction algorithms have been proposed to improve the reconstructed image quality and speed. A careful presentation of different ptychography algorithms and their advantages and benchmarks are outside of this review’s scope, but may be found elsewhere in the literature [19,71–74]. Readers interested in learning about the reconstruction process in more detail should refer to [75] which provides detailed steps along with some Matlab tutorials.

2.5 Resolution

The final resolution of the reconstructed image is proportional to the width of the synthesized bandpass filter in Fourier space. By examining the bandpass filter plot in Fig. 2, it is geometrically clear that the total width of the synthesized bandpass filter is equal to the sum of the imaging system’s objective lens numerical aperture, $NA_{\textrm {obj}}$, and the numerical aperture of the maximum angle of illumination, $NA_{\textrm {ill}}=\sin {\phi _{\textrm {max}}}$ (assuming the sample is in air). A useful measure of resolution $R$ achieved by an FP system is thus given by

2.6 Space-bandwidth product

A primary advantage of FP is its ability to improve the image resolution of large-FOV optics, which typically exhibit a low NA as shown in Fig. 4. After improving the resolution by a factor of up to 5-10 in each dimension over a large FOV, a resulting FP reconstruction can contain up to 1 billion pixels (1 gigapixel). The number of resolvable pixels by an imaging system (i.e., the imaging system’s FOV divided by its half-pitch resolution in two dimensions) is generally referred to as its space-bandwidth product (SBP) [77].

As can be inferred from above, the SBP of an FP system can be in the order of gigapixels; however, the ability of FP to increase the SBP does not come for free - the technique must acquire multiple images of the sample, typically over time, which thus decreases temporal imaging resolution. Likewise, as noted previously, FP also requires a redundancy in the Fourier space, resulting in a measurement overhead of 2-3X (at least). This additional data allows FP to return both the amplitude and the phase at every pixel in the reconstruction, and provides added computational degrees of freedom that can be used to estimate and remove lens aberrations [31], correct for setup misalignment [78], and perform multi-modal reconstructions [79], for example. This additional measurement overhead is a function of the imaging lens NA and the angular shift of the illumination angle, and can be tuned experimentally by moving the LED array closer to or further from the sample.

Multiplexed and parallelized data acquisition methods have been proposed to increase data acquisition speeds, as discussed later in Section 4.4, which introduces the idea of a space-bandwidth-time product (SBTP) [26]. Multiplexed [39], sparse, multi-modal [79] and tomographic FP [48] methods also exploit the large redundancy in FP systems, reconstructing equal or more pixels than the total number of measured pixels across all the captured images. Due to the complexity of such systems, it is not trivial to exactly measure the SBP of many systems, especially in 3D imaging systems where there is a missing cone problem (discussed later in Section 4.5) which makes careful SBP analysis even more complicated. As a simple measure, the total number of reconstructed and assumed resolvable pixels per snapshot is often used to characterize 2D systems, but a more precise measurement strategy remains an open problem for both 2D and 3D imaging systems.

2.7 Practical considerations in Fourier ptychography

To create high-quality FP reconstructions, several experimental parameters must be carefully calibrated. Here we discuss the most common sources of error in FP experiments, and methods to reduce their effects.

1. Stray light is unwanted light in each image that reduces image contrast. Most commercial brightfield microscopes can be operated under room lights, which is not true for most microscopes implementing FP. Experiments should be conducted in a dark room, or the setup should be well enclosed to block any stray room light.

2. Stray reflections contain light from the illumination source (here the LEDs) that reflects from experimental components like the camera casing or optical posts before reaching the image sensor, which can corrupt the image data. To avoid stray reflections, a good first step after building the FP setup is to record a dataset without any sample present. In this calibration step, all darkfield images should be completely dark, as there is no sample to scatter incident light into the collection NA of the imaging optics [80]. Any stray reflection can be easily spotted when the contrast is adjusted for viewing in software such as ImageJ [81]. Black felt or other light absorbent material can help block stray reflections.

3. Dynamic range: In FP, the intensity of the detected images decreases rapidly from brightfield to darkfield illumination. An optimal exposure setting for a brightfield image can thus result in almost no detected signal in a darkfield image. This issue becomes especially problematic when using low-dynamic-range sensors. To mitigate this issue, multiple exposure images can be captured at each unique illumination setting to create a high-dynamic-range image [1], or different exposures can be used for different LED images and the intensities between the images can be re-normalized during reconstruction by converting the images into floating point numbers and dividing by the normalized exposure values [33].

4. LED positioning: It can be difficult to achieve high-precision alignment of optical components, especially in low-cost FP setups, such as when 3D printed enclosures are used [33]. The LED array might not exactly lie in its expected position, and there will likely be deviations due to manufacturing imperfections, especially in custom built illumination modules [21]. Self-calibration algorithms [33,82] can correct for alignment errors in a pre-processing step and simulated annealing methods have been proposed to correct for manufacturing imperfections during FP reconstruction [83,84]. However, efforts should be made to physically align the system as well as possible before these methods are used to correct for minor alignment errors.

5. LED intensity variation: The illumination intensity from different LEDs in the array will vary due to differences in their distances from the sample and their emission profiles, as well as manufacturing imperfections. These effects can be corrected for by measuring the LEDs’ illumination brightness at the sample plane using a photodiode and normalizing each captured image. A method to correct this within the reconstruction process was also proposed [85], which can be used if there aren’t any other sources of error.

6. Partial coherence: LEDs commonly used for illumination in FP are partially coherent sources. To account for their imperfect coherence during image reconstruction, it is helpful to divide the FOV of the captured imagery into small segments and process each segmented FOV independently, before tiling the resulting reconstructions together into a final composite [1]. The segment size should be smaller than the spatial coherence length of the light source at the sample plane, which can be approximated via the Van Cittert-Zernike theorem as $\lambda z/d$, where $z$ is the distance from the LED plane to sample plane and $d$ is the diameter of the LED active area.

7. Pupil errors: Aberrations in the optical system should also be corrected for a good quality FP reconstruction. A detailed discussion on this is provided in 4.3.

8. Vignetting effects: The tube lens aperture of most microscopes typically determines the brightfield FOV in FP. In systems built using singlet lenses [66] this is determined by the lens aperture and the working distance of the system. Between two systems with an identical NA, the system with longer focal length will have larger brightfield FOV. This influences the LED positions and would require a Fresnel diffraction based forward model [86,87] for better reconstruction results.

9. Approximations: The standard FP forward model assumes a thin sample (i.e., applies the projection approximation) and that any deviations in this approximation would result in a failed reconstruction [82]. Example non obvious cases that can cause failure include condensation on coverslips, unintended effects of immersion media, and effects of non-uniform glass slide surfaces. There have been several algorithms proposed to correct system parameters such as aberrations or LED array misalignment. Most of these algorithms are mutually exclusive. For example, attempting to correct for LED position errors (such as due to field-curvature [33]) can alter aberration recovery accuracy. Calibrating these parameters experimentally is preferred for robust FP reconstructions.

3. Parallels with other imaging modalities

Fourier ptychography has similarities with several related computational imaging techniques. In this section, we discuss such similarities and point out key differences that help distinguish it as a unique procedure. Properties of a ptychographic system that help differentiate it from other coherent diffractive imaging methods were previously defined [88]. These definitions are also applicable to FP due to the reciprocity between the two procedures [72]. We restate those properties in the context of FP as follows:

3.1 Connections with ptychography

3.1.1 Imaging by convolution

The term ptychography is made up of two Greek words "ptycho", meaning fold and "graphy", meaning write, which was used to describe the use of convolution - a mathematical process of folding two functions together. In ptychography, a spatially confined light pattern (probe) is used to illuminate the object and the resultant diffraction pattern in the far-field is observed. In the far-field, the Fourier transform of the probe is convolved with the Fourier transform of the object (Fig. 5(a)). The object is translated to several positions and each resultant diffraction pattern is captured, which are then combined using iterative algorithms. In Fourier ptychography, the system pupil (probe) appears in the Fourier plane of the object, which alters the object’s Fourier spectrum, and the resulting band-limited image is recorded. Hence, in the image plane, the Fourier transform of the pupil is convolved with the Fourier transform of the object’s spectrum, which results in a blurred image (Fig. 5(b)). In FP, the object’s spectrum is then translated, typically via tilting the plane wave illumination incident upon the sample, and several band-limited images are recorded, which are then combined using iterative algorithms similar to the ones used in ptychography. Mathematically, the data acquisition and the image reconstruction processes are similar in both cases, with a Fourier transform relationship between the variables of interest being the primary difference [73].

 

Fig. 5. Ptychography word is derived from convolution, hence the relationship between ptychography and FP is explained with the help of convolution. The object and its Fourier spectrum switch places between these two techniques. The object is shifted in ptychography and the object’s Fourier spectrum is shifted in FP accordingly.

Download Full Size | PDF

3.1.2 Experimental differences

Apart from obvious experimental distinctions and their mathematical reciprocity, a key difference with FP is that its imaging FOV is typically divided into multiple small segments and processed independently [72]. The illumination is treated as coherent within each small image segment [27] and this division allows one to correct for spatially varying aberrations [24,31,89]. In ptychography, this type of division has not been demonstrated. At the same time, in ptychography, the object can be scanned to an infinite number positions, in principle, which is only limited in practice by the translation stage. This type of infinite scanning is not possible for FP - the illumination angle cannot be increased beyond $2\pi$ steradians. We note that closely spaced parallel FP systems [29] that image distinct parts of the FOV can be used to increase the FOV when a translation stage is used to scan the sample.

3.1.3 Cross-pollination

While there are numerous physical differences between ptychography and FP, the mathematical similarities between the two techniques have resulted in a significant amount of algorithmic transfer from the these two communities. For example, joint aberration estimation [31] and source position refinement [90] in FP are direct extensions of probe reconstruction [10,16,91] and probe location estimation [17] in ptychography. Other advances in ptychography, such as the reconstruction of mixed states [12] to account for partial coherence, e.g. in bilinear optical systems [92], are still to be translated into FP.

In terms of physical implementation, ptychography has been widely implemented with optics, X-ray, electron microscopy and EUV imaging, and FP has been popular mostly in the optical regime with few demonstrations in the near-infrared [36,93] and X-ray [94,95] modalities. The non-scanning nature of FP due to the use of LED arrays has made FP a popular technique in the optical regime, however, a probe generated using an LED array was recently implemented to demonstrate a non-scanning ptychography in the optical regime [96]. This interesting exchange of experimental concepts has benefited both the fields. Ptychography is currently preferred in X-ray over FP due to experimental challenges such as accurately scanning the illumination angle over a high NA, however, FP provides a complementary imaging solution in X-ray which can be useful in some situations [94]. We expect that the concepts and ideas from these two techniques will inspire a new generation of techniques that combine benefits of each approach.

Since ptychography and FP share such close similarities, a common question that arises is, which of these techniques is better suited for one’s application? While there is no simple answer, looking at their differences might help with a decision. As mentioned above, FP is preferred in the optical regime and ptychography is preferred in X-ray and electron microscopy. This can be due to the fact that FP can be implemented on a commercial optical microscope by simply adding an off-the-shelf LED array. Ptychography typically requires some additional setup. Illumination source arrays also do not readily exist for X-ray or electron microscopy, where a probe illumination and scanning stages with nanometer precision are readily available. Since LEDs can be switched very quickly, sub-second data acquisition times are possible in FP, which is not directly true with ptychography yet. FP can be used to recover additional parameters, such as aberrations of the imaging lens, which can be further used for other applications such as deconvolving fluorescent images, for example. These strengths and shortcomings might, however, change soon with the rapid advancement of these fields.

3.2 Connections with structured illumination microscopy

Structured illumination microscopy (SIM) also offers a reliable means to improve resolution from multiple image measurements [97]. It is most typically used in incoherent imaging setups that are designed to detect the fluorescent emission of microscopic samples at high resolution. To do so, it uses Moire fringes to encode high spatial-frequency information of the object into a lower spatial frequency passband for imaging. Specifically, it projects high-frequency bright/dark stripes onto the sample, whose intensity fluctuates rapidly as a function of position. These stripes are analogous to carrier waves used in radio transmission, typically with a constant spatial frequency near the imaging system’s cut-off frequency. The high spatial frequencies of the object modulate the carrier wave, which results in a shift, as seen in Fig. 6, allowing high-spatial frequency information that was otherwise beyond the cut-off frequency to enter the imaging system.

 

Fig. 6. The cut-off frequency of an incoherent imaging system is twice that of a coherent system with the same imaging optics. Fourier ptychography can be considered as a coherent structured illumination technique under a broad definition of "structuring" the incident plane wave illumination angle.

Download Full Size | PDF

In coherent imaging systems, this modulation can also be achieved by structuring the phase and/or amplitude of the illumination [98–100]. Here, the structured pattern (carrier wave) can have a maximum frequency equal to the imaging system cut-off, similar to incoherent SIM, however, the cut-off frequency of a coherent imaging system is half that of an incoherent system Fig. 6, thus the resolution cut-off of an incoherent SIM system is twice that of a coherent SIM system [101]. SIM reconstruction conventionally uses analytical models to solve for the reconstruction, however, due to their close connection, FP algorithms were recently extended to patterned intensity-modulation in fluorescence microscopy [102]. Following the lines of FP, it was later proposed that a low-NA objective lens can be used for imaging with a high-NA speckle pattern generated using a diffuser placed infront of a high-NA condenser for illumination, resulting in a large SBP system [98,103,104]. Here the condenser illumination covers the entire FOV of the low-NA objective and the diffuser is translated to scan the speckle pattern. Similar work has also mixed SIM for joint fluorescence and coherent imaging [105–108], further speaking to their close connections.

3.3 Connections with synthetic aperture imaging

As noted above, phase-sensitive coherent imaging techniques may also use variable-angle illumination to capture multiple images. To create a higher resolution, these images may simply be linearly combined in Fourier space, effectively via a tiling procedure, before inverse Fourier transforming the tiled composite into the final result. This process is traditionally referred to as synthetic aperture imaging and is widely used in modalities where phase is jointly measured - for example, with radio, microwave and ultrasound [109–111]. There are also demonstrations of these methods in optical wavelengths dating back several decades [59,60,112–117], which are generally referred to as synthetic aperture holographic imaging or microscopy techniques. Such methods typically require highly coherent light, along with a reference beam and phase-stable optics, which prevents easy integration into a standard microscope setup. FP is fundamentally a synthetic aperture technique, the key distinction from traditional techniques being phase recovery, which is done traditionally using the interferometric methods.

4. Demonstrated benefits and recent advances

4.1 High-illumination NA, large SBP systems

From Eq. (2), it is clear that the illumination NA (i.e., the maximum angle of illumination) is an important variable that can be tuned for high resolution, and thus high SBP performance. Early on, it was shown that the combination of high-angle illumination and a relatively large imaging NA can lead to a FP system that offers performance with an NA greater than one [34]. High-angle illumination from a flat LED array, however, can become problematic given that the directionality of LED emission at large angles leads to very low recorded image intensities [118]. To overcome this limitation, several efforts have been made to build hemispherical and quasi hemispherical LED arrays [21,22,36] to provide illumination from across almost a full $2\pi$ steradians. These reported systems have achieved impressive resolutions and SBPs using low-NA objectives. LED illumination units from some of these works are shown in Fig. 7, among them the quasi-Dome LED array is commercially made available by Sci-Microscopy [119].

 

Fig. 7. High-NA illumination systems for FP: (a) Three bowl shaped LED arrays that can achieve illumination NAs close to one in air. (b) Oil immersion condenser was also proposed to provide an illumination NA of 1.2. Such high angle illumination can lead to high-resolution, large SBP image reconstructions. Figures courtesy of [21–23,36].

Download Full Size | PDF

An immersion medium may also be used to increase the illumination NA [23,37], which can also lead to large SBP imaging. One such system is shown in Fig. 7, where an oil immersion condenser coupled with a 10x 0.4NA provides a synthetic NA of 1.6, outperforming the resolution of incoherent images captured by a 100x 2.55 synthetic NA oil-immersion objective commonly used in pathology [23], which is due to the lower contrast in the incoherent system’s optical transfer function. This work offers a compelling example of how FP can provide high-resolution performance, while using low-NA imaging optics. There have also been several novel illumination designs for FP including a ring illuminator for optimal sampling [40,120], laser illumination systems for shorter exposures [40,121], and a waveguide system to provide illumination NAs greater than two [37,122].

4.2 Phase reconstruction applications

Apart from its ability to create large SBP images, a second primary advantage of FP lies in its ability to quantitatively measure phase with a non-interferometric setup. Phase imaging is a critical modality in biomedical imaging [123], wherein many specimens are primarily transparent. It is also useful in metrology, for measuring the topography of 3D surfaces, and in applications requiring digital refocusing or digital aberration characterization, for example. While interferometric phase measurement systems [124,125] can be fast and accurate, they usually require sophisticated optical components, precise alignment and calibration.

By retrieving quantitative sample phase via computational reconstruction from standard microscope images, FP removes the need for complex hardware. In addition, its use of partially coherent LEDs for illumination helps avoid speckle artifacts. As first demonstrated in [25] (Fig. 8), quantitative phase reconstructions from FP agree well with digital holographic measurements. Since this first work, phase imaging with FP has been demonstrated in various biological settings. These include in vitro cell culture studies, and tomographic imaging of C.elegans and cancerous cells [26,35,51,55].

 

Fig. 8. Quantitative phase reconstruction results in FP. Figures courtesy of [25–28].

Download Full Size | PDF

 

Fig. 9. (a) FP can recover and correct for large aberrations within a low-cost imaging system using a mobile phone camera lens as the objective. The image above shows two small segments from the full FOV. (b) FP can also recover and subsequently apply system aberrations for deconvolving fluorescence images. Here, the FP reconstruction phase gradient is overlaid with the fluorescence image for better visualization. Figures courtesy of [24,33].

Download Full Size | PDF

 

Fig. 10. Approaches for faster FP data acquisition. Figures courtesy of [26,44,45,138].

Download Full Size | PDF

FP can also be implemented without LEDs in an aperture-scanning mode, where the aperture is physically moved across the sample’s diffracted spectrum while recording images [27,28,30]. This method can apply to thick samples in either transmission or reflection geometry, while phase recovery provides the ability to digitally refocus and observe features at different depths in these 3D samples (Fig. 8). To further improve the resolution of quantitative phase measurements, a dual-wavelength aperture-scanning based FP method was recently proposed, where a filter was used to select two closely spaced wavelengths from the LED emission spectrum. The resulting phase reconstruction at each wavelength can be used to obtain a phase reconstruction at a synthetic wavelength, which bypasses the need for phase unwrapping and improves the recovered phase accuracy to nanometer-scale precision [28] (Fig. 8). It should be noted that the phase reconstructions are sensitive to misalignments in the imaging system, and thus robust alignment and calibration procedures are required for accurate performance. Several automated algorithms, discussed in Section 2.7, have been proposed to correct for these errors and this is still an active area of research.

4.3 Aberration recovery

Optical aberrations are commonly found in all imaging systems, which, if left uncorrected, degrade spatial resolution and can therefore hinder achievement of the theoretical resolution across the entire FOV (Eq. (2)). While their effects can be minimized with rigorous lens design, lens complexity and cost can quickly balloon [126]. Aberrations are particularly severe when inexpensive lenses are used, or if the FOV being imaged is larger than the lens design specifications. If the effects of aberrations, summarized by the imaging system’s coherent transfer function $a(f_x, f_y)$ in Eq. (1), can be measured, they can be corrected for during FP reconstruction. Following the demonstration of joint probe beam recovery in ptychography [10,16,91], a method termed embedded pupil recovery (EPRY) was proposed in Ref. [31] for measuring and correcting for aberrations within the coherent transfer function from a standard FP data set. This capability has since allowed FP to be implemented with extremely aberrated lenses, such as moldless lenses [47] and mobile phone camera lenses [33] as shown in Fig. 9.

Most imaging systems exhibit shift-variant aberrations that vary in form across the imaging FOV. FP can partially account for shift-variant aberrations by dividing the FOV into multiple small segments and processing them independently. Recovering and removing aberrations for all image segments and then compiling the segments back together allows FP to recover large, high-quality images across the entire FOV. Such recovered shift-variant aberrations can also be used to deconvolve simultaneously acquired fluorescence images [24,93].

EPRY is a nonlinear optimization method and thus can suffer from local minima and stagnation, most notably if aberrations are severe. This issue can be partially addressed by either initializing the iterative solver with an estimate that is close to the expected solution, for example by using a ray-tracing software such as Zemax [42], or by modifying the procedure to recover the difference in aberrations between adjacent sections, which is usually small [33]. Recently, a more robust full-field FP model [127] was proposed to recover the aberrations, where the Zernike coefficients are recovered instead of the entire lens transfer function $a$ from Eq. (1). In general, future research might consider FP as a separate tool for aberration estimate and recovery, whose potential applications can include deconvolution [24,32,93] and adaptive optics.

4.4 High-speed Fourier ptychography

In FP, the sequential illumination of LEDs, the long exposure times for darkfield image capture, and the requirement to capture redundant image information lead to long data acquisition times (few seconds to minutes). Many applications, such as recording videos of cells or other living organisms, require faster temporal sampling rates. There has been significant work towards increasing the speed of FP to open up such application spaces, which we summarize below in three categories and show in Fig. 10.

1. Decreased data redundancy: FP datasets can be up to 10X larger than the number of pixels in the image reconstruction (i.e., have 10X data redundancy). One of the first proposals to improve the speed of FP, demonstrated in Ref. [39], multiplexed the LED acquisition process to greatly decrease both the required data redundancies and image exposure times. By illuminating multiple LEDs at a time for each captured image, it was shown that the incoherent superposition of the contribution of each LED could be separated within a novel reconstruction algorithm. After this demonstration, several strategies were proposed to choose alternative multiplexing LED patterns [128–132], and similar approaches were also later suggested for spectral multiplexing [79,133]. One of the fastest FP systems to date was demonstrated using this multiplexing concept [26]. Other related efforts include LED pattern optimization [120] and the exploitation of sparsity, either in the Fourier domain or within another space, to reduce the number of required images for successful algorithm convergence [134–137].

2. Single-shot FP: There have also been efforts to remove the sequential nature of FP acquisition altogether and instead capture all required data in a single snapshot. The first approach to do so suggested the use of a diffractive grating to separate the light scattered by the sample into several band-limited images that could be captured on a single detector [44,139]. Another approach implemented an experimental configuration similar to a light-field imaging setup, which together with LED multiplexing could provide a single-shot recording for FP [45]. Spectral multiplexing has also been applied for single-shot acquisition [140]. We note that such systems inherently tradeoff either the image FOV or resolution, or both, to achieve FP in a single shot, and thus cannot be expected to increase the system SBP.

3. Multi-camera systems: FP can also be implemented with multiple apertures and sensors to capture different band-limited images in parallel. After demonstrations of FP via time-sequential aperture scanning [27,76], multi-aperture setups were subsequently presented for both far-field imaging [43] and microscopy [42]. Due to the gaps between the cameras, a few different illumination angles are usually required to capture an entire dataset. With the appropriate choice of a multiplexed illumination scheme, it may be possible to create a single-snapshot multi-aperture FP system. To handle the experimental complexities arising from the use of multiple cameras, custom 3D-printed optomechanics may also be used [138]. Parallel FP systems were also developed to image up to 96 well plates simultaneously [29,35], which brings FP closer to applications in cell culture imaging applications.

Finally, it is worth noting that the illumination source intensity must also be considered when examining data acquisition speeds. The limited illumination intensity of off-the-shelf LED arrays at high angles leads to long exposure times, which can be minimized with custom LED arrays [21] or via laser illumination [40,121], the latter of which employed a digital micromirror device for rapid switching between illumination angles.

4.5 Extensions to 3D: Diffraction tomography

With the advent of FP and its success in reconstructing 2D “thin” samples, several works have since extended FP to also reconstruct “thick” samples in 3D using the same simple modified microscope setup, thus revitalizing the field of intensity diffraction tomography (DT) [141–150]. Extension to 3D requires a more sophisticated light propagation model that describes how light interacts with an in-homogeneous 3D medium, relating the 3D object phase or complex-valued refractive index (RI) to the detected 2D intensity. Such models have been applied to DT using holographic or phase-sensitive measurements, and include the first Born and Rytov approximations [151], and the multi-slice approximation (also known as the beam propagation method [152–155]). Recently, FP has been extended to DT with 2D intensity measurements, using the first Born [48–51], the first Rytov [52], and the multi-slice approximations [53–56]. A few selected 3D reconstruction examples are depicted in Fig. 11 under the multi-slice and first Born approximations. It is now clear that even without phase measurements, just as in 2D FP, there is enough information in collected datasets to retrieve the 3D phase computationally using an appropriate phase retrieval algorithm.

 

Fig. 11. Example 3D reconstructions of C. elegans using the first Born model (a) and the multi-slice model (b). Figures courtesy of [51,55].

Download Full Size | PDF

Briefly, the first Born and Rytov approximations provide analytic solutions to the time-independent wave equation under assumptions of weak scattering. Both models describe the 2D field at the detector as orthographic projections of 2D spherical manifolds (Ewald spheres) residing in 3D k-space representation of the 3D object’s scattering potential, which is directly related to its RI. The positions of these partial spherical shells depend on the relationship between the detection and illumination orientations. While in both approximations the scattered field is sampled from Ewald spheres, the difference is that the first Born approximation models the field at the detector as a coherent sum of the incident field and the sample-induced scattered field, while the first Rytov approximation models the field at the detector as the incident field with the sample-induced, complex-valued phase shift. A detailed mathematical treatment of these approximations can be found in Ref. [151,156,157]. The multi-slice approximation, on the other hand, discretizes the sample’s complex RI into a stack of thin 2D layers, for each of which the thin-sample approximation applies. The propagation model involves Fresnel diffraction between successive layers, and can thus model multiple forward scattering but not back-scattering. Certainly, other more sophisticated light propagation models may be used to better model multiple scattering, which is an area of active research [158–162].

It is worth noting that extension to 3D has additional challenges not present in 2D, namely the missing cone problem [163], which produces lower resolution in the axial dimension (Fig. 12). This effect is increasingly severe for lower imaging objective NAs and is not mitigated by increasing the illumination NA. This is in contrast to 2D FP, for which it is very often desirable to use low-NA objectives, which have higher SBPs than high-NA objectives. Thus, the missing cone problem currently hinders 3D FP from obtaining isotropic resolution, and the enhanced SBPs that 2D FP enjoys. However, it has been shown that accounting for multiple scattering can partially fill in the missing cone [154] in DT, highlighting the importance of research in more accurate light propagation models. There has also been extensive research in computationally filling in the missing cone through regularization in tomographic imaging to achieve isotropic resolution that may be applied to 3D FP [89,164–172].

 

Fig. 12. 3D space-bandwidth product (SBP) for various combinations of illumination and imaging NAs. (a) $k_xk_z$ cross-sections of the k-space coverage for various NAs, and the theoretical corresponding $xz$ cross-sections of a reconstructed 0.8-$\mu m$-diameter bead. For low imaging NAs, the illumination NA has a less of an effect than for high imaging NAs. (b) The theoretical 3D SBP for various NAs, calculated as the product of the 3D FOV (assuming arbitrarily a 20-$\mu m$ axial range) and the 3D k-space coverage volume. These calculations were based on specifications from [173]. Figure courtesy of [174]

Download Full Size | PDF

4.6 Deep learning in Fourier ptychography

While the neural network is a decades-old concept, it wasn’t until the advent of accelerated computing and the 2012 landmark paper that demonstrated the superiority of convolutional neural networks (CNNs) in image classification over traditional computer vision techniques [175] that the popularity of deep neural networks (“deep learning”) exploded in numerous fields [176] and FP is no exception. Further accelerating the adoption of deep learning methods are the developments of deep learning frameworks such as TensorFlow [177] and PyTorch [178], Python libraries that have significantly simplified neural network prototyping by using automatic differentiation. A few works have leveraged these software libraries by formulating FP phase retrieval as a gradient descent-based optimization, minimizing the error between the forward prediction and the data [179,180]. One benefit of using a cost-function-minimization-based framework is that it is straightforward to include prior information or reparameterizing the reconstruction as the output of a trained [181] or even untrained neural network [174]. Other works have altogether replaced the iterative phase retrieval algorithm with a neural network that is trained to map the raw intensity dataset to the high-resolution reconstruction, allowing for one-step reconstructions that could decrease the computation time [130,182–184]. Such techniques may be regarded as supervised learning, requiring potentially large datasets, each entry of which is a single raw FP dataset along with its high-resolution reconstruction. Some works have also demonstrated LED multiplexing or compressive sensing to reduce the number of raw measurements needed [130,181,182,184,185]. Finally, a few works have used deep learning to find the optimal illumination pattern for compressive reconstructions [131,186,187] or for application-dependent tasks [188,189]. A comparison among these different approaches is summarized in Fig. 13. Deep learning in FP is a new research direction that has gained a lot of traction in a short time and we expect this will continue.

 

Fig. 13. Comparison of various deep learning approaches in FP. (a) The conventional approach (top) and a gradient-descent-based approach, with the forward model written in an automatic differentiation library. (b) Approaches that train a CNN to learn a mapping between the data and the reconstruction, replacing the forward model. The bottom approach bypasses the reconstruction and optimizes a CNN for a particular task. These approaches may also incorporate optimization of the LED weights in order to design the hardware. (c) These approaches still use the forward model, but reparameterize the reconstruction with a CNN, which may be pretrained (top) or untrained (bottom).

Download Full Size | PDF

4.7 Other implementations of Fourier ptychography

After the first demonstrations of FP in transmission microscopes using visible light LED arrays, a number of new and unique implementations now extend its use to alternative experimental conditions. Examples include the use of reflected light, non-visible wavelengths, and in low-photon conditions. Here, we briefly summarize this work.

To operate FP in a "reflection mode", one can attempt to image the light reflected and/or scattered off a sample’s surface before performing a suitable ptychographic image reconstruction. Such a modality opens up FP for use in metrology, reflective material inspection, and the analysis of thick tissue, for example. Pacheco et al. have reported reconstructing useful FP images using measurements of reflected light [190,191], as has more recent work in Ref. [192]. The use of infrared light, as opposed to visible light, for sample illumination has also been demonstrated [36,93]. Research has also explored the possibility of shifting the illumination to the infrared spectrum as far as the telecommunication band [36,193] for examination of semiconductor material. Finally, FP was also recently demonstrated in the X-ray regime [94], bringing the technique closer to the experimental arena in which ptychography is commonly applied today.

FP has also been studied in regimes where the expected number of detected photons is low. For example, Aidukas et al. proposed a method using heralded imaging (HI) to capture information regarding quantum correlations [194] to enable accurate reconstruction with a limited photon budget. Moreover, FP techniques have also been proposed and adapted to single-pixel microscopy for improved detection sensitivity [195,196]. Furthermore, FP was also recently applied to enhance the resolution of coherent anti-Stokes Raman scattering (CARS) images [197]. We expect FP to slowly broaden in general format as it finds new applications in such unique experimental arrangements.

4.8 Applications and future directions

The unique set of benefits offered by FP makes it an attractive technique for several applications. The following is a non-exhaustive list of such potential applications.

1. Digital pathology and hematology: With an ability to rapidly image at high resolution over large area, and provide digital refocusing, FP has the potential to impact digital pathology. While several preliminary works demonstrate feasibility [198,199], clinical studies are needed to validate the technique. Eventually, one might imagine simple, compact FP systems within hospitals in remote and resource limited settings that cannot afford expensive whole-slide scanning systems.

2. Automated diagnosis: There is a huge interest in radiology, pathology and other medical fields to automate the process of image-based diagnosis with machine learning techniques. FP offers many advantages over conventional imaging approaches, such as higher throughput owing to an enhanced SBP, along with phase sensitivity, which can help improve the diagnosis accuracy of automated decisions.

3. In vitro studies: High-speed imaging of in vitro cultures is a great application to showcase the potential of phase imaging in FP. Several articles have demonstrated successful high-speed performance [26,50], and we expect measurement speeds to continue to improve in the future.

4. Microfluidics: FP can replace holographic imaging methods used to monitor microfluidics [200], potentially offering the ability to image macroscopic areas at sufficient spatial and temporal resolutions in the future.

5. 3D imaging with multiple scattering: While several works have already extended FP to recover 3D images of thick samples, as noted above, additional efforts are required to better understand and model the effects of multiple scattering, of which several recent works have begun to uncover.

6. Semiconductor wafer inspection: FP’s quantitative phase recovery points to its potential for determining surface topography of silicon wafers or other highly reflective surfaces that do not exhibit much contrast otherwise, for example to assist with defect detection [201,202].

7. Learned sensing: Recently, FP experimental setups were used for a “learned sensing" strategy, wherein the hardware setup itself (e.g., the illumination pattern) and the ML classification process were optimized jointly to improve algorithm performance [187–189,203]. This approach can reduce the required data captured in FP and help build high-speed diagnostic systems.

8. Polarization is a useful imaging modality for applications ranging from biomedicine to geology. To the best of our knowledge, there has not been any demonstration of polarization imaging with FP yet, although there are examples within the field of ptychography [204].

9. Reflection: FP has been demonstrated under reflective configurations [190–192] but there still exists several challenges that need to be addressed to open the door for new applications in machine vision and surface imaging of thick biological samples, for example.

10. Non-optical wavelengths: While FP was recently implemented in an X-ray imaging setup [94], there are alternative wavelength regimes (EUV, Terahertz, infrared) and modalities (electron microscopy) where FP might offer new possibilities.

These future directions for FP demonstrate its versatility as an imaging technique. However, there has still been limited application of FP to date for new scientific discovery. This may be partly due to the young age of the technique. Given the various advances and improvements in FP over the past few years, we hope to see FP used more widely within the experimental realm. A collaborative, open-source platform, such as Open-SPIM (for light-sheet microscopy) [205], could potentially improve the uptake of this technique in more labs and research centers around the world.

5. Conclusion

Fourier ptychography is a simple and versatile technique that has potential applications in wide areas of research and industry. Its ability to reconstruct quantitative phase and gigapixel-scale images at low-cost are among the primary advantages that set it apart from other imaging techniques, but new and interesting benefits continue to emerge (e.g., 3D image capture, aberration removal, and novel compact optical arrangements, for example). This article summarized various aspects of FP with an intent to give a quick understanding of the working principle, its relation to other techniques and an overview of its latest advances and future directions.

Due to the rapid progression of the field, there are several things that cannot be explained in detail here so readers are encouraged to visit relevant articles for more information. A short summary of various reconstruction algorithms can be found in the following articles [66,73,74]. Other useful resources for learning about FP include a Matlab tutorial for FP imaging [75], several Ph.D thesis [66,206–210] and there are opensource datasets along with the reconstruction codes available to download at [211,212].