1. Introduction

Ptychography is a form of Coherent Diffractive Imaging (CDI) rapidly gaining in popularity thanks to a simple experimental procedure and robust associated image reconstruction algorithms. In the visible light regime it offers quantitative phase imaging with extremely low noise levels [1]; for X-rays and electrons it also removes the need for (relatively poor) imaging optics [2–4]. A ptychographic experiment involves translating a specimen through a grid of positions and recording at each position the diffraction pattern that results from the specimen’s interaction with a localized coherent illuminating beam. A spacing of the translation grid that is a fraction of the diameter of the beam ensures that a given region of the specimen is illuminated at several specimen positions, thereby introducing redundancy into the recorded data. Structuring the illuminating beam so that a given region of the specimen is illuminated in a different way at each position introduces diversity into the data.

The redundancy within a ptychographic data set provides robustness to the image reconstruction process, whilst the inherent diversity of the data provides a particularly rich source of supplementary information [5–16]. A crucial early example of this richness came with the realization that both the specimen transmission function and the illuminating probe wavefront could be reconstructed from the diffraction data [3, 5, 6]. This made ptychography practical and straightforward. Subsequently, a string of results have shown that within the ptychographic data set lies sufficient information to extract super resolution data [7] and the coherent modes of a partially coherent probe [8], to recover diffraction data lost to shot noise [9] or due to the presence of a beam-stop [10], to separate contributions from a number of probes of different wavelengths [11], to correct errors in the measurement of the specimen positions [5, 12–15] and to divide the reconstructed image of a thick sample into a number of axial sections [17,18]. This last point we will address further here, showing that as many as 34 sections through a thick specimen can be reconstructed using a single ptychographic data set. We will also show how a conventional microscope platform can be adapted to collect the necessary data and reduce the section separation to around 2 μm for penetration depths of up to 150 μm.

Ptychography has been combined with rotational tomography to realize high contrast 3D imaging in the X-ray regime [19], but the success of this method relies on the applicability of the projection approximation, which allows back projection of ptychographically reconstructed images taken at a series of specimen rotation angles. In our method the propagation and scattering of the probe as it passes through a thick sample is taken into account, so that strongly scattering, thick samples can be imaged. Since the technique does not require staining or fluorescent makers, and the illumination intensity required is on the order of 10 nWμm⁻², our work has important applications in the field of label-free live cell imaging [1, 20]. We also envisage it finding applications when combined with rotational tomography to image thick samples with coherent X-rays, and in the electron regime where the majority of samples scatter strongly.

2. Description of the method

Two-dimensional ptychography models the interaction of a transmissive specimen with a localized probe wavefront as a multiplication. This means that at each of the j = 1...J positions within a specimen translation grid, the intensity, I_j(u), of the wavefront incident upon a detector placed a distance d downstream of the specimen is described by:

2D ptychographic reconstruction algorithms solve the inverse problem of determining which specimen and probe, when fed into the forward model of equation 1, would result in the recorded diffraction intensities. However, the multiplicative approximation used in equation 1 is applicable only when the specimen is suitably thin [3,18,21]. As a rough guide, the supplement of [3] derives the following upper limit on the thickness of sample that can be accommodated within the multiplicative approximation:

To increase this thickness limit, the multiplicative model of the specimen-probe interaction can be improved using the multi-slice method well known to electron microscopists [22]. Here the specimen is considered as a series of N slices, O_n(r) (n = 1 ... N), separated by distances z_m (m = 1 ... N − 1), and the wavefront exiting the specimen is calculated using a series of multiply-propagate steps between the slices. For example, in the case N = 3 the j^th recorded intensity may be written

The angular spectrum propagator is used to propagate over the small distances between slices within the specimen, z_m, because this propagator maintains a constant pixel pitch in the axial direction throughout the specimen. The most accurate representation of the propagation of the exit wave to the detector would use the angular spectrum formalism. However, because of sampling constraints which involve the pixel size, binning, and real space size of the probe, the use of the Fresnel propagator is necessitated in the set of experiments reported.

The maximum specimen thickness that can be accommodated by the multi-slice model is increased by a factor equal to the number of slices used, but the model is still approximate because it does not consider reflected components of the wavefront; its accuracy is also dependent upon the accuracy of the propagator 𝒫_z, which will itself depend upon the values of z_m, and the values of a and R [16].

An iterative algorithm [17], similar in principle to those used for 2D ptychography, handles the task of inverting equation 3 given a set of measured diffraction intensities. Each iteration of this algorithm (the 3D Ptychographic Iterative Engine or 3PIE) takes current estimates of P(r) and O_n(r) and uses equation 3 (with the appropriate number of slices) to estimate the wavefronts that were incident upon the detector during the recording of each diffraction pattern. The moduli of these wave-fronts are replaced by the square-root of the corresponding diffraction patterns, and the results propagated back to the plane of the final object slice. A series of update functions then produce new estimates of the probe and the object slices for use in the subsequent iteration: full details of this update procedure are given in [17].

3. Experimental configuration

In this work, the 3PIE algorithm was applied to ptychographic data sets obtained using a modified optical microscope. Figure 1 shows the experimental configuration used for all of the measurements. A λ = 635 nm fiber-coupled laser diode was used as a source of illumination. The collimated beam from this laser was incident upon a 400 μm pinhole covered by a diffuser, which was used to add structure to the illumination and so introduce diversity into the diffraction patterns. The diffuser consisted of two layers of LDPE (low density polythene) which were cut out from a plastic sheet and placed on top of one another with a 90 degree rotation between them to ensure homogeneous scattering. Lenses L1 and L2, of focal lengths 30 mm and 3.1 mm respectively, projected a demagnified image of the pinhole onto the rear of the specimen. The first plane of interest within the specimen was placed a few micrometers beyond this focus, ensuring that the extent of the illumination was well defined and giving a probe diameter of ∼ 75 μm. A motorized x–y translation stage (PI M-686) moved the specimen to positions within a 20 × 20 rectangular grid with a nominal pitch of 5 μm. Each position within the grid was offset by a random component between ±5% in order to prevent raster grid artifacts in the reconstruction [24].

 

Fig. 1 Experimental setup used for all measurements (not to scale). Lenses L1 and L2 project an image of a 400 μm diameter pinhole and diffuser onto the thick specimen which is mounted on an x–y translation stage. The scattered laser light is collected by a conventional optical microscope (lenses L3 and L4). The last plane of the specimen is located at the focal point, f_obj, of the objective lens L3. The CCD is placed at a camera length C_L = 67 mm from the image plane of the microscope, and collects a diffraction pattern from the specimen at each x–y position visited. A typical diffraction pattern is shown at the plane of the CCD.

Download Full Size | PDF

The scattered light was collected by a conventional optical microscope using a 20x objective lens of NA=0.75 (L3), and a 180 mm focal length tube lens (L4). The microscope was equipped with two 14 bit CCD cameras (AVT Pikes) each with dimensions D = (7.6 × 7.6) mm. The first camera was located at the primary image plane of the microscope and was used to accurately position the specimen. In particular, the specimen height was adjusted such that its most downstream plane of interest was in focus on this camera, so ensuring it occupied the front focal plane of the microscope objective. The second camera was attached to a second camera port, and was used to capture the diffraction patterns. It was placed a distance C_L = 67 mm downstream of the primary image plane; calibration measurements using a resolution target gave a M = 21× magnification between this camera and a plane C′_L = 160 μm downstream of the specimen. Consequently this camera and the microscope can be thought of as constituting a ‘virtual detector’, 21× smaller than the actual detector and positioned 160 μm from the specimen.

Each of the 400 recorded diffraction patterns were binned by a factor of two in order to reduce the time for reconstructions and decrease read out noise. The pixel pitch of the reconstructions was calculated according to $\text{dxy}=\frac{\text{M}\lambda {\text{C}}_{\text{L}}^{\prime }}{\text{D}}$, giving a value of dxy = 280 nm,

4. Probe calibration

The convergence and robustness of 3PIE is greatly improved if the initial estimate of the illuminating probe wavefront is reasonably accurate. We thus obtained a good initial probe for our subsequent reconstructions using a calibration sample consisting of two planar layers of polystyrene micro-spheres separated by 40 μm. The spheres in the sample were of diameters 10 μm and 20 μm and were suspended in optical adhesive. Diffraction patterns were collected over a grid of 20×20 positions as described above, and supplied to 3PIE together with an initial probe guess consisting of a mask of ones and zeros forming a circle with a diameter determined by the spatial extent of the illumination. The algorithm was run for 60 iterations using N = 2 to produce a reconstruction of the probe and images of the two layers of micro-spheres.

Figures 2(a) and 2(b) show the resulting reconstructions of the two slices. Here, the hue and color saturation represent the phase and modulus of the reconstructed object respectively. The insets to Figs. 2(a) and 2(b) show the phase profiles across a single bead. Phase shifts of 0.7π and 1.4π are observed for the 10 μm and 20 μm spheres respectively. These values are in good agreement with the values calculated (0.73 π and 1.46 π) using the refractive indices of polystyrene (1.580) and the optical adhesive (1.5568). Figure 2(c) shows the reconstructed modulus of the probe, which was used to seed the reconstruction algorithm in subsequent experiments. For all subsequent measurements, the lowermost plane of interest of the specimen was positioned carefully in the direction of the optical axis so that the incident wavefront matched that of the calibrated probe.

 

Fig. 2 Reconstructions of the complex transfer functions of two planar layers of polystyrene micro-spheres separated by 40 μm. (a) 10 μm spheres in plane one (b) 20 μm spheres in plane two. The insets in (a) and (b) show a cross-sectional profile of the phase of a single bead in each layer. (c) Reconstruction of the complex field of the illumination incident on plane one, namely the probe. The modulus and phase of the three images are represented by the saturation and hue respectively. This is shown with the color map in (b).

Download Full Size | PDF

5. Experimental results

In order to demonstrate the capability of the 3PIE technique, we collected diffraction patterns in the manner described above from a series of standard optical microscope slides. The first was a common Spirogyra algae specimen with a total thickness of ∼ 100 μm. The calibrated probe shown in Fig. 2(c) was incident on the lowermost plane of the algae and was used as the initial estimate of the incident illumination. Initially, the number of planes in the reconstruction was set to two, and the plane separation was set to 100 μm. Sixty iterations of 3PIE were carried out, after which the number of planes was increased to 3 and the plane separation halved. A further 60 iterations were then carried out, after which the number of slices was increased to 6. An additional slice was then added between each existing slice every 60 iterations, until the object totalled 21 slices spaced 5μm apart. Each time new slices were added, all slices were reset to unity. Although the slices of the reconstructed object were not accurate representations of the specimen when the number of slices was small, since their separation was greater than that allowed by equation 2, the process of gradually adding slices assists in the convergence of the algorithm. This we ascribe to the dependence on spatial frequency of the allowable thickness of each object slice: low spatial frequencies are accurately reconstructed when the number of slices is small, whilst progressively better reconstructions of the higher spatial frequencies result as the number of slices increases. The total time taken for the data collection and reconstruction of 21 planes was under 60 minutes using a Matlab implementation of 3PIE accelerated with a GPU.

The modulus and phase of the probe that was reconstructed was found to be very similar to that of the initial probe guess shown in Fig. 2(c). The only noticeable difference being a small defocus (<20 μm) of one with respect to the other. This was a result of the difference in position along the optic axis of the first plane of interest of the specimen with respect to the plane of the calibrated probe.

Figures 3(a)–3(d) show a sample of the reconstructions of the phase, and Figs. 3(e)–3(h) the corresponding modulus of the complex transfer function of a Spirogyra algae which correspond to penetration depths of 65 μm, 50 μm, 40 μm and 25 μm respectively. It is easy to identify features within the helical structure of the Spirogyra such as the central spindle in Fig. 3(d), and the individual cells which make up the outer surface of the helix in Fig. 3(a). The high contrast that can be seen in the phase images in Figs. 3(a)–3(d) illustrates clearly the merits of ptychographic phase imaging, particularly when compared to the modulus images in Figs. 3(e)–3(h). A typical value for the phase shift within the spirogyra cells relative to the background phase shift of zero radians is 0.6 radians. Whereas a typical value of the modulus within the cells is 0.9 (a.u.) (compared to the background value of 1.0). From this we estimate that there is approximately 5 times more signal to noise in the phase images than the modulus images. A weak modulus but strong phase is a typical feature of so called low contrast biological specimens, particularly live cells. An animation of the full stack of 21 phase images can be found in Media 1. Here the full 21 slices in the reconstruction are plotted sequentially and are labeled by their slice number.

 

Fig. 3 Selected slices from a reconstruction of the phase, (a)–(d), and modulus, (e)–(h), of the complex transfer function of a Spirogyra algae specimen. The separations between the slices are 15 μm, 10 μm, and 15 μm for slices (a)–(b), (b)–(c), and (c)–(d) respectively. The color-bars in (a) and (e) indicate the phase (radians) and modulus (a.u.) of the complex transfer function of the specimen relative to the mounting medium (see Media ( Media 1)).

Download Full Size | PDF

In order to better visualize the 3-D structure of the Spirogyra specimen, a volume rendering was performed on the phase images. This involved interpolating the 21 reconstructed slices by a factor of 10 in order to produce a unit voxel with dimensions [dx, dy, dz] = [280, 280, 500] nm. A dynamic threshold was applied to each slice to form a binary mask which was applied to the interpolated data. A series of volume renders of subsets of the masked data are shown in Figs. 4(a)–4(d). Here, phase shift is represented visually by hue. The threshold highlights clearly the strong phase shifts induced by the cells for the renders in Figs. 4(a)–4(c), which show penetration depths of (60 – 80) μm (40 – 60) μm (20 – 40) μm respectively. The threshold for the render (d) is unable to select uniquely the cells which form the outer shell of the helical structure. This render represents a penetration depth of (0 – 20) μm, where 0 μm is the front focal plane of the microscope objective. The reasons for this are discussed in section 6. A rotational view of the full 3-D render can be found in Media 2.

 

Fig. 4 3D volume render of a Spirogyra. Each image represents the total phase shift over 20 μm thick slices within the Spirogyra (see ( Media 2)).

Download Full Size | PDF

Ptychographic scans of a Volvox algae were also collected and reconstructed in the same manner as that described previously for the Spirogya algea. The Volvox algae is a Chlorophyte with a spherical structure and a diameter ranging from ∼ (150 – 500) μm. A small segment of the algae was reconstructed with a total of 33 slices and a slice separation of 4.7 μm. Figure 5 shows a sample of eight selected slices from the reconstructions of the phase of the Volvox algae, with penetration depths ranging from 150 μm to 23 μm. The slices shown are those with the most prominent features, which include the clusters of cells that make up the daughter colonies ((b)–(h)), and the spherical extracellular matrix (a). An animation of the full stack of 33 phase images can be found in Media 3. Here the full 33 slices in the reconstruction are plotted sequentially and are labeled by their slice number.

 

Fig. 5 Selected slices from a reconstruction of the phase of the complex transfer function of a Volvox algae specimen with total thickness 150 μm. The slices (a)–(h) correspond to planes 1, 3, 5, 11, 17, 21, 24 and 28 out of a total of 33. The separation between adjacent planes is 4.7 μm. The color-bar in (a) indicates the phase shift (radians) (see ( Media 3)).

Download Full Size | PDF

In a third experiment, we compared a 3D reconstruction of an Arabidopsis Thaliana plant embryo with a data set collected on a florescence confocal microscope. The embryo was prepared for confocal measurements by clearing the cells and staining the cell walls with propidium iodide to allow for optical sectioning [25]. The leaf tip within the embryo had a maximum thickness of 45 μm and was made up of 6 individual cell layers within the region of interest. Confocal sectioning measurements were made with a 40× microscope objective of NA=0.75. A 2 μm axial step size was used. The label within the cell wall was excited at a wavelength of 543 nm, and the florescence collected via a 560 nm low pass filter.

Figures 6(a)–6(d) show a comparison of selected slices of the reconstructed phase of the leaf tip (upper panel) with the corresponding florescence confocal images in Figs. 6(e)–6(h) (lower panel). The method for data collection and reconstruction was equivalent to that described previously. A total of 34 planes were reconstructed with a plane separation of 2 μm. Shown in Fig. 6 are slices 5, 11, 18 and 28 out of 34, which correspond to penetration depths within the leaf ranging from (10 – 40) μm. Despite the small slice separation, there is an excellent correlation with the confocal data, with very little cross talk between adjacent planes in the 3D ptychographic reconstructions. This can be seen clearly when viewing the animation provided in Media 4. Here the full 34 slices in the reconstruction are plotted sequentially and are labeled by their slice number. Furthermore, Fig. 6 highlights the excellent phase contrast that can be achieved with ptychography. The images in Figs. 6(a)–6(d) are reconstructions of the optical phase shift throughout the whole specimen, whereas Figs. 6(e)–6(h) represent the fluorescence intensity from the fluorophores which are embedded within the cell walls only.

 

Fig. 6 (a)–(d) Selected slices from ptychographic reconstructions of the phase of the complex transfer function of embryonic leaf tip. (e)–(h) The same leaf tip measured on a fluorescence confocal microscope. Individual cells are highlighted in red in each panel in order to aid the comparison. The color-bar in (a) indicates the phase shift (radians). The color-bar in (e) indicates the florescence intensity (a.u.) (see ( Media 4)).

Download Full Size | PDF

6. Discussion

Both the lateral and axial resolutions we obtain in our reconstructed images were limited by several factors. The accuracy of the positioning stage plays a part, as do aberrations introduced by the microscope into the recorded diffraction intensities. (In 2D ptychography these aberrations are, at least in part, accounted for in the form of the reconstructed probe thanks to the multiplicative approximation inherent to the technique, unfortunately this serendipitous situation does not apply to our 3D procedure where the exit wave is not a simple product.) Our use of the Fresnel propagator for ℱ_d, necessitated by sampling constraints and the pixel size of our detector, also limited the accuracy of the high spatial frequency components in our reconstructions.

For all of the 3-D reconstructions presented, there is a degradation in the quality of both the phase and modulus images for the upstream planes. This can be observed in the Media videos, Media 1– Media 4, as an increase in noise of the background. The reasons for this are two-fold.

Firstly, the high NA provided by lens L2 in Fig. 1 results in a large divergence of the probe. This is a requirement of the 3D method, since the large range of illumination angles this high NA provides improves the obtained axial resolution. However, the high NA also results in an increasing probe area relative to its window size of 512 × 512 pixels. For example, the probe shown in Fig. 2(c) occupies approximately half of its 512 × 512 window size. As this probe propagates through the specimen, its size increases such that it eventually fills the entire reconstruction window, causing a degree of aliasing at the probe extremities. This effect can be minimized by focusing the projection of the aperture by lenses L1 and L2 in Fig. 1 onto the central plane of the 3-D specimen. However, for the experimental setup described here, the divergence of the probe restricts the maximum obtainable penetration depth to approximately 200 μm, and is dependent on the refractive index of the mounting medium and the specimen. This number can be increased by reducing the NA of the illumination, but this is at the cost of reduced axial resolution. For example, penetration depths of up to 1 mm were shown in [17] using a 10x objective lens to form illumination with NA = 0.25, and slice separations of 200 μm.

Secondly, the degradation of the image quality for the uppermost layers is induced by a sequential transfer of cross talk between the slices that are reconstructed. Any inconsistencies between the measured diffraction pattern and the reconstruction of the complex transmission functions of slices in the 3-D object propagate sequentially though the algorithm from the lowermost layer to the uppermost layer. This is particularly a problem for areas of the specimen with low spatial frequency variations, because the slice separation is insufficient to remove these features. As a result, inconsistencies are propagated through the slices within the reconstruction, and these grow in size as the wavefront is propagated. This can be seen clearly in Media 1 and Media 3. For example in Media 1, the overall cylindrical outline of the Spirogyra, with low spatial frequency variation, propagates throughout the reconstruction and degrades the image quality for the uppermost slices. This can also been seen in the rotation view of the 3-D render in Media 2, where this effect results in an apparent flattening of the cylindrical structure.

Another feature of the 3D reconstructions is the decrease in phase contrast from the uppermost layers to the lowermost layers. For example, the phase shift in the ptychographic reconstructions of the cell walls of the Arabidopsis Embryo decrease with penetration depth. This can be seen by comparing the phase contrast of Figs. 6(a) and 6(d) which represent penetration depths of 40 μm and 10 μm respectively. Here, the median phase shift observed in the cell walls relative to the background is −0.06 radians and −0.19 radians for slices in Figs. 6(a) and 6(d) respectively. Similar effects are observed for the other data sets. The strength of the decay in phase contrast with penetration depth increases with increasing number of slices and decreases with the number of iterations. Further work is required to investigate the minimum number of iterations required to remove this effect for a given number of slices in a reconstruction.

As a consequence of the limitations discussed above, the results for the phase shifts that are obtained from the reconstructions are not quantitative, and are dependent on the spatial frequency variation of the refractive index of the specimen, the number of slices that are reconstructed, the level of redundancy in the dataset, and the number of iterations of the algorithm. In future work, an investigation of these parameters will be carried out using an engineered calibration specimen with well known spatial variation of refractive index features. Despite these limitations, 3PIE allows optical sectioning that is comparable to conventional florescence confocal microscopy, and has potential applications in the removal of out of focus features in 2D ptychographic imaging of low contrast structures. Furthermore, 2D ptychography has been shown to produce accurate quantitative phase maps, and therefore the current limitations should not prove to be fundamental.

7. Conclusion

We have demonstrated in this paper how a form of 3D ptychography, based on an algorithm called 3PIE, can be implemented on a conventional microscope platform and can produce optically sectioned images of relatively thick samples on the micron scale. We have also demonstrated an increase in the number of sections, or slices, imaged using this method from 5 to 34. Our 3D ptychographic method is unique in that it does not require rotation of the specimen with respect to the illumination source, or any translation of the specimen in the axial direction. Our work has potential applications in high contrast and stain-free live cell imaging, as well as in the imaging of thick, strongly scattering specimens using X-rays and electrons.