Open Access
Add to BibSonomy
Abstract
As the resolution of X-ray tomography improves, the limited long-term stability and accuracy of nanoimaging tools does not allow computing artifact-free three-dimensional (3D) reconstructions without an additional step of numerical alignment of the measured projections. However, the common iterative alignment methods are significantly more computationally demanding than a simple tomographic reconstruction of the acquired volume. Here, we address this issue and present an alignment toolkit, which exploits methods with deep-subpixel accuracy combined with a multi-resolution scheme. This leads to robust and accurate alignment with significantly reduced computational and memory requirements. The performance of the presented methods is demonstrated on simulated and measured datasets for tomography and also laminography acquisition geometries. A GPU accelerated implementation of our alignment framework is publicly available.
© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Computed tomography (CT) and laminography (CL) methods enable 3D imaging and thus make it possible to acquire statistically relevant information about a sample’s internal structure without its destruction. This allows for unique imaging applications broadly applied in material research, medicine, and industry, especially for in situ and ex situ studies. In particular, in combination with high-resolution X-ray microscopy, 3D nano-imaging is important for many applications in biology and materials science [1]. Two-dimensional sub-50 nm spatial resolution can be commonly reached in X-ray microscopy [2–7] for sample thickness of up to tens of microns. However, this high 2D resolution can be translated to comparable 3D resolution only if a sufficient number of measured angular 2D projections of the sample provide mutually consistent information. If the measured 2D projections are misaligned, the reconstruction quality deteriorates and the achievable spatial resolution is limited [8,9]. Since reaching the level of accuracy and mechanical stability required for sub-30 nm 3D resolution is currently not achievable and would drastically increase the complexity of the measurement device, alignment of the measured 2D projections during post-processing is an important step.
In the past decades, various methods for alignment of tomography projections were proposed, such as marker-based methods [10–13], cross-correlation [14–17], bootstrap methods [18–26], common-line [9,27–29], and feature matching [30,31]. However, each of the methods has some drawbacks that hinder their application for high-accuracy alignment of large X-ray datasets. The marker-based methods, often used in electron tomography, require significant effort during the sample preparation and the subsequent marker detection. Furthermore, while for thin samples prepared for electron microscopy a marker is clearly visible in the projection, this is often not the case for majority of X-ray tomography samples. The cross-correlation-based methods suffer from accumulation of subpixel errors resulting in poor sensitivity to misalignment between angularly distanced projections. The common-line methods such as the center-of-mass method [28,29] for estimation of the horizontal shift and the vertical-mass-fluctuation (VMF) method for estimation [28,29] of the vertical shift are limited to the parallel-beam tomography with conventional geometry and cannot be used for more general 3D imaging geometries such as cone-beam tomography, laminography or interior tomography [20,32,33]. Finally, the projection matching alignment (PMA) methods [18–26] provide a very general way of projection alignment. However, PMA applications are often limited by their high computational demands and their convergence may depend on quality of the initial guess.
Here, we describe a framework for alignment of 3D imaging datasets commonly produced by nanotomography, nanolaminography, and interior nanotomography. Our approach is also well suited for other high-resolution 3D imaging methods such as vector and tensor tomography [34–37] and non-rigid tomography [38]. The goal of our toolkit is to provide means for fast, robust and fully automated alignment of large datasets, which are collected in modern synchrotron 3D nano-imaging experiments.
2. Projection alignment strategy
With respect to the broad range of challenges met for high-resolution 3D imaging, the ideal alignment method needs to exhibit high robustness, low computational cost and high alignment accuracy. In order to meet these requirements, we have based our alignment strategy on algorithms with deep subpixel accuracy. The goal is to develop an approach that is able to align projections downsampled to a significantly reduced spatial resolution and simultaneously reach sufficient subpixel accuracy that allows for direct alignment of the full resolution dataset. This approach differs from the conventional pyramidal method [18,19], where the low-resolution alignment is used only as an initial guess in order to improve robustness, but the full resolution alignment step needs to be still performed.
Solving the alignment problem in a lower resolution leads to a significant reduction of the computational time. For example, the algorithmic complexity of alignment of $M$ projections by the cross-correlation-based methods is proportional to $\mathcal {O}(MN^{2} \log N)$ and $\mathcal {O}(MN^{3})$ for the projection matching methods, where $N$ denotes the linear size of the reconstructed volume. Additionally, solving the alignment task in a lower resolution reduces memory requirements, improves angular sampling of the tomographic dataset according to the Crowther criterion [39], and generally increases the average SNR in the downsampled projections. Therefore, as we will demonstrate, our approach provides excellent results even for undersampled datasets with poor signal-to-noise ratio (SNR).
In order to provide maximal alignment robustness under various experimental conditions, our alignment strategy is based on the combined strength of three alignment methods as shown in Fig. 1. After loading the projections, an initial alignment is performed using the cross-correlation alignment (XCA) method. The next alignment step is provided by the vertical-mass-fluctuation (VMF) common-line method. If assumptions of this method are satisfied, it provides a very fast and accurate vertical alignment independent of any horizontal misalignment. If the assumptions are not satisfied, e.g. in laminography, or interior tomography configurations, this prealignment step is skipped. The final alignment is then provided by our implementation of the multi-resolution projection-matching alignment (MR-PMA) method.
3. Common contrast mechanisms in nanotomography
The combination of good penetration and high achievable resolution makes the hard X-ray wavelength range a common choice for nanotomography imaging. X-ray nano-imaging commonly relies on either of two contrast mechanisms: absorption or phase-shift. Due to broad variation of 3D tomography tasks, our alignment methods were designed to be used with either of these imaging modalities.
A consequence of the good penetration length of the hard X-ray photons is weak absorption contrast. The complex-valued transmission $T(x,\;y)$ can be approximated for thin samples as
However, phase-tomography has several major differences compared to the conventional absorption-based approach. Firstly, whereas the attenuation $P_A(x,\;y)$ can be easily calculated from the measured complex-valued projection as
the phase is not uniquely determined by the measured complex-valued transmission $T(x,\;y)$. If the sample is sufficiently weakly phase-shifting, so that the maximal relative phase-shift $\phi (x,\;y)$ is smaller than 2$\pi$, $\phi (x,\;y)$ can be recovered directly from the complex-valued projection as However, if the phase-shift exceeds $2\pi \;\textrm {radians}$, phase unwrapping has to be performed [40] and a unique solution exists only for well-behaved projections, i.e. when the signal to noise ratio is high and the phase evolution is smooth without phase residua or vortices [41].Additionally, each phase projection $\phi (x,\;y)$ can contain an arbitrary phase offset $\phi _0$ and if the projections are reconstructed by ptychography [42], they also include an arbitrary linear phase ramp [8,9]. As we discuss in Section 8, these degrees of freedom become constrained once the aligned projections are adequately assembled in 3D space and thus they do not affect the final imaging quality. However, this limited reliability motivated us to develop alignment methods that are insensitive to these degrees of freedom and other potential systematic errors in the lowest spatial frequencies.
4. Initialization by cross-correlation alignment (XCA) method
Alignment of the adjacent projections by the cross-correlation alignment (XCA) method [14–16] is commonly used in cryo-electron tomography and a similar idea was tested for alignment of X-ray datasets as well [17,24]. Assuming that the average translation between two ideally aligned projections acquired at adjacent angles should be zero, an estimation of the relative shift between the adjacent projections can be used for alignment.
The XCA method can be used with projections that are real- or complex-valued, for generality we denote here the projections that we seek to align $T_\Theta (x,\;y)$. Instead of simply estimating the mutual displacement between the raw projections, our implementation estimates the shift between the local variance fields ${V}_\Theta (x,\;y)$ of the projections defined as
Since XCA serves only as an initial guess, we use the computationally cheaper central differences for calculation of the gradients of the measured field $T_\Theta$ instead of an FFT-based approach.Since $T_\Theta$ denotes a generally complex-valued field, our definition of ${V}_\Theta$ does not require any phase unwrapping. This definition has also other advantages, for example, it makes the XCA insensitive to phase-residues, additional linear phase ramp, and potential errors in the lowest spatial frequencies of ${T}_\Theta$. Although the XCA method can significantly reduce the positioning error between projections acquired at adjacent angles, it is insensitive to a constant offset in the center of rotation. Assuming that the rotation axis is perpendicular to the illumination beam propagation direction and that the full angular range of 180 degrees is acquired, XCA between projections acquired at 0 and 180 degrees can be used to estimate and correct for such offset of the center of rotation [24].
In order to reduce the computational time and increase robustness against noise, the variance fields ${ V}_\Theta$ are downsampled to a 4$\times$ lower resolution grid using the Gaussian pyramid approach [43]. The total displacement for each projection can be calculated as a cumulative sum of the registered displacements between projections measured at adjacent angles. A drawback of the XCA method is accumulation of registration errors, in particular for datasets with large number of projections.
5. Common-line method - vertical mass fluctuation (VMF)
Because the XCA prealignment does not use any knowledge about the imaging geometry, the relative shift between angularly distanced projections can be still far from optimal. Therefore, robustness of the entire alignment process can be further improved if an additional prealignment method is used. Assuming the sample rotation around the vertical axis, parallel beam geometry, and the field of view capturing the entire sample along the horizontal axis in each projection, the vertical alignment can be estimated using a method called vertical mass fluctuation (VMF) [9]. VMF is a powerful and computationally cheap approach based on alignment of an integrated conserved quantity that we denote here $m_\Theta (y)$
The vertical mass distribution $m_\Theta (y)$ is independent of the rotation angle. In conventional transmission X-ray microscopy, the quantity $p_\Theta (x,\;y)$ can be either the attenuation $P_A(x,\;y)$ or the unwrapped phase $\phi (x,\;y)$. A major advantage of the vertical-mass-distribution-based alignment is its independence of the horizontal misalignment.Due to the linearity of Eq. (5), a global offset or a linear slope in the preserved quantity $p_{\Theta (x,\;y)}$ will result in a linear offset in the vertical mass distribution $m_{\Theta (y)}$. The same argument can be also applied for propagation of potential errors in the lowest spatial frequencies of $p_{\Theta (x,\;y)}$. In order to make our analysis insensitive to these errors, the high-pass filtered vertical mass distribution $m_{\Theta (y)}$ by a linear operator is used for alignment, we denote the high-pass filter linear operator $\mathbb {F}_{\textrm {hp}}$. The vertical alignment is then performed as follows:
- 1. $p_{\Theta (x,\;y)}$ is calculated for each projection, either from the unwrapped phase $\phi (x,\;y)$ or attenuation $P_{A(x,\;y)}$.
- 2. $p_{\Theta (x,\;y)}$ is integrated along the horizontal axis and high-pass filtered to calculate the vertical mass distribution. Note that if the phase is used and the selected high-pass filter is a gradient filter, there is no need for phase unwrapping since the vertical phase-gradient $\partial _y \phi$ can be calculated analytically from the complex-valued projections $T_{\Theta }$ using$$\partial_y \phi=\textrm{Im} \left\{ \frac{\overline{T}_\Theta}{|T_\Theta|^{2}} \mathcal{F}^{{-}1} \left[ 2\pi { u}_y \mathcal{F}(T_\Theta) \right] \right\} \quad ,$$where $\textrm {Im}\{\}$ denotes the imaginary part and $\overline {T}_{\Theta }$ denotes the complex conjugate, ${u}_y$ is the vertical reciprocal-space coordinate and $\mathcal {F}\{\}$ is the Fourier transform. However, this strong high-pass filtering increases sensitivity of the VMF method to noise in the data.
- 3. A relative displacement between the one-dimensional VMF vectors is estimated
If the assumptions of the VMF method are fulfilled, the vertical alignment provided by this step can be considered to be final. Otherwise, the vertical alignment has to be refined in the final alignment step.
6. Projection-matching alignment (PMA) method
At this point, our dataset is already prealigned by the cross-correlation method and, if the measured dataset satisfies assumptions of the VMF method, the vertical alignment can be considered to be final. However, in order to keep our alignment strategy general, we will assume that both the vertical and the horizontal alignment have to be refined. Our final alignment step is performed by a multi-resolution projection-matching alignment (MR-PMA) method.
The PMA methods [18–26], also called "iterative reprojection" or "bootstrap methods", are based on the idea that even misaligned projections can be used for a 3D reconstruction, although the quality can be suboptimal. Reprojections of this initial reconstruction will be closer to the optimally aligned projections since only the consistent features are preserved in these synthetic reprojections. Therefore, shifting of the measured projections to match these reprojections will lead to improved reconstruction quality in the next iteration. Iterative application of this principle leads progressive improvement of the reconstruction quality until convergence is reached.
An application of the multi-resolution approach for projection alignment has been already demonstrated in [18,19]. Here, based on the same idea of multi-resolution processing, we demonstrate that with an optimized implementation, this method can reach alignment with so high accuracy in the coarse resolution levels that alignment at the finest resolution level does not further improve the reconstruction quality.
Solving the PMA task in lower resolution not only reduces the computational cost, but also the multi-resolution approach improves the robustness of the optimization. Essentially, assuming that the projections contain some features at each spatial frequency, the coarser resolution makes the optimization less sensitive to local properties of the measured projections and PMA becomes more robust against stagnation. The ability of MR-PMA to reach deep subpixel accuracy is based on the redundant information between the lowest spatial frequencies of the projections in the 3D Fourier space as indicated in Fig. 2. Since mutual displacement of the measured projections results in a mismatch in these spatial frequencies, this discrepancy can be sensed and used for correction by the PMA method.
Fig. 2. Filling of the Fourier space by the tomographic projections (gray stripes). Darker regions in the diagram denote regions with higher information redundancy.
PMA methods generally minimize the difference between the linearized measurements ${p}_{\Theta }$ and their reprojections ${\hat p}_{\Theta }$ defined as
where the variable $\textbf {p}$ denotes a stack of the measurements ${p}_{\Theta _i}$ The linear operator $\mathcal {A}_{\Theta }$ denotes a forward tomographic projection operator at angle $\Theta$ and the inversion operator $\mathcal {A}^{-1}$ denotes generally any common tomography reconstruction method such as filtered backprojection (FBP) or simultaneous algebraic reconstruction technique (SART). In our implementation, we use FBP mainly because of its low computational cost and versatility in non-standard imaging geometries [46]. Since the PMA computation is dominated by the reprojection step, use of Fourier space reconstruction methods such as Gridrec [47] could be beneficial only if the forward tomographic projection $\mathcal {A}_\Theta$ could be implemented efficiently in Fourier space as well.The displacement between the measurements ${p}_\Theta$ and the reprojections ${\hat p}_{\Theta }$ is commonly estimated by cross-correlation methods, which can be implemented with deep subpixel accuracy [44,48,49]. However, we have observed that the cross-correlation-based methods may become unstable if the data contains systematic errors. Additionally, the computational complexity of these methods grows significantly with the aimed level of subpixel accuracy [49]. Therefore, we have adopted an optimization approach inspired by the optical flow methods [50], which we found to be more robust and can iteratively reach deep subpixel accuracy. The generalized PMA optimization task can be expressed by the following cost function
By applying enough downsampling, we can assume without loss of generality that at the coarsest resolution level the displacement errors are much smaller than 1 pixel. The optimal displacements $\Delta x_\Theta ^{(i)}$, $\Delta y_\Theta ^{(i) }$, which minimize the cost function Eq. (9) in the $i$-th iteration, can be then estimated analytically as
The gradients $\partial _x {\hat p}_\Theta$, $\partial _y {\hat p}_\Theta$ are commonly approximated by their finite difference, and displacement of the projections is typically approximated using by bilinear interpolation, which is applied by a change of the projection tomographic geometry [18,19,21,24,25]. However, such strategy is not able to provide the deep subpixel accuracy required to avoid iterative alignment without downsampling. Additionally, it creates numerical noise that hinders convergence [24]. In contrast to other works, we use Fourier transforms to calculate the gradients of the reprojection and to apply the displacements to the measured projections in Eqs. (11) and (12). The discrete Fourier transform is a formally invertible numerical transformation with high numerical accuracy and stability and this is of great importance to achieve deep subpixel alignment accuracy.
7. Interpolation by offset discrete Fourier transform (DFT)
The multi-resolution optimization approach requires a way to upsample and downsample the projections to different interpolation grids. The selection of the optimal interpolation methods is critical to preserve the alignment quality between various interpolation grids and to preserve the deep subpixel accuracy acquired in the coarse resolution levels. In our implementation, we resample of the original signal $x(n)$ of length $N$ to a new signal $\hat x(n)$ of length $M$ using the discrete Fourier interpolation with an offset of half of the sampling step
8. Ambiguities of 3D phase-imaging using ptychography
The global phase offset and linear phase ramp are inherent degrees of freedom of ptychography [9]. Although the discussed alignment methods exhibit high robustness against various low-spatial-frequency errors, the quantitative contrast of the final reconstruction can be guaranteed only if these ambiguities are adequately constrained.
The conventional approach is to select regions in each frame that do not contain any sample contribution [9,24] and subtract the phase ramp in each projection separately. However, this approach may result in inaccuracies if the sample-free regions are not well detected, e.g. when contrast of the sample is low, or if the sample projections do not contain enough empty regions around the sample. Our approach here is based on the fact that not all possible phase ramps are consistent with the 3D reconstruction, in other words tomographic consistency can be used to deal with these ambiguities. Using the projection matching idea in Eq. (7), the optimal phase ramp ${\phi }_\Theta (x,\;y) =\beta _{\Theta ,\;x} { x}+\beta _{\Theta ,\;y} { y}+\beta _{\Theta ,0}$ at angle $\Theta$ can be found by minimization of the following least-squares optimization task
An identical approach can be used for calibration of the absorption tomogram, for which ptychography has an ambiguity in the overall amplitude of the reconstructed complex-valued transmission.
9. Results for simulated tomography datasets
Our alignment framework is implemented in Matlab using the ASTRA framework [21,51] with additional modifications [52]. The reconstructions were performed with CPU Intel Xeon E5-2690 and GPU Nvidia P100 and 512 GB RAM.
Performance of the proposed alignment methods was first verified using a synthetic dataset. In order evaluate our methods for a common class of samples in nanotomography, we have created a phantom of 800$\times$800$\times$800 pixels of a porous structure with two material phases, shown in Fig. 3(a), the structure was inspired in the nanostructure of catalyst materials. This phantom was used to produce 1260 projections, which provide full angular sampling according to the Crowther criterion [39]. The projections were displaced by a combination of a normally distributed random offset with amplitude of 20 pixels and an angularly dependent sinusoidal offset with period of 90 degrees and amplitude of 20 pixels, as shown in Fig. 3(b). Three test datasets were created using these projections. Dataset 1 was fully angularly sampled with noiseless projections, dataset 2 was 8$\times$ angularly undersampled, i.e. it contained only 161 noiseless projections with equal angular sampling, and finally dataset 3 adds 10% of Gaussian noise to the dataset 2.
Fig. 3. (a) An axial slice through our phantom, (b) Horizontal displacement error before (red) and after (blue) initial cross-correlation-based prealignment for Dataset 1.
The XCA method is able to significantly reduce the initial positioning errors. As shown in Fig. 3(b), the position variance between adjacent projections is significantly reduced. However, the overall alignment of the projections, in particular between those with large angular distance, is suboptimal, as expected. The remaining positioning errors for the three methods are detailed in Table 1. The main advantage of the XCA method as a prealignment step is its low computational cost. In this case, the variation calculation, shown in Eq. (4), and downsampling operation required 7 s and the cross-correlation between all 1260 projections required additional 2 s. The alignment quality, summarized in Table 1, is comparable for the first two tested datasets, but the root mean square (RMS) error was increased for Dataset 3 due to noise-sensitivity of our variance projection preprocessing in Eq. (4).
Table 1. Root mean square (RMS) displacement error for all three-alignment methods in pixel units. Displacement error for MR-PMA were estimated after the final full-resolution refinement.
The following optional prealignment method is VMF. In order to provide a more fair comparison, we have not used the cross-correlation alignment from the previous step as the initial guess, and we have applied VMF method directly on the initially displaced projections, as it is used in [9,22,53]. The alignment quality in Table 1 shows that our implementation of the VMF method can reach deep subpixel accuracy, resulting in a displacement accuracy better than 0.2 pixel RMS for all the test datasets. The computational time is comparable with the XCA method, the 2D FFT-based unwrapping [40] of all 1260 projections required 20 s and the cross-correlation alignment of the vertical mass distributions needed only 0.7 s.
The last tested method was our implementation of the MR-PMA method. Similarly, to the previous test, we have not used the advantage of an initial guess provided neither by the XCA nor by the VMF methods. Instead, we perform the reconstruction on multiple resolution levels. Once the MR-PMA method converges according to the criterion in Eq. (13) on the coarser resolution level, the result serves as the initial guess for the progressively finer resolution levels. Our MR-PMA method reached sub-0.2 pixel RMS accuracy for all tested configurations in the final resolution level as shown in Table 1. However, the remarkable feature of the deep-subpixel alignment is the option to finish the alignment already in one of the coarser resolution levels without need to iteratively compute the tomogram and projections at full resolution. This is demonstrated in Fig. 4(a), where we show the evolution of the RMS displacement error in different undersampling levels. The results are very consistent between the tested datasets and only a weak dependence on the angular sampling and the noise in the data can be observed. Note that the errors in Fig. 4(a) refer to the corresponding residual errors at full resolution. Alignment at the 32$\times$ downsampled resolution level was already sufficient to reach the RMS projection displacement error below 0.2 of full resolution pixels, i.e. 0.00625 downsampled pixels, for all the tested synthetic datasets as shown in Fig. 4(a). The level of detail in the sample obtained by the FBP method for different downsampling ratios can be seen in Fig. 5.
Fig. 4. Root mean square (RMS) error of the positions recovered using the MR-PMA method for different downsampling levels in tomography geometry. The RMS error for each downsampling level was calculated in the original pixel size. (b) Number of iterations and time per iteration needed to reach converge threshold in Eq. (13) for Dataset 1. In (a) and (b), the result at previous downsampling level is provided as the initial guess for the next higher resolution level. At downsampling of 4×, 2× and 1× only one iteration is needed due to the deep subpixel accuracy at higher downsampling levels.
Fig. 5. FBP reconstruction quality after alignment by our MR-PMA method. Columns show reconstruction at different downsampling levels from 32 up to 1, and rows correspond to different synthetic datasets. Insets in the last column show detail of the reconstruction quality for the full resolution dataset.
The per-iteration computational time of our PMA implementation is summarized in Fig. 4(b). In the case of highly downsampled resolution, the computational cost is limited by a constant overhead. For downsampling factor below 8, the per-iteration computational cost grows roughly 8$\times$ with each resolution level. On the other hand, the accurate initial guess from previous steps allows reaching the convergence threshold provided by Eq. (13) in a single iteration as show in Fig. 4(b).
10. Results for simulated laminography datasets
Our implementation of the MR-PMA method can be applied for different imaging geometries, e.g. laminography. Laminography is a generalization of conventional tomography with the only difference that the sample rotation axis is not perpendicular to the illumination beam direction. This geometry can be used to image planar 3D objects that would be difficult or impossible to image by conventional tomography in a nondestructive way [54]. However, the laminography geometry brings additional challenges to the projection alignment. Most importantly, the vertical and horizontal alignment cannot be solved separately. Additionally, the tilted axis of rotation implies that a meaningful reconstruction can only be obtained by calculating the full volume or a significant part of it, which leads to a significant increase in computational demands. Furthermore, there are no conserved quantities, such as VMF or center of mass that can be leveraged for alignment.
We have used the phantom from the previous section to generate 1260 projections in parallel beam laminography geometry with a 60 degrees angle between the rotation axis and beam propagation direction. These projections were used to create three datasets in the same way as the tomography datasets. The displacement error after alignment by our PMA method for different downsampling levels is shown in Fig. 6(a). Laminography alignment is also a more complex task due to the missing cone information and typically unlimited support of the imaged sample, which may result into inconsistencies between the measured projections and their reprojections. Despite these limitations, the alignment threshold of 0.2 pixel RMS was reached already for 8$\times$ downsampled resolution for all three test datasets.
Fig. 6. (a) RMS error of the positions recovered using the MR-PMA method for different downsampling levels in laminography geometry. (b) Number of iterations and time per iteration.
11. Alignment of experimental datasets
In order to demonstrate the full potential of the developed methods, we present here the alignment of a ptychographic X-ray laminography (PyXL) dataset. Details of the measurement, acquired at the cSAXS beamline, Paul Scherrer Institut (PSI), Switzerland, can be found in [55]. The measurements were acquired in the laminography nano-imaging instrument (LamNI) [56] equipped with optimized illumination optics [57]. The sample was an integrated circuit (IC) printed using 16 nm node technology. The substrate was mechanically polished to a thickness of about 20 µm, however all the scattering features are confined within a 4 µm thickness. We have collected 2704 ptychographic projections with a circular field of view of 40 µm diameter in the plane of the sample. The dimensions of the reconstructed ptychographic projections were 2549$\times$4155 pixels with a pixel size of 13.0 nm. The corresponding reconstruction volume contains 3800$\times$3800$\times$600 elements.
This dataset presents several challenges, in particular the size of the reconstructed volume. As discussed beforehand, laminography alignment does not allow the vertical and horizontal alignment to be solved separately and, depending on the sample thickness, the entire imaged volume or at least a significant part of it needs to be calculated in each iteration. Since the data volume of the reconstructed ptychographic complex-valued projections is 230 GB in single-precision (32-bit), the alignment task becomes extremely demanding, both in terms of memory and computation. Another complication for this IC sample is that it contains quasi-periodic features, which makes alignment methods more prone to convergence into local minima. The combination of these challenges makes this dataset ideal for demonstration of the unique capabilities of our alignment framework.
As in the previous sections, the alignment was solved in several resolution levels. The coarsest resolution level was 64$\times$ downsampled and the finest was 2$\times$ downsampled. Due to memory limitations, we were not able to perform the alignment step at the finest resolution, therefore the ability to reach alignment with subpixel accuracy was critical for this dataset. Since the ground truth alignment is unknown, in Fig. 7(a) we report the alignment progress in each resolution level using the displacement RMS value.
Fig. 7. (a) RMS of displacement update between subsequent downsampling levels for the experimental nano-laminography dataset. Convergence criterion of 0.2 pixel RMS was reached at 8$\times$ downsampled resolution. (b) Number of iterations and time per iteration required by the MR-PMA method. (c) Total displacement of each projection versus rotation angle.
The displacements per iteration reported in Fig. 7(a) indicate that the 4$\times$ downsampled resolution level already provides sufficient alignment precision, because the update RMS magnitude in the following step is below 0.1 pixel. The required computational time in the 2$\times$ downsampled dataset was 450 s per iteration as shown in Fig. 7(b). The total displacement for each projection angle is shown in Fig. 7(c). The projection displacements are caused among others by tracking the selected field of view in our sample by coarse motors upon rotation, imperfections of the rotation stage, and long-term thermal drifts. The maximal displacement error was up to 4 µm, i.e. 10% of the targeted field of view.
The laminography reconstruction after alignment is shown in Fig. 8. The reconstruction was performed by a filtered back-projection method modified for laminography geometry [58]. The corresponding spatial resolution of 18.5 nm was estimated from the intersection of the Fourier shell correlation (FSC) with the 1/2-bit threshold [59] shown Fig. 8(c). The FSC calculation excluded the Fourier-domain missing cone. This spatial resolution is sufficient to recognize the sample’s fine internal structure. High spatial resolution close to the voxel size and artifact-free reconstructions shown in Figs. 8(a) and 8(b) confirm a good alignment accuracy.
Fig. 8. Laminography slices of the two finest transistor metal layers M0 and M1 are shown in (a) and (b), respectively. The red squares inside of the full circular fields of view denote the regions magnified in the detailed views. (c) Fourier shell correlation of the laminography reconstruction.
12. Conclusion
We have demonstrated here that use of alignment methods for 3D nano-imaging implemented with deep subpixel accuracy can be achieved by a computationally efficient approach for tomography and laminography datasets. The subpixel accuracy of our multi-resolution projection matching method (MR-PMA) makes it possible to avoid sample reconstruction and reprojection at the full spatial resolution during alignment and thus helps to significantly reduce the computational and memory requirements.
Additionally, as we have demonstrated, our alignment methods perform well for noisy and undersampled datasets. The difficulties related to undersampling are alleviated since sufficient downsampling of the projections makes it always possible to satisfy the Crowther angular sampling criterion. Therefore, we have not observed any improvement of the reconstruction performance when iterative reconstruction methods such as SART were used for reconstruction instead of the conventional FBP method.
The combination of all three presented alignment methods allows us to perform the tomography processing in most cases in an unsupervised automatic manner. This helps to avoid the need of user interaction during the alignment process which is a significant bottleneck in nanotomography and nanolaminography imaging.
Our GPU-accelerated MATLAB-based alignment and reconstruction framework is publicly available in 10.5281/zenodo.3360818 [52] including a simple example of an artificial dataset. A high-resolution ptychotomography measurement of nanoporous glass [53] is provided in 10.5281/zenodo.3539513 [60]. The aligned full-resolution complex-valued projections from our experimental laminography dataset and the corresponding reconstruction are provided in 10.5281/zenodo.2657340 [61].
Disclosures
The authors declare no conflicts of interest.
References
1. F. Pfeiffer, “X-ray ptychography,” Nat. Photonics 12(1), 9–17 (2018). [CrossRef]
2. D. A. Shapiro, Y.-S. Yu, T. Tyliszczak, J. Cabana, R. Celestre, W. Chao, K. Kaznatcheev, A. D. Kilcoyne, F. Maia, S. Marchesini, Y. S. Meng, T. Warwick, L. L. Yang, and H. A. Padmore, “Chemical composition mapping with nanometre resolution by soft X-ray microscopy,” Nat. Photonics 8(10), 765–769 (2014). [CrossRef]
3. M. Guizar-Sicairos, I. Johnson, A. Diaz, M. Holler, P. Karvinen, H.-C. Stadler, R. Dinapoli, O. Bunk, and A. Menzel, “High-throughput ptychography using Eiger: scanning X-ray nano-imaging of extended regions,” Opt. Express 22(12), 14859–14870 (2014). [CrossRef]
4. J. Deng, D. J. Vine, S. Chen, Y. S. Nashed, Q. Jin, N. W. Phillips, T. Peterka, R. Ross, S. Vogt, and C. J. Jacobsen, “Simultaneous cryo x-ray ptychographic and fluorescence microscopy of green algae,” Proc. Natl. Acad. Sci. 112(8), 2314–2319 (2015). [CrossRef]
5. J. Deng, D. J. Vine, S. Chen, Q. Jin, Y. S. Nashed, T. Peterka, S. Vogt, and C. Jacobsen, “X-ray ptychographic and fluorescence microscopy of frozen-hydrated cells using continuous scanning,” Sci. Rep. 7(1), 445 (2017). [CrossRef]
6. J. Deng, Y. H. Lo, M. Gallagher-Jones, S. Chen, A. Pryor, Q. Jin, Y. P. Hong, Y. S. Nashed, S. Vogt, J. Miao, and C. Jacobsen, “Correlative 3D x-ray fluorescence and ptychographic tomography of frozen-hydrated green algae,” Sci. Adv. 4(11), eaau4548 (2018). [CrossRef]
7. H. Öztürk, H. Yan, Y. He, M. Ge, Z. Dong, M. Lin, E. Nazaretski, I. K. Robinson, Y. S. Chu, and X. Huang, “Multi-slice ptychography with large numerical aperture multilayer laue lenses,” Optica 5(5), 601–607 (2018). [CrossRef]
8. M. Dierolf, A. Menzel, P. Thibault, P. Schneider, C. M. Kewish, R. Wepf, O. Bunk, and F. Pfeiffer, “Ptychographic X-ray computed tomography at the nanoscale,” Nature 467(7314), 436–439 (2010). [CrossRef]
9. M. Guizar-Sicairos, A. Diaz, M. Holler, M. S. Lucas, A. Menzel, R. A. Wepf, and O. Bunk, “Phase tomography from X-ray coherent diffractive imaging projections,” Opt. Express 19(22), 21345–21357 (2011). [CrossRef]
10. J. C. Fung, W. Liu, W. De Ruijter, H. Chen, C. K. Abbey, J. W. Sedat, and D. A. Agard, “Toward fully automated high-resolution electron tomography,” J. Struct. Biol. 116(1), 181–189 (1996). [CrossRef]
11. D. Ress, M. L. Harlow, M. Schwarz, R. M. Marshall, and U. J. McMahan, “Automatic acquisition of fiducial markers and alignment of images in tilt series for electron tomography,” J. Electron Microsc. 48(3), 277–287 (1999). [CrossRef]
12. R. Han, L. Wang, Z. Liu, F. Sun, and F. Zhang, “A novel fully automatic scheme for fiducial marker-based alignment in electron tomography,” J. Struct. Biol. 192(3), 403–417 (2015). [CrossRef]
13. D. N. Mastronarde and S. R. Held, “Automated tilt series alignment and tomographic reconstruction in IMOD,” J. Struct. Biol. 197(2), 102–113 (2017). [CrossRef]
14. R. Guckenberger, “Determination of a common origin in the micrographs of tilt series in three-dimensional electron microscopy,” Ultramicroscopy 9(1-2), 167–173 (1982). [CrossRef]
15. J. Frank and B. F. McEwen, “Alignment by cross-correlation,” in Electron tomography, (Springer, 1992), pp. 205–213.
16. K. Dierksen, D. Typke, R. Hegerl, and W. Baumeister, “Towards automatic electron tomography ii. implementation of autofocus and low-dose procedures,” Ultramicroscopy 49(1-4), 109–120 (1993). [CrossRef]
17. D. Y. Parkinson, C. Knoechel, C. Yang, C. A. Larabell, and M. A. Le Gros, “Automatic alignment and reconstruction of images for soft x-ray tomography,” J. Struct. Biol. 177(2), 259–266 (2012). [CrossRef]
18. J. Dengler, “A multi-resolution approach to the 3D reconstruction from an electron microscope tilt series solving the alignment problem without gold particles,” Ultramicroscopy 30(3), 337–348 (1989). [CrossRef]
19. F. Bleichrodt and K. J. Batenburg, “Automatic Optimization of Alignment Parameters for Tomography Datasets,” in SCIA, (Springer, 2013), pp. 489–500.
20. M. Guizar-Sicairos, J. J. Boon, K. Mader, A. Diaz, A. Menzel, and O. Bunk, “Quantitative interior x-ray nanotomography by a hybrid imaging technique,” Optica 2(3), 259–266 (2015). [CrossRef]
21. W. van Aarle, W. J. Palenstijn, J. Cant, E. Janssens, F. Bleichrodt, A. Dabravolski, J. De Beenhouwer, K. J. Batenburg, and J. Sijbers, “Fast and flexible X-ray tomography using the ASTRA toolbox,” Opt. Express 24(22), 25129–25147 (2016). [CrossRef]
22. C.-C. Wang, C.-C. Chiang, B. Liang, G.-C. Yin, Y.-T. Weng, and L.-C. Wang, “Fast projection matching for x-ray tomography,” Sci. Rep. 7(1), 3691 (2017). [CrossRef]
23. A. Pryor, Y. Yang, A. Rana, M. Gallagher-Jones, J. Zhou, Y. H. Lo, G. Melinte, W. Chiu, J. A. Rodriguez, and J. Miao, “GENFIRE: A generalized Fourier iterative reconstruction algorithm for high-resolution 3D imaging,” Sci. Rep. 7(1), 10409 (2017). [CrossRef]
24. T. Ramos, J. S. Jørgensen, and J. W. Andreasen, “Automated angular and translational tomographic alignment and application to phase-contrast imaging,” J. Opt. Soc. Am. A 34(10), 1830–1843 (2017). [CrossRef]
25. D. Gürsoy, Y. P. Hong, K. He, K. Hujsak, S. Yoo, S. Chen, Y. Li, M. Ge, L. M. Miller, Y. S. Chu, V. De Andrade, K. He, O. Cossairt, A. K. Katsaggelos, and C. Jacobsen, “Rapid alignment of nanotomography data using joint iterative reconstruction and reprojection,” Sci. Rep. 7(1), 11818 (2017). [CrossRef]
26. H. Yu, S. Xia, C. Wei, Y. Mao, D. Larsson, X. Xiao, P. Pianetta, Y.-S. Yu, and Y. Liu, “Automatic projection image registration for nanoscale X-ray tomographic reconstruction,” J. Synchrotron Radiat. 25(6), 1819–1826 (2018). [CrossRef]
27. Y. Liu, P. A. Penczek, B. F. McEwen, and J. Frank, “A marker-free alignment method for electron tomography,” Ultramicroscopy 58(3-4), 393–402 (1995). [CrossRef]
28. T. Donath, F. Beckmann, and A. Schreyer, “Automated determination of the center of rotation in tomography data,” J. Opt. Soc. Am. A 23(5), 1048–1057 (2006). [CrossRef]
29. M. Scott, C.-C. Chen, M. Mecklenburg, C. Zhu, R. Xu, P. Ercius, U. Dahmen, B. Regan, and J. Miao, “Electron tomography at 2.4-ångström resolution,” Nature 483(7390), 444–447 (2012). [CrossRef]
30. M. V. Wyawahare, P. M. Patil, and H. K. Abhyankar, “Image registration techniques: an overview,” Int. J. Signal Process. Image Process. Pattern Recognit. 2, 11–28 (2009).
31. C.-C. Cheng, C.-C. Chien, H.-H. Chen, Y. Hwu, and Y.-T. Ching, “Image alignment for tomography reconstruction from synchrotron x-ray microscopic images,” PLoS One 9(1), e84675 (2014). [CrossRef]
32. C. Mennessier, R. Clackdoyle, and F. Noo, “Direct determination of geometric alignment parameters for cone-beam scanners,” Phys. Med. Biol. 54(6), 1633–1660 (2009). [CrossRef]
33. J. C. Da Silva, M. Guizar-Sicairos, M. Holler, A. Diaz, J. A. van Bokhoven, O. Bunk, and A. Menzel, “Quantitative region-of-interest tomography using variable field of view,” Opt. Express 26(13), 16752–16768 (2018). [CrossRef]
34. M. Liebi, M. Georgiadis, A. Menzel, P. Schneider, J. Kohlbrecher, O. Bunk, and M. Guizar-Sicairos, “Nanostructure surveys of macroscopic specimens by small-angle scattering tensor tomography,” Nature 527(7578), 349–352 (2015). [CrossRef]
35. F. Schaff, M. Bech, P. Zaslansky, C. Jud, M. Liebi, M. Guizar-Sicairos, and F. Pfeiffer, “Six-dimensional real and reciprocal space small-angle x-ray scattering tomography,” Nature 527(7578), 353–356 (2015). [CrossRef]
36. C. Donnelly, M. Guizar-Sicairos, V. Scagnoli, S. Gliga, M. Holler, J. Raabe, and L. J. Heyderman, “Three-dimensional magnetization structures revealed with x-ray vector nanotomography,” Nature 547(7663), 328–331 (2017). [CrossRef]
37. Z. Gao, M. Guizar-Sicairos, V. Lutz-Bueno, A. Schröter, M. Liebi, M. Rudin, and M. Georgiadis, “High-speed tensor tomography: iterative reconstruction tensor tomography (IRTT) algorithm,” Acta Crystallogr., Sect. A: Found. Adv. 75(2), 223–238 (2019). [CrossRef]
38. M. Odstrcil, M. Holler, J. Raabe, A. Sepe, X. Sheng, S. Vignolini, C. G. Schroer, and M. Guizar-Sicairos, “Ab initio nonrigid X-ray nanotomography,” Nat. Commun. 10(1), 2600 (2019). [CrossRef]
39. R. A. Crowther, D. DeRosier, and A. Klug, “The reconstruction of a three-dimensional structure from projections and its application to electron microscopy,” Proc. R. Soc. London, Ser. A 317(1530), 319–340 (1970). [CrossRef]
40. S. Jeught, J. Sijbers, and J. Dirckx, “Fast Fourier-based phase unwrapping on the graphics processing unit in real-time imaging applications,” J. Imaging 1(1), 31–44 (2015). [CrossRef]
41. R. M. Goldstein, H. A. Zebker, and C. L. Werner, “Satellite radar interferometry: Two-dimensional phase unwrapping,” Radio Sci. 23(4), 713–720 (1988). [CrossRef]
42. J. M. Rodenburg and H. M. Faulkner, “A phase retrieval algorithm for shifting illumination,” Appl. Phys. Lett. 85(20), 4795–4797 (2004). [CrossRef]
43. E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden, “Pyramid methods in image processing,” RCA engineer 29, 33–41 (1984).
44. M. Guizar-Sicairos, S. T. Thurman, and J. R. Fienup, “Efficient subpixel image registration algorithms,” Opt. Lett. 33(2), 156–158 (2008). [CrossRef]
45. J. A. Hartigan and M. A. Wong, “Algorithm AS 136: A k-means clustering algorithm,” J. Royal Stat. Soc. 28(1), 100–108 (1979). [CrossRef]
46. A. Myagotin, A. Voropaev, L. Helfen, D. Hänschke, and T. Baumbach, “Efficient volume reconstruction for parallel-beam computed laminography by filtered backprojection on multi-core clusters,” IEEE Trans. Image Process. 22(12), 5348–5361 (2013). [CrossRef]
47. F. Marone and M. Stampanoni, “Regridding reconstruction algorithm for real-time tomographic imaging,” J. Synchrotron Radiat. 19(6), 1029–1037 (2012). [CrossRef]
48. P. Bing, X. Hui-Min, X. Bo-Qin, and D. Fu-Long, “Performance of sub-pixel registration algorithms in digital image correlation,” Meas. Sci. Technol. 17(6), 1615–1621 (2006). [CrossRef]
49. A. Yousef, J. Li, and M. Karim, “High-speed image registration algorithm with subpixel accuracy,” IEEE Signal Process. Lett. 22(10), 1796–1800 (2015). [CrossRef]
50. J. L. Barron, D. J. Fleet, and S. S. Beauchemin, “Performance of optical flow techniques,” Int. J. Comput. Vis. 12(1), 43–77 (1994). [CrossRef]
51. W. van Aarle, W. J. Palenstijn, J. De Beenhouwer, T. Altantzis, S. Bals, K. J. Batenburg, and J. Sijbers, “The ASTRA Toolbox: A platform for advanced algorithm development in electron tomography,” Ultramicroscopy 157, 35–47 (2015). [CrossRef]
52. M. Odstrcil and M. Guizar-Sicairos, “GPU-accelerated alignment methods for tomography with deep sub-pixel accuracy,” (2019), http://dx.doi.org/10.5281/zenodo.3360818.
53. M. Holler, A. Diaz, M. Guizar-Sicairos, P. Karvinen, E. Färm, E. Härkönen, M. Ritala, A. Menzel, J. Raabe, and O. Bunk, “X-ray ptychographic computed tomography at 16 nm isotropic 3D resolution,” Sci. Rep. 4(1), 3857 (2015). [CrossRef]
54. L. Helfen, T. Baumbach, P. Mikulik, D. Kiel, P. Pernot, P. Cloetens, and J. Baruchel, “High-resolution three-dimensional imaging of flat objects by synchrotron-radiation computed laminography,” Appl. Phys. Lett. 86(7), 071915 (2005). [CrossRef]
55. M. Holler, M. Odstrčil, M. Guizar-Sicairos, M. Lebugle, E. Müller, S. Finizio, G. Tinti, C. David, J. Zusman, W. Unglaub, O. Bunk, J. Raabe, A. F. J. Levi, and G. Aeppli, “Three-dimensional imaging of integrated circuits with macro-to nanoscale zoom,” Nat. Electron. 2(10), 464–470 (2019). [CrossRef]
56. M. Holler, M. Odstrcil, M. Guizar-Sicairos, M. Lebugle, U. Frommerz, T. Lachat, O. Bunk, J. Raabe, and G. Aeppli, “LamNI - an instrument for X-ray scanning microscopy in laminography geometry,” in preparation (2019).
57. M. Odstrčil, M. Lebugle, M. Guizar-Sicairos, C. David, and M. Holler, “Towards optimized illumination for high-resolution ptychography,” Opt. Express 27(10), 14981–14997 (2019). [CrossRef]
58. L. Helfen, A. Myagotin, P. Mikulík, P. Pernot, A. Voropaev, M. Elyyan, M. Di Michiel, J. Baruchel, and T. Baumbach, “On the implementation of computed laminography using synchrotron radiation,” Rev. Sci. Instrum. 82(6), 063702 (2011). [CrossRef]
59. M. Van Heel and M. Schatz, “Fourier shell correlation threshold criteria,” J. Struct. Biol. 151(3), 250–262 (2005). [CrossRef]
60. M. Holler, A. Diaz, and M. Guizar-Sicairos, “High-resolution ptychotomography dataset of nanoporous glass,” (2019), http://dx.doi.org/10.5281/zenodo.3539513.
61. M. Holler, M. Odstrcil, and M. Guizar-Sicairos, “Nanolaminography dataset: Three-dimensional imaging of integrated circuits with macro- to nanoscale zoom,” (2019), http://doi.org/10.5281/zenodo.2657340.
