1. Introduction

Digital image correlation is an established technique for the measurement of in-plane displacement and strain fields. Full-field deformation between two images can be calculated directly by cross-correlating small image regions or subimages [1–7]. The position of the correlation peak indicates the displacement and strain fields can be calculated by differentiation. Image correlation techniques have been applied to many imaging modalities including optical microscopy [1], SEM [1] and STM [2] and used for applications as diverse as elastography of tissue [3] and real-time velocity measurement [4].

The motivation behind the work in this paper was the need for a non-contacting optical system to analyze thermal deformation in microelectronics devices. Microscopic deformation analysis using correlation (MDAC or microDAC [1]) was chosen because, unlike alternative optical techniques such as microscopic moiré [8], no special sample preparation, vibration isolation or complex optics are required. In fact, the algorithms described here could be applied to the output of any imaging system, provided sufficient correlation was present between subsequent images.

Although in principle an optical MDAC system can be used to achieve high strain measurement throughput without requiring special skills of the operator, it does have some inherent disadvantages when compared with microscopic moiré. The first of these results from the need to use a subimage of typically 16×16 to 64×64 pixels to calculate each displacement. Subimages can be overlapped to some extent, but the lateral resolution of MDAC is always much lower than the optical resolution of the imaging system. The second restriction when using MDAC is the computational overhead of interpolating the results to obtain subpixel accuracy. Typically, the true displacement between two digitized subimages will contain some non-integral element and locating the correlation peak directly from the digital correlation is only accurate to ±0.5 pixels. This is insufficient for many purposes, including the analysis of thermal deformation in microelectronics devices. Since the sensitivity of MDAC scales with magnification, lateral resolution can be improved by using an array of high resolution images to cover the required field of view. But naturally this greatly increases the computational demands on the system.

Our requirement was for an accurate system capable of resolving displacements as small as 0.05 pixels without knowledge of the analytical form of the image correlation function. The operating speed of the system was also important as a fast algorithm could allow the analysis of multiple images in a reasonable time. This paper describes and compares two techniques for MDAC which attempt to meet these requirements. The first we call successive approximation nonintegral pixel shifting (SA-NPS). Here, a hill climbing algorithm is used to estimate the subpixel shift from the results of previous correlations. SA-NPS is described in Sect. 2. The second technique, described in Sect. 3, examines the feasibility of using trained artificial neural networks (ANNs) to estimate the subpixel element of the deformation much more quickly. The results of SA-NPS and MDAC using ANNs for translated and thermally stressed microelectronics devices are presented and discussed in Sect. 4. Section 5 contains a summary of the features of each technique and discusses possible improvements for future investigation.

2. MDAC using successive approximation nonintegral pixel shifting

In Ref. 5, Sjödahl describes using a combination of curve-fitting to the correlation peak and nonintegral pixel shifting of subimages to achieve subpixel displacement sensitivity with images of laser speckle patterns. We developed SA-NPS in an attempt to replicate the high accuracy of Sjödahl’s technique without the need for an analytical expression to describe the shape of the correlation peak. It has been demonstrated that NPS can be used to increase the overlap of the initial and deformed images. Aligning the subimages to subpixel accuracy prior to calculating the correlation coefficient is essential if high precision is required. It removes a displacement dependent systematic error from the curve-fitting algorithm and allows displacement resolution of as little as 0.01 pixels [5, 6]. However, laser speckle pattern images are well understood in terms of feature size and shape and are typically of high contrast [9]. The images we wished to analyze, and hence the shape of their correlation functions, are less easily characterized and are typically broader. This is demonstrated in Figure 1 which shows two typical micrographs of cross-sections through a plastic quad flat pack (PQFP) microelectronic device. Sample 1, an area of packaging material, possesses some of the properties of laser speckle patterns, having a fine, homogenous structure. Sample 2 however contains regions of very different image properties. Inspection of correlation peaks generated from the two samples revealed much greater variation in sharpness and d.c. background for sample 2. This fact made it very difficult to select a suitable and reliable function or family of functions to curve-fit to the correlation peaks.

 

Fig. 1. Micrographs (320×320µm) of cross-sections through an PQFP microelectronics device. Sample 1 is packaging material and possesses a fine homogenous structure. Sample 2 contains regions of copper (lower area), silicon (upper right), packaging (upper left) and filler (center).

Download Full Size | PDF

As an alternative to curve-fitting, we applied NPS to iteratively search for the correlation peak. The algorithm proceeds by first aligning the subimages to within 0.5 pixels. The second subimage is then subpixel shifted and re-correlated with the first subimage. Subpixel shifting entails multiplying the spectrum of the subimage by a linear phase gradient before re-correlating [5]. The results of successive correlations are used to make better estimates of the true peak position in a manner analogous to successive approximation analogue-to-digital conversion [10]. Iteration continues until the required accuracy has been achieved or the algorithm converges, indicating a noise limited result. The results in Sect. 4 demonstrate that SA-NPS performs very well with no knowledge of the analytical form of the correlation peaks, but it has a large computational overhead from the repeated correlation calculations. Typically, 20 to 100 iterations of the algorithm are required for each subimage, and each iteration entails calculating a normalized correlation coefficient.

3. MDAC using artificial neural networks

Artificial neural networks are biologically inspired electronic circuit architectures which can be used to build a generalizable model of a data set [11]. Model building consists of ‘training’ the network by exposing it to examples of the data. Training can be ‘supervised’ where the network is repeatedly shown examples of input data together with the desired output, or ‘unsupervised’ where the network is used to find subtle relationships within the input data. Many basic ANN architectures can be simulated on a standard PC with widely available code. The networks used in this paper were trained and tested with code inherited from Ref. 12.

The multi-layer perceptron (MLP), a supervised network, was chosen for this application. MLPs have been used to solve problems such as the detection of ECG spikes [13] which have proved intractable with other available techniques. They consist of highly interconnected layers of nodes or neural processing units. The many connections between nodes each have an associated weight, and these weights store a representation of the training data set. The nodes themselves are simple. They sum their weighted inputs and pass the total through a non-linear function to the next layer. A gradient descent technique algorithm is used to adjust the weights during training in an attempt to minimize the error between the desired and actual network response to the training data [11–13].

We trained and tested a range of MLP architectures with data generated from images of sample 1 and sample 2 in an attempt to create a robust yet accurate model. Training sets were generated by randomly selecting a 24×24 pixel subimage from an image of the sample, subpixel shifting a copy by a known, random amount and cross-correlating with the original subimage. Two MLP input vectors were stored (one each for the x and y displacements) along with the actual subpixel displacements. Training sets of 10,000 examples were generated, along with independent test sets. The MLP input vectors were derived from the central row and column of the cross-correlation peak. Attempts to train MLPs using absolute correlation values failed, however using the gradients of the correlation peak was successful. Preprocessing the MLP input data by calculating the difference between adjacent correlation values reduced the network’s task from differentiating and estimating the intercept with the horizontal axis to simply estimating the intercept.

 

Fig. 2. Test performance of several MLP architectures on sample 1 (lower plots) and sample 2.

Download Full Size | PDF

The test set performance of MLPs for a range of input and hidden node configurations is shown in Fig. 2. It should be noted that the back-propagation algorithm does not guarantee to locate the global error minima, so several networks were trained for each architecture in an attempt to optimize performance. The number of network inputs (I) determines the number of correlation gradient values used to calculate the displacement. The number of hidden nodes (H) in combination with I determines the number of weights, W=I(H+1) and hence the complexity of the network. It can be seen from Fig. 2 that performance increases with both the number of inputs and with the total number of weights, but that a significantly more complex network is needed to analyze sample 2 than sample 1. For sample 1 (lower five plots) a network of just 21 weights (6 input and 3 hidden nodes) performs very nearly as well as one containing 88 weights (10 input and 8 hidden nodes). An optimal ANN has as few weights as possible while achieving the required performance, so for sample 1 a network with 6 input and 3 hidden nodes was selected (denoted MLP-1) and for sample 2 a network with 8 inputs and 8 hidden units was used (MLP-2) for the results in Sect. 4. The average error of these ANNs on computer generated test data was 0.023 and 0.044 pixels respectively.

A useful feature of the MLP architecture is its speed of operation once training is completed. Calculating the output of an MLP requires one forward pass through the network. Mathematically, this amounts to multiplying the input vector (I elements) by a matrix containing the weights between input and hidden nodes (I×H elements). The result is passed through a non-linear operator (typically a hyperbolic tangent or sigmoidal function) and then multiplied by a H element vector containing the weights between the hidden layer and the output node. A final pass though a non-linear operator generates the output.

4. Results and discussion

To assess the baseline performance of SA-NPS and the MLPs on real images, the techniques were first used to measure pure displacement in samples 1 and 2. The results displayed below were generated by imaging the sample at 20 displacements between 1 and 16 pixels (0.6 and 10 µm). For each image the error was calculated as the standard deviation of the displacement results for each subimage. The errors quoted below are the mean errors over the 20 measurements. Imaging was performed using a 0.30 N.A. (x10) microscope objective illumination with an LED (λ_peak=590 nm). The object pixel size was 628 nm to ensure correct image sampling.

Table 1. Performance of successive approximation NPS MDAC for pure translation.

View Table | View all tables in this article

Table 1 illustrates both the high accuracy possible with NPS and the trade-off between lateral resolution (related to the subimage size) and precision. It can also be seen that SA-NPS performs better on the more homogenous sample. Table 2 summarizes the results obtained using MLP-1 and MLP-2. A single application of a network (1 iteration) produces inferior results to SA-NPS, but with a reduction in the computational overhead of greater than an order of magnitude. It was found that more accurate measurements were obtained by using the results of the first MLP iteration to subpixel shift the deformed subimage and improve overlap. A second iteration with the MLP reduced the mean error and also greatly attenuated displacement dependent systematic errors as reported in Ref. 5. The results in Table 2 show a simple but specialized filter (MLP-1) can achieve high precision on a homogenous sample, but performs very badly on a more complicated image. A more complex, general-purpose filter (MLP-2) can attain reasonable accuracy on a inhomogeneous sample without sacrificing significant performance on simple images.

Table 2. Performance of ANN MDAC for pure translation.

View Table | View all tables in this article

The results of applying SA-NPS and MLP-2 (with 2 iterations) to images of a thermally stressed PQFP are shown in Figs. 3 and 4. It can be seen that SA-NPS appears to have performed poorly on the packaging area of the device. This is probably due to the very low light levels in this region. The MLP seems to have less problem here as its training data contained examples from this region. The MLP seems to work least well on the filler area and it is possible a specially trained network may help in this region.

5. Summary

Two techniques for subpixel MDAC have been described and tested. The first of these uses SA-NPS to realize high accuracy without the need for an analytical function to describe the form of the correlation peak. However, its use is limited by the amount of computer processing required to implement it. The second technique uses specially trained artificial neural networks to achieve subpixel MDAC of comparable accuracy to SA-NPS but with a much reduced computational overhead. The results in Sect. 4 establish the feasibility of the neural approach. They indicate that an MLP trained over a wide range of input data can perform well, but suggest that more specialized filters may perform better. Future work will investigate the potential of real-time MDAC using ANNs and adaptive MDAC using a categorization unit with a bank of specialized filters.

 

Fig. 3. Unfiltered in-plane displacement map calculated using NPS with a 24×24 subimage for a temperature rise from 20 °C to 50 °C. The length and direction of the arrows indicate the magnitude and direction of the displacement for each subimage.

Download Full Size | PDF

 

Fig. 4. Unfiltered in-plane displacement map calculated using MLP-2 for a temperature rise from 20 °C to 50 °C. The displacement fields of Figs. 3 and 4 may be smoothed and differentiated to produce strain maps.

Download Full Size | PDF