1. Introduction

In many applications, complicated boundaries or shapes are represented using polygons [1–4]. In this paper, we consider a new application of the polygonal representation of shapes in the field of analysis and statistic generation for the microscopy images of fabricated structures [5–8] or optical effects like fringing [9, 10]. The microscopy images are inherently digitized and amenable for digital image processing. Thus methods using polygonal approximation of digital curves can be used for shape analysis in microscopy images as well. While this paper represents shapes as polygons, the use of higher order polynomial curves like cubic splines is also common. It is notable that curves of order 2 or higher are smooth (second order derivatives being continuous) curves. Thus, they are unsuitable for representing shapes with corners and sharp curvature changes. On the other hand, polygonal representation is quite generic and can represent a wide variety of shapes such as used in the fabrication industry. Further, polygonal representation of shapes helps in the determination of geometrical properties like inflexion points, perimeter, and tangent estimation. Lastly, it filters away the effect of digitization and retains the shape characteristics of fabrication.

There are several methods for polygonal representation of digital curves [11–17]. However, not all of them are suitable for detecting fabrication imperfections and quantifying them. For example, Nguyen [15] is not suitable for this purpose because it uses the concept of blurred segments to obtain the polygonal representation. It is interesting to note that there are some methods that use smoothing kernels [16, 17], like a Gaussian kernel, to remove the digitization effects while detecting dominant points. Such methods are somewhat similar to the used of Fourier components in [6]. There are relatively more direct methods like [11–14] which work in the digital image domain itself and use some form of distance metrics within the digital image domain as control parameters. As discussed in [18], [14] focuses on representing the global nature of the curve better while [13] focuses on representing the local curvature changes better. Nevertheless, both of them are computation expensive. On this account, [11, 12] have better characteristics, since a larger value of control parameter results in a polygonal fit more representing of the global curvature properties and vice versa. However, there is no direct or strict correspondence between the nature of fit and the value of control parameter.

Recently, precision and reliability metrics specifically representing the local and global natures of polygonal fit were proposed [19] and used in precision and reliability optimization (PRO) method [18] for obtaining the polygonal fit. The control parameter of PRO ${\epsilon }_0$ behaves quite regularly in representing the nature of polygonal fit. PRO provides the user with good flexibility regarding the nature of fit. For example, the user may choose a very close fit, which closely follows the digital curve and retains all the small perturbations in the curve. Or, the user may choose a curvature following fit that removes the local effects (due to noise, etc.) and retains the large scale features of the digital curve. Thus, it is quite suitable to identify the general shape (large/global scale) properties as well as local imperfection (local scale).

This paper applies PRO to microscopy images of various types of fabricated structures in order to analyze and assess the quality of features fabricated in the sample. It is shown that the imperfections can be better analyzed using PRO even in diverse applications like photonic crystal arrays (which are essentially electromagnetic band gap structures, albeit for optical frequencies), brightness enhancement films for CCD arrays, and electromagnetic band gap structures. For example, some imperfections that look visually unobvious can be easily found and located using PRO. Further, optical effects and alignments can also be easily studied, as demonstrated in the images for alignment and inspection of aspherical mirror. Aspects of image qualities like sharpness etc. can also be easily recognized. Using these examples of wide variety, the utility of PRO for analyzing microscopy images and determining and assessing fabrication imperfections is clearly demonstrated. The example images used in this paper are copyright of the authors of [6–8, 10] and the Optical Society of America. The images have been used with the kind permission of their respective authors and OSA.

The outline of the paper is as follows. Section 2 presents the PRO method briefly and analyzes the nature of its control parameter. It also demonstrates the applicability of PRO for complicated digital shapes. Section 3 considers various examples of microscopy images and how PRO is applied to these images for analysis and assessment. Section 4 concludes the paper.

2. Precision and reliability optimization method

2.1 The method

The precision and reliability measures were defined for a line segment in [19]. Suppose, for a sequence of connected pixels $S=\left\{P_i\left(x_i,y_i\right)\right\}$, $i=1\text{to}M$, we intend to fit a line $ax+by=1$. Then, the coefficients of the line, $a$ and $b$, can be determined by casting the problem of fitting into the following matrix equation as expressed in Eq. (1):

On the other hand, for the global characteristics of fit, another measure called the reliability measure is needed. Generally, reliability of a fit refers to how well the fit is expected to satisfy at least two conditions:

A combination of both these properties can be sought by defining a reliability measure as shown in Eq. (3):

Let us consider a digital curve $e=\left\{\begin{array}{cccc}{P}_1& P_2& \dots & P_N\end{array}\right\}$, where $P_i$ is the $i$th edge pixel in the digital curve $e$. The line passing through a pair of pixels $P_a(x_a,y_a)$ and $P_b(x_b,y_b)$ is given by Eq. (4):

Then the deviation $d_i$ of a pixel $P_i(x_i,y_i)\in e$ from the line passing through the pair $\left\{P_1,P_N\right\}$ is given as Eq. (5):

Accordingly, the pixel with maximum deviation can be found. Let it be denoted as $P_{\max }$. Then considering the pairs $\left\{P_1,P_{\max }\right\}$ and $\left\{P_{\max },P_N\right\}$, we find two new pixels from $e$ using the concept expressed in Eqs. (4) and (5). It is evident that the maximum deviation decreases as one chooses newer pixels of maximum deviation between a pair.

For each pair of line segment and its associated curve, the precision and reliability measures ${{\epsilon }^{\prime }}_p$ and ${\epsilon }_r$ are computed using Eqs. (2) and (3). Then, for each segmented curve, if the precision and reliability measures are not below a certain threshold given by Eq. (6):

 

Fig. 1 Pseudocode for PRO algorithm.

Download Full Size | PDF

2.2 Demonstration of the nature of PRO and the control parameter ${\epsilon }_0$

We consider the 17 examples used in [19] for illustrating the nature of the control parameter ${\epsilon }_0$. Such analysis is helpful for choosing a suitable value of PRO’s control parameter for various images and applications. Each example is a snippet of 20 pixels dimensions containing only one digital curve, which may be similar to a digital line or a more complicated curve. The images are kept small so that the performance can be seen in close up for various curvature conditions of the digital curves. The proposed method is called PRO<number>, where <number> denotes the value of ${\epsilon }_0$. For example, PRO0.2 refers to the use of ${\epsilon }_0=0.2$ in the PRO method. Comparison with a very popular method, RDP [11, 12], and two recent methods, Carmona-Poyato [14] (referred to as Carmona for simplicity) and Masood [13] is also given. For RDP, we use the notation RDP<tol>, where tol indicates the value of the control parameter of RDP. The value of control parameter for Carmona [14] and Masood [13] are 0.4 and 0.9 respectively, as suggested in [14] and [13] respectively. Carmona and Masood are chosen for comparison because of two reasons. The first reason is that they are quite recent and represent the state-of-the-art method for dominant point detection problem. The second reason is that while Masood uses local nature of fit as the main criterion for dominant point detection, Carmona uses the global nature of fit as the main criterion. Thus, it is interesting to compare the performance of these methods against PRO, which considers both the local and global natures of fit.

The actual lines fitted by various algorithms are shown in Fig. 2. Further, several performance parameters are listed in Table 1. It can be seen in the row PRO0.2 of Fig. 2 that PRO0.2 tends to follow the digital curves very closely. As a consequence it generates numerous small line segments to represent the curve, strongly evident in columns (e-h) of Fig. 2. Next, we see that PRO0.6 tends to follow the curvature of the digital curve, better than PRO0.2. We highlight the results in column (m) of Fig. 2. While PRO0.2 generated many line segments for the right side of the curve, PRO0.6 is more selective in fitting the line segments and fits the line segments focusing at the location of changes in curvature, rather than following every small-scale feature of the curve. This is significantly evident in the results in columns (i), (j), and (q) of Fig. 2. In PRO1.0, instead of focusing on the small features in the digital curve, it tends to follow the general characteristics of the digital curve on a relatively larger-scale, see columns (m) and (n) of Fig. 2. As a consequence of this characteristic, PRO1.0 has significantly better dimensionality reduction as compared to other PRO algorithms (see dimensionality reduction ratio in Table 1). PRO algorithms demonstrate at least two well-defined patterns in terms of performance and the nature of fit. The first pattern is that small value of ${\epsilon }_0$ provides very close fits that are highly precise. The second pattern is that large value of ${\epsilon }_0$ tends to smooth out small variations and retain large scale curvature changes. Based on this analysis, depending on the requirements, a suitable value of ${\epsilon }_0$ can be chosen or PRO with more than one values of ${\epsilon }_0$ can be executed.

 

Fig. 2 Results for the 17 snippets used in [19]. The lines fitted by various algorithms are shown for each snippet.

Download Full Size | PDF

Table 1. The values of performance parameters for various algorithms. The entries with gray background are the minimum values of the performance parameters in a group of algorithms.

View Table

RDP1 algorithm gives a performance comparable to PRO0.6, both qualitatively (specifically note the columns (i), (m), and (o) of Fig. 2) and quantitatively (see Table 1). RDP2 and RDP3 perform poorer than RDP1 for all the parameters except the dimensionality reduction ratio. Even RDP does not give a well-defined performance for fitting polygons on digital curves with various shapes and curvature. The results of Masood [13] show a mixed performance which cannot be classified as either adhering to local shape variations or global shape variations. Carmona’s method [14] attempts to fit the line segments using a relative measure such that the allowable maximum deviation increases with the length of the digital curve. This results in over-fitting for small digital curves and under-fitting for large digital curves.

As compared to RDP, Carmona, and Masood, the controlled and well-behaved nature of PRO is evident for all the examples. In addition, we note that the time complexities of Carmona and Masood are significantly higher than RDP and PRO due to the iterative optimization procedure conducted for each dominant point. The capability of PRO in representing the local and global features of diverse digital curves is demonstrated in Fig. 3 for 7 standard curves used in the image processing community [13–15].

 

Fig. 3 The digital curves used in Masood [13] and Carmona-Poyato [14] and the lines fitted using PRO0.2, PRO0.6, and PRO1.0.

Download Full Size | PDF

3. Fabrication analysis using PRO

3.1 High resolution SEM images of 2D planar slab photonic crystal arrays

Statistical analysis of imperfections in the fabrication of 2D planar slab photonic crystal arrays of circular shape was done in [6] using high resolution SEM images and representing the shape in terms of Fourier components. We denote the circular elements in the image as $e=1\text{to}E$. Here, we demonstrate our approach for this example.

First the images are binarized using simple threshold on the gray level (user-specified) from a range 0-255. The digital contours are obtained from the boundaries in the binarized images. Then PRO is applied on the digital contours using the following values of control parameter ${\epsilon }_0\in 0.2n,n=1\text{to}\text{5}$. The vertices of the polygons obtained using PRO are used in an ellipse-fitting method. Here, we have used recently proposed ElliFit method [20] for its selectivity of elliptic shapes. Let the parameters of ellipses be denoted as: $a_{e,n},b_{e,n},{\stackrel{⌢}{x}}_{e,n},{\stackrel{⌢}{y}}_{e,n},{\stackrel{⌢}{\theta }}_{e,n}$, which represents the lengths of semi-major and semi-minor axes, the x and y-coordinates of the center of ellipse, and the orientation (the counter-clockwise angle from x-axis to the semi-major axis).

The ellipses found for all elements of the crystal array and all values of ${\epsilon }_0$ are used to generate the statistical average size of the elements, i.e. $a={\displaystyle \sum _{\forall e,\forall n}{a}_{e,n}}/\left(5e\right)$, $b={\displaystyle \sum _{\forall e,\forall n}{b}_{e,n}}/\left(5e\right)$. The average coordinate and orientation of each element is determined by taking the averages for all value of ${\epsilon }_0$, i.e. ${\stackrel{⌢}{x}}_e={\displaystyle \sum _{\forall n}{\stackrel{⌢}{x}}_{e,n}}/5$, ${\stackrel{⌢}{y}}_e={\displaystyle \sum _{\forall n}{\stackrel{⌢}{y}}_{e,n}}/5$, ${\stackrel{⌢}{\theta }}_e={\displaystyle \sum _{\forall n}{\stackrel{⌢}{\theta }}_{e,n}}/5$. Then the normal distance of each digital curve from the corresponding average ellipse $\left[a,b,{\stackrel{⌢}{x}}_e,{\stackrel{⌢}{y}}_e,{\stackrel{⌢}{\theta }}_e\right]$ is computed and used as the measure of the fabrication imperfection of the circular photonic crystal array. Let us denote the normal distance of $p$th pixel in a digital curve from the average ellipse be denoted as $d_p$, then we use $\text{mean}\left(d_p/a;\forall p\right)$, $\text{max}\left(d_p/a;\forall p\right)$, and $\text{std}\left(d_p/a;\forall p\right)$ as the statistics of imperfections, where these expressions denote the average, maximum, and standard deviation of the normal distances of the pixels from the ellipse corresponding to their digital curve.

We note that the above procedure works for both partially and completely present elements in the image. However, the digital curves of partially present elements will result into large normal distance for the chord of the partial element. So, they are neglected in the computation of imperfection. A simple example shown in Fig. 4 illustrates the procedure.

 

Fig. 4 Illustration of the computing average feature and statistics of a 2D planar array of circular photonic crystals. The original images are the copyright of the authors of [6] and have been used with their permission.

Download Full Size | PDF

Two more elaborate examples are shown in Fig. 5, where the first to third columns respectively showing the actual microscopy image, its digital curves, and the detected features. In addition, the top 10 features with the highest value of $\text{mean}\left(d_p/a;\forall p\right)$ are highlighted in the middle column using red triangles.

 

Fig. 5 Two examples of photonic crystal array. In the middle column, the triangles denote the top 10 features with highest values of $\text{mean}\left(d_p/a;\forall p\right)$. The average ellipses are shown in white (overlapped with the original image) in the third column. The original images are the copyright of the authors of [6] and have been used with their permission.

Download Full Size | PDF

The statistics of the features in these two examples are plotted in Fig. 6. The comparison shows that the features in the second example have lesser imperfections (lesser value of $\text{max}\left(d_p/a;\forall p\right)$ as well as $\text{mean}\left(d_p/a;\forall p\right)$). Also the distribution of imperfections is more uniform in the second example. It is seen that though the distribution of $d_p/a;\forall e,p$ can be approximated using Gaussian distribution, the fit is closer for example 2. These statistics allow for identifying the features with more imperfections as illustrated using the triangles in the middle column of Fig. 5.

 

Fig. 6 The statistics of the two examples of photonic crystal array shown in Fig. 5. (a) statistics of individual features in example 1. (b) histogram and equivalent Gaussian distribution for the values of $d_p/a;\forall e,p$ for example 1. (c) and (d): same as (a) and (b) but for example 2.

Download Full Size | PDF

3.2 Fabrication fidelity in brightness enhancement films with prism arrays fabricated using roll-to-roll process

Wang and Tseng [7] demonstrated the use of roll-to-roll process for fabricating brightness enhancement films (for CCD arrays) in the form of prism arrays fabricated on a PET substrate. In [7], SEM images of the fabricated prism arrays were shown to demonstrate the good quality of fabricated array. However, minor edge defects did exist which may or may not be easily perceivable by naked eye using the microscopy image. Here, we use PRO to highlight these effects. The edges in the image are detected using Canny edge detector [21] and the PRO 1.0 is applied. Two images and their PRO results are shown in Fig. 7. The first image shows a SEM image with good contrast and sharpness. The PRO result shows the boundaries of the prisms very well. However, the upper side (towards left) shows a lot of clutter which occurs due to the darker shadowed region in the upper left side. Further, besides the clean edges of the prisms, the defects can also be easily identified. The second image is a good example of a low contrast blur image. While the image looks visually acceptable, it is not suitable for identifying or localizing the defects or even finding the edges of the fringes. This can be easily seen in the PRO result. While the Canny edge detector does not identify any edges in the lower half of the image, the PRO result shows several dominant points along most edges. Nevertheless, a big defect (highlighted using an ellipse) can still be identified from the image easily.

 

Fig. 7 Two examples of fabrication fidelity in brightness enhancement films with prism arrays fabricated using roll-to-roll process. The defects are highlighted using red arrows and red ellipse. The example images are taken from [7] with the permission of its authors.

Download Full Size | PDF

3.3 Electromagnetic band gap structures and their fabrication quality

Wu et. al [8] demonstrated a fast and inexpensive technique for fabricating electromagnetic band gap structures using a polymer-jetting technique. While the advantages of this fabrication technique are easily identified, it is of significant practical interest to assess the fabrication quality. Two images reported in [8] are considered here in Fig. 8. The features in the images appear reasonably well-fabricated. However, use of PRO1.0 shows some anomalies as shown using red arrows in Fig. 8. We highlight that some of the anomalies are the structures from the lower layer. We note that it may also indicate the need of better image quality.

 

Fig. 8 Two examples of electromagnetic band gap structures. The defects not easily visible are highlighted using red arrows. The example images are taken from [8] with the permission of its authors.

Download Full Size | PDF

3.4 Aspherical mirror quality and alignment

Pan et. al [10] reported some methods for assessing the quality and alignment of aspheric mirrors. Some images were presented in [10] to demonstrate the fringes, the aberrations, misalignments, etc. Here, PRO1.0 is used to analyze those images as shown in Fig. 9. Image 1 in Fig. 9 shows the fringes obtained using interferogram. The result is identified as good due to good contrast. Notwithstanding this fact, the result of PRO1.0 showed in Fig. 9 indicates that the fringes are non-uniform and irregular. Image 2 in Fig. 9 shows a difference image obtained by removing lower order coma and astigmatism. The PRO1.0 result (and the center of the outer circle computed using [20]) show that the inner circles are shifted from the geometric center of outer circle. This indicates the misalignment of the mirror.

 

Fig. 9 Example images from [10] (taken with the authors’ permission) and the results of PRO. The images are shown in the upper row and PRO1.0 results are shown in the lower row. For the images 2 and 4, the center of the outer circle computed using [20] is shown in a red colored ‘ + ’ sign.

Download Full Size | PDF

The averages images obtained using 168 independent images are shown as images 3 and 4 in Fig. 9. Image 3 corresponds to a prior traditional method and image 4 corresponds to a the CGH method proposed in [10], and it is reported in [10] that the CGH method is better for alignment. PRO1.0 and the center of the outer circle computed using [20] show more conclusively that the CGH method is better for alignment while the traditional methods do not provide any cursor (in the form of inner circles) for alignment.

7. Conclusion

This paper has demonstrated that the precision and reliability optimization (PRO) method, that uses precision and reliability measures in the optimization goals, is a useful tool for assessing fabrication or feature qualities from the microscopy images for a variety of applications. The PRO method explicitly considers both local and global qualities of fit and its control parameter ${\epsilon }_0$ behaves in a very regular manner. Smaller values of ${\epsilon }_0$ result in very close fitting of polygons such that small local features can be represented well. On the other hand, larger values of ${\epsilon }_0$ result in better representation of the large scale features. An elaborate example of photonics crystal array with circular elements shows how PRO can be used for identifying imperfect features and generating statistics for large arrays. Other examples of prism arrays for brightness enhancement in CCD arrays, electromagnetic band gap structures, and imaging based assessment techniques for aspherical mirrors are also considered and PRO is used for detecting the image quality, fabrication defects, and misalignments. It is seen that PRO aids in assessing the image quality easily and identify the effects that are not easily obvious to human eye. Further, PRO also assists in easily assessing and/or quantifying the fabrication quality and localizing the imperfections. Thus, PRO is shown to be a useful tool for analysis and image processing of microscopy images. Nevertheless, if the shape information is known a priori, methods dedicated to those shape priors are expected to perform better (for example [20] for elliptic shapes).

Two interesting future directions of this work are discussed here. Developing quantitative criteria for the measurement of fabrication error is an important future direction. For developing such criteria, the knowledge of the structures, geometries, and positions of the features to be fabricated is essential. Further, Jaccard index and other shape overlap measurement techniques are some possible approaches suitable for quantifying the fabrication errors. The second future direction is to combine the proposed method with inverse reconstruction approaches. It is well-known that the microscopy image is never a true one-to-one map of the object being imaged. It is rather a complicated map of the object which combines the input illumination and the response of the microscopy system. Further, unless confocal system is used, the image is always a 2-dimensional projection of the 3-dimensional object features (thus we see the features from lower layer in Fig. 8). The mapping from focal plane to the image is approximately linear if the microscopy system is paraxial and confocal. In such a case, the proposed method is expected to represent the fabrication details and errors quite realistically. However, in general, for separating the optical response of the microscope from the fabricated structure, it is essential to solve an optical or electromagnetic inverse (object reconstruction) problem (for example, using deconvolution, holography, etc [22–25].). It is therefore an interesting future direction to combine the proposed method with a reconstruction approach. In this regard, a possible motivation is to detect the shapes using the proposed method and use them as initial guess or a deformable shape [26] which gets adapted in the iterative procedure of reconstruction.