1. Introduction

One major goal of computer vision processes is the characterization of industrial (quality control) or medical (diagnosis) objects using automatic surface inspection methods. In other words, this research field tackles the processing of different surfaces to be characterized, by means of different types of illuminations and/or recording techniques, for the visual enhancement of defective surface parts. Common and main requirements of proposed vision methods are their ability to characterize the surface to be inspected and their rapidity in terms of inspection time. Thus, effective lightings, and algorithms, but also efficient handling and software frameworks have to be involved, in order to address more and more challenging tasks, like the recognition of different types of surface defects in real-time.

In general, the visual enhancement of a certain type of defective surfaces is directly dependent on the lighting and the recording technology. Typically, depth defects related to geometric deformations of the surface, or textural defects synonymous of different surface roughness have to be visually enhanced by means of a structural and a diffuse lighting. Different automatic inspection systems have been recently proposed, as the measurement of electronic devices by the companies Aceris [1] or Comet [2], or the wafer inspection by the company Solvision [3].

With the same objective of increasing different defect types with one system, an alternative surface inspection procedure has been proposed in [4]. It has been demonstrated that the characterization of a projected structured pattern serves the direct surfaces interpretation. The adapted and textural feature-based content description method of the corresponding stripe images rely on the characterization of the depicted bright/dark structures [5]. The advantages of such a method are manifold, especially in terms of real-time processing, “simple” algorithmic procedure, and process simplification, enhancement of different defect types in one camera shot.

However, in order to simplify the algorithmic processing of the stripe images, periodical and vertical bright/dark structures have been considered. Hence, the described inspection task in [5] can be applied to the same approaches, i.e. the inspection of specular and cylindrical surfaces by means of an adapted illumination technique, or to further inspection tasks necessitating the interpretation of similar vertical bright/dark structures. Such results are therefore only applicable in case of the characterization of vertical patterns. There are two possible steps toward the generalization of the proposed inspection method to further non-cylindrical free-form surfaces.

The first possible alternative could be to adapt the structured illumination to the inspected surface shape so that a periodical vertical pattern is depicted in the recording sensor. This approach is difficult or even impossible to implement in case of free-formed surfaces, in particular if these are highly specular which are more difficult to record: the camera does not observe the surface itself, but the reflection of the light on it. This problem is addressed in detail in [6]. The second possible solution could be to consider the characterization of non-vertical and non-periodical bright/dark structures which are produced when a light pattern is projected onto free-form surfaces. This approach is tackled in this paper.

Hence, one major purpose of this paper is to define and to optimize a free-form stripe pattern recognition process, in terms of retrieving the most relevant set of features that accurately classifies the reference image sets. According to Raudys and Jain [7], the main steps defining a typical pattern recognition system are the data collection, the pattern class formations, the characteristic feature selections, and the classification algorithm specifications. With the proposed free-form surface inspection task a successive optimization of these stepwise procedures will be addressed.

At first, various reference stripe image sets defining the free-form surfaces to be characterized will be introduced. Each considered set of patterns will be classified in two formations, corresponding to two distortion types. Then, two different stripe feature-based image content description methods will be considered: a method specially adapted for the characterization of such stripe patterns, and a general textural Fourier-based approach. The optimization of the feature selection will be addressed by means of specific feature fusion and one optimal feature selection method. Finally, the determination of the optimal pattern recognition process is achieved by means of the classification rate.

Hence, on the basis on the previous experiments related to the characterization of cylindrical specular surfaces, the purposes of this paper are:

The rest of this paper is organized as follows: The use of structured illumination for surface quality control is introduced in Sec. 2. The two considered image content description methods, namely the general Fourier and the adapted stripe approaches are presented in Sec. 4. Section 5 describes the involved proposed four steps procedure for the determination of optimal feature subsets in case of free-form bright/dark structure characterization. Finally, a summary is given in Sec. 6.

2. Structured Light for Surface Characterization

2.1. General approach

The general purpose of structured light is the shape recovery of objects or scenes to be inspected, see [9]. Concerning object reconstruction principles, the adequate method is related to surface reflectivity, i.e. wether the surface is rough (diffuse reflection) or reflective (specular reflection).

An approved method for specular surface inspection is the deflectometric-based approach [10], which is used in many industrial inline inspection processes [11–13]. According to light source intensity and surface diffuse reflection proportion, 3D triangulation-based methods were defined for rough surface reconstruction [14]. However, all cited 3D-shape recovery methods necessitate a preliminary recording set-up calibration [6, 15]. Complete 3D reconstruction can be avoided by adapting the set-up elements (light, surface, sensor) to visually enhance defective geometrical surface parts [16–18]. Depending on the involved lighting, geometrical and/or textural surface information can be directly recovered: light-sectioning [13], image fusion [19], or photometric stereo [20] for example.

However, all these methods do not address the inline inspection (complex calibration procedure) for the simultaneous detection of geometrical and textural defects (non-adapted lighting) in real-time environments (complex handling or recording processes).

2.2. Adapted Approach for Limited Surface Geometries

In addition to previously cited conventional approaches, and as stated in the introduction, a new cylindrical surface interpretation principle, based on the projection of a structured light pattern, has been defined [4]. This surface inspection task is a $2\frac{1}{2}D$ approach, no 3D depth information is required, as all the relevant information is contained in the image. Indeed, patterns corresponding to a geometrical deformation of the surface can be discriminated by “only” interpreting the stripe disturbance degree, their real depth is not retrieved. Figure 1 depicts the pattern arrangement for the three considered classes Ω_A, Ω_R,3D, and Ω_R,2D.

Such regular patterns depicted in Fig. 1 are only a part of the non-vertical and non-periodical bright/dark stripe structures that would be depicted in case of free-form surfaces, i.e. when the light cannot be adapted to the surface geometry.

2.3. Adapted Approach for Free-Form Surfaces

The aim of this paper is to apply the inspection principle to a broader range of surface types and geometries. Figure 2 depicts two examples of free-form surfaces, a rough and a specular one, being illuminated with a “non-adapted” structured light pattern.

Although the recording principle of rough and specular objects are different [6], the camera focusses on the surface in case of the former and on the lighting screen for the latter, both defective 3D depth defects depicted in Fig. 2(b) can be visually enhanced by means of the depicted bright/dark structures in the images.

 

Fig. 1. Six examples taken from the reference initial set of images Φ ₀ ⁰ made of 252 patterns. This set has been used for the qualification of the industrial system for the cylindrical specular surfaces inspection. All the patterns have been classified into distinct classes: Acceptable Ω_A, rejected (non-acceptable) Ω_R,3D, and Ω_R,2D. All other 246 patterns depict similar structures, i.e. correspond to similar geometry and/or grey level changes/perturbations.

Download Full Size | PDF

 

Fig. 2. (a) Surface inspection principle: A camera C records the object surfaces to be inspected S, illuminated by a lighting L. (b) Recordings of one free-form rough surface and one free-form specular surface, both are illuminated by a structured light pattern.

Download Full Size | PDF

The major difference with the considered images in [4], where regular periodical stripe structures are observed, is that, as both objects have a non-planar surface, and a conventional planar structured lighting is used, the bright and dark stripes in the images of Fig. 2(b) are neither vertical nor periodical. Rather, their geometries depend on the shape of the inspected objects.

As a consequence, with the purpose of generalizing the inspection principle to free-form rough and specular surfaces, an extensive range of stripe structures have to be considered. Such stripe geometries can be obtained by means of different surface shapes, structured light positions or sensor types. Our task is not to enumerate all possible combinations of these components and to compute the corresponding stripe geometries. This would hardly be possible. Hence, it is preferable to focus our investigations on a restricted and predefined number of non-vertical and non-periodical bright/dark stripe deformations.

Figure 3 depicts two examples of bright/dark geometries. Each geometry can be obtained when a rough or a specular surface is inspected. The case of a linear moving object with speed V⃗ recorded with a static line-scan camera is considered. One application example for a spherical object is also shown.

For the purpose of clarity, only the case of surfaces recorded by means of line-scan sensors has been considered in Fig. 3. However, this does not restrict the application of the proposed surface inspection principle, as similar patterns can be obtained when matrix cameras are used.

 

Fig. 3. Left and middle: Two different examples of stripe deformations arising when (a1) and (b1) free-formed rough or (a2) and (b2) free-formed specular objects are recorded. These two examples show that depending on the surface geometry, similar stripe deformations can be observed for rough or specular objects. These two examples show the inspection of a surface S illuminated by a lighting L and recorded by a line-scan camera C during its linear movement along V. Right: One possible bright/dark structure example in case of a round (spherical) object. Upper image shows a sphere with 3 surface portions of different sizes. Lower images show the corresponding image structures if a light pattern is projected.

Download Full Size | PDF

On the basis of these two examples, it can be demonstrated that it is possible to obtain the same bright/dark structures for specular and rough surfaces.

The underlying assumption is that the disturbances induced by the depicted bright/dark image patterns are always distinguishable from the undisturbed pattern. Thus, as stripe structure geometry and/or gray level is used for the defect localization and characterization, this means that in the vicinity of a defective region, the “background” variations, i.e. of the surface geometry and/or the surface texture, are below a certain level. The illumination is considered as ideal and projects a homogeneous bright/dark light structure on all the surfaces to be inspected.

Hence, the same reference stripe patterns can be used for the characterization of rough and specular surfaces. The considered reference patterns used for retrieving the ideal free-form stripe pattern recognition process, and the influence of such different bright/dark structures on the classification rate, will be tackled in the next sections.

As it is not possible to consider all possible stripe disturbance induced by object geometry, for the rest of the paper, the investigations will be “restricted” to the two types of stripe geometrical deformations depicted in Fig. 3(a) and 3(b), the perspective and the geometrical ones. Indeed, Fig. 3(c) shows an example of bright/dark stripe geometry obtained for a round object. It can be seen how these disturbances encompass, i.e. can be described by the two considered.

Thus, perspective and geometrical disturbances will serve for the generalization of the proposed inspection method based on the direct interpretation of “almost” free-form stripe patterns, i.e. non-vertical and non-periodical ones, for the characterization of free-form surfaces. The terminology “almost” is used, as it is assumed that the complex bright/dark structures to be characterized, must permit the localization and description of all the defects situated completely on the surface to be inspected.

The limiting factors for this statement is the bright/dark structures disturbance degree. Figure 3(c) clearly shows by means of a concrete example, that the recorded surface size must be adapted to object geometry. Content description with the proposed method will be more robust if it is applied to image (1) than to image (3). Thus, in case of spherical objects, it would be preferable to increase the number of recordings in order to permit a robust inspection of the whole object.

2.4. Defining the Reference Image Sets

The primary condition to evaluate the proposed inspection method for the characterization of free-form surfaces, is to define a set of reference stripe image patterns, where number and type of reference stripe patterns depend on the considered inspection task. For example, the reference Brodatz database is used for the evaluation of various textural analysis approaches, so that it encompasses different textural gray level pattern types, as regular and stochastic ones. A good alternative therefore is to consider the reference stripe patterns that have been involved for the qualification of the industrial system [4].

Each stripe pattern, which depicts one type of surface to be characterized, has been recorded by an adapted stripe illumination, producing vertical and periodical stripe structures. The whole set of reference patterns, named Φ₀ ⁰ , is made of 252 elements manually annotated and classified into three distinct classes Ω_A, Ω_R,3D, and Ω_R,2D. These classes correspond to acceptable surfaces, rejected 3D geometrical, and rejected 2D textural surfaces.

Further pattern structures have therefore to be defined. The easiest and simplest way consists of using the patterns of Φ₀ ⁰ and to “transform” or “adapt” them, so that these can be used for the characterization of free-form surfaces.

Thus, the stripe-illumination-based free-form surface inspection task will be addressed by means of different image sets: The reference initial set Φ₀ ⁰, previously introduced, and eight further derived sets. The four sets Φ₁ ¹-Φ⁴ ₁ correspond to the warping of all patterns of Φ₀ ⁰ with increasing projective transformations. The four sets Φ¹ ₂-Φ⁴ ₂ correspond to the warping of all patterns of Φ₀ ⁰ with increasing cylindrical transformations. Both projective -1- and cylindrical -2- transformations correspond to the types of stripe pattern geometries depicted in Figs. 3(a1), 3(a2) and 3(b1), 3(b2). All sets are made of 252 patterns.

Figure 4 shows 3 of the 9 considered stripe image data sets. Φ₀ ⁰ is the reference set where the stripe structures are periodical and vertical, Φ⁴ ₁ is the set corresponding to the warped patterns of set Φ₀ ⁰ with a maximum perspective distortion -1- and Φ⁴ ₂ is the set corresponding to the warped patterns of set Φ⁴ ₂ with a maximum cylindrical distortion -2-.

Such nine sets of reference patterns can be used to address the proposed inspection task within a general approach, if following conditions are fulfilled:

The first condition addresses the necessary minimal size and intensity of the defect to be detected, whereas the second condition is related to the capacity of the illumination to enhance all the surface to be inspected. In the following, the lighting is considered to be ideal, i.e. produces a homogeneous bright/dark structure, for all possible surface geometries and reflectance.

 

Fig. 4. Left: Reference patterns for the classification of free-form rough and specular surfaces. These image patterns correspond to three different surface shapes illuminated with a regular periodical structured illumination: -0- for surfaces inducing no deformations, and -1- and -2- for surfaces inducing perspective and cylindrical distortions. Φ₀ ⁰ corresponds to patterns without distortion related to the shape of the object. These patterns have been measured. Φ⁴ ₁ and Φ⁴ ₂ corresponds to patterns with a maximal perspective distortion of type -1- and a maximal cylindrical distortion of type -2-. These patterns have been simulated by transforming patterns Φ₀ ⁰ with perspective and cylindrical distortions. All the patterns have a size of 64 × 64 pixel. Right: Bright/dark geometry and reflectance characteristics: Surface: Period d_L,P and coefficient ρ_S ; Defect: Size [d_D,u × d_D,v] and coefficient ρ_D.

Download Full Size | PDF

Defect minimal size and intensity is directly linked to the bright/dark structure deformations and sensor sensitivity.

In order to be detected, each defect D must be at least as huge as the minimal depicted bright/dark structure period and have a significant reflectance coefficient. Hence, if d_D,u and d_D,v are the defect width and height, r_C,u and r_C,u the sensor resolution in u- and v- directions, d_L,P the projected light stripe period, ρ_D and ρ_S the reflectance coefficients of the defect D and the neighboring surface S, following equation holds:

Factor 2 comes from the Shannon theorem, linking the sampling frequency, r_C,u ^-1 and r_C,u ^-1, with the signal frequency in u-, (d_P/2)^-1, and in v-, d_v. Factor α depends on the sensor sensitivity, the lighting intensity, and surface reflectance.

Hence, for the further stripe image content description, Eq. (1) must be verified for all the considered reference patterns. For the measured Φ₀ ⁰ patterns, used for the qualification of the industrial system [4], above conditions are fulfilled, as these correspond to the customer’s requirements. Concerning the other reference artificial patterns, Φ⁴ ₁ and Φ⁴ ₂, obtained after simulating a perspective and a cylindrical transformation, the constrain was that all depicted transformed defects are still characterized by above Eq. (1).

Thus, the necessary assumption in case of real images obtained with a surface inspection system based on the proposed researches, is that the requirements defined by Eq. (1), are fulfilled. It is therefore assumed, that the considered lighting technique, described in [4], can always be adapted and applied for the characterization of free-form surfaces by means of almost free-form, i.e. non-vertical and non-periodical bright/dark patterns. Necessary set-up optimizations for optimal components spatial arrangements are not tackled here.

3. Stripe Image Content Description

The next important step now consists of defining the most appropriate algorithmic procedures that best characterize non-vertical and non-periodical stripe patterns, i.e. to search for adequate feature sets and groups that best describe such bright/dark structures. As stated in the introduction, a hierarchical method is proposed to optimize the retrieval of the most appropriate non-vertical and non-periodical stripe pattern characterization features. The steps are: (i) evaluation of two different methodologies, (ii) individual evaluation of different feature groups, and determination of the most appropriate feature subsets by means of appropriate feature (iii) fusion and (iv) selection approaches.

Concerning the considered feature families/methodologies, two approaches will be evaluated. An adapted one, previously defined in [5], and a general one, based on Fourier analysis, see [21]. The reasons for involving these two procedures are described hereafter.

Concerning the adapted features, a set of 14 stripe features has been evaluated in a previous paper. It has been shown that these characteristics outperform in terms of classification rates five further different types of textural feature families. Moreover, 8 of these 14 features were adapted from previously defined features describing fringe patterns [22], which are also used for non-destructive inspection purposes, based on interferometric approaches. A particularity of such fringe patterns is that they have a more complex geometrical structure, see [5]. This is also the case of the depicted bright/dark stripes that will be considered in case of the free-form surface inspection, see Fig. 4. In addition, the classification results in [5] showed that the adapted stripe characterization approach mostly outperforms the general textural methods. It is therefore strongly assumed that such adapted features are particularly suited for the free-form surface inspection task considered in this paper.

Fourier-based approaches are very attractive methods in case of real-time applications, where low computation costs are demanded. [23] proposes a Fourier-based approach for the description of industrial surface defects, and demonstrates that such a technique is accurate and computationally light. Ünsalan [24] uses Fourier-based features for the description of steel surfaces and the Fast Fourier Transform (FFT) to increase the speed of the transformation. [8] et al. propose two efficient algorithms to analyze large scale periodic structures by means of FFT-based methods. Then, the Fourier transform has the property of periodic features description, which makes such an approach very attractive in terms of stripe pattern characterization depicting periodic or almost periodic structures. Several authors use this property to describe images depicting periodic structures. Within the field of surface inspection, [25] uses the inverse Fourier transform to remove the repetitive periodic patterns of statistical features. Qian et al. [26] propose a fault detection method by means of interferometric fringe patterns based on a windowed Fourier transform approach. As such fringes can be considered as non-vertical and non-periodical stripe structures, it is strongly believed that such an approach will also be suited for this paper’s purposes.

All these facts concerning the adapted and the Fourier-based transformation, are strong arguments in favor of using adapted stripe features and textural Fourier-based features for the characterization of non-vertical and non-periodical stripe images.

3.1. Textural Analysis with Fourier Features

The characterization of the stripe structures by means of the textural Fourier analysis will be based on the approach of Weska [21]. The author uses the power spectrum P as an image signature for the discrimination of different types of image patterns F. P, defined as the square of the spectral’s magnitude, is a matrix of same size as the matrix F. The major goal of [21] was to use the particularity of the spectral domain by selecting different frequency subbands, which is equivalent to retaining certain levels of details and directions in the patterns to be analyzed. Weska considers the radial and the angular spectral distributions, saying that the former is sensitive to texture coarseness and the latter to texture directionality. He also uses the distributions corresponding to the principal spectral image axes, the u- and v-directions.

The features are directly computed from amounts of values in the Fourier spectrum for different spectral regions. [21] defines various radial, directional, horizontal and vertical frequency regions. The assumption is here that the use of different spectral regions characterizing different image frequencies, would be more appropriate for the description of almost free-form patterns, corresponding to spatial frequency variations of bright/dark structures.

3.2. Defining Fourier Feature Groups

The assumption in using different parts of the power spectrum for Fourier-based image content description is that some regions may be more discriminative or representative of certain classes of stripe images.

A major part of the considered bright/dark stripes are characterized by a vertical pattern whose disturbances are synonymous of defective surfaces. Hence, a first hypothesis could be that filtering out the power spectrum regions which correspond to the vertical stripe pattern, could lead to an increase of the signal (defective surface) to noise (vertical patterns) ratio, and therefore lead to higher classification rates. Such a filtering could be obtained by considering for example only the horizontal or the directional frequency.

The considered disturbed stripe patterns are also characterized by local variations of the pattern, corresponding to high frequency changes (geometric disturbances) or low frequency changes (grey-level disturbances), see Fig. 1. Thus, the signal (disturbance) to noise (vertical pattern) ratio could be increased using different pass-band radial frequency filters.

Next Fig. 5 illustrates the relation of the frequency distribution and the image contents with three examples, and shows the mathematical expression of the considered spectral regions.

The images show that disturbances in the spatial domain have typical signatures in the frequencies representation. The above figure illustrates three different cases.

As depicted in the left image in Fig. 5, a regular non-disturbed pattern is represented with two peaks related to the pattern frequency. If the pattern intensity changes along a certain direction in the spatial domain, its transformed counterpart in the Fourier domain is orthogonal, as depicted in the middle image in Fig. 5. Then, if pattern geometrical disturbance is characterized by a local variation of pattern frequency, a broader energy distribution is observed at two peak positions, see right image in Fig. 5.

For these three examples, it is highly probable that the directional or the horizontal components in the frequency domain may be strongly discriminative in terms of stripe pattern characterization. Hence, with a generalization purpose, this approach can be applied for all the considered stripe disturbances. In case of the stripe pattern analysis, feature vectors integrating different subbands of the frequency domain were taken into consideration. The following five different feature vectors of lengths N_c will be used:

The length of each feature vector depends on the considered frequency regions. The vector c _{F,r,θ,v, u}, which considers all possible regions has a maximal length of N_c = 33.

 

Fig. 5. Image characterization from amounts of values in the Fourier spectrum, according to [21]. Four different spectral regions are considered: The radial P̄_r1,r2, directional P̄_θ1θ2, horizontal P̄_v1,v2 and vertical P̄_u1,u2 spectral ones.

Download Full Size | PDF

3.3. Use of Adapted Stripe Features

In [5], 14 features for the characterization of the bright and dark stripes have been introduced. Six features were specially developed for the purpose of vertical bright and dark structure characterization, eight features were adapted from fringe features originally defined by Zhi [22] for the purpose of free-form bright fringes description. In order to propose a homogeneous adapted approach, we also applied these eight features to the dark stripes of the considered non-vertical and non-periodical patterns.

The notations and names of the 20 considered stripe features are listed in the table below:

Table 1. Notation and names of the 20 considered adapted features of the stripe feature vector c ^S. These features characterize the bright and the dark stripe depicted in a pattern.

View Table | View all tables in this article

Hence, a total of 20 adapted features are computed for each stripe pattern F. The algorithmic procedure to retrieve these feature images is described in [5]. The mathematical expression of related feature vectors of lengths N_c are described in [5].

3.4. Defining Adapted Feature Groups

Within the context of defining optimal adapted feature groups, the above described features can be classified in two main groups: The 6 first features specially developed for vertical stripe description, and the remaining 14 features defined for fringe structure description. Figure 6 illustrates these two groups, depicting three “projected” stripe patterns and three “interferometric” fringe patterns examples, and showing the computation of “adapted stripe” feature deviation and “adapted fringe” feature shape.

 

Fig. 6. Typical “projected” and “interferometric” bright/dark structures. The three upper images correspond to ideal “projected” structures, the three lower ones to more complex “interferometric” fringe structures. The first “vertical stripes” feature group was defined for the characterization of the former, whereas the second “free-form” feature group was developed for description of the latter. For illustration purpose, the equations of one adapted “vertical stripes” “minimum distance” feature c ₀₂ and one adapted “free-form stripe” “tangent” feature c ₀₈ are listed. The results of corresponding operators are written for one central blue marked pixel ^(B) sⁱ_c. Both are the average results of operators O₀₂ and O₀₈ applied to all bright stripes central pixel elements ^(B) sⁱ_c of the considered image F.

Download Full Size | PDF

The major difference between the two feature groups is that for the former group the assumption is made that the stripe structures are vertical, i.e. that the result depends on the main direction of the structures. For the latter group, the feature value is independent of the stripe direction, with the assumption that it is more adapted for the characterization of non-vertical and non-periodical structures.

The following three different feature vectors of lengths N_c will be used:

The considered feature vector c ^S encompasses two different feature groups or types, c ^S ₀₆, c ^S ₁₄, each defined for similar bright/dark image structure characterization tasks. Former group consists of vertical bright/dark structures, whereas the latter of more free-form bright/dark structures. It therefore seems appropriate, to fuse these two feature groups into one describing vector, as the considered inspection task consists of describing and characterizing disturbed “projected” stripe patterns, whose disturbance degrees are in between the disturbance degrees of the considered bright/dark image groups, see Fig. 4. Thus, we are convinced that such a fused feature vector should lead to optimal image classification rates, as non-vertical and non-periodical bright/dark structures must be interpreted.

4. Feature Selection and Pattern Classification

This section addresses the involved feature subset selection (FSS) and classification procedures applied for the general inspection problem stated in this paper, i.e. the inspection of free-form objects using structured illumination. As stated in Sec. 2.4 it is assumed that for all considered free-form surfaces, it is possible to define an adapted lighting which produces “almost” free-form bright/dark structures, so that the requirements defined by Eq. (1), are fulfilled.

Testing the influence of various feature selection and pattern classification approaches would be beyond the scope of this paper. Previous investigations will therefore be considered.

Concerning the feature-based interpretation of stripe structures, three different rules were considered in [5]: the Naive Bayes, the One-Nearest-Neighbor and the Three-Nearest-Neighbor. Their influence on the classification of vertical periodical bright/dark structures was evaluated. The comparison of these three classificators showed that, in general better classification rates were obtained using the One-Nearest-Neighbor approach. This is a strong argument in terms of using “only” this approach for our purposes. Furthermore, Cover [27] and Guttierez [28] show that the k-NN method approaches the results of the Naive Bayes classifier in case of a large data set as we have here.

With the methodology, a 10-fold stratified validation, which is certainly the mostly used approach within the pattern classification community, was addressed in [5]. Various n-fold cross-validation approaches for different values of n have been evaluated and compared with the bootstrap technique by Kohavi [29]. He shows that a stratified 10-fold cross-validation is the more appropriate model in terms of classification accuracy. Moreover, Witten [30], referred that a ten times sampling is the right number of folds to get the best estimation error.

Thus, as our aim is to evaluate the two involved feature families, Fourier and adapted stripe features, and not a certain stripe pattern classification, in the following considered feature selection will be a 1-NN-wrapper-based procedure, whereas a 1-NN classification rule will be combined with a stratified 10-fold cross-validation for supervised pattern classification.

5. Proposed Four Steps Procedure: Results for Free-Form Surfaces

This section addresses the involved proposed four steps procedure for the determination of optimal feature subsets using feature evaluation, grouping, fusing, and selection in case of the general inspection problem stated in this paper, i.e. the inspection of free-form objects using structured illumination.

As stated before this evaluation is based on two feature families: 33 Fourier and 20 adapted stripe features. It has been demonstrated that these two sets of 33 and 20 features can be divided into four and two groups, see Eqs. (2) and (6). The evaluation criteria for each feature group is the rate R of correctly classified non-vertical and non-periodical bright/dark stripe patterns. This rate is expressed in percent.

The optimal feature groups are determined in the first subsection by means of the reference image data set Φ₀ ⁰. Then, the second section addresses evaluation of these considered feature groups in case of problem generalization, i.e. free-form surface quality control. The third section addresses the evaluation of FSS methods by considering the further eight reference databases defined for problem generalization to free-form surfaces. Finally, the last and fourth section is dedicated to the evaluation, in terms of types and number, of the previously selected features.

Evaluation criterion for all these investigations is the classification rate R, expressed in percent, which corresponds to the amount of correctly classified patterns for the 3 considered classes, Ω_A, Ω_R,3D, and Ω_R,2D. These 3 distinct pattern classes and the annotated data set Φ₀ ⁰ were considered, as these correspond to the qualification requirements of the reference industrial system [4].

Thus, changing this predefined pattern distinction and/or the reference data set, would have a direct impact on the results. In case of other applications, the number n of distinct pattern classes, but also the number n_i,i = {0,..,n - 1} of reference pattern for each considered class i, must be determined in accordance. Typical industrial applications for example consider 2-classes problems n = 2 with the same proportion of reference pattern n_i ≅ n_j ∀i ≠ j, {i,j} = {0, ..,n- 1}.

However, in case of the considered inspection task, other different approaches would have been possible, such as the consideration of two consecutive 2-classes procedures. It would have been possible to first classify all good and all bad patterns, Ω_A, {Ω_R,3D;Ω_R,2D}, and then to classify all 3D- and 2D-bad ones, Ω_R,3D, Ω_R,2D.

5.1. Fourier and Adapted Feature Groups Evaluation

Concerning the determination of optimal feature groups, Table 2 shows the classification results of image set Φ₀ ⁰ by means of vector c ^S made of the 20 adapted stripe features, vector c ^F _r,θ,v,u made of the 33 Fourier features, and the four and two feature group vectors described in Eqs. (2) and (6). Classification rates C_P and corresponding root mean square errors E_rms are listed.

Table 2. Rates R of correctly classified patterns for image set Φ₀ ⁰ with Fourier’s textural features and stripe adapted features by means of a 1-NN classifier.

View Table | View all tables in this article

In Sec. 3.1 the assumption was made that some frequency subbands could be more representative of the stripe patterns to be characterized. This is clearly observable in case of the results Rates for increasing distortions of types -1- and -2- for feature vectors c^F _r,θ,v,u c _θ ^F and c ^S.

 

Fig. 7. The detection rates were computed for different image sets and correspond to increasing distortions of type -1- and of type -2-. Left to right values: detection rates for image set Φ⁰ ₀ to image sets Φ⁴ ₁ and Φ⁴ ₂.

Download Full Size | PDF

listed in Table 2. A high discrepancy in the classification results is observable concerning the Fourier-based approach. Best classification rates of 92.4% are obtained when only the 10 directional Fourier features c _θ ^F are used. With the adapted features, the best rate could be obtained when all the 20 features are used. It is however noticeable that the 14 “free-form” features outperform the 6 “adapted” ones.

These first results show that from the 33 Fourier features and the 20 stripe features, the feature group made of the 10 directional Fourier features is particulary relevant in terms of stripe pattern characterization. Thus, further investigations are dedicated to the characterization of free-form surfaces using the 33 Fourier, the 10 directional Fourier, and the 20 adapted features sets and groups.

5.2. Feature Groups Evaluation for Free-Form Surfaces

The previously determined optimal feature groups for the reference patterns are now evaluated on all considered pattern sets, in order to tackle the inspection of free-form objects. These nine pattern sets were introduced in Sec. 2.4.

Figure 7 shows the classification rates for image distortions of type -1- and of type -2- by means of the three feature vectors c ^F _r,θ,v,u, c ^F _θ, c ^S.

For both types of distortions the 20 adapted stripe features lead to higher classification rates than when the complete 33 Fourier features are considered. The directional Fourier features are more characteristic of certain types of distortions. In case of the results depicted in Fig. 7, stripe distortions of type -2- (cylindrical) are better characterized using the directional Fourier features.

Rates for increasing distortions of type -1- and -2- for feature vectors c ^F _r,θ,v,u ^S,c ^FS _θ, and 1-NN c^FS _θ

 

Fig. 8. The detection rates were computed for different image sets and correspond to increasing distortions of type -1- and of type -2-. Left to right values: detection rates for image set Φ⁰ ₀ to image sets Φ⁴ ₁ and Φ⁴ ₂.

Download Full Size | PDF

The next step now consists of attempting to improve these results in terms of increasing the classification rates and decreasing the number of necessary features, by fusing and selecting the involved features.

5.3. Feature Groups Fusion and Selection for Free-Form Surfaces

This section investigates to what extend an appropriate fusion and selection of the Fourier and the adapted stripe feature sets can lead to a better quality control of the free-form surfaces. For this purpose, three different feature vectors will be considered. c ^F _r,θ,v,u ^S is a vector combining the 33 Fourier and the 20 adapted features, and c ^F _θ ^S is a vector made of the 10 directional Fourier and the 20 adapted features. Vector ^1-NN c ^F _θ ^S is made of the selected features of vector c ^F _θ ^S using a 1-NN-wrapper-based feature subset selection (FSS) method.

Figure 8 shows the classification rates for image distortions of type -1- and of type -2- by means of these three feature vectors.

On the whole, the reported classification rates in Fig. 8 are higher than those depicted in Fig. 7. Indeed, in the first case, more features or relevant selected features by means of a wrapper approach are considered.

These results show that fusing optimal feature groups leads to higher classification rates, which are in case of the considered problem, of approximately 2 % (difference between the maximal detection rates of both considered graphics).

Concerning the use of a feature selection method, both graphics of Fig. 8 show that for the considered sets of patterns, the considered FSS method does not improve the classification rates, but leads to similar classification results when the combined 10 directional Fourier and 20 adapted stripes are used.

Thus, the last investigation is dedicated to a more detailed depiction of the considered FSS method, in order to determine the relevant features.

5.4. Evaluation of Selected Features

The influence of increasing distortions of type -1- and of type -2- in the number and types of selected features using a wrapper 1-NN approach with a 10-Fold cross-validation are depicted in Tables 3 and 4.

Table 3. Selected features when a wrapper 1-NN approach is used, for increasing distortion of type -1-. The maximum number of times a feature can be selected is 10. The variables N_c,sub on the left give the total number of selected features after the 10 runs. The 10 time, 9 time and 8 time selected features are marked with ^***, ^** and ^*. Results for all relevant features are marked in bold.

View Table | View all tables in this article

Table 4. Selected features when a wrapper 1-NN approach is used, for increasing distortion of type -2-. The maximum number of times a feature can be selected is 10. The variables N_c,sub on the left give the total number of selected features after the 10 runs. The 10 time, 9 time and 8 time selected features are marked with ^***, ^** and ^*. Results for all relevant features are marked in bold.

View Table | View all tables in this article

An important parameter is the variable N_c,sub, which is the total number of selected features after the 10 runs of the 10-Fold cross-validation. As 10 is the maximum number of times a feature can be selected, N_c,sub/10 is the average measure of feature relevance. For both tables, increasing the distortion of the bright/dark structures, leads to an increase of the necessary relevant features.

A general remark for both tables concerns the types and the number of selected features, which are approximately the same. It appears that approximately seven features, i.e. only a fourth of the initial 30 ones, are relevant. Most of the selected features are adapted ones, whereas mainly the directional 90° Fourier features have a strong relevance.

It is also noticeable, that feature relevance is related to the bright/dark structure distortion degree. As an example, in case of both tables, the importance of feature c ^S ₁₃ is proportional to the distortion degree, whereas the contrary is observed for feature c ^S ₀₇.

6. Summary

In this paper, a general structured-illumination-based method for the characterization and interpretation of free-form and rough surfaces is proposed. Such a procedure was initially defined for the inspection of cylindrical specular industrial objects.

Starting from the reference image set defined for the qualification of the industrial process, eight additional image sets, corresponding to various types of free-form rough and specular shapes recorded with a structured illumination could be defined. All these image sets are used to search for the most appropriate pattern recognition process, in terms of retrieving the most adequate subset of features.

In order to address such an inspection task within a general approach, two different image content description methods, a Fourier-based approach and an adapted stripe-based technique, were considered. These methods necessitate the computation of a huge amount of 33 and 20 features, which signifies high computational costs. Hence, in order to propose a competitive solution adapted to real-time processes, extensive investigations were done to retrieve only the most relevant features that accurately classify the reference image sets.

A four steps feature evaluation, grouping, fusion, and selection procedure was taken into consideration. At first, each feature group is evaluated and compared individually. Then, the influence of various feature combination and selection techniques on the detection accuracy was evaluated. Finally, it has been demonstrated that feature grouping leads to an increase of at least 2 % of the classification rates, and that on average approximately a fourth of the initial features are relevant for free-form surfaces characterization by means of a structured illumination.