Flexible dynamic measurement method of three-dimensional surface profilometry based on multiple vision sensors

Zhen Liu; Xiaojing Li; Fengjiao Li; Guangjun Zhang

doi:10.1364/OE.23.000384

1. Introduction

Industrial production sites are characterized by strong vibrations, narrow measurement space, and other complex environmental factors. A single vision sensor cannot measure an entire object because of its limited field of view. Therefore, multiple vision sensors need to be developed for the large-scale synchronous measurement of three-dimensional (3D) surface profilometry. However, multiple vision sensors are sensitive to strong vibrations, which reduce their measurement accuracy. Thus, the development of a dynamic measurement method for 3D surface profilometry with a high tolerance to strong field vibrations and other complex environmental conditions is crucial.

The current measurement methods for 3D surface profilometry can be classified as contact and non-contact. Contact measurement methods include manual measurements using fixtures and three-coordinate measuring machines. However, such methods have the low measurement efficiency and limited measurement range. Meanwhile, touch probes can damage some surfaces of measured objects. Non-contact measurement methods involve the use of laser trackers, 3D laser range finders, theodolites, and vision inspection technology. And this methods have high accuracy and widely application.

Vision inspection technology [1–4] has rapidly developed with the constant advancements in computer technology, electronics, optical technology, and image processing and pattern recognition technology. It has gradually become the principal measurement method for 3D surface profilometry because of its high degree of automation, large measurement range, high accuracy, and non-contact. Commonly used vision inspection technologies include passive and active vision inspection components.

Passive vision inspection [5–8] requires relatively simple equipment to obtain 3D surface information under natural lighting. Therefore, this technology has been widely applied to measure, for example, auto-body construction, large-scale antenna, and large aircraft. However, it is sensitive to the light intensity on site and to the texture of the measured object. Therefore, auxiliary methods such as adhesive mark points and spraying speckle patterns must be employed. Without these auxiliary means, passive vision inspection is inappropriate for the dynamic measurement of the 3D surface profilometry of objects with complex curved surfaces because there is not enough features for matching .

By projecting specially-controlled beams on the measured object, active vision technologies can obtain high-density 3D surface data. Active vision technologies include, for example, Fourier-transform profilometry [9,10], phase-measuring profilometry [11–13], and structured light vision inspection [14–19]. Fourier-transform profilometry can determine the high-density 3D surface profilometry of the measured object with only one image. This technology is suitable for dynamic measurements, but it requires a long operation time and has poor automatic performance. Despite its high measurement accuracy, phase-measuring profilometry is unsuitable for dynamic measurements because it requires images with some phase angle differences to be repeatedly projected at the same area. Some studies used color cameras to measure 3D surface profilometry with phase-measuring profilometry . However, this method is influenced by the color of the object itself and thus unsuitable in industrial fields. The structured light method is a 3D dynamic vision inspection method commonly used in industrial fields at present because of its simple equipment requirement, high degree of automation, and suitability for dynamic measurements. In this paper, the structured light method would be used to obtain 3D surface profilometry of objects.

A single vision sensor cannot measure the whole 3D surface profilometry of object because of occlusion. The measured area is generally divided into many subareas, and the 3D data of these subareas are integrated into the global coordinate system to obtain the 3D surface profilometry of an entire object. Vision inspection can be classified as flow vision inspection and multi-sensor vision inspection.

Flow-vision inspection can measures the 3D surface profilometry of an entire large-scale mechanical component by moving a single vision sensor around the measured object. All measured data of all subareas are integrated into the global coordinate system by using fiducial marks attached to the component. A typical example is ATOS developed by GOM (US). However, fiducial mark use is limited because they cannot be attached to soft objects, liquid, or high-accuracy machinery. In addition, this method is time consuming, and the fiducial marks are easily deformed. By contrast, flow-vision inspection has the advantages of simple equipment requirement and easy operation; it is also suitable for static measurements in industrial fields. However, it is unsuitable for the dynamic measurement of 3D surface profilometry.

Multi-sensor vision inspection [20–22] requires multiple vision sensors to perform global calibration before measurements and then integrates the data of all subareas measured by all vision sensors into the global coordinate system based on the calibration results. This method is commonly used in industrial fields, such as auto-body geometry measurement systems from Perceptron (US) and full-profile of rails measurement systems from MERMEC (Italy). Although multi-sensor vision inspection has a simple measurement principle, the global calibration of multiple vision sensors is difficult to conduct on site with this method. Furthermore, the measurement accuracy of the system after global calibration is largely influenced by strong vibrations on site.

To realize the dynamic measurement of the 3D surface profilometry of an object in the presence of strong vibrations, we combine a raster binocular stereo vision sensor and a wide-field camera to form a 3D optical probe. Then, multiple 3D optical probes and planar targets are arranged to form a flexible multi-sensor vision measurement system. Similar to the multi-sensor vision inspection, the proposed measurement system contains multiple vision sensors, but they are not rigidly fixed. The remainder of the paper is organized as follows. Section 2 introduces the structures and mathematical models of 3D optical probes and describes the basic principle of the algorithm. Section 3 presents a physical experiment in which rotating fans are measured using two 3D optical probes to verify the algorithm.

2. Basic measurement principle of the system

Three 3D optical probes are served as examples in the following to introduce the proposed method. The structural schematic of the measurement system is shown in Fig. 1 The system mainly consist of three 3D optical probes, three planar targets, a high-speed image acquisition system, a computer, system software, and the corresponding mechanisms. The basic principle of the measurement system is as follows. As required, the 3D optical probes are arranged around the measured object at a certain angle to ensure that the measurement range can cover the entire measured object. Then, the raster binocular stereo vision sensors of the 3D optical probes are operated to measure the surface of the local component in real time, and wide-field cameras are used to shoot the planar targets around the measurement site. Finally, the planar targets in the common field of view of the wide-field cameras are considered as the mediator, and the local 3D data obtained by all 3D optical probes are integrated into the global coordinate system.

Fig. 1 Structural schematic of the dynamic vision measuring system for the 3D surface profilometry of a large-scale component under complex site conditions.

Download Full Size | PDF

The specific measurement steps of the system are listed below.

Step 1: The coordinate systems of the 3D optical probes are set up below those of raster vision sensors, and the optical probes are calibrated. Specifically, the raster binocular stereo vision sensors, intrinsic parameters of the wide-field cameras, and the transformation matrices between the coordinate systems of the wide-field cameras and those of the raster binocular stereo vision sensors are calibrated.

Step 2: According to the size and shape of the measured object, the 3D optical probes are arranged around the measured object. The planar targets are set up in the common field of view of the different wide-field cameras. The global coordinate system is established below one of the coordinate systems of the 3D optical probes.

Step 3: During the actual measurement, the raster binocular stereo vision sensor of each 3D optical probe is used to measure the local 3D profilometry of the measured object. Meanwhile, the wide-field cameras are used to shoot planar targets. Then, the transformation matrix from the coordinate system of the 3D optical probe to that of the global coordinate system is calculated. Finally, the local 3D data measured by each 3D optical probe are integrated into the globe coordinate system.

Step 4: 3D optical probes are used to constantly shoot the measured object in dynamic change and planar targets. Step 3 is repeated to realize the dynamic vision measurement of the 3D surface profilometry of the object.

2.1 Mathematical model of the measurement system

The mathematical model of the measurement system principally consist of mathematical models of 3D optical probes and the global measurement model in the paper. Specifically, models of the raster binocular stereo vision sensor and the wide-field camera are used.

2.1.1 Mathematical model of a 3D optical probe

In this paper, each 3D optical probe principally consist of a raster binocular stereo vision sensor and a wide-field camera. The raster binocular stereo vision sensor is used to measure local 3D data, whereas the wide-field camera is used to integrate global data. The structural schematic of the 3D optical probe is shown in Fig. 2. The raster binocular stereo vision in each probe sensor is composed of two cameras and a projector, and the wide-field camera is composed of a camera and a mirror with four surfaces. The coordinate system $O_{o} x_{o} y_{o} z_{o}$ of the 3D optical probe is established below the $O_{l} x_{l} y_{l} z_{l}$ of the wide-field camera, whereas the coordinate system $O_{s} x_{s} y_{s} z_{s}$ of the raster binocular stereo vision sensor is set up below $O_{c1} x_{c1} y_{c1} z_{c1}$ of the left camera. $T_{os}$ is the transformation matrix from $O_{o} x_{o} y_{o} z_{o}$ to $O_{s} x_{s} y_{s} z_{s}$ .

Fig. 2 Structural schematic of the 3D optical probe.

Download Full Size | PDF

1) Mathematic model of a raster vision sensor

As shown in Fig. 3, the coordinate systems of the left and right cameras are $O_{c 1} x_{c 1} y_{c 1} z_{c 1}$ and $O_{c 2} x_{c 2} y_{c 2} z_{c 2}$ , respectively. The transformation matrix from the coordinate system of the left camera to that of the right camera is $T_{21} = [\begin{matrix} R_{21} & t_{21} \\ 0 & 1 \end{matrix}]$ ( $R_{21}, t_{21}$ are the rotation matrix and the translation vector, respectively). $r_{21}$ is the rodrigues representation of the rotation matrix $R_{21}$ . The coordinate system $O_{s} x_{s} y_{s} z_{s}$ of the raster binocular stereo vision sensor is set up above $O_{c 1} x_{c 1} y_{c 1} z_{c 1}$ . $p_{1} = {[u_{1}, v_{1}, 1]}^{T}$ and $p_{2} = {[u_{2}, v_{2}, 1]}^{T}$ are homogeneous coordinates of non-distorted images of the raster stripe point $P$ in the coordinate system of the image by the left and right cameras, respectively. $l_{1}$ is the epipolar line of $p_{2}$ in the image of the left camera, and $l_{2}$ is the epipolar line of $p_{1}$ in the image of the right camera. The raster stripe point P from the raster binocular stereo vision sensor is projected by the left and right cameras, respectively. Then, the binocular stereo vision model is used to calculate the 3D coordinates $P_{s} = [x_{s}, y_{s}, z_{s}, 1]$ of P in $O_{s} x_{s} y_{s} z_{s}$ :

{\begin{cases} ρ_{1} p_{1}^{} = K_{1} [\begin{matrix} I & 0 \end{matrix}] P_{s} \\ ρ_{2} p_{2}^{} = K_{2} [\begin{matrix} R_{21} & t_{21} \end{matrix}] P_{s} \end{cases}

where

K_{m} = [\begin{matrix} f_{x} & γ & u_{0} \\ 0 & f_{y} & v_{0} \\ 0 & 0 & 1 \end{matrix}] (m = 1, 2)

are the intrinsic parameters of the left and right cameras, respectively.

u_{0}

and

v_{0}

are the coordinates of the principal point.

f_{x}

and

f_{y}

are the scale factors in the image u and v axes and the parameter

γ

is the skew of the two image axes [23].

Fig. 3 Binocular stereo vision model.

Download Full Size | PDF

In the actual measurement, lens distortion occurred in the camera’s imaging system. $p_{d} = {(u_{d}, v_{d}, 1)}^{T}$ is the homogeneous coordinate of the distorted image, $p = {(u, v, 1)}^{T}$ is the homogeneous coordinate of the non-distorted image, and $p_{n} = {(x_{n}, y_{n}, 1)}^{T}$ is the homogeneous coordinate of the normalized image. The lens distortion model can be expressed as

\begin{array}{l} u_{d} = u + (u - u_{0}) (k_{1} r^{2} + k_{2} r^{4}) \\ v_{d} = v + (v - v_{0}) (k_{1} r^{2} + k_{2} r^{4}) \end{array}

where

r = \sqrt{x_{n}^{2} + y_{n}^{2}}

,

k_{1}

, and

k_{2}

are the coefficients of radial lens distortion.

2) Mathematical model of a wide-field camera

Existing wide-field cameras mostly adopt hyperboloid, spherical, and cylindrical mirrors. These cameras are difficult to use for modeling and possess low measurement accuracy. Hence, they are unsuitable for high-accuracy industrial measurements and principally used for video monitoring and target recognition. To realize high-accuracy wide-field measurements, we use a wide-field, high-resolution camera with a four-surface mirror. The camera is suitable for multi-angle measurements, as shown in Fig. 4. A picture of the wide-field camera with a four-surface mirror is shown in Fig. 4(b). The wide-field camera can be designed to have a flat mirror depending on actual needs.

Fig. 4 Structural schematic of the wide-field camera. (a) Sketch map of the wide-field camera with a four-surface mirror. (b) Physical picture of the wide-field camera with a four-surface mirror.

Download Full Size | PDF

As shown in Fig. 4(a), $\begin{matrix} O_{m i} x_{m i} y_{m i} z_{m i} & (i = 1, 2, 3, 4) \end{matrix}$ is the coordinate system of the four-mirror camera of the wide-field camera. $T_{m 21}, T_{m3 1}, T_{m4 1}$ are the transformation matrixes from the coordinate system of mirror cameras 2, 3, and 4 to that of mirror camera 1, respectively. The coordinate system $O_{l} x_{l} y_{l} z_{l}$ of the wide-field camera is established based on the coordinate system $O_{m 1} x_{m 1} y_{m 1} z_{m 1}$ of mirror camera 1. According to Formula 3, the 3D coordinates $P_{o} = [x_{o}, y_{o}, z_{o}, 1]$ of the measuring point P of the raster binocular stereo vision sensor can be obtained under the coordinate system of the 3D optical probe:

P_{o} = T_{so} P_{s}

Therefore, using Eqs. (1)-(3), we can obtain the 3D coordinates $P_{o}$ of the measuring point P of the raster binocular stereo vision sensor under the coordinate system $O_{o} x_{o} y_{o} z_{o}$ of the 3D optical probe.

2.1.2 Global measurement model of the system

The global integration of the measured data of the system is shown in Fig. 5. In the figure, $T_{o i, t j}$ is the transformation matrix from the coordinate system of the 3D optical probe to that of the planar target (oi is the i-th 3D optical probe, and tj is the j-th planar target). $T_{o i, t j}$ can be solved as previously described in [23] using the planar targets captured by the mirror camera of the wide-field camera.

Fig. 5 Schematic of the global integrated model.

Download Full Size | PDF

The global coordinate system is set up on the basis of the coordinate system of 3D optical probe 1. The local 3D data measured by 3D optical probes 2 and 3 can be integrated into the global coordinate system through the following equation:

{\begin{cases} P_{G1}^{} = T_{N} P_{o1}^{} \\ P_{G2}^{} = T_{o 1, t 1}^{- 1} T_{o 2, t 1} P_{o2}^{} \\ P_{G3}^{} = T_{o 1, t3}^{- 1} T_{o 3, t 3} P_{o3}^{} \end{cases}

where

P_{o1}^{}

,

P_{o2}^{}

, and

P_{o3}^{}

are 3D data of the local topographies measured by the raster binocular vision sensors of 3D optical probes 1, 2, and 3, respectively, under the corresponding coordinate systems;

P_{G1}^{}

,

P_{G2}^{}

, and

P_{G3}^{}

are the coordinates of

P_{o1}^{}

,

P_{o2}^{}

, and

P_{o3}^{}

under the global coordinate system, respectively; and

T_{N}

is the unit matrix.

2.2 Rapid extraction and recognition of light stripe center

2.2.1 Rapid extraction of light stripe center

In [24], Steger proposes an algorithm for extracting a light stripe center based on the Hessian matrix. In this method, the Hessian matrix used to calculate every image point needs to use a Gaussian convolution mask to convolute the image. In this paper, a method similar to Steger’s method [24] is proposed to realize the automatic extraction of light stripe.

1) Extracting the sub-pixel coordinate of the light stripe center

The ideal light stripe center is the vertex of its grayscale surface. The maximum eigenvalue corresponding to the Hessian matrix is related to the local maximum of the second-order derivative of the grayscale surface.

H (u, v) = [\begin{matrix} r_{u u} & r_{u v} \\ r_{u v} & r_{v v} \end{matrix}]

Where r_uu, r_uv, and r_vv are second order partial derivatives of image intensity function

I (u, v)

. They can be computed by convolving the image with the corresponding differential forms of Gaussian kernel as follows:

\begin{array}{l} r_{u u} = g_{u u} (u, v) \otimes I (u, v) \\ r_{u v} = g_{u v} (u, v) \otimes I (u, v) \\ r_{v v} = g_{v v} (u, v) \otimes I (u, v) \end{array}

Where

g_{u u}

,

g_{u v}

and

g_{v v}

are second order Gaussian convolution kernel.

The eigenvector of H(u, v) corresponding to the largest eigenvalue represents the normal direction of light stripe, denoted by n(t) = (n_u, n_v)^T. The grayscale distribution function of the neighborhood, where the initial point $(u_{i}, v_{i})$ of the ith light stripe center lies, is subject to second-order Taylor expansion. The first-order derivative of the light stripe center in the normal direction of the light stripe is taken as zero, and the sub-pixel coordinates of this point are calculated.

{\begin{matrix} u_{i}^{'} = u_{i} - \frac{n_{u} r_{u} + n_{v} r_{v}}{n_{u}^{2} r_{u u} + 2 n_{u} n_{v} r_{u v} + n_{v}^{2} r_{v v}} n_{u} \\ v_{i}^{'} = v_{i} - \frac{n_{u} r_{u} + n_{v} r_{v}}{n_{u}^{2} r_{u u} + 2 n_{u} n_{v} r_{u v} + n_{v}^{2} r_{v v}} n_{v} \end{matrix}

$(t n_{u}, t n_{v}) \in [- \frac{1}{2}, \frac{1}{2}] \times [- \frac{1}{2}, \frac{1}{2}]$ ( $t = - \frac{n_{u} r_{u} + n_{v} r_{v}}{n_{u}^{2} r_{u u} + 2 n_{u} n_{v} r_{u v} + n_{v}^{2} r_{v v}}$ ) is determined, i.e., if the point with zero first-order derivative lies within the current pixel, this point $(u_{i}^{'}, v_{i}^{'})$ is the sub-pixel coordinate of the light stripe center. The details of the process are introduced in [24].

The extracting result of the sub-pixel coordinate of the light stripe center is shown in Fig. 6. There are some false light stripe center points at the ends of light stripes except for the right light stripe center points, as shown in Fig. 6(a).

Fig. 6 Extracting result of the light stripe. (a) Extracting result of the sub-pixel coordinate of the light stripe center. (b) Extracting result of the light stripe center after linking.

Download Full Size | PDF

2) Linking the light stripe center

In the above, the sub-pixel coordinates of light stripe center points and the orientation of light stripe at these points are obtained. Direction constraint is added in the process of linking light stripe center points in order to eliminate false sub-pixel coordinates of light stripe, as shown in Fig. 6(a).

The whole images are searched for light stripe center points, from top to bottom and left to right. The first pixel with qualified light stripe center point is called the initial pixel of a line to be linked. Then this pixel is appointed as the current pixel. Three neighboring pixels compatible with the line orientation at current pixel are examined. One of the qualified neighboring pixels is added to the line, and becomes the new current pixel. This process is repeated until there’s no qualified light stripe center point to add.

There are two criterions for a qualified neighboring pixel: 1) The angle between the line orientation at the pixel and the current pixel is less than a threshold.2) The distance between the sub-pixel center point locations of the pixel and that of the current pixel is less than a threshold. If there exist more than one qualified neighboring points, the point with minimum distance to the current point is selected.

The linking process of a light stripe ends when there’s no qualified neighboring pixel for the current pixel. Then, initial point of a new light stripe is searched, and the linking process is repeated for the new light stripe. This process continues until all the pixels of the image have been examined.

As shown in Fig. 6(b), false light stripe center points and background light stripes are removed after forcing the direction constraints in the light stripe linking process. The linking method not only eliminates false sub-pixel coordinates of light stripe, but also links the light stripe centers to form multiple line segments, as shown in Fig. 6(b).

2.2.2 Fast matching of raster light strips

In Section 2.2.1, light stripe centers are extracted and connected to obtain multiple light stripe segments. The following content describes a fast method for matching light stripes from the left and right cameras of the raster binocular stereo vision sensor.

Suppose the raster binocular stereo vision sensor has been calibrated, then the plane equation $a_{i} x + b_{i} y + c_{i} z + d_{i} = 0$ (i = 1, 2, 3…n) of all light stripes projected by the raster projector in the coordinate system of the raster binocular stereo vision sensor are also given.

According to the principle of epipolar constraint in the binocular stereo vision measurement, in the light stripe image shot by the right camera, the epipolar $l_{2}$ corresponding to any point $p_{1}$ in the raster light stripe from the left camera can be determined. Point $p_{2}$ corresponding to point $p_{1}$ must be the intersection point of the epipolar $l_{2}$ and the raster light stripe, as shown in Fig. 7.

Fig. 7 Schematic of epipolar constraint.

Download Full Size | PDF

If point $p_{2}$ corresponding to $p_{1}$ can be found, then $p_{1}$ and $p_{2}$ are substituted into Formula (1) to calculate the 3D coordinates of this point under the coordinate system of the raster binocular stereo vision sensor. Moreover, if the plane equation of the point is given, then the 3D coordinates $P = {[x, y, z]}^{T}$ of this point can be calculated through Eq. (8) under the coordinate system of the raster binocular stereo vision sensor.

{\begin{array}{l} ρ p = K_{1} (I \begin{matrix}  \end{matrix} 0) P \\ a x + b y + c z + d = 0 \end{array}

where

(a, b, c, d)

is the coefficient of plane equation of the light plane under the coordinate system of the raster binocular stereo vision sensor,

p

is the homogeneous coordinate of the raster light stripe point under the coordinate system of image by Camera 1, and

K_{1}

is the intrinsic parameter of Camera 1.

Based on the above conditions, the concrete steps for the fast matching of light stripes are as follows. First, any light stripe is selected from the light stripes by the left camera, denoted as Stripe 1; any point $p_{1}$ is chosen in Stripe 1. Second, based on the solved light plane equations, set $Q_{1}$ of all possible 3D coordinate points of $p_{1}$ is solved by Eq. (8). Third, according to Eq. (1), the set of 3D coordinate points $Q_{2}$ where $p_{2}$ in the epipolar $l_{2}$ of the right camera intersects with all raster light stripes is calculated. Fourth, a pair of points closest in the set is found. This pair of points is the corresponding points of left and right cameras, and the matching of light stripes of the left and right images is realized. Fifth, steps 1–4 are repeated to realize the matching of all light stripes by left and right images.

2.3 Calibration of a 3D optical probe

In this paper, the calibrations of a 3D optical probe mainly include the calibration of a raster binocular stereo vision sensor, a wide-field camera, and transformation matrix T_so between the coordinate system of the wide-field camera and that of the raster binocular stereo vision sensor. The calibrations of a raster binocular stereo vision sensor include the calibration of intrinsic parameters of a wide-field camera and the calibration of transformation matrixes $T_{m 21}, T_{m3 1}, T_{m4 1}$ between the coordinate systems of a four-mirror camera. In this paper, the calibrations of a raster binocular stereo vision sensor and the intrinsic parameters of a wide-field camera are performed using the method in [23,25]. The calibrations of $T_{m 21}, T_{m3 1}, T_{m4 1}$ , and T_os are performed using the method in [26], as shown in Fig. 8.

Fig. 8 Schematic of calibration process of $T_{m4 1}$ .

Download Full Size | PDF

The coordinate system of a wide-field camera is established below the coordinate system 1 of the mirror camera, and the calibration process of $T_{m 21}, T_{m3 1}, T_{m4 1}$ , and T_os is as shown below.

Step 1: Two planar targets are placed in the fields of view of mirror cameras 1 and 4, the 3D optical probe is moved at least twice. Each mirror camera captures the image of the corresponding planar target. Using the constraint that the relative positions of the two planar targets are invariable, T_m41 can be computed. By repeating the above process, T_m21, T_m31and T_so also can be computed.

The flatness and the coating method of the mirror can affect the measurement accuracy. To get high measurement accuracy, the first surface mirror with high accuracy would be used in the real measurement system. The flatness of the first surface mirror with four surfaces used in our physical experiments is about 1 um.

3. Physical experiments

Two 3D optical probes are used in this experiment to verify the effectiveness of the proposed algorithm. The setup of the physical experiment is shown in Fig. 9. The configurations of the two 3D optical probes are identical in our experiment. The raster binocular stereo vision sensor of the 3D optical probes consists of two cameras (AVTGC1380H, 17 mm Schneider lenses, with a resolution of 1360 × 1024 and a field of view of 500 mm × 380 mm × 400 mm) and one projector (DellM110, with a resolution of 1360 × 768). The wide-field camera of the 3D optical probes consists of one camera (Pointgray, 12 mm Schneider lenses, with a resolution of 2448 × 2048) and one mirror with four surfaces. The characteristic point of the planar target is unified as 10 × 10, with a machining accuracy of 0.05 mm.

Fig. 9 Layout of physical experiment.

Download Full Size | PDF

3.1 Result of system calibration

The two 3D optical probes are calibrated according to Section 2.3. The characteristic point of planar target is unified as 10 × 10, with a machining accuracy of 0.05 mm.

Typical images used for the calibration of 3D optical probes are shown in Figs. 10(a)-10(d). As shown in Fig. 10(a), the planar target images captured by the cameras on the two sides of the raster binocular stereo vision sensor of probe 1 are used to calibrate the intrinsic parameters of cameras and the transformation matrix between the coordinate systems of the two cameras, respectively. As shown in Fig. 10(b), the images captured by the wide-field camera are used to calibrate the intrinsic parameters of wide-field camera. The target images captured by the wide-field camera, also shown in Fig. 10(c), are used to calibrate the transformation matrixes of the coordinate system of four mirror cameras. The images captured by the raster binocular stereo vision sensor and the wide-field camera are shown in Fig. 10(d) and used to calibrate the transformation matrix between the coordinates of the two. The result of parameter calibration of probe 1 is shown in Table 1.

Fig. 10 Typical images used for the calibration of 3D optical probes.(a) Images captured by the raster binocular stereo vision sensor.(b) and (c) Images captured by the wide-field camera.(d) Images captured by the raster binocular stereo vision sensor and the wide-field camera.

Download Full Size | PDF

Table 1. Result of parameter calibration of 3D optical probe 1

View Table | View all tables in this article

The typical images used for the calibration of probe 2 are similar to those used for probe 1.The result of parameter calibration of probe 2 is shown in Table 2.

Table 2. Result of parameter calibration of 3D optical probe 2

View Table | View all tables in this article

The abovementioned result is the calibration result of the two 3D optical probes. The following accuracy evaluation test and dynamic measurement test are performed to verify the effectiveness of the proposed algorithm.

3.2 Evaluation of global measurement accuracy

A self-designed method is used to evaluate global measurement accuracy. The specific experimental procedure is as follows: The characteristic points attached to the mechanical part are measured by the high accuracy 3D measurement device(HAMD)(the measurement accuracy of HAMD is better than 0.01 mm), as shown in Fig. 11(a). The mechanical part is place behind two 3D optical probes. The 3D optical probes 1 measures the characteristic points attached to the left part of the mechanical part and 3D optical probes 2 measures the characteristic points attached to the right part of the mechanical part. Finally, all characteristic points are transformed to the global coordinate system by the planar target, as shown in Fig. 11(b).Fourteen characteristic points are selected to form seven distances between two characteristic points measured by probe 1 and probe 2, respectively. The distance obtained by the HAMD is calculated as the ideal distance d_t and the distance obtained by the proposed method is the measurement distance d_m. The deviation $Δ d$ between d_m and d_t and the root–mean–square (RMS) error are calculated to evaluate the global accuracy of the measuring system.

Fig. 11 (a) Schematic of global measurement accuracy evaluation. (b) Mechanical part

Download Full Size | PDF

The images captured by probe 1 are shown in Fig. 12(a). The images captured by probe 2 are shown in Fig. 12(b).

Fig. 12 Images captured by two probes for the evaluation of global measurement accuracy.(a) Images captured by probe 1.(b) Images captured by probe 2.

Download Full Size | PDF

All measurement results listed in Table 3 show that the RMS error is about 0.12 mm. All characteristic points measured by the HAMD and the proposed method are transformed to the same coordinate system by the ICP method [27], and the RMS error of the corresponding characteristic points distances is 0.11 mm. All characteristic points transformed to the same coordinate system are shown in Fig. 13.

Table 3. Evaluation result of global measurement accuracy

View Table | View all tables in this article

Fig. 13 3D coordinates of all characteristic points transformed to the same coordinate system.

Download Full Size | PDF

3.3 Dynamic measurement test

A physical experiment is designed to verify the effectiveness of the algorithm in dynamic measurement. The specific procedure is as follows. Two 3D optical probes are placed before and behind the electric fan, respectively. One planar target is placed within the common field of view of wide-field cameras of two probes, as shown in Fig. 9. Turning on the electric fan, the 3D surface profilometry of the two surfaces of electric fan is measured with the two 3D probes, respectively. Using the planar target in the common field of view of wide-field cameras, the local 3D data measured by the two probes are integrated into one coordinate system.

Given that our cameras are not high speed, the rotating speed of fan blades is set at a low speed. Images of fan blades captured by probe 1 and 2 are shown in Fig. 14.

Fig. 14 Images captured by probe 1 and 2. (a) Images captured by probe 1. (b) Images captured by probe 2.

Download Full Size | PDF

The 3D surface profilometry of fan blades is obtained based on images of Fig. 14, as shown in Fig. 15. The 3D surface profilometry of six groups of continuously rotating fan blades is shown in Fig. 16.

Fig. 15 3D surface profilometry of fan blades.(a) 3D surface profilometry of fan blades measured by probe 1.(b) 3D surface profilometry of fan blades measured by probe 1.(c) integration result of (a)and(b).

Download Full Size | PDF

Fig. 16 3D surface profilometry of six groups of continuously rotating fan blades.

Download Full Size | PDF

4. Conclusion

When measuring the dynamic 3D surface profilometry of the object surface, the accuracy of the measurement system composed of several rigidly fixed vision sensors is vulnerable to the environmental vibration. To promote the accuracy of on-site measurement, a novel dynamic vision measurement system of 3D surface profilometry is introduced in this paper.

Compared with the existing methods, the highlight of our method is the use of 3D optical probe composed of a raster binocular stereo vision sensor and wide-field cameras. On the measurement site, the positions of 3D optical probes can be flexibly regulated, and need not be rigidly fixed. Therefore, the influence of on-site vibration on the measurement accuracy is reduced. Meanwhile, the proposed method is simple and flexible, and need not the global calibration on site. Physical experiment confirms that when the size of the measured object is about 0.7 m and the measuring range of raster binocular stereo vision sensor is about 0.5 m × 0.38 m × 0.4 m, the accuracy of the proposed method could reach 0.12 mm. Moreover, the test with rotating fan blades also confirms the effectiveness of the method in dynamic measurement.

Acknowledgments

The authors acknowledge the support from National Natural Science Foundation of China under Grant No. 51175027 and the Beijing Natural Science Foundation under Grant No. 3132029.

References and links

1. S. Shirmohammadi and A. Ferrero, “Camera as the instrument: the rising trend of vision based measurement,” IEEE Trans. Instrum. Meas. 17(3), 41–47 (2014). [CrossRef]

2. S. Zhang, “Recent progresses on real-time 3D shape measurement using digital fringe projection techniques,” Opt. Lasers Eng. 48(2), 149–158 (2010). [CrossRef]

3. E. N. Malamas, E. G. M. Petrakis, M. Zervakis, L. Petit, and J. D. Legat, “A survey on industrial vision systems, applications and tools,” Image Vis. Comput. 21(2), 171–188 (2003). [CrossRef]

4. F. Chen, G. M. Brown, and M. Song, “Overview of three-dimensional shape measurement using optical methods,” Opt. Eng. 39(1), 10–22 (2000). [CrossRef]

5. S. W. Jung, J. Y. Jeong, and S. J. Ko, “Sharpness Enhancement of Stereo Images Using Binocular Just-Noticeable Difference,” IEEE Trans. Image Process. 21(3), 1191–1199 (2012). [CrossRef] [PubMed]

6. M. Ceccarelli, A. Speranza, D. Grimaldi, and F. Lamonaca, “Automatic detection and surface measurements of micronucleus by a computer vision approach,” IEEE Trans. Instrum. Meas. 59(9), 2383–2390 (2010). [CrossRef]

7. Z. G. Ren, J. R. Liao, and L. L. Cai, “Three-dimensional measurement of small mechanical parts under a complicated background based on stereo vision,” Appl. Opt. 49(10), 1789–1801 (2010). [CrossRef] [PubMed]

8. W. M. Li and Y. F. Li, “Single-camera panoramic stereo imaging system with a fisheye lens and a convex mirror,” Opt. Express 19(7), 5855–5867 (2011). [CrossRef] [PubMed]

9. X. Y. Su, W. J. Chen, Q. C. Zhang, and Y. P. Chao, “Dynamic 3-D shape measurement method based on FTP,” Opt. Lasers Eng. 36(1), 49–64 (2001).

10. E. Zappa and G. Busca, “Static and dynamic features of Fourier transform profilometry: A review,” Opt. Lasers Eng. 50(8), 1140–1151 (2012). [CrossRef]

11. L. Lu, J. T. Xi, Y. G. Yu, and Q. Guo, “New approach to improve the accuracy of 3-D shape measurement of moving object using phase shifting profilometry,” Opt. Express 21(25), 30610–30622 (2013). [CrossRef] [PubMed]

12. P. S. Huang, C. P. Zhang, and F. P. Chiang, “High-speed 3-D shape measurement based on digital fringe projection,” Opt. Eng. 42(1), 163–168 (2003). [CrossRef]

13. Y. J. Fu and Q. Luo, “Fringe projection profilometry based on a novel phase shift method,” Opt. Express 19(22), 21739–21747 (2011). [CrossRef] [PubMed]

14. X. Zhang, Y. F. Li, and L. M. Zhu, “Color code identification in coded structured light,” Appl. Opt. 51(22), 5340–5356 (2012). [CrossRef] [PubMed]

15. H. R. A. Basevi, J. A. Guggenheim, H. Dehghani, and I. B. Styles, “Simultaneous multiple view high resolution surface geometry acquisition using structured light and mirrors,” Opt. Express 21(6), 7222–7239 (2013). [CrossRef] [PubMed]

16. R. Q. Yang, S. Chen, Y. Wei, and Y. Z. Chen, “Robust and accurate surface measurement using structured light,” IEEE Trans. Instrum. Meas. 57(6), 1275–1280 (2008). [CrossRef]

17. J. Vargas, M. J. Terrón-López, and J. A. Quiroga, “Flexible calibration procedure for fringe projection profilometry,” Opt. Eng. 46(2), 023601 (2007). [CrossRef]

18. J. Vargas, T. Koninckx, J. A. Quiroga, and L. V. Gool, “Three-dimensional measurement of microchips using structured light techniques,” Opt. Eng. 47(5), 053602 (2008). [CrossRef]

19. J. Vargas and J. A. Quiroga, “Novel multiresolution approach for an adaptive structured light system,” Opt. Eng. 47(2), 023601 (2008). [CrossRef]

20. R. S. Lu, Y. F. Li, and Q. Yu, “On-line measurement of straightness of seamless steel pipe using machine vision technique,” Sens. Actuators A Phys. 94(1), 95–101 (2001). [CrossRef]

21. Q. Li and S. Ren, “A real-Time visual inspection system for discrete surface defects of rail heads,” IEEE Trans. Instrum. Meas. 61(8), 2189–2199 (2012). [CrossRef]

22. Y. Li, Y. F. Li, Q. L. Wang, D. Xu, and M. Tan, “Measurement and defect detection of the weld bead based on online vision inspection,” IEEE Trans. Instrum. Meas. 59(7), 1841–1849 (2010). [CrossRef]

23. Z. Y. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). [CrossRef]

24. C. Steger, “An unbiased detector of curvilinear structures,” IEEE Trans. Pattern Anal. Mach. Intell. 20(2), 113–125 (1998). [CrossRef]

25. J. Y. Bouguet, Camera calibration toolbox for Matlab. Available from: http://www.vision.caltech. edu/ bouguetj/ calib_doc/.

26. Z. Liu, G. J. Zhang, Z. Z. Wei, and J. H. Sun, “A global calibration method for multiple vision sensors based on multiple targets,” Meas. Sci. Technol. 22(12), 125102 (2011). [CrossRef]

27. P. J. Besl and N. D. McKay, “A method for registration of 3-D shapes,” IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992). [CrossRef]

	Parameter
Binocular stereo vision sensor	Intrinsic parameters of left camera	fx = 2737.3; fy = 2737.4; $γ$ = 0; u₀ = 703.8; v₀ = 533.5 k₁ = −0.20; k₂ = 0.28
	Intrinsic parameters of right camera	fx = 2732.7; y = 2732.9; u₀ = 693.7; v₀ = 540.3 k₁ = −0.21; k₂ = 0.46
	T ₂₁	r ₂₁ = [ 0.004,0.368,0.007]; t ₂₁ = [ −448.608,-12.586,52.362]
Wide-field camera	Intrinsic parameters	fx = 3687.4; fy = 3688.1; $γ$ = 0; u₀ = 1231.5; v₀ = 1000.2 k₁ = −0.15; k₂ = 0.14
Wide-field camera	T _m21	r _m21 = [-0.012,-1.559,-0.812]; t _m21 = [85.021,-57.214,95.326]
	T _os	r _os = [1.302,1.123,-0.974]; t _os = [223.711, −188.514, 52.276]

	Parameter
Binocular stereo vision sensor	Intrinsic parameters of left camera	fx = 2719.7; fy = 2719.8; $γ$ = 0; u₀ = 691.2; v₀ = 482.2 k₁ = −0.21; k₂ = 0.26
	Intrinsic parameters of right camera	fx = 2735.4; fy = 2735.5; $γ$ = 0; u₀ = 684.5; v₀ = 533.9 k₁ = −0.21; k₂ = 0.35
	T ₂₁	r ₂₁ = [ 0.032,0.387,0.076]; t ₂₁ = [ −483.616,-17.343,83.233]
Wide-field camera	Intrinsic parameters	fx = 3705.8; fy = 3706.5; $γ$ = 0; u₀ = 1222.5; v₀ = 997.4 k₁ = −0.14; k₂ = 0.26
Wide-field camera	T _m21	r _m21 = [-0.032,-1.546,-0.833]; t _m21 = [85.934, −54.729, 95.498]
	T _os	r _os = [1.456,1.003,-1.337]; t _os = [97.625, −103.644, 261.717]

Index	HAMD			The proposed method			dt(mm)	dm(mm)	$Δ d$ (mm)
Index	x(mm)	y(mm)	z(mm)	x(mm)	y(mm)	z(mm)	dt(mm)	dm(mm)	$Δ d$ (mm)
1	161.00	−171.84	228.42	88.97	3.99	1148.79	445.17	444.92	0.25
1	133.31	269.09	283.10	455.40	13.90	1400.96	445.17	444.92	0.25
2	114.60	208.87	171.45	33.99	64.95	1154.05	481.84	481.83	0.01
2	108.99	269.67	227.48	444.21	73.24	1406.64	481.84	481.83	0.01
3	106.70	218.99	143.40	21.39	92.55	1148.35	551.69	551.67	0.02
3	48.60	323.72	223.76	460.56	97.79	1482.18	551.69	551.67	0.02
4	127.90	201.97	129.71	46.79	103.09	1135.32	509.97	510.03	−0.06
4	84.59	301.32	199.65	459.49	109.84	1434.92	509.97	510.03	−0.06
5	45.69	−272.85	140.09	−55.78	103.08	1171.96	658.95	658.85	0.10
5	−10.99	379.52	213.54	478.84	128.71	1556.17	658.95	658.85	0.10
6	94.25	233.46	112.05	2.81	123.86	1143.26	598.43	598.47	−0.04
6	24.69	354.95	196.01	475.71	134.18	1509.89	598.43	598.47	−0.04
7	54.00	270.68	95.18	−49.53	144.67	1153.97	608.40	608.57	−0.17
7	60.56	332.90	171.33	474.87	146.86	1462.76	608.40	608.57	−0.17
							RMS error		0.12

	Parameter
Binocular stereo vision sensor	Intrinsic parameters of left camera	fx = 2737.3; fy = 2737.4; $γ$ = 0; u₀ = 703.8; v₀ = 533.5 k₁ = −0.20; k₂ = 0.28
	Intrinsic parameters of right camera	fx = 2732.7; y = 2732.9; u₀ = 693.7; v₀ = 540.3 k₁ = −0.21; k₂ = 0.46
	T ₂₁	r ₂₁ = [ 0.004,0.368,0.007]; t ₂₁ = [ −448.608,-12.586,52.362]
Wide-field camera	Intrinsic parameters	fx = 3687.4; fy = 3688.1; $γ$ = 0; u₀ = 1231.5; v₀ = 1000.2 k₁ = −0.15; k₂ = 0.14
Wide-field camera	T _m21	r _m21 = [-0.012,-1.559,-0.812]; t _m21 = [85.021,-57.214,95.326]
	T _os	r _os = [1.302,1.123,-0.974]; t _os = [223.711, −188.514, 52.276]

	Parameter
Binocular stereo vision sensor	Intrinsic parameters of left camera	fx = 2719.7; fy = 2719.8; $γ$ = 0; u₀ = 691.2; v₀ = 482.2 k₁ = −0.21; k₂ = 0.26
	Intrinsic parameters of right camera	fx = 2735.4; fy = 2735.5; $γ$ = 0; u₀ = 684.5; v₀ = 533.9 k₁ = −0.21; k₂ = 0.35
	T ₂₁	r ₂₁ = [ 0.032,0.387,0.076]; t ₂₁ = [ −483.616,-17.343,83.233]
Wide-field camera	Intrinsic parameters	fx = 3705.8; fy = 3706.5; $γ$ = 0; u₀ = 1222.5; v₀ = 997.4 k₁ = −0.14; k₂ = 0.26
Wide-field camera	T _m21	r _m21 = [-0.032,-1.546,-0.833]; t _m21 = [85.934, −54.729, 95.498]
	T _os	r _os = [1.456,1.003,-1.337]; t _os = [97.625, −103.644, 261.717]

Flexible dynamic measurement method of three-dimensional surface profilometry based on multiple vision sensors

Abstract

1. Introduction

2. Basic measurement principle of the system

2.1 Mathematical model of the measurement system

2.1.1 Mathematical model of a 3D optical probe

1) Mathematic model of a raster vision sensor

2) Mathematical model of a wide-field camera

2.1.2 Global measurement model of the system

2.2 Rapid extraction and recognition of light stripe center

2.2.1 Rapid extraction of light stripe center

1) Extracting the sub-pixel coordinate of the light stripe center

2) Linking the light stripe center

2.2.2 Fast matching of raster light strips

2.3 Calibration of a 3D optical probe

3. Physical experiments

3.1 Result of system calibration

3.2 Evaluation of global measurement accuracy

3.3 Dynamic measurement test

4. Conclusion

Acknowledgments

References and links

Cited By

Figures (16)

Tables (3)

Equations (8)

Optics Express