1. Introduction

Profile recovery is one of the most important studies in the optical detection field in view of the widely application prospects [1, 2]. Some typical cases include face recognition [3], vehicle or industrial parts inspection [4–6], reverse engineering [7], building crack inspection [8], cardiac optical mapping [9], etc. Two main methods are appropriate to solve the problem of profile recovery. Binocular vision is previously studied by many researchers [10–12]. The method adopts two cameras to capture the feature points on the measured object. Based on the triangulation among two cameras and the measured object, the object is reconstructed by the extracted feature points in the images [13, 14]. The difficulty of binocular vision is to match the corresponding feature points from two images. However, it is often a challenge to obtain the accurate feature point pairs, even though the epipolar constraint is considered in the matching process. The point matching is more difficult for an object with the smooth surfaces. The other reconstruction method is the active vision consisting of the line, plane or coded structured light [15, 16]. The line structured light takes the advantage of the high illumination of the laser projection, whereas only the point information is obtained from one captured image. The coded structured light utilizes the DLP projector to project a coded image on the measured object so that the pixels are able to be distinguished in the profile reconstruction. The coded structured light reconstructs the object profile with the high efficiency. As the illumination of the coded structured light generated from the DLP projector is weaker than the one of the laser projector, the coded-structured-light reconstruction tends to be impacted by the environmental light. Therefore, the profile reconstruction with the plane structured light from the laser projector is the moderate method considering both of the illumination and the reconstruction efficiency.

In the profile reconstruction method with a camera and the structured light, 1D [17], 2D [18–20] and 3D [21, 22] calibration references are studied in the previous works. J. Santolaria et al. [22] explained an approach to calibrate the laser sensors based on the 3D crenellated reference with feature points. The method is performed on the laser triangulation sensor in articulated arm coordinate measuring machine. J. E. Ha [23] interpreted a calibration method for the extrinsic parameters of the laser sensor and the camera. A 2D calibration reference with a triangle hole is designed for the calibration. S. Zhang et al. [24] explored a 2D-reference method to calibrate the structured light system for the 3D shape measurement. The projector is considered as a camera in the calibration. Therefore, the structured light system consisting of a camera and a projector is similar to the binocular vision system including two cameras. Furthermore, the calibration technology of the binocular vision is well established. J. Heikkila et al. [25] proposed a three step calibration method by employing 3D reference with the circular points. The mapping of the circle on the reference is reviewed to realize the camera calibration. Although the simple structure of the 1D reference is beneficial for the manufacture and operation, the 1D reference is rarely used for the lack of the calibration information. 2D reference is widely reported due to its flexible calibration process. However, the 2D reference method requires more than 3 images to perform the calibration. Hence, the 3D reference, taking the advantage of only one needed image, is employed as the basis of the system calibration and the profile reconstruction. In order to cover the large view-field of the camera, two cuboid references are designed to achieve the purpose of the profile recovery. Correspondingly, the reconstruction method with two 2D references can be considered as the same 2D reference on two different positions. In the 2D reference case, three or more reference images provide a global intrinsic parameter matrix. However, in the 3D reference case, a 3D reference image is enough to provide a local intrinsic parameter matrix. As there should be only one intrinsic parameter matrix for one camera to represent the essential characteristics of the camera, two or more 3D references in an image take the problem of data redundancy. Therefore, it is meaningful to solve the global intrinsic parameter matrix by a reasonable balance method, considering the data of both 3D references.

The rest of the paper is outlined as follows: In Section 2, a two-step method is presented to balance the intrinsic parameter matrices generated from two cuboid references. The first step is to calibrate the world-image transform matrices by two cuboid references and decompose the two extrinsic parameter matrices from the two world-image transform matrices. In the second step, the world coordinates of the two references are transformed to the coordinates in the same camera coordinate system by the decomposed extrinsic parameter matrices. The global intrinsic parameter matrix is solved by the coordinates in the camera coordinate system and the image projections of them. In Section 3, the flexible laser plane is determined by the homographies from the intersection laser lines on the two references to the projection laser lines in the image. The optimization laser plane and intrinsic parameter matrix are generated from the cost function that is contributed by the re-projection errors of the parameterized laser lines. The points on the measured object are reconstructed by the laser plane and the projection transform. Section 4 provides the comprehensive experiment results, which investigate the impact factors of the measurement distance from the camera to the measured object, the reference distance between two cuboid references, and the test distance on the benchmark ruler. Section 5 concludes the paper.

The reconstruction method takes two main benefits at the same time. The first benefit is to realize the profile reconstruction, camera calibration and laser plane calibration with the information from only one image. The second one is the flexible laser plane without strict movement constraint. The method is profitable to enhance the profile measurement in both fundamental research and potential applications.

2. Balance model of intrinsic parameter matrix

The profile reconstruction system, which consists of two cuboid references, a camera and a laser plane, is shown in Fig. 1. Two world coordinate frames O^W1-X^W1Y^W1Z^W1, O^W2-X^W2Y^W2Z^W2 are defined on the two cuboid references. The camera coordinate frame O^C-X^CY^CZ^C and the image coordinate frame o-xy are defined by the camera and the image, respectively. The flexible laser plane scans the measured object and the cuboid references. Therefore, the image of four intersection laser lines on the references and an intersection laser curve on the measured object is captured by the camera.

 

Fig. 1 Profile reconstruction method with two cuboid references and a flexible laser plane.

Download Full Size | PDF

According to the projection of the 3D point set on the two cuboid references $X_i^{(W,m)}=\left\{X_i^{(W,m,n)}\right\}$, the projection point on the image plane is given by [26]

The projection matrix P_m is decomposed by the method in [26] to

The intrinsic parameter matrix ${\text{K}}_1^{\text{(L)}}$ of the first cuboid reference should be theoretically identical to the intrinsic parameter matrix ${\text{K}}_2^{\text{(L)}}$ of the second cuboid reference. However, ${\text{K}}_1^{\text{(L)}}$ is actually unequal to ${\text{K}}_2^{\text{(L)}}$, as there are noises in the experimental image. In order to balance the noise impacts on the image of two cuboid references, we solve the globally intrinsic parameter matrix according to the feature point coordinates of the two references in the camera coordinate system. Based on the local extrinsic parameters ${\text{R}}_m^{\text{(L)}}$, $t_m^{\text{(L)}}$ solved by Eq. (2), the 3D point $X_i^{\text{(W,m})}$ on the m-th cuboid reference is transformed to

The image coordinates $x_i^{\text{(I,}1)}$ and $x_i^{\text{(I,2)}}$ are combined into $x_i^{\text{(I)}}$ while the 3D coordinates $X_i^{\text{(C,}1)}$, $X_i^{\text{(C,2)}}$ are combined into $X_i^{\text{(C)}}$. Considering the pinhole model in the camera coordinate system [26], the transform from the 3D coordinate $X_i^{\text{(C)}}$ to the 2D image coordinate $x_i^{\text{(I)}}$ is comprehensively expressed by

Let $x_i^{\text{(I)}}={(x_i^{\text{(I)}},y_i^{\text{(I)}},1)}^{\text{T}}$, $X_i^{\text{(C)}}={(X_i^{\text{(C)}},Y_i^{\text{(C)}},Z_i^{\text{(C)}},1)}^{\text{T}}$. Equation (4) is transformed to the non-homogenous equation as

The global intrinsic parameter matrix of ${\text{K}}^{(G)}$ can be determined by

From Eq. (2), the global extrinsic parameter matrix is estimated by

The solution process of the global intrinsic parameter matrix and the global extrinsic parameter matrix is shown in Fig. 2.

 

Fig. 2 The solution diagram of the global intrinsic parameter matrix and the global extrinsic parameter matrix.

Download Full Size | PDF

3. Profile reconstruction model

In the left plane of the reference m, $X_i^{(W,m,1)}=0$. In the right plane of reference m, $Y_i^{(W,m,1)}=0$. Therefore, for the point on the left plane of the cubical reference, $X_i^{(W,m,1)}={\left[0,Y_i^{(W,m,1)},Z_i^{(W,m,1)},1\right]}^{\text{T}}$. For the point on the right plane of the cubical reference, $X_i^{(W,m,2)}={\left[X_i^{(W,m,2)},0,Z_i^{(W,m,2)},1\right]}^{\text{T}}$. According to Eqs. (1), (2), the point-point homography matrix from the left plane of the reference m to the image is ${\text{H}}^{(W,m,1)}{\text{=K}}^{(G)}[r_2^{(G,m)},r_3^{(G,m)},t_m^{\text{(G)}}]$. The point-point homography matrix from the right plane of the reference m to the image is ${\text{H}}^{(W,m,2)}{\text{=K}}^{(G)}[r_1^{(G,m)},r_3^{(G,m)},t_m^{\text{(G)}}]$. $r_1^{(G,m)},r_2^{(G,m)},r_3^{(G,m)}$ are the three column vectors of ${\text{R}}_m^{\text{(G)}}$.

The line-line homograhpy from the left plane or right plane of the reference m to the image is ${{\text{(H}}^{(W,m,n)}\text{)}}^{\text{-T}}$ [26, 28]. Then, the 2D line on the left or right plane of the reference m is

Let ${\text{J}}_m\text{=}\left[\begin{array}{cc}{\text{R}}_m^{\text{(G)}}& t_m^{\text{(G)}}\\ 0^{\text{T}}& 1\end{array}\right]$. In the camera coordinate system, the left and right planes of the reference m transformed by J_m are

In the camera coordinate system, the two planes passing the line $l_j^{(m,n)}$ in Eq. (8) and vertical to the left and right planes of the reference m are

Thus, the 3D laser line on the reference m is represented by the Plücker matrix

The 3D laser line ${\text{L}}_j^{(m,n)*}$ represented by two planes is transform to the point-form Plücker matrix ${\text{L}}_j^{(m,n)}$ with the relationship of ${(l_j^{(m,n)\ast })}_{34}:{(l_j^{(m,n)\ast })}_{42}:{(l_j^{(m,n)\ast })}_{23}:{(l_j^{(m,n)\ast })}_{14}:{(l_j^{(m,n)\ast })}_{13}:{(l_j^{(m,n)\ast })}_{12}={(l_j^{(m,n)})}_{12}:$${(l_j^{(m,n)})}_{14}:{(l_j^{(m,n)})}_{23}:{(l_j^{(m,n)})}_{42}:{(l_j^{(m,n)})}_{34}$ [26]. The coordinate of the laser plane Π_j passing ${\text{L}}_j^{(m,n)}$ is

The laser plane is then solved by the singular value decomposition (SVD) method [30]. The solution process of the initial solution of the laser plane is shown in Fig. 3.

 

Fig. 3 The solution diagram of the initial solution of the flexible laser plane.

Download Full Size | PDF

In the general reconstruction case, the laser plane Π_j can be used to estimate the 3D points of the intersection laser curve on the measurement object. In order to improve the reconstruction accuracy, in the camera coordinate system, the intersection laser line is further parameterized by the laser plane Π_j, ${\text{R}}_m^{\text{(G)}}$ and $t_m^{\text{(G)}}$ as

The plane-form Plücker matrix ${\text{L}}_j^{(C,m,n)*}({\Pi }_j,R_m^{\text{(G)}},t_m^{\text{(G)}})$ is also transformed to the point-form Plücker matrix ${\text{L}}_j^{(C,m,n)}({\Pi }_j,R_m^{\text{(G)}},t_m^{\text{(G)}})$. Then, from Eq. (2), the point-form Plücker matrix ${\text{L}}_j^{(C,m,n)}({\Pi }_j,R_m^{\text{(G)}},t_m^{\text{(G)}})$ is projected to the image by the parameterized projection matrix as

The re-projection intersection laser lines are parameterized by Eq. (16). The cost function is constructed by the re-projection errors of the intersection laser lines on the two cuboid references and described by

The solution process of the optimization solution of the laser plane is shown in Fig. 4.

 

Fig. 4 The solution diagram of the optimization solution of the flexible laser plane.

Download Full Size | PDF

The optimization solutions ${\widehat{\Pi }}_j,{\widehat{K}}^{\text{(G)}},{\widehat{R}}_1^{\text{(G)}},{\widehat{t}}_1^{\text{(G)}},{\widehat{R}}_2^{\text{(G)}},{\widehat{t}}_2^{\text{(G)}}$ are derived from the minimum value of the cost function. As the intersection point $X_i^{\text{(O})}$ between the laser plane and the object is located on the optimization laser plane ${\widehat{\Pi }}_j$, it satisfies [24]

Therefore, the intersection laser point $X_i^{\text{(O})}$ on the measured object is reconstructed by Eq. (18), Eq. (4), the optimized intrinsic matrix ${\widehat{K}}^{\text{(G)}}$ and image coordinate $x_i^{\text{(O)}}$.

4. Experiments and discussion

A camera with 2048 × 1536 image resolution and two 50 mm × 50 mm × 400 mm references with 20 mm × 20 mm checkerboard pattern are employed to test the performance of the profile reconstruction method. Harris corner recognition [31] contributes the image coordinates of the feature points on the bi-cuboid references. 40 feature points are selected on each cuboid reference. The performance of the optimization method is verified by changing the measurement distance, the reference distance and the test distance in the experiments. The reconstruction errors of the optimization method are compared with the ones of the initial method to test the accuracy of the reconstruction method.

The verification method with the benchmark ruler is shown in Fig. 5. In order to evaluate the accuracy of the reconstruction method, a target with 5 feature points is considered as the benchmark ruler and the distance between two feature points is regarded as the test distance. The test distances are 20 mm, 30 mm, 40 mm and 50 mm, respectively. The reference distances between two cuboid references are 150 mm, 200 mm, 250 mm and 300 mm, respectively. In addition, the measurement distances from the measured object to the camera are 700 mm, 800 mm, 900 mm and 1000 mm. The reconstruction error between the reconstructed test distance and the real test distance on the ruler is adopted to verify the accuracy of the profile reconstruction method [32].

 

Fig. 5 Verification method with the benchmark ruler.

Download Full Size | PDF

The reconstruction results adopting the bi-cuboid references are shown in Fig. 6. Theexperiment setup of the reconstruction method is shown in Fig. 6(a). The verification experiment is shown in Fig. 6(b). Figures 6(c), 6(e), 6(g) and 6(i) are the images captured in the experiments. Four different objects are located between two cuboid references. Figures 6(d), 6(f), 6(h) and 6(j) are the reconstruction results of the four objects. The red spheres represent the points on the intersection lines between the laser plane and the objects in the camera coordinate system. The blue stars indicate the feature points of the bi-cuboid references. The reconstruction results accurately agree with the shape of the four objects.

 

Fig. 6 Profile reconstruction results adopting the parameterized re-projection errors of laser lines from the bi-cuboid references. (a) The experiment setup of the reconstruction method. (b) The verification experiment. (c) The captured image of the car model. (d) The reconstruction results of the car model. (e) The captured image of the square box. (f) The reconstruction results of the square box. (g) The captured image of the cylindrical tube. (h) The reconstruction results of the cylindrical tube. (i) The captured image of the flat surface. (j) The reconstruction results of the flat surface.

Download Full Size | PDF

In order to quantitatively analyze the accuracy of the reconstruction, the reconstruction distance errors λ_j of the optimization method and the initial method are demonstrated in Fig. 7. The statistic data are shown in Table 1.

 

Fig. 7 Reconstruction errors of the initial method and the optimization method. λ_j is the reconstruction error. IN indicates the initial method. OP indicates the optimization method. MD indicates measurement distance, mm. RD indicates reference distance, mm. (a) MD = 700, RD = 150. (b) MD = 700, RD = 200. (c) MD = 700, RD = 250. (d) MD = 700, RD = 300. (e) MD = 800, RD = 150. (f) MD = 800, RD = 200. (g) MD = 800, RD = 250. (h) MD = 800, RD = 300. (i) MD = 900, RD = 150. (j) MD = 900, RD = 200. (k) MD = 900, RD = 250. (l) MD = 900, RD = 300. (m) MD = 1000, RD = 150. (n) MD = 1000, RD = 200. (o) MD = 1000, RD = 250. (p) MD = 1000, RD = 300.

Download Full Size | PDF

Table 1. Statistic data of the reconstruction experiments.

View Table

When the measurement distance is 700 mm, the means of the reconstruction errors of the initial method are 1.13 mm, 1.16 mm, 1.18 mm and 1.22 mm, related to the reference distances of 150 mm, 200 mm, 250 mm and 300 mm. Moreover, the means of the optimization method are 1.07 mm, 1.09 mm, 1.12 mm and 1.15 mm. When the measurement distance increases to 800 mm, the means of the initial method are 1.13 mm, 1.15 mm, 1.18 mm and 1.20 mm, corresponding to the reference distance of 150 mm, 200 mm, 250 mm and 300 mm. then, the means of the optimization method are 1.06 mm, 1.09 mm, 1.11 mm and 1.13 mm. When the measurement distance grows up to 900 mm, the means of the initial method are 1.14 mm, 1.16 mm, 1.19 mm and 1.22 mm, under the reference distances of 150 mm, 200 mm, 250 mm and 300 mm. Furthermore, the means of the optimization method are 1.08 mm, 1.10 mm, 1.13 mm and 1.16 mm. When the measurement distance is 1000 mm, the means of the initial method are 1.15 mm, 1.18 mm, 1.21 mm and 1.25 mm with the reference distances of 150 mm, 200 mm, 250 mm and 300 mm. Moreover, the means of the optimization method are 1.09 mm, 1.11 mm, 1.14 mm and 1.18 mm.

The relative errors of the reconstruction distances are illustrated in Fig. 8. When the measurement distance is 700 mm, the means of the relative errors of the optimization method are 2.96%, 3.04%, 3.10% and 3.21%, under the reference distances of 150 mm, 200 mm, 250 mm and 300 mm. While the measurement distance is 800 mm, the means of the relative errors of the optimization method are 2.93%, 3.02%, 3.08% and 3.16%. When the measurement distance is 900 mm, the means of the relative errors of the optimization method are 3.00%, 3.06%, 3.14% and 3.23%. For the measurement distance of 1000 mm, the means of the relative errors of the optimization method are 3.03%, 3.10%, 3.19% and 3.29%.

 

Fig. 8 Relative errors of the initial method and the optimization method. δ_j is the reconstruction error. IN indicates the initial method. OP indicates the optimization method. MD indicates measurement distance, mm. RD indicates reference distance, mm. (a) MD = 700, RD = 150. (b) MD = 700, RD = 200. (c) MD = 700, RD = 250. (d) MD = 700, RD = 300. (e) MD = 800, RD = 150. (f) MD = 800, RD = 200. (g) MD = 800, RD = 250. (h) MD = 800, RD = 300. (i) MD = 900, RD = 150. (j) MD = 900, RD = 200. (k) MD = 900, RD = 250. (l) MD = 900, RD = 300. (m) MD = 1000, RD = 150. (n) MD = 1000, RD = 200. (o) MD = 1000, RD = 250. (p) MD = 1000, RD = 300.

Download Full Size | PDF

Finally, we can come to the conclusion that the absolute errors and relative errors of the experimental reconstruction errors obtained by the optimization method are smaller than those of the initial method. Therefore, the optimization method adopting parameterized re-projection errors of laser lines reduces the experimental errors effectively. The relative error slightly increases with the growth of the test distance. In addition, with the test distance increasing from 20 mm to 50 mm, the reconstruction errors present steady growing trends. The reconstruction errors show steady growing trends with the increasing reference distances from 150 mm to 300 mm. Therefore, when the distance between the bi-cuboid references is 150 mm, the reconstruction values are closer to the true valves. It is because that the references approach to the center of the image, where the image distortion is indistinctive. Besides, the measurement distance between the camera and the measured object is also considered as an impact factor. It is proved that the reconstructed errors are the smallest when the measurement distance is 800 mm. When the measurement distance is 700 mm, the reconstruction errors are slightly larger than the errors under the measurement distance of 800 mm, and are slightly smaller than the errors under the measurement distance of 900 mm. While the measurement distance is 1000 mm, the reconstructed errors are the largest in the four groups of experiments. In summary, the experimental results indicate that the reconstruction values are closest to the true values on the condition of the measurement distance of 800 mm, the test distance of 20 mm and the reference distance of 150 mm.

5. Conclusion

A profile reconstruction method based on the line re-projection errors of the laser plane and bi-cuboid references is proposed in this paper. Two world coordinate systems are built on the bi-cuboid references. The intrinsic parameter matrix of the camera is solved by the balance model of the two 3D references in one image. The flexible laser plane is modeled by the homographies from the references to the images and refined by the cost function of the line re-projection errors. The accuracy of the optimization method and the initial method are verified by comparing the reconstruction distance errors. The impact factors of the measurement distance, the reference distance and the test distance are investigated in this paper. The mean of the reconstruction errors of the initial method is 1.18 mm while the mean of the reconstruction errors of the optimization method is 1.11 mm. The mean of the relative errors of the initial method is 3.30% while the mean of the relative errors of the optimization method is 3.10%. The comparison results show that the optimization method is valuable for the vision-based measurement systems, due to the advantages of high accuracy and good stability. The potential applications include the inspection fields of vehicle manufacture, medical operation, robot navigation, etc. In the future, the object scale in the inspection will be studied to extend the application scopes.