Open Access
Add to BibSonomy
Abstract
Large-field-of-view stereo vision system lacks flexible and high-precision calibration methods. To this end, we proposed a new distance-related distortion model based calibration method combining 3D points and checkerboards. The experiment indicates that the proposed method has a root mean square of fewer than 0.08 pixels for the reprojection error on the calibration dataset, and the mean relative error of length measurement in a volume of 5.0 m × 2.0 m × 16.0 m is 3.6‰. Compared with other distance-related models, the proposed model has the lowest reprojection error on the test dataset. Besides, in contrast to other calibration methods, our method offers enhanced accuracy and greater flexibility.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Vision-based large-field-of-view (LFOV) precision measurement is widely used in many fields, including large aviation components measurement [1], propeller measurement [2], shield machine size measurement [3], and other fields [4,5]. The calibration of the LFOV stereo vision system plays an important role in ensuring measurement accuracy. There has been a lot of research into the calibration of vision sensors. For example, Abdel-Aziz et al. proposed the direct linear transformation method (DLT) [6]. This method requires at least 6 control points, and can directly solve the internal and external parameters of the camera according to the 3D coordinates of the control points and the 2D image coordinates. However, the DLT method does not consider the influence of camera distortion, and the calibration accuracy cannot be satisfied. Yakimovsky [7] improved the DLT method by optimizing the camera distortion parameters. As 3D targets are difficult to manufacture, calibration methods based on 2D calibration objects are more flexible. Moreover, 2D targets can provide considerable reference points to effectively constrain the camera parameters. Tsai [8] proposes a two-step approach that implements camera calibration using radial constraints and optimization. And the Zhang calibration method [9], which uses planar checkerboard targets for calibration, is wildly used in close-range photogrammetry.
Calibration of the LFOV stereo vision system requires large calibration objects. However, large-size planar targets or 3D targets have the problems of high cost, suffering in manufacturing, and being inconvenient to carry. Therefore, many researchers focus on the calibration of the LFOV system based on virtual stereo targets. Those virtual 3D targets are constructed by measuring the pre-arranged targets through three-coordinate instruments (total station [10], laser tracker [1,11], and visual three-coordinate system [2]). For example, Marta Róg used a total station to obtain a large number of control points to calibrate the camera. According to the calibration report, the overall mean accuracy of this process is 0.24 mm, which was automatically calculated by the software based on residuals obtained on test points in the directions of coordinate system axes [10]. Yang used 3D targets which are obtained by laser tracker and cube-corner prism for calibration. This work constructs a virtual three-dimensional flexible control field and realizes the precise calibration of the binocular vision system. The experimental results show that under the conditions that the baseline is 120 mm and the working distance is 3m∼5 m, the residual error of spatial points is 0.3654 mm and the root mean square (RMS) error of the reconstructed distance is 1.0728 mm [11]. Hu calibrated the camera by using a digital close-range industrial photogrammetry system. With a camera resolution of 2448 × 2048 pixels, the reprojection error is less than 0.11 pixels. For the measuring volume of 10.0 m × 8.4 m × 3.6 m, the maximum absolute error of 3-D measurement is 0.42 mm [2]. Previous studies on the calibration methods of the LFOV stereo vision system have achieved calibration accuracies that can satisfy the measurement requirements in specific environments. However, these methods also have limitations, such as the considerable amounts of control points and the complexity of operation. These limitations can make it challenging to apply the method to specific scenarios, such as a scene of tens of meters.
Small targets have many advantages over large ones [9], such as ease of use and lower cost. Therefore, it is of great significance to be able to calibrate an LFOV stereo vision system with a small target. However, to cover the imaging view, small targets must be used close to the vision sensors. As a result, the measurement field and the calibration field are not unified, and the calibration result would be affected. Magill and Brown's study [12,13] of the optical imaging model shows that the lens distortion coefficient, primarily the radial distortion coefficient, is related to the magnification of imaging. However, in the traditional camera model [8,9], camera distortions are described using constant parameters. This theoretically explains that when the constant distortion model (CDM) is used, the inconsistency between the calibration field and measurement field will lead to inaccurate results. Therefore, new distortion models are established to solve the inconsistency. For example, Magil et al. proposed a distance-related model for different focusing planes [12]. By calibrating the distortion in two focal planes (these planes being perpendicular to the optical axis), Brown [13] et al. improved the generalization performance of the model based on Magil, and they gave a numerical derivation of the distortion coefficients in the unfocussed and focused planes. Based on Brown et al., Fryer [14] proposed a new distance-related model that overcomes the difficulty of Brown's model in describing large image distortions. However, there are empirical coefficients in this model that make it difficult to obtain good results in terms of generalization to different types of lenses. Alvarez [15] et al. derive a radial distortion model suitable for planar scenes that eliminates empirical coefficients and improves calibration flexibility compared to Fryer's model. Also, Alvarez proposed a camera calibration method by using the straight line information on the planar scene. However, this method needs to estimate the location of the focal plane, which introduces some parameters that are not necessary to solve. Sun [16] et al. proposed a distance-related model for solving the camera distortion coefficients in an arbitrary plane, and the 3D measurement accuracy in a 7.0 m × 3.5 m × 2.5 m space at 6 m object distance was greatly improved. However, the calibration method proposed by Sun requires an industrial robot and a large linear beam calibration plate to provide precise constraints. This approach is not flexible. Combining the work of Frye and Alvarez et al, Li [17] et al. proposed a model based on the equal-radius, which divides the image into several annular regions and models the camera distortion coefficients in each annular region, and they demonstrated the validity of the model experimentally in a close-up small depth of field (DOF) experimental condition. They proposed a calibration method that can reduce the position requirements for the checkerboard. But this method needs to fully occupy the field of view (FOV) and DOF with a checkerboard, which is almost impossible for LFOV measurement.
Therefore, to improve the calibration flexibility and calibration precision of the LFOV stereo vision system, a new distance-related distortion model and a new flexible calibration method are proposed in this paper. Firstly, compared to the existing distance-related models [16,17], the proposed model has the lowest reprojection error on the test dataset and it can alleviate overfitting. Secondly, our method can achieve satisfactory results in terms of re-projection error on 2D test datasets and 3D length measurement when using 5 3D points and 3 checkerboard images for calibration. Finally, our method has greater flexibility compared to the CDM-based method. The improvement in camera calibration accuracy with an increasing number of chessboards for the CDM-based method would be slight when a sufficient number of 3D points are available. In contrast, our approach can enhance the calibration accuracy further by increasing the number of checkerboards placed in close range when the 3D points are adequate, and the increase in calibration accuracy achieved by adding checkerboards nearby is equivalent to that of adding 3D points at a long-range distance.
The rest of the paper is organized as follows. Section 2 introduces our distance-related model. Section 3 gives the details of the procedures of the proposed calibration method. In Section 4, the calibration experiments and the accuracy verification are designed to verify the validity of our distance-related distortion model and calibration method. Section 5 presents the conclusion.
2. Mathematical model
2.1. Camera pinhole model with lens distortion
The pinhole imaging mode is the most common camera imaging model [9]. It can be described as:
In Eq. (2), x and y are the normalized image coordinates without distortion. $\Delta x$ and $\Delta y$ represent the amount of distortion. ${k_1}$ and ${k_2}$ are radial distortion coefficients. p1 and p2 are tangential distortion coefficients. r is the radial distance on the normalized plane.
2.2. Distance-related distortion model
The effect of object distance on imaging distortion, as shown in Fig. 1, can be divided into two categories [13]. One is caused by the focus position of the camera, as shown in Fig. 1(a); the other is caused by the position of the object in the DOF, as shown in Fig. 1(b). Figure 1(a) shows that while the points below the object distance s1 and s2 are respectively imaged on their focal plane, and the radial distance ${r_{{s_1}}}$ is equal to ${r_{{s_2}}}$ numerically, there is a clear gap between the distortion $\Delta {r_{{s_1}}}$ and distortion $\Delta {r_{{s_2}}}$. Figure 1(b) illustrates that while the focal length of the camera is fixed and the camera is focused at distance s, the object at distance s1 defocuses on the image plane. In this case, the distortion $\Delta {r_s}_{,{s_1}}$ at defocus position is different from that $\Delta {r_{{s_1}}}$ at the focus position. Sun [16] and Li [17] have conducted modeling on the aforementioned scenario. Despite both of the models exhibiting superior performance than constant parameter models on calibration datasets, they underperform when applied to test datasets. Also, as the number of calibrators decreases, the RMS of reprojection errors on calibration datasets decreases while the RMS of reprojection errors on test datasets increases. These phenomena are analogous to that of model overfitting in machine learning. To alleviate this phenomenon and improve the calibration accuracy, we took inspiration from hyperparameter tuning in integrated learning and proposed a new DRDM model by introducing a pair of hyperparameters based on the Sun and Li models.
Fig. 1. Effect of object distance on imaging distortion. (a) Influence of the camera’s focus position. (b) Influence of object distance in the DOF.
According to the previous work [13], the distortion parameters of two focal planes s1 and s2 can be used to describe the distortion of any focal plane sx:
When the focus position s is fixed, the distortion coefficient at any position sx in the DOF can be expressed by the coefficients of position s1 and position s2:
According to Eq. (8), we obtain:
If we eliminate the parameters related to s, such as ${\alpha _{s,{s_1},{s_x}}}$, we obtain:
3. Calibration method
This paper proposes a new LFOV stereo vision system calibration method that integrates checkerboards with 3D points. The internal parameters of the LFOV camera and the external structural parameters between the vision sensors are the main solving objectives. The calibration principle is shown in Fig. 2. In our method, checkerboards are used as close-range targets, and 3D calibration points are used as long-range targets, and both targets are guaranteed to be in the DOF of the LFOV camera. The 3D calibration points are placed in the public FOV of the LFOV stereo vision system, while the checkerboard targets are placed in the non-public FOV.
The entire calibration procedure, shown in Fig. 3, can be divided into four steps. (1) We process the target images in the DOF and then use the image data to achieve the initial calibration of the LFOV camera. (2) Next, we calculate the initial value of the external parameters and obtain the initial value of the distance-related distortion coefficients. (3) Then, we use the nonlinear optimization algorithm Levenberg-Marquardt (LM) [18] to obtain the optimal parameters of the LFOV camera based on DRDM. As shown in Fig. 3, ${{\boldsymbol R}_{\textrm{CB}}}$, ${{\boldsymbol t}_{\textrm{CB}}}$ are external parameters of checkerboard, and ${{\boldsymbol R}_{\textrm{TS}}}$, ${{\boldsymbol t}_{\textrm{TS}}}$ are external parameters of the total station. (4) Finally, after completing the above three steps, we globally optimize the parameters of the left and right cameras. ${{\boldsymbol R}_{LR}}$, ${{\boldsymbol t}_{LR}}$ in Fig. 3 are the rotation matrix and translation vector from the left to the right camera frame. Here, the calibration work of the LFOV stereo vision system is completed.
3.1. Initial parameters estimation
The initial parameter estimation is a two-step method. First, when the image coordinates of points p1, p2, and p3 are given, it is easy to calculate the external camera parameters using the p3p method in the literature [19], where the manufacturing parameters of the camera are used as the internal parameters. However, this p3p method has to solve a quadratic equation, and the solution of the quadratic equation is not unique. Therefore, verification points p4i are added in this paper, and the reprojection errors of these points are used to filter the external camera parameters. Usually, two test points are sufficient. Second, since the camera distortion is not considered in the calculation of the external parameters, further optimization is required. The form of the optimization function is as follows:
3.2. Distance-related parameters estimation
According to the initial camera parameters estimated in the previous section, the depth value of the standard point can be obtained from Eq. (14).
If there are n points, the matrix equation can be obtained by combining all the equations like Eq. (15), as follows:
Then, the least square method is used to solve the problem.
3.3. Optimization based on geometric constraints
To obtain more reliable camera parameters, optimization is essential. Therefore, how to design the minimizing objective function becomes the primary problem. This paper proposes a multi-constraint optimization method based on the imaging model and geometric plane constraints.
First, the distance from the model projection point to the extraction point should be minimum, therefore, for a single camera, its minimum objective function is:
Second, according to the literature [20], the projection of a straight line is still a straight line. Therefore, we use the straightness of the correction point as a geometric constraint. In this paper, two equations are used to describe the straightness:
- (1) As shown in Fig. 4(a), the distance ${d_s}$ between the correction point and its fitted line ${l_w}$ can be expressed by:
where ${p_{w,n}}$, ${p_{w,m}}$, ${p_{w,s}}$ are the three non-overlapping points on the wth line. All these undistort points can be obtained by Eq. (15). Meanwhile, checkerboards can provide b straight lines and there are t characteristic points on a line. Therefore, the first minimal line-constraint objective function is:
- (2) As shown in Fig. 4(a), the vector angle θ of any three points ${p_{w,n}}$, ${p_{w,m}}$, ${p_{w,s}}$ on the line segment can be expressed by:
Therefore, the second minimal line-constraint objective function is:
Thus, the minimum optimization function for the monocular camera has the form:
- (1) As shown in Fig. 4(b), the vectors $\overrightarrow {{O_L}{p_1}}$, $\overrightarrow {{O_R}{p_1}}$, $\overrightarrow {{O_L}{O_R}}$ satisfy the following relationship:$$\overrightarrow {{O_L}{O_R}} \cdot ({\overrightarrow {{O_L}{p_1}} \times \overrightarrow {{O_R}{p_1}} } )= 0$$where ${O_L} = R_{\textrm{TS}}^LO + T_{\textrm{TS}}^L$ and ${O_R} = R_{\textrm{TS}}^RO + T_{\textrm{TS}}^R$. O is [0,0,0]T. $R_{\textrm{TS}}^L$, $T_{\textrm{TS}}^L$ represent the rotation-translation relationship between the left camera frame and the world frame, and $R_{\textrm{TS}}^R$, $T_{\textrm{TS}}^R$ represent the rotation-translation relationship between the right camera frame and the world frame. Therefore, Eq. (25) only refers to external parameters and is not coupled with internal parameters. Thus, for n points, we have:
- (2) As shown in Fig. 4(b), the vectors $\overrightarrow {{O_L}{p_2}}$, $\overrightarrow {{O_R}{p_2}}$, $\overrightarrow {{O_L}{p_3}}$, $\overrightarrow {{O_R}{p_3}}$, $\overrightarrow {{p_2}{p_3}}$ satisfy the following relationship:
Therefore, the global optimization function for LFOV stereo vision system is as follows:
4. Experiment and discussions
4.1. Calibration experiment
To verify the effectiveness of the calibration method and the accuracy of the proposed model, the following calibration experiment is conducted. First, a LFOV stereo vision system is built as shown in Fig. 5. Then, we set up a test field with a space of 5m × 2m × 16 m which is around 5 m away from the LFOV stereo vision system. In this test field, 54 3D points were evenly arranged. The 3D view of the experimental space is shown in Fig. 6. Next, the images of calibrators are recorded, and part of them are shown in Fig. 7, including 6 pictures of checkerboards and 2 pictures of 3D points. Simultaneously, the pixel coordinates of the checkerboard corners and 3D points are extracted by the Harris algorithm [21]. Finally, by using the calibration method introduced in Section 3, the calibration of the LFOV stereo vision system is completed.
3D points were divided into two parts, of which 15 were chosen randomly as 3D calibration datasets and the remaining 39 were chosen as 3D test datasets. In Fig. 6 and Fig. 7, the test dataset and calibration dataset are marked in cyan and blue, respectively, and the data marked in red are used for the proposed calibration method. The physical part of the experiment consists of two AVT1920 CCD cameras (with a resolution of 1920*1080, and a cell size of 5.5µm), two 8 mm Schneider fixed focus lenses, a laptop computer, a total station with an accuracy of 3mm + 2 ppm, a 40 cm glass luminous mosaic grid target with a processing accuracy of 0.05 mm, several 3D points which measured by total station and several power supply cables. Several algorithms are compared in this paper to verify the accuracy of our model and the effectiveness of the proposed method. (1) Zhang + Epnp algorithm, where the internal camera reference is solved by the Zhang method [9] and the external reference between the world coordinate system and the camera coordinate system is solved by the Epnp algorithm [22]. (2) Constant distortion model-based method (CDMM), which is based on CDM, and calibrated by the method in Section 3. (3) DRDM-Ours, which is based on our DRDM proposed in Section 2, and calibrated by the calibration method proposed in Section 3. (4) DRDM-Sun, which is based on the DRDM proposed in literature [15], and calibrated by the calibration method proposed in Section 3. (5) DRDM-Li, which is based on the DRDM proposed in literature [16], and calibrated by the calibration method proposed in Section 3.
Since the Zhang method requires at least 12 checkerboard images to accurately calibrate the internal parameters of the camera [9], and the Epnp algorithm requires at least 10 points to obtain accurate external parameters [22], a total of 30 checkerboard images and 15 3D points are used in the Zhang + Epnp method. The calibration method proposed in this paper uses 5 3D points and 6 checkerboard images. The type and quantity of calibrators for each method are listed in Table 1. The interior parameters of the LFOV stereo vision system are shown in Table 2. As for the left camera, s1 = 542.32 mm, s2 = 22810.26 mm, and for the right camera, s1 = 661.69 mm, s2 = 22742.84 mm. The external parameters between the left and right cameras are shown in Table 3, where RLR and tLR are the construction parameters of the LFOV stereo vision system.
Table 1. Calibrators types
Table 2. Internal parameter calibration results
Table 3. External parameter calibration results
4.2. Model performance analysis
To demonstrate that our method can indeed mitigate some of the overfitting problems, we conducted three groups of calibration experiments A (5p + 1pic, which means using 5 3D calibration points and 1 checkerboard picture), B (10p + 1pic), and C (15p + 1pic) for comparison, and each group includes 100 calibration experiments, and the experiment environment is the same as Section 4.1. Figure 8 shows the calibration results for A, B, and C separately. For each group, row (1) represents the performance of different methods on the calibration datasets, and row (2) represents their performance on the test datasets. Columns (a), (b), (c), and (d) respectively represent the performance of CDMM, DRDM-Ours, DRDM-Sun, and DRDM-Li on different datasets.
From Fig. 8, we find that (1) As the number of calibration points decreases, the reprojection error of all methods on the calibration datasets becomes smaller while the reprojection error on the test datasets increases. This indicates that all methods suffer from overfitting issues. (2) In all experiment groups, DRDM-Ours, DRDM-Sun, and DRDM-Li have smaller reprojection errors on the calibration datasets than CDMM, with DRDM-Ours having the smallest reprojection error on the test datasets compared to DRDM-Sun and DRDM-Li. This illustrates that our model has better resistance to overfitting compared to the Sun and Li models.
4.3. Calibration accuracy analysis
To establish the effectiveness of the proposed method, we conducted 10 LFOV camera calibration experiments. The calibration objects used in each calibration experiment are shown in Table 1. Our experimental procedure involved constructing the space illustrated in Fig. 6, fixing the monocular camera (this camera features a lens with an 8 mm focal length and a resolution of 1920 × 1080), capturing images of the calibration object, and extracting the image points with the Harris method. We categorize the points into three distinct datasets: the 3D point test set, the checkerboard calibration grid dataset, and the 3D calibration dataset. Calibration was performed using the approach outlined in Section 3. In this paper, the RMS of the reprojection error of each method on the 3D test dataset is used as the evaluation standard. The smaller the RMS, the higher the calibration accuracy of the method. Figure 9 shows the RMS of reprojection error of each method on the 3D test dataset and 3D calibration test in all monocular camera calibration experiments. Rows (1) and (2) respectively represent the reprojection errors of the 3D point calibration dataset and the 3D point test set. Columns (a) to (e) in Fig. 9 represent the reprojection errors of different methods.
Firstly, the RMS of reprojection errors of CDMM is chosen as a baseline to evaluate the performance of DRDM-Sun, DRDM-Li, and DRDM-Ours. Our findings in Fig. 9 reveal that the Sun and Li models exhibited a considerable decrease in RMS on the calibration dataset, which is nearly 50%. However, there is an increase in RMS on the test dataset for these models. To be specific, DRDM-Sun shows an increase of 9.6%, while DRDM-Li has an increase of 55% in the RMS on the test dataset. In contrast, DRDM-Ours exhibits a decline in the RMS on both the calibration and test datasets when compared to CDMM. Specifically, there is a decrease of 21% in the calibration dataset and a 10% reduction in the test dataset for DRDM-Ours. The experimental result further proves that our model can resist the overfitting issues of DRDM-Sun and DRDM-Li.
Secondly, it can be seen from Fig. 9 that: DRDM-Ours has the lowest RMS on the 3D test dataset. DRDM-Ours demonstrates a reduction in the test dataset's reprojection error RMS by at least 18% when compared to other methods like DRDM-Sun and DRDM-Li. In addition, DRDM-Ours has a better performance than CDMM on both the calibration dataset and test dataset. Compared with CDMM, the RMS of DRDM-Ours is reduced by 21% and 10% on the calibration dataset and the test set, respectively. In comparison to the Zhang + Epnp method, we achieve a reduction of 9% in RMS on the test dataset. Consequently, the calibration accuracy of the proposed method on the test set is verified.
Finally, we observe that the reprojection error on the u-axis is consistently higher than that on the v-axis across all calibration methods. We hypothesize that this discrepancy may be attributed to additional errors introduced in the u-axis direction during the image feature point extraction process.
4.4. Length measurement accuracy verification
To verify the accuracy of DRDM-Ours, we reconstruct all points in 3D test datasets and evaluate the accuracy of different methods such as DRDM-Sun, DRDM-Li, CDMM and Zhang + Epnp by comparing the length between any two points. DRDM-Ours, DRDM-Sun and DRDM-Li adopted the reconstruction method in literature [16], while CDMM and Zhang + Epnp adopted the reconstruction method in literature [11]. In this experiment, the measurement of the total station is taken as the true value. By comparing with the true value, the relative measurement error ${e_i}$ of each method can be easily calculated. There are 39 points in the 3D test set and 741 possibilities to take 2 points from the test set. In such a condition, 741 relative measurement errors are obtained. The RMS of the 741 relative measurement errors ${e_{rms}}$ is used as the accuracy criterion in this thesis.
The smaller the ${e_{rms}}$, the more accurate the algorithm. To ensure the reliability of the experiment, 6 measurement experiments were performed randomly. The ${e_{rms}}$ of the measurement is shown in Table 4, where Li (i = 1∼6) represents the serial number of the ith experiment. Table 4 shows that DRDM-Ours has the highest measurement accuracy, which improves the measurement accuracy by around 25% compared to Zhang + Epnp or CDMM. Other models, such as DRDM-Sun and DRDM-Li, do not perform well in terms of measurement errors. Although other DRDMs perform well on the reprojection error of the calibration dataset, it appears that these models tend to overfit and therefore perform poorly in terms of relative measurement error. Notably, the standard deviation (std) of our method is larger than DRDM-Sun’s method. However, this does not mean that our method is inferior to Sun's. The std in Table 4 is main indicator for evaluating the quality of the average error of multiple measurements, which is a statistical measure of various errors and represents the stability of the errors. Moreover, as shown in Table 4, even our highest RMS value is lower than the lowest RMS value of Sun, which demonstrates that our method is better than Sun.
Table 4. Relative measurement errors
4.5. Calibration efficiency analysis
To verify the efficiency of our method, We conduct two sets of experiments. First, we fix the number of 3d calibration points at five and divide the experiments into 15 groups according to the number of checkerboards. Second, we fix the number of 3d calibration points at fifteen and divide the experiments into 15 groups according to the number of checkerboards. Each group contains 100 random LFOV camera calibration experiments, which are the same as the experiment in 4.2. We record the standard deviation of the 100 calibration parameters (including focal length, principle points, and radial distortion coefficient) and the RMS of projection errors on the 3D test data in each group. In order to illustrate the rationality of DRDM-Ours, we also conducted 100 calibration experiments on the Zhang + Epnp algorithm, which uses 15 3D calibration points and 30 pictures of checkerboards. The correlation between the number of checkerboards and the standard deviations of the principal distance and principal point coordinates is illustrated in Fig. 10(a) and Fig. 10(c), respectively. Figure 10(b) and Fig. 10(d) depict the association between the number of checkerboards and the standard deviations of the distortion coefficients. The relationship between the number of checkerboards and the RMS of projection errors on the 3D test dataset is presented in Fig. 11(a) and Fig. 11(b) for two separate experiments, respectively.
Firstly, we analyze the correlation between the number of checkerboards and parameter uncertainty in the proposed method. Figure 10 indicates that as the number of checkerboards increases, the standard deviations of parameters decrease. The decreasing trend gradually levels off and most of them show convergence at 14 checkerboards. Figure 10(a) reveals that when the number of calibration objects is limited (5p + 3pic), our approach can yield acceptable parameter uncertainty. Notably, the comparison between Fig. 10(a) and (c) shows that integrating distant 3D points (15p + 3pic) can significantly diminish calibration uncertainty. Nevertheless, for LFOV measurements, integrating 3D points would incur considerable operational expenses and is relatively rigid. By increasing the number of nearby checkerboards (5p + 15pic), our method achieves identical results compared to incorporating a greater number of distant 3D points. Consequently, our methodology offers a more adaptable alternative whereby the calibration stability of parameters can be enriched simply by including additional checkerboards.
Secondly, we investigate the correlation between the number of checkerboards and the RMS of the reprojection error on the test set. Figure 11(a) illustrates that with five 3D calibration points, both DRDM-Ours and CDMM exhibit RMS convergence around 14 checkerboard grids. In contrast, Fig. 11(b) shows that when using 15 3D calibration points, the RMS in CDMM methodology varies insignificantly with an increase in the number of checkerboards, while the RMS in DRDM-Ours stabilizes around 12 checkerboards. If the number of checkerboards exceeds 2, our method consistently outperforms CDMM and Zhang + Epnp, suggesting that our approach can attain exceptional precision even with a limited amount of calibration objects.
Finally, we further analyze the distinctive nature of our approach by selecting four scenarios, namely A(5p + 3pic), B(5p + 15pic), C(15p + 3pic), and D(15p + 15pic). Table 5 shows the RMS of reprojection errors and the calibrators component for each scenario. Through the comparison of A, B, and C, we found that DRDM-Ours can achieve the effect of adding 3D points in distant areas by increasing the number of nearby checkerboards. However, the CDMM is unable to achieve a similar outcome. Moreover, C and D highlight that while additional checkerboards fail to enhance the accuracy of the CDMM when numerous 3D points are involved in calibration, our method can further improve the precision by increasing the number of checkerboards. Consequently, it is capable to improve calibration stability and accuracy by adding close-range checkerboards. Compared to arranging a large number of 3D points in the external field, our method keeps the calibration accuracy and improves the efficiency of calibration to a certain extent.
Table 5. Calibration efficiency analysis
5. Conclusion
This study proposes a calibration method for the LFOV stereo vision system based on DRDM. The main contributions of our works are as follows. (1) We propose a new DRDM in our paper. Compared to DRDMs proposed by Sun and Li, our model has a better performance on reprojection errors in the 3D test dataset and it can alleviate overfitting. (2) We propose a new calibration method that utilizes two types of calibration objects. Compared to the Zhang + Epnp, our method requires fewer calibration objects and achieves satisfactory results in the experiments. Compared to the CDMM, our approach offers greater flexibility in improving calibration precision. (3) Our method introduces geometric constraints such as straight lines and vector cross products to improve calibration accuracy. Experiments demonstrate the effectiveness of the proposed method.
Although our method reduces the number of 3D calibration points compared with Zhang + Epnp, it still needs at least 5 3D calibration points at long distances to calibrate accurately. Also, our method can alleviate overfitting, but it cannot eliminate this phenomenon entirely. Hence, several potential research directions can be explored in the future, such as: (1) optimizing 3D calibration points to achieve high measurement accuracy with the minimal number of points, and (2) addressing overfitting issues that stem from inadequate calibration objects. Those research can improve the calibration efficiency and calibration accuracy further, which is particularly meaningful for LFOV stereo visual systems.
Funding
National Natural Science Foundation of China (52005028, 52127809).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. Y. Zhang, W. Liu, F. Wang, Y. Lu, W. Wang, F. Yang, and Z. Jia, “Improved separated-parameter calibration method for binocular vision measurements with a large field of view,” Opt. Express 28(3), 2956–2974 (2020). [CrossRef]
2. H. Hu, B. Wei, S. Mei, J. Liang, and Y. Zhang, “A two-step calibration method for vision measurement with large field of view,” IEEE Trans. Instrum. Meas. 71, 1–10 (2022). [CrossRef]
3. J. Huo, H. Zhang, Z. Meng, F. Yang, and G. Yang, “A flexible calibration method based on small planar target for defocused cameras,” Opt. Lasers Eng. 157, 107125 (2022). [CrossRef]
4. D. Li, X. Pan, Z. Fu, L. Chang, and G. Zhang, “Real-time accurate deep learning-based edge detection for 3-D pantograph pose status inspection,” IEEE Trans. Instrum. Meas. 71, 1–10 (2022). [CrossRef]
5. X. Chen, J. Lin, Y. Sun, H. Ma, and J. Zhu, “Analytical solution of uncertainty with the GUM method for a dynamic stereo vision measurement system,” Opt. Express 29(6), 8967–8984 (2021). [CrossRef]
6. Y.I. Abdel-Aziz, H.M. Karara, and M. Hauck, “Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry,” Photogramm. Eng. Remote Sens. 81(2), 103–107 (2015). [CrossRef]
7. Y. Yakimovsky and R. Cunningham, “A system for extracting three-dimensional measurements from a stereo pair of TV cameras,” Comput. Graph. Image Process. 7(2), 195–210 (1978). [CrossRef]
8. R. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE J. Robot. Automat. 3(4), 323–344 (1987). [CrossRef]
9. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Machine Intell. 22(11), 1330–1334 (2000). [CrossRef]
10. M. Róg and A. Rzonca, “The impact of photo overlap, the number of control points, and the method of camera calibration on the accuracy of 3D model reconstruction,” Geomatics Environ. Eng. 15(2), 67–87 (2021). [CrossRef]
11. S. Yang, Y. Gao, Z. Liu, and G. Zhang, “A calibration method for binocular stereo vision sensor with short-baseline based on 3D flexible control field,” Opt. Lasers Eng. 124, 105817 (2020). [CrossRef]
12. A.A. Magill, “Variation in distortion with magnification,” J. Opt. Soc. Am. 45(3), 148–152 (1955). [CrossRef]
13. C.B. Duane, “Close-range camera calibration,” Photogramm. Eng. 37(8), 855–866 (1971).
14. J.G. Fryer and D.C. Brown, “Lens distortion for close-range photogrammetry,” Photogramm. Eng. Remote Sens. 52, 51–58 (1986).
15. L. Alvarez, L. Gómez, and J.R. Sendra, “Accurate depth dependent lens distortion models: an application to planar view scenarios,” J. Math. Imaging Vis. 39(1), 75–85 (2011). [CrossRef]
16. P. Sun, N. Lu, and M. Dong, “Modelling and calibration of depth-dependent distortion for large depth visual measurement cameras,” Opt. Express 25(9), 9834–9847 (2017). [CrossRef]
17. X. Li, W. Li, X. Ma, X. Yin, X. Chen, and J. Zhao, “Spatial light path analysis and calibration of four-mirror-based monocular stereo vision,” Opt. Express 29(20), 31249–31269 (2021). [CrossRef]
18. A. Ranganathan, “The Levenberg-Marquardt algorithm,” Tutorial on LM algorithm 11(1), 101–110 (2004).
19. G. Nakano,. “A Simple Direct Solution to the Perspective-Three-Point Problem,” In British Machine Vision Conference (BMVC) (2019), pp. 26.
20. F. Devernay and O. Faugeras, “Straight lines have to be straight,” Mach. Vis. Appl. 13(1), 14–24 (2001). [CrossRef]
21. C. Harris and M. Stephens, “A combined corner and edge detector,” In Alvey Vision Conference (AVC) (1988), pp. 10–5244.
22. V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An accurate O(n) solution to the PnP problem,” Int. J. Comput. Vis. 81(2), 155–166 (2009). [CrossRef]
