Open Access
Add to BibSonomy
Abstract
Calibrating the extrinsic parameters on a system of 3D Light Detection And Ranging (LiDAR) and the monocular camera is a challenging task, because accurate 3D-2D or 3D-3D point correspondences are hard to establish from the sparse LiDAR point clouds in the calibration procedure. In this paper, we propose a geometric calibration method for estimating the extrinsic parameters of the LiDAR-camera system. In this method, a novel combination of planar boards with chessboard patterns and auxiliary calibration objects are proposed. The planar chessboard provides 3D-2D and 3D-3D point correspondences. Auxiliary calibration objects provide extra constraints for stable calibration results. After that, a novel geometric optimization framework is proposed to utilize these point correspondences, thus leading calibration results robust to LiDAR sensor noise. Besides, we contribute an automatic approach to extract point clouds of calibration objects. In the experiments, our method has a superior performance over state-of-the-art calibration methods. Furthermore, we verify our method by computing depth map and improvements can also be found. These results demonstrate that our method performance on the LiDAR-camera system is applicable for future advanced visual applications.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Multi-sensor systems equipped with cameras are vital modules in many visual and interactive applications, such as geometric multi-camera imaging system [1,2], the light field imaging system [3,4], binocular stereo vision measurement system [5], spacecraft optical system [6] and 3D Light Detection And Ranging (LiDAR) and camera (LiDAR-camera) system [7]. Among these imaging systems, the LiDAR-camera system is widely used in the field of robotic vision, such as detecting 3D objects [8] and solving navigation tasks [9]. In these applications, based on the distances measured by light beams, a LiDAR sensor can generate the sparse point cloud of the surroundings. The main advantage of LiDAR is the active illumination which can work independently of ambient light. However, the disadvantages of LiDAR are its expensive cost, limited low resolution, such as Velodyne-64 LiDAR that only measures 64 channels with a low refresh rate. Besides, a LiDAR sensor cannot measure RGB information. RGB camera is relatively cheap, and it can produce high resolution, color images with a high frame rate. But it cannot measure depth information directly. In a word, a LiDAR sensor produces sparse 3D information while the camera captures 2D dense information. Fortunately, by fusing measurements of LiDAR and camera, most of the shortcomings of LiDAR sensors can be compensated by RGB cameras and vice versa. After that, the LiDAR-camera system can precept and analyze the target objects with a more advanced and intelligent view. However, sensor fusion requires the extrinsic parameters of the LiDAR and the camera. So, it is essential to calibrate the extrinsic parameters of these sensors in advance.
The key of LiDAR-camera system calibration is to find geometric relationships from co-observable features [10]. Although feature points in the 2D image can be easily detected, the corresponding 3D points from the sparse LiDAR point cloud are hard to identify. So, the core problem of LiDAR-camera calibration is to exploit point correspondences. To solve this problem, a general approach is to design calibration objects, the corner points of which can be used to establish point correspondences. Calibration objects can be divided as 1D objects, such as line objects with aligned points [11,12], 2D objects, such as polygonal planar boards [10], ordinary planar boards [13], planar boards with chessboard patterns [14], planar boards with rectangle holes [15], planar boards with circle holes [16], and 3D objects, such as ordinary boxes [17]. Besides, there exist methods using no calibration objects. According to literature [18], the information of visual odometry and LiDAR odometry are fused to estimate the relative pose of both sensors. LiDAR odometry is obtained via iterative closest point (ICP) algorithm [19]. Other methods find 3D-2D correspondences using mutual information [20]. Although lots of works have been proposed above, they might face challenges to calibrate the low-resolution LiDAR-camera system, because the point cloud is sparse enough, making it difficult to establish point correspondences.
Motivated by this, we propose a novel calibration object combination and geometric calibration method for estimating the extrinsic parameters of the LiDAR-camera system. Firstly, we design a new combination of calibration objects, which contains rectangular planar boards with chessboard patterns and auxiliary calibration objects. An improved planar chessboard can provide 3D-2D and 3D-3D point correspondences. In this paper, auxiliary calibration objects are defined as some temporarily found 2D planar board in the field of view (FOV) of the camera, which can provide extra 3D-2D point correspondences. After that, a novel geometric optimization framework is proposed, which considers all 3D-2D and 3D-3D point correspondences. Extrinsic parameters can be solved via Bundle Adjustment (BA) [21]. By fusing the information of 3D-2D and 3D-3D point correspondences, the proposed method can estimate the positions of corner points with accuracy, thus leading calibration results robust to LiDAR sensor noise. Besides, we also present an automatic approach to extract LiDAR point clouds of planar calibration objects. Experimental results and 3D reconstruction demonstrate that the proposed planar calibration object and geometric calibration method is helpful for LiDAR-camera system. We believe that it will contribute to future advanced visual applications.
The remainder of this paper is organized as follows. We briefly survey the field of LiDAR-camera system calibration in Sec. 2. After that, the proposed LiDAR-camera system calibration based on novel polygonal planar board with chessboard patterns is presented by details in Sec. 3. In Sec. 4, experiment settings, calibration results, reconstruction are presented and discussed. Finally, we conclude our work in Sec. 5.
2. Related works
The purpose of LiDAR-camera calibration is for two kinds of parameters, including intrinsic and extrinsic parameters of this system. The intrinsic parameters of most LiDAR sensors are calibrated in advance by manufactures. One can estimate the intrinsic parameters of LiDAR via methods [13]. The intrinsic parameters of cameras can be calibrated with the traditional method [22]. In this paper, we assume that the intrinsic parameters of the LiDAR and the camera are both known in advance. As for extrinsic parameters of LiDAR-camera systems, many insightful approaches have been proposed, such as methods based on calibration objects [10–17], the method based on odometry fusion [18], the method based on mutual information [20], and the method based on deep learning [23]. As for methods based on 2D and 3D calibration objects, the corner points of the calibration objects are used to establish point correspondences, and then the relative pose of LiDAR-camera can be computed by Effective Perspective-n-Points (EPnP) algorithm [24]. However, the accuracy of EPnP algorithm is tightly dependent on the precise of 3D points. Among the calibration objects, 1D calibration object is the most special due to its simplicity. Previous works [11,12] have been done to calibrate the intrinsic and extrinsic parameters of the camera using 1D calibration objects. These methods can be extended for LiDAR-camera calibration. But detecting the endpoints of 1D calibration objects in the sparse point cloud with accuracy is a challenging task. Another methods [15,25] attempt to establish 3D-3D point correspondences, and then apply ICP algorithms [19,26] to solve the problem. But these method needs to be calibrated binocular cameras to obtain the positions of 3D points, which cannot be used for monocular cameras. The method based on odometry fusion needs to optimize the odometry information of the camera and LiDAR [18], respectively. Odometry of LiDAR is obtained via ICP algorithm [19]. However, for low-resolution LiDAR, ICP algorithm might obtain bad even not a convergent estimation, because point clouds are sparse enough to find sufficient matched points. The method uses mutual information to match the camera image and images generated by LiDAR [20]. However, sparse point clouds generate sparse images, which decreases the accuracy of matching results. Graphics Processing Unit (GPU) is also required to accelerate the process of calibration [20]. The methods based on deep learning can calibrate the LiDAR-camera system without using calibration objects, odometry information or mutual information, but the accuracy of this kind of method is dependent on the size of the training data set and the framework of deep learning networks. Through the above discussion, it is still a challenging to calibrate the LiDAR-camera system with high precision. Therefore, we propose a novel combination of calibration objects and geometric calibration method robust to LiDAR sensor noises. Our work are proved to be helpful in calibrating these multi-sensor systems.
3. Method
3.1 Problem statement
The problem of calibration on the LiDAR-camera system is presented in Fig. 1. A model of calibration objects and the LiDAR-camera system is shown in Fig. 1(a). In this system, the LiDAR and the camera are both mounted on a base, and the relative pose of them are fixed. We propose a novel combination of calibration objects, which contains planar chessboards and auxiliary calibration objects. Planar chessboards are shown as the pink boards in Fig. 1(a). $P_{i}(i=1,\ldots ,8)$ denotes the corner point on planar chessboards. The coordinate system of i-th planar chessboard is $O_{pi}-X_{pi}Y_{pi}Z_{pi}$. $O_{pi}$ denotes the up and left corner point of i-th planar chessboard. It is discussed in detail in Sec.3.2.1. Auxiliary calibration object is an arbitrary planar board, shown as the orange board in Fig. 1(a). $A_{i}(i=1,2)$ means the corner point on the auxiliary calibration object. For stable calibration results, it is recommended to use at least two planar chessboards and one auxiliary calibration objects, and the reason is shown in Sec.4.2.5. Auxiliary calibration object, such as the wooden table in Fig. 1(b), is temporarily found and might be used for calibration by providing extra constraints. These calibration objects are discussed in Sec.3.2.1. LiDAR coordinate system is represented as $O_{d}-X_{d}Y_{d}Z_{d}$. The origin of $O_{d}-X_{d}Y_{d}Z_{d}$ means the position of LiDAR. The camera coordinate system is denoted as $O_{c}-X_{c}Y_{c}Z_{c}$. The origin of $O_{c}-X_{c}Y_{c}Z_{c}$ is the optical center of the camera. The extrinsic parameters of the LiDAR-camera system are the rotation matrix $\mathbf {R}$ and the transformation vector $T$, which are shown in Fig. 1(a). $\mathbf {R}$ and $T$ mean the rigid transformation between LiDAR coordinate system $O_{d}-X_{d}Y_{d}Z_{d}$ and camera coordinate system $O_{c}-X_{c}Y_{c}Z_{c}$. The proposed calibration method aims to estimate $\mathbf {R}$ and $T$.
Fig. 1. Representation of calibration objects and LiDAR-camera system. (a) Geometric Representation of planar chessboards, auxiliary calibration objects, and LiDAR-camera system. $\mathbf {R}$ and $T$ are extrinsic parameters for calibration. (b) Two planar chessboards and one auxiliary calibration object. It is recommended to use at least two planar chessboard and one auxiliary calibration object. Point $P_{i}(i=1,\ldots ,8)$ is the corner point of planar chessboards. Point $A_{i}(i=1,2)$ is the corner point of auxiliary calibration object.
A projection model of the camera in LiDAR-camera system is shown by Fig. 2. In $O_{c}-X_{c}Y_{c}Z_{c}$, $Z_{c}-axis$ coincides with the camera optical axis. $X_{c}-axis$ and $Y_{c}-axis$ coincide with the vertical and horizontal axis of the image plane $c-uv$[27]. Let $\mathbf {K}$ be the intrinsic matrix of camera, and it is shown as:
Fig. 2. Projection model of the camera in a LiDAR-camera system. $P$ and $I$ represent the 3D-2D point correspondence.
where $f_{u}$ is the focal length along the $u-axis$ in pixels, $f_{v}$ is the focal length along $v-axis$ in pixels [22]. $(u_{0},v_{0})$ denotes pixel coordinates of the principal point of the image plane (i.e., the intersection of the image plane and the optical axis) [27].
Mathematically, let us assume that there is an arbitrary point $P$ in the free 3D space as in Fig. 2. $P$ is visible to the camera, which can also be measured by LiDAR. Let $P^{d}=[X^{d},Y^{d},Z^{d}]^{T}$ represent the coordinates of $P$ in $O_{d}-X_{d}Y_{d}Z_{d}$, and $I=[u,v,1]^{T}$ represent the corresponding image coordinates. The basic assumption here is that we do not consider the lens distortion problems. According to geometric properties of optical imaging [28], a 3D-2D point correspondence of $P^{d}$ and $I$ can be established as:
where $Z$ is the third element of vector $\mathbf {K}(\mathbf {R}P^{d}+T)$. If the coordinates of $P$ in $O_{c}-X_{c}Y_{c}Z_{c}$ are known as $P^{c}$, a 3D-3D point correspondence of $P^{d}$ and $P^{c}$ is established as: In the following discussion, 3D-2D and 3D-3D point correspondences, as shown in Eqs. (2) and (3), are both considered in the proposed geometric optimization framework to calibrate the LiDAR-camera system.3.2 Methodology
3.2.1 Combination of calibration objects for LiDAR-camera system
A novel combination of calibration objects is proposed in this paper, which contains planar chessboards and auxiliary calibration objects. We use planar chessboard $1$, shown in Fig. 3, for illustration. The sizes of the planar board and chessboard patterns are known in advance. $X_{p1}-axis$ and $Y_{p1}-axis$ coincide with the vertical and horizontal axis of the planar chessboard, respectively. The vertical and horizontal edges of chessboard patterns are parallel with $X_{p}-axis$ and $Y_{p}-axis$, respectively. $C_{1}$ is the first feature point in the planar chessboard, and remained feature points are marked as red points shown in Fig. 3. The configuration of the planar chessboard $2$ is the same as the chessboard $1$. However, the planar chessboard in this paper is slightly different from the general planar chessboard, for bias vector $V_{b}$ is known. For planar chessboard $1$, $V_{b}$ means the coordinates of vector $O_{p1}C_{1}$ in $O_{p1}-X_{p1}Y_{p1}Z_{p1}$. If $V_{b}$ is known, the coordinates of all feature points of planar chessboard $1$ in $O_{p1}-X_{p1}Y_{p1}Z_{p1}$ are known, and then 3D-3D point correspondences are established. It is discussed in detail in Sec.3.2.3. Hence, compared with the general planar chessboard which only provides 3D-2D point correspondences, our planar chessboard can provide not only 3D-2D but also 3D-3D point correspondences, thus achieving more accurate calibration results.
Fig. 3. Representation of the planar chessboard. Slightly different with the general planar chessboard, vector $V_{b}$ is known with accuracy.
In the practical applications, there might exist 3D and 2D objects temporally found in the surrounding of the LiDAR-camera system. The object can be used as auxiliary calibration object if it satisfies two conditions: $(i)$ It is fallen into the field of view (FOV) of the camera; $(ii)$ Its corner points can be separated from the background point cloud. Auxiliary calibration objects can provide extra 3D-2D point correspondences, thus increasing the stability of calibration results. However, it is difficult to detect the corner points of 3D object in the low-resolution LiDAR point cloud with accuracy [17]. Compared with 3D objects, the corner points of 2D objects in the sparse LiDAR point cloud are easier to detect automatically with precision. Hence, in this paper, we only consider 2D planar auxiliary calibration objects. For instance, the auxiliary calibration object in Fig. 1(b) is the blue wooden table. Its two corner points, such as points $A_{1}$ and $A_{2}$, can provide two 3D-2D point correspondences. A further reason for auxiliary calibration objects is that corner points on the planar chessboard are distributed on a limited 2D plane, and might lead inaccurate calibration results [15]. 3D calibration objects can solve this problem, for corner points are distributed on a 3D space. However, compared with the 2D calibration object, it is more difficult to produce a sufficiently accurate 3D calibration objects. Fortunately, combinations of planar chessboards and auxiliary calibration objects can be regarded as a special type of 3D calibration object, for they are distrusted in a 3D space. Besides, planar chessboards and auxiliary calibration objects are convenient to obtain in practical applications. Therefore, the combinations of planar chessboards and auxiliary calibration objects can achieve stable and robust calibration results.
3.2.2 Corner points extraction
It is essential to extract the corner points of calibration objects before the procedure of calibration. Finding the corner points by human operations is time-consuming and even with mistakes [15]. Therefore, we propose an approach to obtain the corner points of planar target point clouds more automatically, as shown in Fig. 4(a). LiDAR sensor generates raw point cloud, as shown in Fig. 4(b). The image captured by the camera is shown in Fig. 1(b). By manually setting several threshold values, threshold cut algorithm is applied to obtain the point clouds that only contains calibration objects, as shown in Fig. 4(c). Using Kmeans cluster algorithm [29], we obtain the point clouds of each planar calibration object. For each planar point cloud, its corresponding point clouds are projected into a plane estimated by RANSAC plane fit algorithm [30]. The results are shown in Fig. 4(d). Hull extraction algorithm can obtain the hull of point clouds, shown in Fig. 4(e). Hough line detection algorithm is applied to find lines on the hulls, and the corner points are the intersection of the detected lines. The results are shown in Fig. 4(f). However, due to the sparsity of LiDAR point clouds, corner points estimated via the pipeline in Fig. 4(a) are not accurate enough. Hence, we propose a geometric optimization framework to achieve an accurate calibration results, which is discussed in the following.
Fig. 4. Representation of corner points extraction. (a) The pipeline of corners points extraction. (b) Raw point cloud produced by LiDAR. (c) Point cloud by threshold cut algorithm. (d) Point cloud by RANSAC plane fit algorithm. (e) Hull of the projected point cloud. The left is the hull of planar chessboard 1. The middle is the hull of planar chessboard 2. The right is the hull of auxiliary calibration object 1. (f) Lines are found by the Hough line detection algorithm. Corner points are the intersection of lines. Points $No.i(i=1,\ldots ,4)$ are the corner points of planar chessboard 1. Points $No.i(i=5,\ldots ,8)$ are the corner points of planar chessboard 2. Points $No.i(i=9,10)$ are the corner points of auxiliary calibration object 1.
3.2.3 Geometric LiDAR-camera calibration
We consider that the information of 3D-2D and 3D-3D point correspondences are helpful to obtain calibration results with high precision. With the proposed combination of calibration objects shown in Fig. 1(a), we present a geometric calibration method with considering 3D-2D and 3D-3D point correspondences. The framework is shown in Fig. 5. The geometric calibration method is divided into three steps, such as 3D-2D optimization, 3D-3D optimization, and points merging. The proposed method needs at least four corner points, because the scheme of 3D-2D optimization exploits Effective PnP (EPnP) algorithm [24], which requires at least four corner points.
The scheme of 3D-2D optimization is discussed first. The coordinates of corner points on planar chessboards in $O_{d}-X_{d}Y_{d}Z_{d}$ are marked as $\{P_{i,p}^{d}\}_{i=1,\ldots ,N_{p}}$. Their corresponding pixel coordinates are $\{I_{i,p}\}_{i=1,\ldots ,N_{p}}$. The coordinates of corner points on auxiliary calibration objects in $O_{d}-X_{d}Y_{d}Z_{d}$ are marked as $\{P_{i,a}^{d}\}_{i=1,\ldots ,N_{a}}$. Their corresponding pixel coordinates are $\{I_{i,a}\}_{i=1,\ldots ,N_{a}}$. $P_{i,d}$ and $P_{i,a}$ are obtained via the pipeline in Fig. 4(a). $I_{i,p}$ and $I_{i,a}$ are directly obtained from the image captured by the camera. 3D-2D point correspondence is established from $I_{i,p}$, $P_{i,p}^{d}$, and $I_{i,a}$, $P_{i,a}^{d}$, where the relation is described as Eq. (2). The representation of 3D-2D point correspondence is shown in Fig. 2. Raw estimations of $\mathbf {R}$ and $T$, marked as $\mathbf {R}_{raw}$ and $T_{raw}$, are computed via these 3D-2D point correspondences by means of EPnP algorithm [24]. After that, $\mathbf {R}_{raw}$ and $T_{raw}$ are refined by minimizing following function [21]:
In the scheme of 3D-3D optimization, camera extracts the feature points on the chessboard patterns, shown in Fig. 3. Only the planar chessboard can be used for 3D-3D optimization, for auxiliary calibration objects do not have chessboard patterns. We first use one planar chessboard for illustration. As the size of pattern is known, the coordinates of all feature points in $O_{p}-X_{p}Y_{p}Z_{p}$ are known. Then the rotation matrix $\mathbf {R}_{p}$ and translation vector $T_{p}$ of camera in $O_{p}-X_{p}Y_{p}Z_{p}$ are computed via EPnP algorithm and BA optimization. After that, the coordinates of feature point $C_{1}$ in $O_{c}-X_{c}Y_{c}Z_{c}$, marked as $C_{1}^{c}$, are determined using $\mathbf {R}_{p}$ and $T_{p}$. As the bias vector $V_{b}$ and the size of the planar chessboard are known with accuracy, the coordinates of corner points $\{P_{i,p}\}_{i=1,\ldots ,4}$ in $O_{c}-X_{c}Y_{c}Z_{c}$, marked as $\{P_{i,p}^{c}\}_{i=1,\ldots ,4}$, can be computed via $C_{1}^{c}$ with high precision. Finally, 3D-3D point correspondences, as shown in Eq. (3), can be established from two point sets $\{P_{i,p}^{d}\}_{i=1,\ldots ,4}$ and $\{P_{i,p}^{c}\}_{i=1,\ldots ,4}$. Representation of 3D-3D point correspondences of one planar chessboard in the ideal situation are show in Fig. 6. Moreover, we introduce the general case that $N$ planar chessboards are used for calibration, which is presented as $algorithm 1$ in Appendix A.
Fig. 6. Representation of 3D-3D point correspondences for one planar chessboard in the ideal case. One planar chessboard provide four corner points. Red and blue arrows denote the coordinates of corner points in LiDAR and camera coordinate system, respectively. 3D-3D point correspondences can be established from $P_{i,p}^{d}$ and $P_{i,p}^{c}$ via Eq. (3).
Using $algorithm 1$, 3D-3D point correspondences of $\{P_{i,p}^{d}\}_{i=1,\ldots ,N_{p}}$ and $\{P_{i,p}^{c}\}_{i=1,\ldots ,N_{p}}$ are established. According to Eq. (3), extrinsic parameters $\mathbf {R}$ and $T$ can be obtained by minimizing the following function [15]:
The scheme of points merging is discussed now. Using the result $\mathbf {R}_{c}$ and $T_{c}$ obtained in 3D-3D optimization scheme, we can back-reconstruct the coordinates of corner points $\{P_{i}^{d}\}_{i=1,\ldots ,N_{p}}$, marked as $\{P_{i,back}^{d}\}_{i=1,\ldots ,N_{p}}$, on $O_{d}-X_{d}Y_{d}Z_{d}$ as:
Let $P^d_{i,true}$ be the coordinates of the true i-th corner point in the LiDAR coordinate system. As discussed above, due to the measurement noise in LiDAR point cloud, $P^d_{i,p}$ and $P^d_{i,back}$ are not accurate enough. For more accurate calibration results, we attempt to estimate the optimal approximation of $P^D_{i,true}$ from $P^d_{i,p}$ and $P^d_{i,back}$. In this paper, we consider the first-order optimal approximation of $P^d_{i,true}$, and then $P^d_{i,opt}$ can be represented as: where $w_{i}$ is the weight parameter needed to be estimated. The geometrical meaning of $P_{i,opt}^{d}$ is shown in Fig. 7 where we use the i-th corner point in a planar chessboard for illustration. The initial value of $w_{i}$ can be set as:Fig. 7. Representation of point merging for i-th corner point in a planar chessboard. In the LiDAR coordinate system, green and red points denote the coordinates of detected and back-reconstruct i-th corner points, respectively. Purple point $P^d_{i,true}$ means the coordinate of the true i-th corner point. Yellow point $P^d_{i,opt}$ is on the line of $P^d_{i,back}$ and $P^d_{i,p}$. $P^d_{i,opt}$ is the closest point to $P^d_{i,true}$, which can be regarded as the first-order optimal approximation of $P^d_{i,true}$
where $e_{i,p}$ is the reprojection error of i-th corner point computed in the scheme of 3D-2D optimization. If $e_{i,p}$ is smaller, $w_{i}$ is closer to $1.0$, meaning that $P_{i,opt}^{d}$ is closer to $P_{i,p}^{d}$. $\sigma$ can be equal to $0.1$ in practical applications. Finally, parameters $\{w_{i}\}_{i=1,\ldots ,N_{p}}, \mathbf {R}, T$ can be optimized by minimizing the following function:
4. Experiments
4.1 Simulations
4.1.1 Experiment settings
Several simulated experiments are implemented to evaluate the performance of the proposed method. We set up a virtual LiDAR-camera system. The intrinsic parameters of the virtual camera and the extrinsic parameters of the virtual LiDAR-camera system are shown in Table 1. The rotation matrix $\textbf {R}$ of this LiDAR-camera system is set as a $3\times 3$ identity matrix. Let $N (N \geq 4)$ denote the numbers of the corner points of all calibration objects. The low bound of $N$ is set as four because EPnP algorithm [24] in the scheme of 3D-2D optimization requires at least four corner points. In the following experiments, we evaluate the accuracy of calibration results by measuring the reprojection errors of all corner points. For point $No.i (i=1,2,\ldots ,N)$, reprojection error $E_{i}$ is computed as:
Table 1. Parameters of the virtual LiDAR-camera system.
where $I_{i,est}$ is computed via $\mathbf {K}$, $P_{i}^{d}$, $\mathbf {R}$ and $T$ via Eq. (2). The mean reprojection error is consider as $\bar {E}_{p} = \frac {1}{N}\sum _{i=1}^{N}E_{i}$. Besides, in simulations, the ground truths of $\textbf {R}$, $T$ are known in accuracy. Let $\xi \in se(3)$ represent extrinsic parameters $\textbf {R}$, $T$. The norm-2 $se(3)$ error of $\xi$ is also used for evaluations, which is marked as $E_{\xi }$.
4.1.2 Performance with respect to noise level
This experiment investigates the performance with respect to measurement noise. In the calibration process, there are two sources of measurement noise, such as noise on the pixel and noise on the point cloud. Pixel noise comes from corner points detection in the image. The accuracy of extrinsic parameters in calibration will be affected by these errors. According to the pinhole model in Eq. (2), noise on the point cloud can be converted as pixel noise. Hence, we use Gaussian pixel noises with zero mean and $\delta$ standard deviation added on corner points to verify the robustness performance of our method. The noise level (i.e., $\delta$) varies from $[0.1, 0.5]$ pixels. 500 independent trials are preformed, and the mean error are recorded as $\bar {E}_{p}$ and $E_{\xi }$, respectively. The results of $\bar {E}_{p}$ and $E_{\xi }$ are shown in Fig. 8 and Fig. 9, respectively. The tendencies of curves are shown that calibration errors increase as noise level increases. It means that improving the accuracy of the coordinates of corner points in LiDAR coordinate system is a essential key to obtain calibration results in high precision.
4.1.3 Performance with respect to corner points
This experiment investigates the performance with respect to numbers of corner points. In the actual applications, as the number of calibration objects is limited, the number of corner points is also limited. Exploring the relationship between the number of feature points and the calibration error is important. The results are shown in Fig. 8 and Fig. 9. When $N$ increase from 4 to 10, the errors of calibration are decreased obviously, because the more the number of corner points, the more robust it is to noise [22]. However, the change rate of mean reprojection error is not obvious when $N\geq 14$, because the measurement errors cannot be eliminated completely. Due to the measurement noise, using larges of calibration objects might not obtain extremely accurate calibration results. So, it is wise to use appropriate numbers of calibration objects to get relatively accurate results.
4.2 Real data experiments
4.2.1 Experiment settings
To verify the performance of our method, we set up a LiDAR-camera system as shown in Fig. 10. It consists of Velodyne-64 LiDAR and Kinect v2 camera with a resolution of $1920 \, pixels\;\times\;1080 \, pixels$. The intrinsic parameters of LiDAR have been calibrated via manufacturers in advance. The intrinsic matrix $\mathbf {K}$ of the camera is calibrated using chessboard patterns via the classical method [22], which is shown in Table 2. In the experiment, a proposed combination of calibration objects, containing two planar chessboards and one auxiliary calibration object, are used to calibrate LiDAR-camera system, as shown in Fig. 1(b). Planar chessboards are manually placed on the FOV of the camera. Feature points of these planar chessboards are extracted with the accuracy of sub-pixel. Auxiliary calibration object is temporarily found in the surroundings. We obtain the corner points of these calibration objects by means of the approach discussed in Sec.3.2.2, and mark these points as $No.i \,\, (i=1,\ldots ,10)$.s For point $No.i$, its coordinates in $O_{d}-X_{d}Y_{d}Z_{d}$, as $P_{i}^{d}$, is obtained via method in Sec.3.2.2, and its coordinates in image coordinate system, as $I_{i}$, is extracted from Fig. 1(b). In the following experiments, we still evaluate the accuracy of calibration results by measuring the reprojection errors, as shown in Eq. (10).
Table 2. Intrinsic parameters of the camera in LiDAR-camera system.
4.2.2 RANSAC planar fit versus reprojection error performance
This experiment investigates the performance with respect to RANSAC planar fit. Specifically, we compare the calibration methods with or without the RANSAC planar fit algorithm. First, the corner points of calibration objects are estimated in the pipeline shown in Fig. 4(a) without the part of plane fit. Extrinsic parameters can be estimated via EPnP algorithm and BA optimization. This raw method is marked as $3D2D$. Then, the corner points of calibration objects are estimated in the whole pipeline shown in Fig. 4(a), and extrinsic parameters are estimated via EPnP algorithm and BA optimization. This method is marked as $3D2DFit$. The reprojection errors of two methods are both shown in Table 3. The reprojection errors of $P_{8}$ computed via $3D2DnoFit$ is larger than $90.0$ pixels, while $E_{8}$ computed via $3D2DFit$ is smaller than $30.0$ pixels. Compared with $3D2D$, reprojection errors of most of points are decreased in $3D2DFit$, expect $P_{4}$. The reprojection errors of $3D2DFit$ are decreased by $52.57\%$ than $3D2D$ in average. Although planar chessboards and auxiliary calibration objects are both planar, raw point clouds of them are not planar in fact, as shown in Fig. 11. Processed by the RANSAC plane fit algorithm, point clouds of these calibration objects are smoother and more closed to the true point clouds, thus increasing the accuracy of positions of corner points, and eventually leading more accurate calibration results. Therefore, it is recommended to denoise the planar point clouds by the RANSAC plane fit algorithm.
Fig. 11. Three-view drawing of raw point clouds of the first planar chessboard. (a) Frontal plane of this calibration object. (b) Profile plane of this calibration object. (c) Top view of this calibration object. The raw point clouds of the planar object are obviously not planar in practical applications.
Table 3. Reprojection errors of different LiDAR-camera calibration methods discussed in Sec.4.2, Sec.4.3 and Sec.4.4 (Unit: Pixels). $P_{i}$ means $i-th$ corner points of calibration objects. Term $none$ means the point has no reprojeciton error. $St.Dev$ denotes the standard deviation of reprojection errors. $Gain$ denotes the current method’s accuracy improvement rate relative to method $3D2D$, calculating by thier reprojection errors.
4.2.3 Point filter strategy versus reprojection error performance
This experiment investigates the performance with respect to point filter strategy. From Table 2, reprojection errors of $P_{4}$, $P_{7}$ and $P_{8}$ in $3D2DFit$ are larger than mean reprojection error in this method. The reason is that coordinates of $P_{4}^{d}$, $P_{7}^{d}$ and $P_{8}^{d}$ in $O_{d}-X_{d}Y_{d}Z_{d}$ are estimated not accurately. As shown in Fig. 12, due to the sparsity of LiDAR point clouds, the distances between these measured points and the corresponding true corner points are relatively large. For stable calibration results, these bad estimated points are rejected, and extrinsic parameters are refined again via EPnP algorithm and BA optimization. This method is named as $3D2DFilter$, which is essentially $3D2DFit$ plus point filter strategy. Besides, $3D2DFilter$ represents the scheme of 3D-2D optimization. Using $3D2DFilter$, the improvement is shown in Table 3. Compared with $3D2DFit$, reprojection errors of $71.4\%$ corner points of calibration objects in $3D2DFilter$ are decreased. Also, the reprojection errors of $3D2DFilter$ is decreased by $66.96\%$ than $3D2DFit$ in average. However, the reprojection errors of $P_5$ and $P_{10}$ in $3D2DFilter$ are larger than the corresponding reprojection errors in $3D2DFit$. Due to the tiny lens distortion of the camera, the pixel coordinates of several corner points far from the image center, are not accurate enough, such as $P_5$, $P_6$, $P_9$ and $P_{10}$. Although we have decreased the sum reprojection errors of all corner points via Eq. (4), the optimized result reaches a comprimise between the accurate and inaccurate corner points. So, there exist several corner points, such as $P_5$ and $P_{10}$, that have larger reprojection errors in $3D2DFilter$ than $3D2DFit$. Hence, the strategy of filtering bad corner points can reduce the mean reprojection error, but it fails to achieve more robust and accurate calibration results.
Fig. 12. Points with large reprojection errors. (a) In the first planar chessboard, $P_{4}$ has large reprojection error. (b) In the second planar chessboard, $P_{7}$ and $P_{8}$ have large reprojection errors. The distances between these measured points and the corresponding true corner points are large.
4.2.4 Optimization framework versus reprojection error performance
This experiment investigates the performance with respect to the proposed optimization framework. Although $3D2DFilter$ achieves more stable calibration results than the previously discussed method, $3D2DFilter$ is difficult to obtain more accurate results because the information of rejected corner points, such as $P_{4}$, $P_{7}$ and $P_{8}$, are unable to use. It is noted that the proposed planar chessboards can provide 3D-3D point correspondences. The extrinsic parameters of the LiDAR-camera system can be computed via Eq. (5). This method is named as $3D3D$. In this experiment, a trick is used to improve the accuracy of coordinates of corner points in $O_{d}-X_{d}Y_{d}Z_{d}$ by fusing size information of planar chessboards and auxiliary calibration objects, thus leading stable calibration results. In Table 3, the reprojection errors of $3D3D$ are decreased by 39.92$\%$ than $3D2DFilter$. More importantly, $3D3D$ refines $P_{4}^{d}$, $P_{7}^{d}$ and $P_{8}^{d}$, which provides probabilities for achieving more accurate calibration results. After that, we evaluate the performance of the scheme of points merging. The method is named as $MergeSimple$ if weights of corner points are not optimized and used only as initial guesses in Eq. (8). Mean reprojection error of $MergeSimple$ is decreased by 15.68$\%$ than $3D3D$, because merged information is robust to LiDAR sensor noise. If we optimize weights of all corner points, this method is named as $MergeOpt$. Obviously, $MergeOpt$ is the proposed calibration method. As we can see in Table 3, reprojection errors of $90\%$ corner points of calibration objects in $MergeOpt$ are less than 2.0 pixels. And the mean reprojection error of $MergeOpt$ is the minimum of all discussed methods. The result of $MergeOpt$ is also shown in Fig. 13. It can be found that using the proposed optimization framework, the distances between estimated corner points and the corresponding true corner points are decreased. Therefore, it means that calibration results are more stable and accurate by applying our optimization framework.
Fig. 13. Results of our optimization framework. White points are the hull of planar chessboards. Red points are the back-reconstruct results of corner points of planar chessboards. (a)-(b) Front and vertical views of the calibration results without using our optimization framework. (c)-(d) Front and vertical views of the calibration results using our optimization framework. Using our optimization framework, the distances between estimated corner points and the corresponding true corner points are decreased.
4.2.5 Numbers of calibration objects versus reprojection error performance
This experiment investigates the performance with respect to numbers of calibration objects. We have two planar chessboards and one auxiliary calibration objects in the experimental configuration. The mean error of each combination of calibration objects are shown in Table 4. It can be found that the accuracy of calibration increases as the numbers of calibration objects increase, which is verified the conclusions in Sec. 4.1.3. Compared with real experimental results in Table 4 and the theoretical results (Noise level $\sigma$=0.5) in Sec. 4.1.3, curves about the relationship between the number of corner points and mean reprojection errors are shown in Fig. 14. Due to the measurement error, errors of real experiment are larger than theoretical errors. For theoretical curve, the change rate (CR) of reprojection errors is less than $10\%$ when numbers of corner points are larger than 12. Besides, CR of reprojection errors decreases as the number of corner points increases. According to the trend of the theoretical error curve, it can be concluded that CR of reprojection errors in real experimental curve is less than $10\%$ when numbers of corner points are larger than 10. Therefore, in practical applications, using a combination of calibration objects, which provides at least 10 corner points, is helpful to obtain stable and relatively accurate calibration results. So, it is recommended to use at least two planar chessboards and at least one auxiliary calibration objects for LiDAR-camera system calibration.
Fig. 14. Relationship between numbers of corner points and reprojection error in LiDAR-camera calibration. Blue curve is the theoretical result($\sigma =0.5$). Black dotted curve is the real experimental result. $CR$ denotes the change rate of reprojection errors.
Table 4. Reprojection errors (Unit: Pixels) of different combinations of calibration objects. Term $none$ means that the current combination of calibration objects fails to compute extrinsic parameters of LiDAR-camera system.
4.2.6 Model comparisons versus reprojection error performance
This experiment investigates the performance with respect to different LiDAR-camera calibration method. Our method is compared with Park et al. method [10] and Dhall et al. method [15]. Park method [10] uses 3D-2D point correspondences provided by planar chessboards and auxiliary calibration objects for estimating the intrinsic parameters of the camera and extrinsic parameters of the LiDAR-camera system. Dhall method [15] uses 3D-3D point correspondences of the proposed planar chessboards for calibrating extrinsic parameters of LiDAR-camera system. In his method, a stereo camera is used to provide the coordinates of corner points in the camera coordinate system. Instead of using a stereo camera, the camera of the LiDAR-camera system is a depth camera in our experiment configuration, which can also provide 3D information. Reprojection errors of these methods are shown in Fig. 15. It is found that the reprojection errors of all corner points in our method are all smaller than other two methods, because Park method needs to estimate extra intrinsic parameters which are known in advance, and Dhall method only uses 3D-3D point correspondences. The mean reprojection errors of our method and other two methods are 1.392, 2.078 and 1.902 pixels, respectively. Our method fuses 3D-2D and 3D-3D point correspondences, which can achieve accurate calibration results.
4.3 Verification on depth map
Computing depth maps from raw Lidar point clouds is the natural applications of LiDAR-camera systems. After estimating the extrinsic parameters using the proposed method, we can obtain the depth map from the LiDAR point clouds using the pinhole model, as shown in Eq. (2). During the procedure of camera projection, the point clouds, which are out of the FOV of the camera, are filtered out. The results of depth maps are shown in Fig. 16. The color changes from green to red, meaning the depth increases. Two red boxes ($i$=1,2) are shown in the RGB image, as Fig. 16(a). Without using any calibration methods, the depth map is computed via the initial extrinsic parameters, as shown in Fig. 16(c). The depth map computed via the extrinsic parameters calibrated by our method is shown in Fig. 16(b). Compared to the depth map without calibration, the depth map using our method are highly aligned with the RGB image. Referred with the RGB image, the pixel coordinates of boxes $1$ and $2$ in Fig. 16(c) exist significant offsets, in which the mean rate of pixel offset is nearly 40$\%$. For instance, $P$ is the corner points in RGB image, and $P_{1}$, $P_{2}$ are the corresponding points in two depth maps. It can be found that $P$ is highly aligned with $P_{1}$, and there exists offset between $P$ and $P_{2}$. After LiDAR-camera calibration using our method, there are nearly no pixel offsets. Besides, it can be seen that the most of edge information remains in the depth map via our method. These results demonstrate that the proposed calibration method contributes to the advanced applications of LiDAR-camera systems.
Fig. 16. Results of depth map using the proposed calibration method. (a) RGB image captured by the camera in LiDAR-camera system. (b) Result of the depth map with using our calibration method. (c) Result of the depth map without using any calibration method. $P$, $P_{1}$, and $P_{2}$ are the corresponding points in three images.
4.4 Limitations
There are several limitations in our work. First, the proposed method requires at least four corner points for calibration, for EPnP algorithm [24] requires at least four corner points. During the procedure of calibration, the number of corners points should be larger than four. Second, we do not consider the optical distortion of the camera. We can calibrate the lens coefficients of the camera in advance, and undistort the image via the lens distortion model [22]. Third, we suppose that the auxiliary calibration objects are only 2D planar objects. However, in the practical applications, they might exist 3D objects, such as boxes, which provide 3D-2D point correspondences. We can automatically detect the corner points of 3D auxiliary calibration objects via a geometrical based optimization scheme [17].
5. Conclusion
In this paper, we proposed a novel geometric method for estimating the extrinsic parameters of the LiDAR-camera system. In this work, we first design a new combination of calibration objects, containing planar chessboards and auxiliary calibration objects, which can provide 3D-2D and 3D-3D point correspondences. Auxiliary calibration objects can be temporarily found in the surroundings. After that, a novel geometric optimization framework is proposed to merge the information of all 3D-2D and 3D-3D point correspondences. Simulations, real data experiments and depth map applications demonstrate that our method is stable, accurate, and outperforms other state-of-the-art methods. Our method can benefit for advanced applications based on the LiDAR-camera system. For achieving more precise calibration results, we can carry on further study on the calibration scheme with using 2D and 3D auxiliary calibration objects in the following work.
Appendices
A. Establish 3D-3D point correspondences for $N$ planar chessboards
The perse-code of finding 3D-3D point correspondences in $N$ planar chessboards is shown in Algorithm. 1.
|
B. Closed-form solution in Eq. (5)
$\mathbf {R}, T$ in Eq. (5) can be computed via method [26]. The derivation is shown in the following. Let $\bar {P}_{p}^{c}$ and $\bar {P}_{p}^{d}$ denote $\frac {1}{N_{p}}\sum _{i=1}^{N_{p}} P_{p,i}^{c}$, $\frac {1}{N_{p}}\sum _{i=1}^{N_{p}} P_{p,i}^{d}$, respectively. After that, $Q_{p,i}^{c}$ and $Q_{p,i}^{d}$ can be computed as:
Funding
Equipment pre-research project (305050203, 41415020202, 41415020404); National Natural Science Foundation of China (U1913602).
Acknowledgments
The authors thank Siying Ke for providing many print suggestions. The authors also appreciate anonymous reviewers for providing valuable and inspiring comments and suggestions.
Disclosures
The authors declare no conflicts of interest.
References
1. F. Abedi, Y. Yang, and Q. Liu, “Group geometric calibration and rectification for circular multi-camera imaging system,” Opt. Express 26(23), 30596–30613 (2018). [CrossRef]
2. L. Lilin, P. Zhiyong, and T. Dongdong, “Super multi-view three-dimensional display technique for portable devices,” Opt. Express 24(5), 4421–4430 (2016). [CrossRef]
3. Z. Cai, X. Liu, X. Peng, and B. Z. Gao, “Ray calibration and phase mapping for structured-light-field 3D reconstruction,” Opt. Express 26(6), 7598–7613 (2018). [CrossRef]
4. Z. Cai, X. Liu, X. Peng, Y. Yin, A. Li, J. Wu, and B. Z. Gao, “Ray calibration and phase mapping for structured-light-field 3D reconstruction,” Opt. Express 24(18), 20324–20334 (2016). [CrossRef]
5. Y. Cui, F. Zhou, Y. Wang, L. Liu, and H. Gao, “Precise calibration of binocular vision system used for vision measurement,” Opt. Express 22(8), 9134–9149 (2014). [CrossRef]
6. M. Wang, Y. Cheng, B. Yang, S. Jin, and H. Su, “On-orbit calibration approach for optical navigation camera in deep space exploration,” Opt. Express 24(5), 5536–5554 (2016). [CrossRef]
7. H. Di, H. Hua, Y. Cui, D. Hua, B. Li, and Y. Song, “Correction technology of a polarization lidar with a complex optical system,” J. Opt. Soc. Am. A 33(8), 1488–1494 (2016). [CrossRef]
8. H. Cho, Y.-W. Seo, B. V. K. V. Kumar, and R. Rajkumar, “A multi-sensor fusion system for moving object detection and tracking in urban driving environments,” in Proceedings of IEEE International Conference on Robotics and Automation, (IEEE, 2014), pp. 1836–1843.
9. P. Moghadam, W. S. Wijesoma, and D. Feng, “Improving path planning and mapping based on stereo vision and lidar,” in Proceedings of International Conference on Control, Automation, Robtics and Vision, (Academic, 2008), pp. 384–389.
10. Y. Park, S. Yun, C. Won, K. Cho, K. Um, and S. Sim, “Calibration between color camera and 3d lidar instruments with a polygonal planar board,” Sensors 14(3), 5333–5353 (2014). [CrossRef]
11. Z. Zhang, “Camera calibration with one-dimensional objects,” IEEE Trans. Pattern Anal. Machine Intell. 26(7), 892–899 (2004). [CrossRef]
12. F. Wu, Z. Hu, and H. Zhu, “Camera calibration with moving one-dimensional objects,” Pattern Recognit. 38(5), 755–765 (2005). [CrossRef]
13. F. M. Mirzaei, D. G. Kottas, and S. I. Roumeliotis, “3d lidar-camera intrinsic and extrinsic calibration: Identifiability and analytical least-squares-based initialization,” Int. J. Robotics Res. 31(4), 452–467 (2012). [CrossRef]
14. A. Geiger, F. Moosmann, O. Car, and B. Schuster, “Automatic camera and range sensor calibration using a single shot,” in Proceedings of IEEE International Conference on Robotics and Automation, (IEEE, 2012), pp. 3936–3943.
15. A. Dhall, K. Chelani, V. Radhakrishnan, and K. M. Krishna, “Lidar-camera calibration using 3d-3d point correspondences,” in arXiv:1705.09785, (2017), pp. 1–19.
16. C. Guindel, J. Beltrán, D. Martin, and F. Garcia, “Automatic extrinsic calibration for lidar-stereo vehicle sensor setups,” (IEEE, 2017), pp. 1–6.
17. Z. Pusztai and L. Hajder, “Accurate calibration of lidar-camera systems using ordinary boxes,” in Proceedings of IEEE International Conference on Computer Vision Workshops, (IEEE, 2017), pp. 394–402.
18. I. Ryoichi, O. Takeshi, and I. Katsushi, “Lidar and camera calibration using motion estimated by sensor fusion odometry,” in Proceedings of International Conference on Intelligence Robots and Systems, (IEEE, 2018), pp. 7342–7349.
19. Y. Ge, C. R. Maurer, and J. M. Fitzpatrick, “Surface-based 3-d image registration using the iterative closest point algorithm with a closest point transform,” Proc. SPIE 2710, 358–367 (1996). [CrossRef]
20. G. Pandey, J. R. Mcbride, S. Savarese, and R. M. Eustice, “Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information,” in Proceedings of the Twenty-Sixth Conference on Artificial Intelligence, (Academic, 2012), pp. 1–7.
21. B. Triggs, P. F. Mclauchlan, R. I. Hartley, and A. W. Fitzgibbon, “Bundle adjustment - a modern synthesis,” in Proceedings of Workshop on Vision Algorithms, (Academic, 2000), pp. 298–372.
22. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Machine Intell. 22(11), 1330–1334 (2000). [CrossRef]
23. G. Iyer, K. R. R J. K. Murthy, and K. M. Krishna, “Calibnet: Self-supervised extrinsic calibration using 3d spatial transformer networks,” in Proceedings of International Conference on Intelligence Robots and Systems, (IEEE, 2018), pp. 1–8.
24. V. Lepetit, F. Moreno-Noguer, and P. Fua, “EPnP: An accurate O(n) solution to the PnP problem,” Int. J. Comput. Vis. 81(2), 155–166 (2009). [CrossRef]
25. M. Hassanein, A. Moussa, and N. El-Sheimy, “A new automatic system calibration of multi-cameras and lidar sensors,” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XLI-B1, 589–594 (2016). [CrossRef]
26. B. K. P. Horn, H. M. Hilden, and S. Negahdaripour, “Closed-form solution of absolute orientation using orthonormal matrices,” J. Opt. Soc. Am. A 5(7), 1127–1135 (1988). [CrossRef]
27. Y. Li, J. Zhang, and W. Hu, “Method for pan-tilt camera calibration using single control point,” J. Opt. Soc. Am. A 32(1), 156–163 (2015). [CrossRef]
28. R. Harltey and A. Zisserman, Multiple view geometry in computer vision (2. ed) (Cambridge University Press, 2006).
29. Y. Zhao and G. Karypis, “Empirical and theoretical comparisons of selected criterion functions for document clustering,” Mach. Learn. 55(3), 311–331 (2004). [CrossRef]
30. B. Oehler, J. Stueckler, J. Welle, D. Schulz, and S. Behnke, “Efficient multi-resolution plane segmentation of 3d point clouds,” in Proceedings of International Conference on Intelligent Robotics and Applications, (Academic, 2011), pp. 145–156.
31. J. More, “The levenberg-marquardt algorithm, implementation and theory,” Numer. Analysis 630, 105–116 (1977). [CrossRef]
