1. Introduction

With the rapid development in image processing technology and sensor accuracy, depth measurement technology has been widely used in various fields, such as 3D reconstruction [1–4], industrial detection [5,6], medical imaging [7–9] and so on. It is a non-contact method to measure the distance between the target and depth sensor. The depth measurement technology is mainly divided into three categories: Time of Flight (ToF), binocular stereo vision (without structured illumination), and structured-light methods.

ToF obtains the depth information by calculating the flight time or the phase value of the modulated light [10]. It has great advantage in long-distance measurement, but its accuracy is relatively low [11]. Different from ToF, the binocular stereo vision without structured illumination acquires the disparity map of the target by matching the image pair captured at two perspectives. Block matching or semi-global block matching are always utilized to calculate the similarity of the searching windows, which is relatively easy to achieve [12,13]. However, the binocular stereo vision method still has great difficulty in extracting features in textureless areas, which may lead to failure in feature-matching. In addition, the computational complexity caused by feature extraction and matching limits its application in real-time measurement [14,15].

To overcome the limitations of binocular stereo vision, researchers proposed the structured-light methods [16–18]. The structured-light systems can be categorized into two kinds: systems with controllable or designed projections [19] and systems with random speckle projections [20–22]. Examples of the former kind include systems with software-controlled projector, designed projection patterns, etc. These systems usually measure via pattern-coding, e.g., temporal coding or spatial coding [23], and provide high accuracy. Furthermore, benefit by the controllable or specific patterns, they can be easily calibrated, e.g, by phase-shift method [24]. On the other hand, the latter kind of system, also called the laser speckle projection system, is based on laser passing through diffraction optical elements (DOE) like ground glass and only generates highly-random speckle pattern. Due to its compact structure, it provides advantages in system modularization and miniaturization [25,26] compared to the former kind.

The laser speckle projection system is generally composed of rigidly-fixed camera and laser speckle projector. Abundant feature information provided by the speckle patterns significantly improves the accuracy of image matching for textureless areas [25,27]. According to the number of cameras, the laser speckle projection system is generally divided into two categories: binocular system and monocular system. The binocular one [28,29], which contains a laser speckle projector and two cameras, is more commonly-used and performs well in accuracy and stability. However, its cost is comparatively high and calibration process is complex. The monocular system only consists of a camera and a laser speckle projector, and hence is more compact and cost-effective. So far, there are many commercial depth-measurement products based on monocular laser speckle projection system, such as Microsoft KinectV1 [30,31], Intel RealSenseR200 [32,33], Orbbec Astra-Pro [34,35] and iPhone X [36].

Unfortunately, few calibration methods exist for the monocular laser speckle projection system and they are still far from perfect. The traditional method is still applied in most of the systems. In this method, reference speckle images at several standard distances are captured and then employed for image matching during measurement. It requires high-precision auxiliary equipment to obtain several reference images, making the calibration expensive and complex [11]. Moreover, optical axes of the camera and projector are assumed to be parallel for the traditional method. Therefore, it is sensitive to the angle error of the optical axes (by manufacturing imperfection, collision, etc.), which however cannot be found and corrected by the method itself.

Recently, a calibration method has been proposed by Zhu and Zhang [37]. It calibrates the system with the help of reference speckle images (i.e., the datum image in [37]) and a calibration board. After taking images of the calibration board with and without speckles pattern, it extracts features’ coordinates in the camera images and finds their correspondences in the reference images by image-matching. Since features’ world coordinates are known, by changing calibration board’s pose and repeating the above step, it calibrates the system as a binocular vision system. This method avoids the usage of high-precision auxiliary equipment and needs much fewer reference images, leading to more convenient calibration. However, it still requires taking a reference image in advance. Furthermore, since the reference image plays as projector’s pseudo image plane, ideally it should be taken with a whiteboard perpendicular to the camera’s optical axis. Angle error easily occurs here as no high-precision equipment is involved, and then propagates to subsequent steps and grows with target distance.

For more flexible and accurate calibration, our previous work [38] further exploits the hidden constraints for the speckles on different-pose calibration boards, i.e., matches of them among calibration boards with different poses should lie on the same ray emitted from the projector and all these rays intersect at projector’s optical center. Given these hidden constraints, projector’s optical center is fitted and then utilized to compute the rigid transformation between the camera and projector. However, our previous work is time-consuming since all sets of matched speckles on different-pose calibration boards should be fitted to rays of the projector. Moreover, the fitted optical center for the projector is noise-sensitive.

To address these problems and further improve the accuracy, a novel calibration method has been proposed in this work. Instead of fitting all the matched speckle points to lines passing through projector’s optical center, the proposed method focuses only on the matched speckles on the projector’s optical axis, which significantly reduces computational cost. More importantly, it illustrates that the fitted projector’s optical axis can be used to efficiently compute the rotation between the camera and projector, after which the translation is efficiently estimated in closed form. Finally, the virtual speckle image of the projector is generated though the homography composed by the rotation and translation, and the system can be viewed as a binocular vision system with speckle pattern. All these give rise to an efficient, accurate and convenient calibration method without the need of any high-precision auxiliary equipment and reference image of speckle.

The rest of this paper is organized as follows: Section 2 introduces some preliminaries. Section 3 gives a detailed description of the architecture of the proposed method. Results for synthetic and real-data experiments are analyses in Section 4 and Section 5 summaries this paper.

2. Preliminaries

2.1 Depth-estimation model for idealized monocular laser-speckle projection system

The idealized monocular laser-speckle projection system estimates the depth via parallax. As drawn in Fig. 1, an idealized system is comprised of a camera and a laser speckle projector, with their optical axes paralleled to each other and known baseline length $s$. The camera is calibrated to obtain its focal length $f$. Other parameters in Fig. 1 are defined as follows:

By comparing the target speckle image with the reference speckle image, the deviations $\delta$ in the two images are derived. According to similarity of triangles, it is trivial that:

 

Fig. 1. Schematic diagram of the depth-estimation model for idealized monocular laser-speckle projection system.

Download Full Size | PDF

Obviously, $Z_r$ in Eq. (2) should be known in advance and this is the reason why the traditional calibration method requires reference speckle images captured at known distance in high-precision.

2.2 Camera calibration

The camera imaging model we adopt is the pinhole model [39] extended with radial and tangential distortion [40]. Suppose that $[x_w,y_w,z_w]^T$ is the coordinate of a 3D point in the world coordinate system and its distortion-free and distorted (observed) image coordinates are $[u,v]^T$ and $[u_d,v_d]^T$, respectively. The pinhole model is described as follows:

3. Proposed method

3.1 Projector optical axis calibration

In this paper, the laser speckle projector is regarded as an inverse camera, which is also modeled by Eq. (3) but with inverse optical properties. The intrinsic matrix of the projector is the same as that of the camera. In the proposed method, the virtual image of the projector is used for subsequent reconstruction tasks. This virtual image is an image that meets the ideal perspective projection model and directly derived using extrinsic parameters between the camera and projector. Therefore, the actual distortion of the projector need not be considered in the proposed method. This subsection introduces the method to derive projector’s optical axis, which is essential for the recovery of the rotation between the camera and projector.

We have designed a simple calibration board with features only covering small edge-areas of the calibration board, as shown in Fig. 2(a). Note, the requirement on this optical target’s flatness is not high and all targets with normal diffuse reflecting surface could be used for our method. In fact, in our real-data experiments, the optical target is printed by commercial printer using A4 paper sheet and then we can achieve depth-estimation with error around 0.1 mm. The position of the calibration board in a specific frame is determined by its unit normal vector $\boldsymbol {n}$ and the distance between the plane and frame’s origin, e.g., $d_{cam}$ in Fig. 2(a). Here, features of the type same as our previous work [38,42] have been adopted due to their sub-pixel detection accuracy and, more importantly, ability to be detected under poor-contrast conditions at a small size. However, the feature type is not restricted and other types able to provide accurate and stable detection can also be adopted. As the actual distance between features is known, their 3D coordinates in the target frame can be obtained.

 

Fig. 2. (a) Schematic diagram of the calibration board. (b) Speckle image of the calibration board.

Download Full Size | PDF

First, we place the calibration board in front of the system and take the image with the camera. Based on the sub-pixel coordinates of features in the images [43], we can calculate the extrinsic relationship between the target and camera frames, as well as features’ 3D coordinates $\boldsymbol {P}_F^i \in \mathbb {R}^3$ in the camera frame [44]. Then, position of the calibration board in the camera frame is fitted. Specifically, the board (plane) can be expressed as $\boldsymbol {n}_{cam}^T \boldsymbol {P}_F^i=d_{cam}$, where $\boldsymbol {n}_{cam} \in \mathbb {R}^3$ is the plane’s unit normal vector in the camera frame. The plane is found by least-squares and several-step Gauss-Newton refinement on the penalty function as:

Next, we project speckle pattern onto the calibration board and capture the images using the camera. Exploiting the fitted plane for the calibration board, it is easy to obtain 3D coordinates of these speckles in the camera frame. During the calibration, pose of the calibration board needs to be adjusted several times and the above steps are repeated. After that, the digital image correlation (DIC) [25,45] is used to match the corresponding speckles among the speckles images.

In a standard laser speckle projector, the optical axis is designed strictly perpendicular to the DOE. Moreover, the optical axis passes through the center or other fixed positions of the DOE. In general, the closer to the optical axis, the stronger the intensity of the beam, and thus the brighter the speckle in the image. As shown in Fig. 2(b), there is a larger and brighter speckle in the image, which is the projection point of the optical axis.

To calibrate projector’s optical axis, we select an image and extract the coordinate $[u_0,v_0]^T$ of this brighter speckle, whose correspondences in other images are already got by DIC before. Subsequently, their 3D coordinates $\boldsymbol {P}_o^i$ in the camera frame are derived using the fitted plane for the calibration board. Given that they all lie on projector’s optical axis $\boldsymbol {L}_o$, 3D-line fitting method is utilized to fit it in the camera frame. Specifically, accurate image coordinates of the brighter speckle are derived by: 1) labeling the area by image thresholding [46] and connected component analysis [46,47]; 2) taking the centroid of labeled area. Besides, the line-fitting is achieved using Opencv [46] fitLine() function, in which the iteratively reweighted least squares [48] are applied to minimize the following cost function:

3.2 System calibration

3.2.1 Homography transformation

For readability, variables are defined before derivations:

Fig. 3 demonstrates the principle of the calibration. Then the relationship between $\boldsymbol {P}_{cam}$ and $\boldsymbol {P}_{proj}$ can be expressed as:

 

Fig. 3. Principle of the calibration.

Download Full Size | PDF

Since the laser speckle projector projects the speckles onto the calibration board and the spatial plane equation of the calibration plane is solved in Section 3.1, coordinates of the speckles in the camera frame satisfy the following equation:

By substituting Eq. (8) into Eq. (7), we can get:

According to Eq. (9), the homography transformation between the camera image and the virtual image of projector is constructed:

3.2.2 Rotation matrix

Keeping the monocular laser speckle system stationary, and changing the pose of the calibration plane, homography transformation exists between any two of the images of the calibration board under different poses. According to Eq. (10), the homography relationship between the image of camera and the virtual image of projector is constructed:

By substituting Eq. (11) into Eq. (13), we can get:

It can be seen from the Eq. (16) that $\boldsymbol {H}^2_{pc}\boldsymbol {H}^1_{cp}$ is only related to $\boldsymbol {T}_c$ and independent of the rotation matrix $\boldsymbol {R}_{cp}$. However, this does not indicate that $\boldsymbol {R}_{cp}$ is completely unsolvable. In fact, two degrees of freedom of it, i.e., $A_x$ and $A_y$, can be determined by aligning the directions of projector’s optical axis in the camera and projector frames. The detailed derivations are explained as follows.

In a standard laser speckle projector, the optical axis is designed strictly perpendicular to the DOE and passes through its center. To make the virtual image of the projector contains the speckle pattern, we set the obtained optical axis as the $z$ axis of the projector coordinate system. Moreover, we take the direction pointing to the target as its positive direction. Suppose that the unit direction vector of the optical axis is $\boldsymbol {V}_c =\left [v_x,v_y,v_z\right ]^T$ in the camera coordinate system and $\boldsymbol {V}_p =\left [0,0,1\right ]^T$ in the projector coordinate system. The relationship between $\boldsymbol {V}_c$ and $\boldsymbol {V}_p$ can be expressed as:

Then, $A_x$ and $A_y$ can be solved:

It should be emphasized that the spatial position of the lines emitted by the projector is uniquely determined when the relative position between the camera and the laser speckle projector is fixed. If the projector coordinate system takes the laser optical center as the origin, directions of the axes of the projector coordinate system determine the position of the projector virtual image plane. However, the virtual image planes at different positions are equivalent and do not affect the depth estimation.

The Euler angle $A_z$ determines the $x$ and $y$ axes of the projector coordinate system, which will affect the coordinates of the speckle pattern in the virtual image. So we need to select an appropriate value of $A_z$ to ensure that the speckle pattern can be distributed in the center of the virtual image. Then, the rotation matrix $\boldsymbol {R}_{cp}$ is obtained by Euler angles $A_x$, $A_y$ and $A_z$.

3.2.3 Translation vector

According to Eq. (10), relationship between the speckles in the camera and projector frames can be formulated as:

Assuming that the pose of calibration board has been changed twice, we can obtain:

Suppose that $\boldsymbol {R}_{cp} = \left [ \boldsymbol {r}_0,\boldsymbol {r}_1,\boldsymbol {r}_2 \right ]^T$ and $\boldsymbol {T}_{cp} = \left [t_0,t_1,t_2 \right ]^T$, we combine Eq. (20) and Eq. (21):

Then, Eq. (22) can be equivalently described as follow:

To simplify the calculation, Eq. (23) can be changed into:

Let $\boldsymbol {M}=\begin {bmatrix} b & 0 & m_0 \\ 0 & b & m_1 \end {bmatrix}$ and $\boldsymbol {d}=\left [n_0,n_1\right ]^T$ , we select the first two lines of the Eq. (24) as:

Equation (25) contains two equations, but there are three translation factors to be solved. Therefore, we need to change the pose of the calibration board at least three times to determine the solution of $\boldsymbol {T}_{cp}$. Assuming that there are $n$ groups of calibration board poses in pairs, the Eq. (25) can be converted into:

Hence, translation vector ${\boldsymbol {T}_{cp}}$ can be calculated as:

3.2.4 Virtual image

With the estimated rotation matrix and translation, the homography from the image plane of the camera to the virtual image plane of the projector can be estimated according to Eq. (12). Then the virtual image of the projector can be recovered by Eq. (11).

4. Experiments

4.1 Numerical simulations

To verify the feasibility of the proposed method, several numerical simulations comparing the methods proposed in [38] (named as Fitting Method) and this work (named as Proposed Method), have been conducted. All numerical simulations were written and run in C++.

In the numerical simulations, speckle projector and camera were modeled as pinhole cameras, whose resolutions were set to $1920\times 1080$ (pixel) and $1280\times 1024$ (pixel), respectively. Besides, their focal length and field of view were 2100 (pixel) and $30^{\circ }$. In each trail, $A_x$, $A_y$, $A_z$ and the three elements of $\boldsymbol {T}_{cp}$ were randomly generated in the ranges of $[-5^{\circ },5^{\circ }]$, $[0^{\circ },15^{\circ }]$, $[-5^{\circ },5^{\circ }]$, $[-200,-150]$ (mm), $[-10,10]$ (mm) and $[-10,10]$ (mm), respectively. Speckle points were first randomly generated in the image plane of the projector. Then they were projected onto the calibration board (spatial plane) with different appropriate poses and, subsequently, the camera image. Based on the generated scene, performances of the two methods were studied. During the calibration process, there are many factors that affect the calibration accuracy of the monocular laser speckle projection system. We investigated the influence of the image matching accuracy, the speckle points number and the poses number of calibration board.

First, accuracy of the two methods were evaluated with varying image matching error from 0.1 to 1 (pixel) at interval of 0.1 (pixel). For each level of matching error, 500 trails were executed and the mean data were recorded. As shown in Fig. 4(a)(b)(c), the calibration errors of Euler angle, translation vector and RMS of speckle points become significantly larger as the image matching error increases, which illustrates decisive effect of the image matching error on the calibration results. In this paper, DIC is utilized to match the speckle images and the matching error is normally lower than 0.3 pixels.

 

Fig. 4. Results with respect to the simulated data. The influence of image matching error: (a) Euler angle; (b) Translation vector; (c) RMS of the speckle points. The influence of the number of the speckle points: (d) Euler angle; (e) Translation vector; (f) RMS of the speckle points. The influence of the number of calibration board poses: (g) Euler angle; (h) Translation vector; (i) RMS of the speckle points.

Download Full Size | PDF

Analogously, the two methods were tested under varying number of speckle points (100 to 1000 at interval of 100) and poses of calibration board (3 to 20 at interval of 1), respectively. Fig. 4(d)(e)(f) showcases that the accuracy of the two methods grows with increased number of speckles. They also demonstrate that adding more speckles is more helpful for improving accuracy of the translation, while has less effect on the rotation and RMS. Further, in Fig. 4(g)(h)(i), as the pose number of the calibration board increases, the calibration error gradually decreases. Such downward trend nearly stabilizes when there are more than 12 poses for the calibration board. In these figures, the curves of two methods are basically coincident, with Proposed Method slightly better than Fitting Method. The numerical simulations preliminary proves the effectiveness and accuracy of the proposed method.

4.2 Real-data experiments

To further demonstrate the calibration accuracy the proposed method, three experiments with real-data were conducted. Throughout these experiments, the optical targets on the calibration board were printed by commercial printer using common paper sheet and a single shot was used for each reconstruction.

4.2.1 Comparison between fitting method and proposed method

In the first two experiments, accuracy of Fitting Method and Proposed Method were compared on a monocular laser speckle projector system, as shown in Fig. 5(a), where an ANHUA M10EGF(LC) [49] speckle projector (resolution: $1920\times 1080$) and a MindVision MVSUA134GM camera [50] (resolution: $1280\times 1024$) are rigidly fixed together. The focal length of the camera was 2089.1 (pixel). In the first experiment, calibrated by the two methods, respectively, the projector system was used to measure the displacement of a whiteboard installed on a two-dimensional displacement table (precision: 0.01mm), which provided the ground-truth displacement. Thirteen trails in the range of 2-9 mm were conducted. The second experiment followed a similar process, though instead, measured two customized spheres [51] with known radius of 28.56 mm (precision: 0.003 mm) as shown in Fig. 5(b).

 

Fig. 5. Experimental set up for the first two real-data experiments. (a) Monocular laser speckle projector system; (b) Images of the spheres.

Download Full Size | PDF

In calibration process, the projector projected the random speckle pattern to the surfaces of the calibration board in different poses and the camera captured the corresponding speckle images. With the help of DIC, we matched the speckle images and determined the corresponding speckle sets. Then, we solved the relationship between the camera frame and the projector frame, i.e., the rotation matrix and the translation vector. With the help of the extrinsic parameters of the monocular speckle projection system and plane equations of the calibration board in different poses, we built the homography transformation between the camera image and projector’s virtual image. The projector was treated as the inverse camera by constructing the virtual image. As a result, $[A_x, A_y, A_z]$ and $\boldsymbol {T}_{cp}$ were calibrated as $[-1.87^{\circ }, 13.30^{\circ }, -0.84^{\circ }]$ and $[-178.93,-1.98,-5.91]$ (mm), respectively. Obviously, the resolutions, camera’s focal length, $[A_x, A_y, A_z]$ and $\boldsymbol {T}_{cp}$ of the two real-data experiments were close to that values set in the simulations. Therefore, the results are also expected to illustrate how well the simulations "track" the true performances of the two methods.

The results of the first experiment are shown in Fig. 6(a), the measurements of Fitting Method and Proposed Method are basically coincident with the real value, which illustrate that they can realize high-precision displacement measurement. In addition, Fig. 6(b) shows that the results of Proposed Method are obviously better than those of Fitting Method. The average errors of the two methods are 0.069 mm and 0.037 mm respectively. Compared to Fitting Method, accuracy of Proposed Method has been significantly improved.In the second experiment, by matching the virtual image of projector and speckle image of the spheres, we obtained the coordinates of the spheres in the camera frame and constructed their models. The three-dimensional models of the two spheres are shown in Fig. 7. We compare the measured radii of the spheres with their actual values in Table 1. The absolute deviations for the two spheres are 0.076 mm and 0.098 mm in Proposed Method while 0.092 mm and 0.127 mm in Fitting Method. The results of Proposed Method are better than Fitting Method significantly.

 

Fig. 6. Displacement measurement results for the first real-data experiment.

Download Full Size | PDF

 

Fig. 7. Sphere models (dark blue) fitted by the obtained point clouds (gray) for the spheres in the second real-data experiment. (a) Left sphere; (b) Right sphere.

Download Full Size | PDF

Table 1. Radius measurement of the spheres in the second real-data experiment. (unit: mm).

View Table | View all tables in this article

4.2.2 Comparison between the proposed and tradition methods

In the third experiment, Proposed Method was compared with the traditional method using Orbbec Astra-s and Astra-pro [34]. The resolutions of their cameras (infrared) are $1280\times 1024$ and $640\times 480$, respectively. Depth-estimation accuracy of the two methods were measured via the reconstruction of a plane. In particular, with the help of the proposed and traditional methods respectively, depths of the speckles on the plane are estimated and employed to fit the plane. Then, the average off-plane error and the depth distribution range were recorded. The trail number for Astra-s and Astra-pro are five and ten, respectively.

Astra-s and Astra-pro are both pre-calibrated using the traditional method. In addition, we followed the proposed calibration process to calibrate the two Astra systems and constructed the virtual image of the projector for measurement. Fig. 8 shows a sample speckle image captured by the camera and its corresponding virtual image of the projector. As shown in Table 2 and Table 3, the proposed method obtains much better results for Astra-s and Astra-pro than that measured by themselves, i.e., the traditional method, with performances on the average off-plane error and depth distribution range improved by about 10 times.

 

Fig. 8. Sample images of the third real-data experiment. (a) Speckle image of camera; (b) virtual speckle image of projector.

Download Full Size | PDF

Table 2. Results w.r.t the third real-data experiment: comparison between the traditional and proposed methods on Astra-s (unit: mm).

View Table | View all tables in this article

Table 3. Results w.r.t the third real-data experiment: comparison between the traditional and proposed methods on Astra-pro (unit: mm).

View Table | View all tables in this article

5. Conclusion

An efficient and accurate calibration method for the monocular laser speckle projection system has been proposed in this paper, which does not require any expensive high-precision auxiliary equipment or reference speckle image. The key to such improvement is the hidden colinearity of the correspondences among the speckles on different-pose calibration boards. To avoid redundant computation, only the projector’s optical axis is fitted using these correspondences. The rotation is calibrated by aligning the optical axis in the camera and projector frame. After that, the translation and virtual image of the projector are recovered easily. The superior performance of the proposed method has been validated both by the simulated and real-data experiments. Experimental results on Astra-s and Astra-pro show that the proposed method obviously outperforms the traditional method in depth-estimation, with the accuracy improved by about 10 times. There are still future works to improve the proposed method further, such as utilizing the precise stereo calibration devices to improve the calibration accuracy. Although the proposed method focuses on the laser speckle projector, it can also be applied to software-controlled projectors. To achieve this, designed patterns satisfying the requirements of the proposed method, or more conveniently, speckle patterns, should be used.