1. Introduction

Structured light 3D modeling systems have been widely used in quality inspection, medicine, robotics and other fields [1–7]. In general, the structured light 3D modeling system consists of one camera and one projector. When structured light patterns are emitted by the projector onto an object, these patterns are distorted by the surface of the measured object and then captured by the camera. Once the camera and projector is calibrated, the three-dimension coordinates of the measured object can be obtained from the distorted structured light patterns via triangulation.

The conventional projectors used in the structured light 3D modeling system are DLP projectors with large volume and heavy weight. In addition to the DLP projector, the galvanometric [8,9] and MEMS mirror [10,11] are used as new projection technologies with the advantages of small size, light weight, and no demand of focusing optics [12,13]. In this paper, a one-axis MEMS mirror system containing a laser diode and a one-axis MEMS mirror was adopted to replace the DLP projector for realizing miniaturization of the structured light system. It can project laser stripes in one direction, but with slight bending caused by assembly error. By incorporating the one-axis MEMS mirror, this compact structured light system is easy to achieve a large measuring range, due to the system’s depth of field is only up to the camera’s focus range.

The measurement accuracy of the structured light 3D modeling system mainly depends on the accuracy of the calibration of the camera and projector. By being studied for several decades, there are lots of methods proposed for the camera calibration [14–17]. One of the most widely used methods is proposed by Zhang [17], which can obtain an accurate calibration result just by utilizing the camera to capture a planar calibration target at different locations. The projector calibration is a more complicated task compared to the camera calibration because the projector is unable to capture images. Numerous calibration methods were proposed to obtain the system parameters used in 3D reconstruction [18–20]. These methods are generally time consuming and complicated. Reference-plane-based methods, with simple process, were widely applied to the systems that with no demand for high accuracy [21–24]. Zhang and Huang proposed a method that enables the projector to capture images by treating the projector as a reverse camera [25], from which the camera calibration method can be applied to the projector. This method can achieve highly accurate calibration through a simple process, but can only be applied to the projector with the ability of projecting orthogonal patterns. By treating each structured light stripe as an ideal light plane, some Light Plane Model (LPM) methods were proposed to calibrate each structured light stripe directly [26,27]. These methods neglect the distortion of the light stripes, so it is difficult to obtain high-precision calibration results in severely distorted systems. To correct the distortion of the light stripes, Yang [28] proposed a method of fitting the light stripes using the general conicoid equation. This method works well on correcting the distortion, but the calibration procedures are complicated with demand of two cameras, one of which is not necessary in monocular systems. Another problem with this method is that it is prone to overfitting, so the reconstructed surface is probably not smooth enough. Lu [29] proposed a method to correct the distortion by directly creating the relationship between the sub-pixel coordinates of each stripe and 3D coordinates. In order to achieve high accuracy, the adopted order of polynomial is 5th, so the calculation process is time-consuming.

In this paper, we proposed a calibration method for the structured light 3D modeling systems incorporating the one-axis MEMS mirror that can only project light stripes in one direction. By treating each light stripe as a curved light surface, a curved surface equation with seven parameters was proposed to replace the ideal plane equation as the mathematic model of the light stripes, called curved light surface model (CLSM), and was used in the projector calibration and 3D reconstruction to reduce the effect of the light stripe’s distortion. The main idea of the calibration is to fit the light stripes one-by-one using CLSM. Experiments were conducted to demonstrate that the proposed method can significantly reduce the effect of the projection distortion, and the accuracy of the system is 0.11 mm when measuring a dumbbell-shape two spheres object with a center-to-center distance of 201.10 mm.

2. Curved light surface model

The schematic diagram of the one-axis MEMS mirror system is illustrated in Fig. 1. A mirror with a diameter of 3 mm is supported by a coaxial cantilever, and it can sinusoidal vibrate at a frequency of 1.15 kHz when actuated by an AC voltage. By emitting a line laser onto the rotation axis of the mirror and controlling the laser on and off in the time domain as the mirror vibrates with a frequency much higher than camera capturing frequency, a structured light pattern can be generated and captured by the camera. The line laser is achieved by letting a collimated spot laser incident perpendicularly on a cylindrical lens. The size of the entire system above is less than 30 mm, which is much smaller than the size of DLP projectors. In practical applications, since slight assembly errors cannot be avoided, it is almost impossible to ensure that the line laser and the rotation axis of the MEMS mirror completely coincide, nor can it ensure that the spot laser is perpendicularly incident on the cylindrical lens. Therefore, the line laser emitted from the cylindrical lens has slight bending, forming a curved light surface spatially.

 

Fig. 1. Schematic diagram of the one-axis MEMS mirror system.

Download Full Size | PDF

In this paper, the structured light pattern produced by the one-axis MEMS mirror system consists of 1024 vertical stripes, each of which is formed independently. They are numbered as stripe numbers $I = 1,2,3,\ldots ,1024$. Due to the distortion of these stripes, the ideal plane equation is unable to accurately describe them. The general conicoid equation can describe the distorted light [28] but is prone to overfitting. Therefore, we proposed a curved surface equation as the mathematic model of the light stripes for the projector calibration and 3D reconstruction and the derivation process is as follows. Since the formation processes of all the 1024 curved light surfaces are similar, any curved light surface can be chosen to demonstrate the concept of curved light surface model. Here, the curved light surface with I = 512 is selected. In Fig. 2, ${O_p}{x_p}{y_p}{z_p}$ is the projector coordinate system (PCS), where is fixed on the center of the mirror, ${O_c}{x_c}{y_c}{z_c}$ is the camera coordinate system (CCS). Except the cylindrical lens used to generate the line laser, there are no other optical lenses in the one-axis MEMS mirror system. Therefore, after being reflected by the MEMS mirror, there will be no distortion such as radial distortion and tangential distortion in the projected light. In such a system, PCS can be set freely and we set it to be parallel to CCS to facilitate model building. When the laser stripe was projected onto the mirror at a specific angle, this stripe is reflected by the mirror to form $Surfac{e_{512}}$. Let $Plan{e_1}$ and $Plan{e_2}$ be arbitrary planes in PCS, ${L_1}$ is the intersection curve of $Surfac{e_{512}}$ on $Plan{e_1}$, ${L_2}$ is the intersection curve of $Surfac{e_{512}}$ on $Plan{e_2}$, ${L_3}$ is a light ray from ${O_p}$ that belongs to $Surfac{e_{512}}$, and respectively intersects $Plan{e_1}$ and $Plan{e_2}$ to form points ${P_1}({x_1},{y_1},{z_1})$ and ${P_2}({x_2},{y_2},{z_2})$ in PCS respectively.

 

Fig. 2. Geometric diagram of the curved light surface with I = 512.

Download Full Size | PDF

The design working distance of the proposed system is more than 300 mm, which is much larger than the diameter of the MEMS mirror. Therefore, the MEMS mirror is assumed to be a point. It means that all the paths of the projected light converge at the point of ${O_p}$, so it is easy to derive Eq. (1):

3. System calibration and 3D reconstruction

3.1. Camera calibration

The pinhole imaging model is widely used to describe the camera model. As shown in Fig. 3, (P) is a spatial point, ${P_i}(u,v)$ is the ideal projection of P onto the image plane Ouv, ${O_w}{x_w}{y_w}{z_w}$ is the world coordinate system (WCS). The transformation between $P({x_w},{y_w},{z_w})$ and ${P_i}(u,v)$ is expressed as:

 

Fig. 3. Pinhole imaging model.

Download Full Size | PDF

Influenced by the distortion of lens, the measured projection of P on the image plane is ${P_d}({u_d},{v_d})$. In general, the lens distortion is composed of the radial distortion and tangential distortion, which can be expressed by the following equation [14]:

Camera calibration estimates the intrinsic parameters $({f_u},{f_v},{u_0},{v_0},\gamma )$, extrinsic parameters $({R_C},{T_C})$, and distortion coefficients $({k_1},{k_2},{k_3},{k_4},{k_5})$. The method proposed by Zhang [17] is adopted with the advantages of convenience and high-precision, and a camera calibration toolbox is available in OpenCV.

3.2. Projector calibration based on curved light surface model

Projector calibration is to estimate the projector parameter COE. Based on CLSM proposed in section 2, this paper introduced a calibration method for the projectors that can project stripes in one direction. The schematic diagram of this method is shown in Fig. 4. A planar chessboard fixed on a removable shelf is required and the board is large enough to allow all the 1024 curved light surfaces to be projected onto it. The main idea is to obtain 3D coordinates of sufficient spatial points belonging to these curved light surfaces by placing chessboard at random positions and adjusting the one-axis MEMS mirror system to project these curved light surfaces onto the chessboard. Then, we apply these data to fit the 1024 curved light surfaces one-by-one using Eq. (5). It is time-consuming to project only one stripe onto the chessboard at a time and repeat this process 1024 times to obtain the data of all the 1024 curved light surfaces. Therefore, the structured light encoding technique combining 8-bit gray coding and four steps line-shift strategy is adopted to assist the calibration [30,31], by which the data of all the 1024 curved light surfaces can be obtained from only 12 images.

 

Fig. 4. Schematic diagram of the projector calibration.

Download Full Size | PDF

The structured light patterns used in the calibration are composed of 1024 vertical stripes, and their stripe number is $I = 1,2,3,\ldots ,1024$. The conventional coding methods are random pattern, gray coding, and sinusoidal phase coding [32]. Due to the one-axis MEMS mirror can only project light stripes in one direction and it is difficult to accurately control the light intensity, 8-bit gray coding which only contains white and black striped patterns is adopted. As shown in Fig. 5(a), the entire projection image is divided into ${2^8}$ sub-regions by 8-bit gray coding patterns, corresponding to the decoding number ${I_G} = 0,1,2,\ldots ,255$ which can be converted by the 8-bit gray code words. The finest resolution of 8-bit gray coding is 4 pixels, which is not fine enough for high-precision 3D reconstruction. In theory, 10-bit gray coding has a resolution of 1 pixel. However limited by the width of the line laser, it is difficult for the one-axis MEMS mirror to ensure a good contrast in the patterns whose finest resolution is less than 4 pixels. Therefore, 8-bit gray coding pattern is recommended.

 

Fig. 5. Structured light coding strategy. (a) 8-bit gray coding divide the image into 256 sub-regions; (b) four steps line-shift patterns use stripe edges to encode each gray coding sub-region.

Download Full Size | PDF

The conventional methods to further decompose the sub-region are line-shift strategy and phase-shift strategy. Stripe edges are robust and not easily affected by laser speckle and surface texture, so we adopt the four steps line-shift strategy to encode these sub-regions. As shown in Fig. 5(b), the first pattern of the line-shift strategy is a binary stripe pattern with a stripe width of 4 pixels and is shifted three times in steps of 1 pixel to produce the three other patterns. In the line-shift patterns, edges of stripes can be detected with line-shift number ${I_S} = 1,2,3,4$ after binaryzation. By matching these edges with the gray coding sub-regions and then combining ${I_G}$ and ${I_S}$, the stripe number I can be ascertained by the follow equation:

During the calibration, the one-axis MEMS mirror system is controlled to project the structured light sequence onto the chessboard, meanwhile the camera is used to capture the image sequence. The projected curved light surfaces intersect the chessboard, generating abundant intersection points. By performing sub-pixel stripe edge detection [33] and structured light decoding, the stripe number I and sub-pixel coordinates $(u,v)$ of these intersection points are ascertained. The 3D coordinates of these intersection points in CCS are required for fitting, which can be calculated by combining the equations of camera rays and the chessboard plane. To simplify calculation, the origin ${O_w}$ of WCS is fixed on the corner of the chessboard, the axis ${O_z}$ is perpendicular to the chessboard, and the ${x_w}{y_w}$ plane coincides with the chessboard. The equation of the chessboard plane can be expressed by its extrinsic parameters, which is shown as:

A convenient approach to ascertain the extrinsic parameters of the chessboard plane is to generate the floodlighting images containing the chessboard from the image sequence and add it into the camera calibration procedures. The camera ray is a ray passing through the spatial point ${P_c}({x_c},{y_c},{z_c})$ and the projection point ${P_i}(u,v)$ on the normalized image plane, which can be expressed as:

By applying the calculations above, the coordinates of all the intersection points in CCS can be obtained from the image sequence captured at one position. Since the chessboard is placed approximately perpendicular to the axis ${z_c}$, the axis ${z_c}$ coordinate range of the intersection points from the chessboard placed at one position is much smaller than the depth of the measuring range. It is difficult to obtain good fit results by such data, so we move the chessboard to another position along the axis ${z_c}$ to expand the axis ${z_c}$ coordinate range of the intersection points. The greater the distance between the two positions, the higher accuracy of the calibration. The data obtained from two random positions is sufficient to do the fitting. Therefore, in order to simplify the calibration process as much as possible, no more positions are used. We place the chessboard at the forefront and the last end of the measuring range to obtain high accuracy within the entire measuring range. These intersection points are divided into 1024 groups according to the stripe number I to fit each column of COE. The fitting is a non-linear least squares problem which can be solved by minimizing the following function:

3.3. 3D reconstruction

To obtain the 3D coordinates of an object, the one-axis MEMS mirror system projects the same structured light sequence used in the projector calibration onto the measured object, meanwhile, the camera captures these stripe images. For a point to be measured, by performing the sub-pixel stripe edge detection and structured light decoding on these stripe images, the stripe number I and sub-pixel coordinates $(u,v)$ of this point can be ascertained, and then the camera ray equation and curved light surface equation of the measured point can be expressed by Eq. (11) and Eq. (5) respectively. Combine the two equations, then:

After solving Eq. (14) to obtain the resolution of s, the 3D coordinates of the measured point in CCS can be expressed by Eq. (11). Applying the above calculation to all points detected from the stripe images, the 3D reconstruction can be implemented.

4. Experiments and results

The structured light 3D modeling system proposed in this paper is shown in Fig. 6, (a) camera (IDS UI-3240CP-NIR-GL, $1280 \times 1024$) assembling a 6 mm lens and a projector are assembled in a structure with a horizontal distance of 80 mm. The projector consists of a line laser with a wave length of 830 nm, and a one-axis MEMS mirror with a diameter of 3 mm (commercially available from Zhisensor Tech.). The measuring range of the axis ${x_c}$ and axis ${y_c}$ increases with working distance and is 490 mm (${x_c}$) ${\times}$ 425 mm (${y_c}$) at focus distance (about 500 mm). A planar chessboard (800 mm ${\times}$600 mm) with 10 ${\times}$10 square patterns (30 mm ${\times}$ 30 mm), which is fixed on a movable shelf, was used in system calibration. The flatness error of the chessboard is 0.002 mm.

 

Fig. 6. The composition of the proposed structured light 3D modeling system.

Download Full Size | PDF

4.1. System calibration

In the camera calibration, the chessboard needs to be captured at many different positions so that all detected points are distributed uniformly. Generally speaking, when the number of images is less than 20, the more images captured, the greater the accuracy of the camera calibration. As the number of photos increases, the accuracy will gradually stabilize. It is recommended to use 20 images, because when the number of images exceeds 20, adding images will no longer produce significant benefits for improving accuracy. We first placed the chessboard at 20 different positions to capture 20 images, named ${I_1} \sim {I_{20}}$. Then, the chessboard was moved to the forefront and the last end of the measuring range, meanwhile, we adjusted the one-axis MEMS mirror system to project the structured light sequence onto the chessboard and utilized the camera to capture two image sequences, named ${S_{21}}$ and ${S_{22}}$. The floodlighting images ${I_{21}}$ and ${I_{22}}$ containing the chessboard were created from ${S_{21}}$ and ${S_{22}}$. The camera calibration was conducted by applying Zhang’s method [17] to ${I_1} \sim {I_{22}}$ and the results are as follows:

The extrinsic matrixes [${R_{21}},{T_{21}}$] and [${R_{22}},{T_{22}}$] obtained from the camera calibration were required in the projector calibration. The procedures described in section 3.2 were performed to ${S_{21}}$ and ${S_{22}}$ to ascertain the 3D coordinates and stripe numbers I of 1,599,584 intersection points. The Levenberg-Marquardt Algorithm were used to calculate the projector parameter COE, with an initial guess of [1,1,1,1,1,1,1]$^T$. The fitting of all the 1024 curved light surfaces are similar, therefore, the result of any curved light surface can be discussed to evaluate the performance of CLSM. Here, the result of the curved light surface with the stripe number $I = 512$ is selected, which is shown as:

Moreover, we used the ideal plane equation to fit the same curved light surface to evaluate the performance of Light Plane Model (LPM) method, the obtained equation is shown as:

Figure 7 shows the residuals of the intersection points of the curved light surface whose stripe number is I = 512 and the chessboard placed at the farther position from the two equations. The maximum absolute values of the residuals and the root mean square errors are 0.1597 mm and 0.0558 mm for CLSM method, and 0.8996 mm and 0.414 mm for LPM method. The results demonstrated that the fitting in CLSM method does not require an high-precision initial guess and the proposed curved surface equation performs much better in fitting the curved light surface compared to the ideal plane equation.

 

Fig. 7. Residuals of the intersection points of the curved light surface with I = 512 and the chessboard placed at the farther position from the two equations.

Download Full Size | PDF

4.2. Experiment of 3D reconstruction

By performing the 3D reconstruction procedures described in section 3.3, the chessboard used in the calibration was measured to compare CLSM method and LPM method. The measurement was conducted at eight different distances to evaluate the accuracy over the entire measuring range. The obtained point clouds were used to fit ideal planes and calculate the errors between the measured points and the best-fit planes, and the results are shown in Table 1. We can see that in the entire measuring range, the root mean square errors of CLSM method are significantly small than LPM method. Another phenomenon that can be seen is that the errors of the two methods increase along with the working distance. The main reason is that CLSM and LPM are approximate models, so there are deviations from the true model in the two methods, although the deviation of CLSM method is much smaller than LPM method, and the influence of the deviation in triangulation increases along with the working distance. To compare the two methods in more detail, Fig. 8 shows the errors of all measured points from the chessboard placed at position No. 4, No. 6, and No. 8. It can be easily observed that the errors of LPM method are distributed unevenly, those of the upper and lower parts of the chessboard are greater than 3 mm, while the errors in the middle are less than -1.5 mm. These uneven error distributions indicated that the measurements using LPM method were strongly impacted by the distortion of the light stripes. A better performance is shown in CLSM method that the errors are evenly distributed and significantly smaller than LPM method. The results verified that CLSM method can greatly reduce the effect caused by the distortion of the light stripes.

 

Fig. 8. Analysis of measurement results at different positions. (a) Photograph of the chessboard captured at position No. 4, and the measurement errors using the two methods; (b) the corresponding results obtained at position No. 6; (c) the corresponding results obtained at position No. 8.

Download Full Size | PDF

Table 1. Measurement results of the chessboard using CLSM method and LPM method.

View Table | View all tables in this article

To further analyze the accuracy of CLSM method and LPM method, a standard dumbbell-shaped object was measured at six different positions over the measuring range. The measured object is shown in Fig. 9(a), with a diameter of 38.10 ${\pm}$0.01 mm, and with a center-to-center distance of the two spheres of 201.10 ${\pm}$ 0.01 mm. Figure 9(b) shows one of the obtained point clouds of the standard dumbbell-shaped object. We used the point cloud to fit spheres by least squares approach to calculate the center-to-center distance. Table 2 shows the results, the root mean square error and the standard deviation of the center-to-center distance are 0.11 mm and 0.11 mm for CLSM method, and 0.55 mm and 0.41 mm for LPM method. The standard deviation of the error is equal to the standard deviation of the center-to-center distance. Any one of them can be computed to compare the stability of the two methods, so the standard deviation of the error is not computed here. The results showed that the accuracy of CLSM method is better than LPM method.

 

Fig. 9. Measurement of the standard dumbbell-shaped object. (a) Photograph of the standard dumbbell-shaped object; (b) a obtained point cloud.

Download Full Size | PDF

Table 2. Measurement results of the standard dumbbell-shaped object.

View Table | View all tables in this article

Moreover, we conducted a 3D reconstruction of a plaster statue containing complex details to evaluate the system’s performance. After removing the useless points of the obtained point cloud, the results are shown in Fig. 10. It can be observed that the surface of reconstruction is smooth and the details such as eyes, nose defect, and hair were recovered well.

 

Fig. 10. Reconstruction of the plaster statue using CLSM method. (a) Photograph of the plaster statue; (b) six-th gray coding pattern; (c) first stripe-shift pattern; (d) reconstruction result.

Download Full Size | PDF

In summary, by performing the three experiments above, the root mean square errors of the chessboard plane and standard dumbbell-shaped object were obtained to evaluate the accuracy of our method and LPM method. The results demonstrated that CLSM method perform better than LMP method. The plaster statue was reconstructed to evaluate the performance of the proposed system on object containing complex details.

5. Conclusion

In this paper, a calibration method was proposed for the calibration of the structured light 3D modeling system incorporating a one-axis MEMS mirror that can only support unidirectional stripe projection. By analyzing the geometry of the projected light stripes, a curved surface equation called curved light surface model was proposed as the mathematic model of the light stripes for the projector calibration and 3D reconstruction. Due to the projection distortion has been considered in CLSM, no other processing is required to rectify the projection distortion in the 3D reconstruction. The experiment results verified that CLSM method can significantly reduce the effect of the light stripe’s distortion, and an accuracy of 0.11 mm was achieved when measuring a dumbbell-shaped object with a center-to-center distance of 201.10 mm. This method can also be adaptable to DLP projectors and other systems that can project stripes in one direction with distortion caused by lens or assembly error.