Flexible calibration method for microscopic structured light system using telecentric lens

Beiwen Li; Song Zhang

doi:10.1364/OE.23.025795

1. Introduction

With recent advances in precision manufacturing, there has been an increasing demand for the development of efficient and accurate micro-level 3D metrology approaches. A structured light (SL) system with digital fringe projection technology is regarded as a potential solution to micro-scale 3D profilometry owing to its capability of high-speed, high-resolution measurement [1]. To migrate this 3D imaging technology into micro-scale level, a variety of approaches were carried out, either by modifying one channel of a stereo microscope with different projection technologies [2–6], or using small field-of-view (FOV), non-telecentric lenses with long working distance (LWD) [7–11].

Apart from the technologies mentioned above, an alternative approach for microscopic 3D imaging is to use telecentric lenses because of their unique properties of orthographic projection, low distortion and invariant magnification over a specific distance range [12]. However, the calibration of such optical system is not straightforward especially for Z direction, since the telecentricity will result in insensitivity of depth changing along optical axis. Zhu et al. [13] proposed to use a camera with telecentric lens and a speckle projector to perform deformation measurement with digital-image-correlation (DIC). Essentially the Z direction in this system was calibrated using a translation stage and a simple polynomial fitting method. To improve the calibration flexibility and accuracy, Li and Tian [12] has formulated the orthographic projection of telecentric lens into an intrinsic and an extrinsic matrix, and successfully employed this model to an SL system with two telecentric lenses in their later research [14]. This technology has shown the possibility to calibrate a telecentric SL system analogously to a regular pin-hole SL system.

These aforementioned approaches for telecentric lens calibration have been proven successful to achieve different accuracies. However, Zhu’s approach [13] is based on a polynomial fitting method along Z direction using a high-accuracy translation stage. And the use of a high-accuracy translation stage for system calibration is usually difficult to setup (e.g., moving direction perfectly perpendicular to Z axis) and expensive. While Li’s method [12, 14] increases the calibration flexibility and simplifies the calibration system setup, it is difficult for such a method to achieve high accuracy for extrinsic parameters calibration due to the strong constraints requirement (e.g. orthogonality of rotation matrices). Moreover, the magnification ratio and extrinsic parameters are calibrated separately, further complicating the calibration process and increasing the uncertainties of modeling accuracy since the magnification and extrinsic parameters are naturally coupled together and are difficult to separate.

To address the aforementioned limitations of the state-of-art system calibration, we propose to use a LWD pin-hole lens to calibrate a telecentric lens. Namely, we developed a system that includes a camera with telecentric lens and a projector with small FOV and a projector with a LWD pin-hole lens. Since the pin-hole imaging model has been well-established and its calibration is well studied, we can use a calibrated pin-hole projector to assist the calibration of a camera with a telecentric lens. To the best of our knowledge, there are no flexible and accurate approach to calibrate an SL system using a camera with telecentric lens and a projector with a pin-hole lens.

In this research, we propose a novel framework to calibrate such a structured light system using a camera with a telecentric lens and a projector using a pin-hole lens. The pin-hole projector calibration follows the flexible and standard pin-hole camera calibration procedures enabled by making a projector to capture images like a camera, a method developed by Zhang and Huang [15]. With the calibrated projector, 3D coordinates of those feature points used for projector calibration are then estimated through iterative Levenberg-Marquardt optimization. Those reconstructed 3D feature points are further used to calibrate the camera with a telecentric lens. Since the same set of points are used for both projector and camera calibration, the calibration process is quite fast; and because a standard flat board with circle patterns are used and posed flexibly for the whole system calibration, the proposed calibration approach of a structured light system using a telecentric camera lens is very flexible.

Section 2 introduces the principles of the telecentric and pinhole imaging systems. Section 3 illustrates the procedures of the proposed calibration framework. Section 4 demonstrates the experimental validation of this proposed calibration framework. Section 5 summaries the contributions of this research.

2. Principle

The camera model with a telecentric lens is demonstrated in Fig. 1. Basically, the telecentric lens simply performs a magnification in both X and Y direction, while it is not sensitive to the depth in Z direction. By carrying out ray transfer matrix analysis for such an optical system, the relationship between the camera coordinate (o^c; x^c, y^c, z^c) and the image coordinate ( $o_{0}^{c}$ ; u^c, v^c) can be described as follows:

[\begin{matrix} u^{c} \\ v^{c} \\ 1 \end{matrix}] = [\begin{matrix} s_{x}^{c} & 0 & 0 & 0 \\ 0 & s_{y}^{c} & 0 & 0 \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} x^{c} \\ y^{c} \\ z^{c} \\ 1 \end{matrix}],

where

s_{x}^{c}

and

s_{y}^{c}

are respectively the magnification ratio in X and Y direction. While the transformation from the world coordinate (o^w; x^w, y^w, z^w) to the camera coordinate can be formulated as follows:

[\begin{matrix} x^{c} \\ y^{c} \\ z^{c} \\ 1 \end{matrix}] = [\begin{matrix} r_{11}^{c} & r_{12}^{c} & r_{13}^{c} & t_{1}^{c} \\ r_{21}^{c} & r_{22}^{c} & r_{23}^{c} & t_{2}^{c} \\ r_{31}^{c} & r_{32}^{c} & r_{33}^{c} & t_{3}^{c} \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} x^{w} \\ y^{w} \\ z^{w} \\ 1 \end{matrix}],

where

r_{i j}^{c}

and

t_{i}^{c}

respectively denotes the rotation and translation parameters. Combining Eq. (1) – (2), the projection from 3D object points to 2D camera image points can be formulated as follows:

[\begin{matrix} u^{c} \\ v^{c} \\ 1 \end{matrix}] = [\begin{matrix} m_{11}^{c} & m_{12}^{c} & m_{13}^{c} & m_{14}^{c} \\ m_{21}^{c} & m_{22}^{c} & m_{23}^{c} & m_{24}^{c} \\ 0 & 0 & 0 & 1 \end{matrix}] [\begin{matrix} x^{w} \\ y^{w} \\ z^{w} \\ 1 \end{matrix}]

Fig. 1 Model of telecentric camera imaging.

Download Full Size | PDF

The projector model respects a well-known pin-hole imaging model is illustrated in Fig. 2. The projection from 3D object points (o^w; x^w, y^w, z^w) to 2D projector sensor points ( $o_{0}^{p}$ ; u^p, v^p) can be described as follows:

s^{p} [\begin{matrix} u^{p} \\ v^{p} \\ 1 \end{matrix}] = [\begin{matrix} α & γ & u_{0}^{c} \\ 0 & β & v_{0}^{c} \\ 0 & 0 & 1 \end{matrix}] [\begin{matrix} R_{3 \times 3}^{p} & t_{3 \times 1}^{p} \end{matrix}] [\begin{matrix} x^{w} \\ y^{w} \\ z^{w} \\ 1 \end{matrix}] .

Here, s^p represents the scaling factor; α and β are respectively the effective focal lengths of the camera u and v direction. The effective focal length is defined as the distance from the pupil center to the imaging sensor plane. Since the majority software (e.g., OpenCV camera calibration toolbox) gives these two parameters in pixels, the actual effective focal lengths can be computed by multiplying the camera pixel size in u and v direction, respectively. (

u_{0}^{c}

,

v_{0}^{c}

) are the coordinates of the principle point;

R_{3 \times 3}^{p}

and

t_{3 \times 1}^{p}

are the rotation and translation parameters.

Fig. 2 Model of pinhole projector imaging.

Download Full Size | PDF

3. Procedures

The calibration framework includes seven major steps:

Step 1: Image capture. Use a 9 × 9 circle board [see Fig. 3(a)] as the calibration target. Put the calibration target at different spatial orientations. Capture a set of images for each target pose, which is composed of the projection of horizontal patterns, vertical patterns and a pure white frame.
Step 2: Camera circle center determination. Pick the captured image with pure white fringe (no pattern) projection, extract the circle centers (u^c, v^c) as the feature points. An example is shown in Fig. 3(b).
Step 3: Absolute phase retrieval. To calibrate the projector, we need to generate a “captured” image for the projector since the projector cannot capture images by itself. This is achieved by mapping a camera point to a projector point using absolute phase. To obtain the phase information, we use a least-square phase-shifting algorithm with 9 steps (N = 9). The k-th projected fringe image can be expressed as follows: $I_{k} (x, y) = I^{'} (x, y) + I^{″} (x, y) cos (ϕ + 2 k π / N),$ where I′(x, y) represents the average intensity, I″(x, y) the intensity modulation, and ϕ(x, y) the phase to be solved for, $ϕ (x, y) = {tan}^{- 1} [\frac{\sum_{k = 1}^{N} I_{k} sin (2 k π / N)}{\sum_{k = 1}^{N} I_{k} cos (2 k π / N)}] .$ This equation produces wrapped phase map ranging [−π, +π). Using a multi-frequency phase-shifting technique as described in [16], we can extract the absolute phase maps Φ_ha and Φ_va without 2π discontinuities respectively from captured images with horizontal [see Fig. 3(c)] and vertical pattern projection [see Fig. 3(d)].
Step 4: Projector circle center determination. Using the absolute phases obtained from step 3, the projector circle centers (u^p, v^p) can be uniquely determined from the camera circle centers obtained from Step 2: $u^{p} = ϕ_{h a}^{c} (u^{c}, v^{c}) \times P_{1} / 2 π,$ $v^{p} = ϕ_{v a}^{c} (u^{c}, v^{c}) \times P_{2} / 2 π,$ where P₁ and P₂ are respectively the fringe periods of the horizontal and vertical patterns used in Step 3 for absolute phase recovery, which are 18 and 36 pixels in our experiments. This step simply converts the absolute phase values into projector pixels.
Step 5: Projector intrinsic calibration. Using the projector circle centers extracted from the previous step, the projector intrinsic parameters (i.e. α, β, γ, $u_{0}^{c}$ , $v_{0}^{c}$ ) can be estimated using standard OpenCV camera calibration toolbox.
Step 6: Estimation of 3D target points. Align the world coordinate (o^w; x^w, y^w, z^w) with the projector coordinate (o^p; x^p, y^p, z^p), then the projector extrinsic matrix [ $R_{3 \times 3}^{p}$ , $t_{3 \times 1}^{p}$ ] becomes [I_3×3, 0_3×1], which is composed of a 3 × 3 identity matrix I_3×3 and a 3 × 1 zeros vector 0_3×1. To obtain the 3D world coordinates of target points, we first define the target coordinate (o^t; x^t, y^t, 0) by assuming its Z coordinate to be zero, and assign the upper left circle center (point A in Fig. 3(a)) to be the origin. For each target pose, we estimate the transformation matrix [R_t, t_t] from target coordinate to world coordinate using iterative Levenberg-Marquardt optimization method provided by OpenCV toolbox. Essentially this optimization approach iteratively minimizes the difference between the observed projections and the projected object points, which can be formulated as the following functional: $min_{R_{i}, t_{i}} ‖ [\begin{matrix} u^{p} \\ v^{p} \\ 1 \end{matrix}] - M_{p} [R_{t}, t_{t}] [\begin{matrix} x^{t} \\ y^{t} \\ 0 \\ 1 \end{matrix}] ‖,$ where ‖ · ‖ denotes the least square difference. M_p denotes the projection from the world coordinate to image coordinate. After this step, the 3D coordinates of the target points (i.e. circle centers) can be obtained by applying this transformation to the points in target coordinate.
Step 7: Camera calibration. Once the 3D coordinates of the circle centers on each target pose are determined, the camera parameters $m_{i j}^{c}$ can be solved in the least-square sense using Eq. (3).

Fig. 3 Illustration of calibration process. (a) Calibration target; (b) camera image with circle centers; (c) capture image with horizontal pattern projection; (d) capture image with vertical pattern projection; (e) mapped circle center image for projector; (f) estimated 3D position of target points.

Download Full Size | PDF

After calibrating both the camera and the projector, using Eq. (3) and Eq. (4), we can compute the 3D coordinate (x^w, y^w, z^w) of a real-world object based on calibration.

4. Experiments

We have conducted some experiments to validate the accuracy of our calibration model. The test system includes a digital CCD camera (Imaging Source DMK 23U274) with a pixel resolution of 1600 × 1200, and a DLP projector (LightCrafter PRO4500) with a pixel resolution of 1140 × 912. The telecentric lens used for the camera is Opto-engineering TC4MHR036-C with a magnification of 0.487. It has a working distance of 102.56 mm a field depth of 5 mm. The LWD lens used for the projector has a working distance of 700 mm and an FOV of 400 mm × 250 mm. Since both lenses used for the camera and the projector have a distortion ratio less than 0.1%, therefore, we ignored the lens distortion from both the camera and the projector lenses for simplicity. To validate the accuracy of our model, we examined the reprojection error for both the camera and the projector, as shown in Fig. 4. It indicates that our model is sufficient to describe both the camera and the projector imaging, since the errors for both the camera and the projector are mostly within ±5μm. The root-mean-square (RMS) errors are respectively 1.8 μm and 1.2 μm. Since the camera calibration was based on a calibrated projector, one may notice that the projector calibration has slightly higher accuracy than camera calibration. This could be a result of the coupling error from the mapping besides optimization error, or because the camera has smaller pixel size than the projector.

Fig. 4 Reprojection error of the calibration approach. (a) Reprojection error for the camera (RMS: 1.8 μm); (b) reprojection error for the projector (RMS: 1.2 μm)

Download Full Size | PDF

We first measured the lengths of two diagonals $\bar{AC}$ and $\bar{BD}$ [see Fig. 3(a)] of the calibration target under 10 different orientations. The two diagonals are formed by the circle centers on the corner. The circle centers on this calibration target was precisely manufactured with a distance d of 1.0000 ± 0.0010 mm. Therefore, the actual length of $\bar{AC}$ and $\bar{BD}$ can be expressed by $8 \sqrt{2} d$ , or 11.3137 mm. We reconstructed the 3D geometry for each target pose, and then extracted the 3D coordinates of the 4 points (i.e. A, B, C, D). Finally, the euclidean distances $\bar{AC}$ and $\bar{BD}$ will be computed and compared with the actual value (i.e. 11.3137 mm). The results are shown in Table 1, from which we can see that the measurement results are consistently accurate for different target poses. On average, the measurement error is around 9 μm with the maximum being 16 μm. Considering the length of the measured diagonal, the percentage error is quite small (around 0.10%), which is even comparable to the manufacturing uncertainty. The major error sources could come from the error introduced by circle center extraction or the bilinear interpolation of 3D coordinates.

Table 1. Measurement result of two diagonals on calibration board (in mm).

View Table | View all tables in this article

We then put this calibration target on a precision vertical translation stage (Model: Newport M-MVN80, sensitivity: 50 nm) and translated it to different stage heights (spacing: 50 μm); We measured the 3D coordinates of the circle center point D [see Fig. 3(a)] at each stage, noted as D_i(x, y, z), where i denotes the i-th stage position. Then we computed the rigid translation t_i from D₁(x, y, z) to D_i(x, y, z). The magnitude of ‖t_i‖ are then compared with the actual stage translation. The results are shown in Table 2. The results indicate that our calibration is able to provide a quite accurate estimation of a rigid translation. On average, the error is around 1.7 μm. Considering the spacing for the stage translation (i.e. 50 μm), this error is quite small.

Table 2. Measurement result of a linearly translated calibration target point in mm.

View Table | View all tables in this article

To further examine measurement uncertainty, we measured the 3D geometry a flat plane and compare it with an ideal plane obtained through least-square fitting. Figure 5(a) shows the 2D color-coded error map, and Figure 5(b) shows one of its cross section. The root-mean-square (RMS) error for the measured plane is 4.5 μm, which is very small comparing with the random noise level (approximately 20 μm). The major sources of error could come from the roughness of this measured surface, or/and the random noise of the camera. This result indicates that our calibration method can provide a good accuracy for 3D geometry reconstruction.

Fig. 5 Experimental result of measuring a flat plane. (a) 2D error map, with an RMS error of 4.5 μm; (b) a cross section of (a).

Download Full Size | PDF

To visually demonstrate the success of our calibration method, we measured two different types of objects with complex geometry. We first measured a ball grid array [Fig. 6(a)] and then measured a flat surface with octagon grooves [Fig. 6(d)]. The reconstructed 3D geometries and the corresponding cross sections are shown in Fig. 6(b)–(c) and Fig. 6(e)–(f). From those results, it indicates that our calibration algorithm works well for different types of geometry (e.g. spheres, ramps, planes), which further confirms the success of our calibration framework. One may notice that there is a small slope in the cross section shown in Fig. 6(c). This is owing to the fact that the ball grid array sample has a little tilted bottom surface, which deviates a little bit from Z plane.

Fig. 6 Experimental result of measuring complex surface geometry. (a) Picture of a ball grid array; (b) reconstructed 3D geometry; (c) a cross section of (b); (d)–(f) corresponding figures for a flat surface with octagon grooves

Download Full Size | PDF

5. Conclusion

In this research, we have presented a novel calibration method for a unique type of microscopic SL system, which is comprised of a camera with telecentric lens and a regular pin-hole projector using LWD lens with small FOV. The proposed calibration approach is flexible since only standard flat board with circle patterns are used and the calibration targets are posed freely for the whole system calibration. The experimental results have demonstrated the success of our calibration framework by achieving very high measurement accuracy: approximately 10 μm accuracy with a calibration volume of 10(H) mm × 8(W) mm × 5(D) mm.

Acknowledgments

This study was sponsored by the National Science Foundation (NSF) under Grant No. CMMI-1300376. Any opinion, findings, and conclusions or recommendations expressed in this article are those of the authors and do not necessarily reflect the views of the NSF.

References and links

1. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Lasers Eng 48, 133–140 (2010). [CrossRef]

2. R. Windecker, M. Fleischer, and H. J. Tiziani, “Three-dimensional topometry with stereo microscopes,” Opt. Eng. 36, 3372–3377 (1997). [CrossRef]

3. C. Zhang, P. S. Huang, and F.-P. Chiang, “Microscopic phase-shifting profilometry based on digital micromirror device technology,” Appl. Opt. 41, 5896–5904 (2002). [CrossRef] [PubMed]

4. K.-P. Proll, J.-M. Nivet, K. Körner, and H. J. Tiziani, “Microscopic three-dimensional topometry with ferroelectric liquid-crystal-on-silicon displays,” Appl. Opt. 42, 1773–1778 (2003). [CrossRef] [PubMed]

5. R. Rodriguez-Vera, K. Genovese, J. Rayas, and F. Mendoza-Santoyo, “Vibration analysis at microscale by talbot fringe projection method,” Strain 45, 249–258 (2009). [CrossRef]

6. A. Li, X. Peng, Y. Yin, X. Liu, Q. Zhao, K. Körner, and W. Osten, “Fringe projection based quantitative 3d microscopy,” Optik 124, 5052–5056 (2013). [CrossRef]

7. C. Quan, X. Y. He, C. F. Wang, C. J. Tay, and H. M. Shang, “Shape measurement of small objects using lcd fringe projection with phase shifting,” Opt. Commun. 189, 21–29 (2001). [CrossRef]

8. C. Quan, C. J. Tay, X. Y. He, X. Kang, and H. M. Shang, “Microscopic surface contouring by fringe projection method,” Opt. Laser Technol. 34, 547–552 (2002). [CrossRef]

9. J. Chen, T. Guo, L. Wang, Z. Wu, X. Fu, and X. Hu, “Microscopic fringe projection system and measuring method,” in “Proc. SPIE,” (Chengdu, China, 2013), pp. 87594U. [CrossRef]

10. D. S. Mehta, M. Inam, J. Prakash, and A. Biradar, “Liquid-crystal phase-shifting lateral shearing interferometer with improved fringe contrast for 3d surface profilometry,” Appl. Opt. 52, 6119–6125 (2013). [CrossRef]

11. Y. Yin, M. Wang, B. Z. Gao, X. Liu, and X. Peng, “Fringe projection 3d microscopy with the general imaging model,” Opt. Express 23, 6846–6857 (2015). [CrossRef] [PubMed]

12. D. Li and J. Tian, “An accurate calibration method for a camera with telecentric lenses,” Opt. Lasers Eng. 51, 538–541 (2013). [CrossRef]

13. F. Zhu, W. Liu, H. Shi, and X. He, “Accurate 3d measurement system and calibration for speckle projection method,” Opt. Lasers Eng. 48, 1132–1139 (2010). [CrossRef]

14. D. Li, C. Liu, and J. Tian, “Telecentric 3d profilometry based on phase-shifting fringe projection,” Opt. Express 22, 31826–31835 (2014). [CrossRef]

15. S. Zhang and P. S. Huang, “Novel method for structured light system calibration,” Opt. Eng. 45, 083601 (2006). [CrossRef]

16. Y. Wang and S. Zhang, “Superfast multifrequency phase-shifting technique with optimal pulse width modulation,” Opt. Lett. 19, 5149–5155 (2011).

Pose No.	$\bar{AD}$	Error	$\bar{BC}$	Error

1	11.2976	0.0161	11.2981	0.0156
2	11.3049	0.0088	11.3047	0.0090
3	11.3061	0.0076	11.3033	0.0104
4	11.3030	0.0107	11.3028	0.0109
5	11.3057	0.0080	11.3068	0.0069
6	11.2972	0.0165	11.2993	0.0144
7	11.3009	0.0128	11.2986	0.0151
8	11.3049	0.0088	11.3091	0.0046
9	11.3166	0.0029	11.3005	0.0132
10	11.3057	0.0080	11.3063	0.0074

Actual	11.3137	NA	11.3137	NA
Average	11.3043	0.0094	11.3029	0.0108

Stage No.	‖t_i‖	Actual	Error

1	0	0	NA
2	0.0473	0.0500	0.0027
3	0.0997	0.1000	0.0003
4	0.1470	0.1500	0.0030
5	0.1990	0.2000	0.0010

Pose No.	$\bar{AD}$	Error	$\bar{BC}$	Error

1	11.2976	0.0161	11.2981	0.0156
2	11.3049	0.0088	11.3047	0.0090
3	11.3061	0.0076	11.3033	0.0104
4	11.3030	0.0107	11.3028	0.0109
5	11.3057	0.0080	11.3068	0.0069
6	11.2972	0.0165	11.2993	0.0144
7	11.3009	0.0128	11.2986	0.0151
8	11.3049	0.0088	11.3091	0.0046
9	11.3166	0.0029	11.3005	0.0132
10	11.3057	0.0080	11.3063	0.0074

Actual	11.3137	NA	11.3137	NA
Average	11.3043	0.0094	11.3029	0.0108

Stage No.	‖t_i‖	Actual	Error

1	0	0	NA
2	0.0473	0.0500	0.0027
3	0.0997	0.1000	0.0003
4	0.1470	0.1500	0.0030
5	0.1990	0.2000	0.0010

Flexible calibration method for microscopic structured light system using telecentric lens

Abstract

1. Introduction

2. Principle

3. Procedures

4. Experiments

5. Conclusion

Acknowledgments

References and links

Cited By

Figures (6)

Tables (2)

Equations (9)

Optics Express