## Abstract

Accurate in situ measurement of complex optical surfaces is desired for precision manufacturing, and deflectometry is a promising measuring method. However, deflectometry’s measurement accuracy depends heavily on the geometric calibration’s reliability. Existing calibration methods suffer from the limitations of low efficiency, complicated operations, and expensive equipment. Therefore, a self-calibration method is proposed for the monoscopic deflectometry in order to determine the the camera and screen positions. A flat mirror without markers, rather than high-precision metrological instruments, is employed to assist with the self-calibration work, and a re-projection model is adopted in order to trace the rays from the screen to the camera. Numerical optimization is performed on the global parameters—including the camera intrinsic parameters, the screen intrinsic parameters, and the geometric transformations—in order to improve the geometric calibration’s reliability. This self-calibration method only requires that four images be captured, hereby reducing the operating complexity and improving the measuring efficiency. Experiments demonstrate that the re-projection error can be significantly decreased, and also that the measuring accuracy of specular surfaces is comparable to high-precision interferometry.

© 2019 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

## 1. Introduction

In situ measurement of complex optical surfaces is urgently demanded in precision optical manufacturing. Integrating a measurement instrument into the optical manufacturing machine can greatly improve the manufacturing reliability and efficiency, because the repetitive positioning error of the workpiece can be avoided. Deflectometry is potential to achieve in situ measurement of specular surfaces, which measures the surface slopes (or equivalently the normals) by pattern projection and then reconstructs the surface form by numerical integration [1]. Unfortunately, deflectometry suffers from the problem of ‘height ambiguity’, i.e., the solution satisfying the correspondence between the camera and the screen is not unique, as revealed in Fig. 1(a). Consequently, an extra imaging/projecting equipment or measuring procedure is demanded to eliminate the ambiguity, e.g. by stereoscopic deflectometry [2,3] or by screen shifting [4–8] as depicted in Figs. 1(b) and 1(c). While the monoscopic deflectometry based on the software configurable optical test system (SCOTS) [9–12] resolves the problem of ‘height ambiguity’ by providing a pre-knowledge surface to assist the iterative reconstruction, and an additional equipment is not required, as shown in Fig. 1(d).

The measured surface slopes are extremely sensitive to the geometric calibration error. Currently, there are mainly two kinds of geometric calibration methods, one is by third-party metrological instruments, and the other is by numerical calculation based on the self-consistency of the ray tracing between the camera and the screen. The former [10,13] generally uses a laser tracker or a coordinate measuring machine to specify the positions of the camera, the screen, and the measured mirror. Their geometric parameters are obtained in a unified coordinate system with the assistance of certain markers. This kind of methods is of high accuracy but it is difficult to operate, and in addition, it is costly inefficient and inconvenient to be automatized. While the latter [1,5,14–18] usually optimizes the geometrical parameters globally by a re-projection model to reduce the systematic error, and this approach is called bundle adjustment. Unfortunately, it is difficult to be implemented in deflectometry, because the camera and the screen face to the measured surface simultaneously, making the camera unable to capture the screen directly. In order to resolve this problem, a flat mirror is introduced to assist the calibration work. However, a new issue arises that the mirror’s pose is ambiguous because the mirror is ‘not visible’ to the camera, thus researchers proposed to label some markers on the mirror [1,18] or locate the mirror at different poses [14,16,17]. Markers on the mirror can interrupt the projection pattern, while the poses insufficiently distinguishable can make the solution ill-posed. In addition, the field of view will be seriously limited because the projected pattern has to been captured simultaneously at all of the mirror’s poses, then the calibration reliability will be greatly reduced.

In order to improve the efficiency and reliability of geometrical calibration, a self-calibration method composed of a ray-tracing model and parameter optimization is proposed for the in situ SCOTS, as presented below.

## 2. Principle of slope measurement in SCOTS

Figure 2 schematically shows how the normal of a single point on the surface is measured. A screen, facing to the mirror, is placed on the right, and on the left a camera captures the image of the screen reflected via the measured surface. Due to the pin-hole camera model, only the principal ray through the camera optical center *C* is considered. As the incidence angle from the screen pixel *S* is equal to the reflection angle to the camera, the angular bisector of the incident and reflected rays is the normal **n** of the surface at the measured point *K*.

Accordingly, the normal vector **n** is along the sum of the unit vector of the incident ray **i** and the unit vector of the reflected ray **r**,

*x–*slope and the

*y–*slope at the measured point

*K*, denoted as ${W}_{x}$ and ${W}_{y}$, respectively, are calculated as

*x*,

*y*and

*z*coordinates of the measured point

*K*; ${x}_{s}$, ${y}_{s}$ and ${z}_{s}$ are the coordinates of the screen pixel

*S*; ${x}_{c}$, ${y}_{c}$ and ${z}_{c}$ are the coordinates of the camera optical center

*C*; ${d}_{k2s}$ is the distance from the screen pixel

*S*to the measured point

*K*, and ${d}_{k2c}$is the distance from the camera optical center

*C*to the measured point

*K*, respectively. All the above geometric positions need to be determined accurately except the coordinates of the measured point

*K*, which are obtained by the intersection between the pre-knowledge surface and the reflected ray.

The key to SCOTS is to iteratively calculate the surface form *W* and its slopes ${W}_{x}$ and ${W}_{y}$ using Eq. (2), until the slopes match well to the surface form.

## 3. Self-calibration for SCOTS

It is worth mentioning that the deflectometric measuring system is designed for the in situ measurement of the single point diamond turning (SPDT). As a result, the world coordinate system (WCS) in measurement should be in accordance with the SPDT machine, and a rotary stage is employed to simulate the spindle of the machine. A flat mirror without markers is introduced as a reference mirror to assist the self-calibration rather than using third-party metrological instruments, and a re-projection model is applied to trace the rays from the screen to the camera. The calibration procedure is composed of three parts, calibrating the position of the rotary stage in WCS, tracing the rays by re-projection and optimizing the geometrical calibration result. In this paper capital letters **R** and **T** are used to denote the rotation matrix and the translation vector, respectively. The subscript refers to coordinate transformation between the source and the target coordinate systems, e.g. ${R}_{w2c}$ means the rotation matrix from WCS to the camera coordinate system (CCS). The coordinate systems in use are listed in Table 1. The inverse transformation of $\left[{R}_{w2c},{T}_{w2c}\right]$ satisfies a relationship $\{\begin{array}{l}{R}_{c2w}={R}_{w2c}^{-1}\\ {T}_{c2w}=-{R}_{w2c}^{-1}{T}_{w2c}\end{array}$. The rest coordinate systems have a similar relationship.

Figure 3 schematically shows the in situ measurement system to be calibrated. Here the ${X}_{w}{O}_{w}{Y}_{w}$–plane of WCS is defined at the rotary stage with the origin *O _{w}* defined at the rotational center. A flat mirror without markers is introduced as a reference mirror and placed on the rotary stage. The mirror’s upper surface is defined as the ${X}_{m}{O}_{m}{Y}_{m}$–plane of MCS, but the origin

*O*is identical to

_{m}*O*. It is difficult to guarantee the parallelism between the ${X}_{w}{O}_{w}{Y}_{w}$–plane and the ${X}_{m}{O}_{m}{Y}_{m}$–plane, as a result only a rotation matrix ${R}_{m2w}$ is used to describe their geometric transformation. Subsequently, the calibration needs to link CCS, MCS and SCS to WCS, and to solve the geometric transformations between them.

_{w}#### 3.1 Calibrating rotary stage in WCS

The calibration of WCS can be thought as two issues, one is to identify the axes directions of WCS and the other is to determine the origin *O _{w}*. A calibration board placed on the rotate stage is used to assist calibrating WCS. The stage is rotated by an arbitrary angle, and BCS before rotation is marked as

*b*, while BCS after rotation is marked as

*b’*, as depicted in Fig. 4(a). The rotational axis from

*b*to

*b*’ is along the

*z*–axis of WCS, consequently the origin

*O*and the

_{w}*z*–axis of WCS can be worked out accordingly, as shown in Fig. 4(b).

The camera, calibrated using the Zhang’s method [19], captures these two positions to obtain their extrinsic parameters $\left[{R}_{b2c},{T}_{b2c}\right]$ and $\left[{R}_{b\text{'}2c},{T}_{b\text{'}2c}\right]$. Supposing that WCS has the same axis directions with *b* before rotation, we have ${R}_{w2c}={R}_{b2c}$. Besides, the origin ${O}_{w}$ is the intersection of two perpendicular bisectors of $\overrightarrow{{P}_{b}{P}_{b\text{'}}}$ and $\overrightarrow{{Q}_{b}{Q}_{b\text{'}}}$, where *P*, *Q* are two points selected on the calibration board. As a result, the translation vector ${T}_{b2c}$ is determined by the extrinsic parameters $\left[{R}_{b2c},{T}_{b2c}\right]$ and $\left[{R}_{b\text{'}2c},{T}_{b\text{'}2c}\right]$.

#### 3.2 The re-projection model for ray tracing

Once WCS is determined, ray tracing with a re-projection model will be performed from the screen to the camera. The geometric transformation $\left[{R}_{s2w},{T}_{s2w}\right]$ from SCS to WCS is worked out using the extrinsic parameters $\left[{R}_{s2c},{T}_{s2c}\right]$ and the geometric transformation $\left[{R}_{c2w},{T}_{c2w}\right]$,

However, the camera and the screen face to the measured surface simultaneously and the camera is not able to capture the screen directly, but instead capturing the virtual image of the screen reflected via a reference mirror, thus the screen extrinsic parameters $\left[{R}_{s2c},{T}_{s2c}\right]$ need to be calibrated using the transformation concerning the virtual image $\left[{R}_{v2c},{T}_{v2c}\right]$. Henceforth a re-projection model is developed for the ray tracing. In addition, other related variables including the screen intrinsic parameters and the camera intrinsic parameters are taken into account in the model as well to improve the calibration accuracy.

As illustrated in Fig. 5, the virtual image pixel *V* and the screen pixel *S* are symmetrical with respect to the reference mirror. *h* indicates the distance from the origin *O _{m}* to the mirror along the

*Z*–direction. The stage is rotated by an angle

_{m}*ε*, then the orientation matrices of the mirror before rotation (called pose 0) and after rotation (called pose 1) are denoted with ${R}_{w2m0}$ and ${R}_{w2m1}$, respectively, and their relative difference is a rotation around the

*Z*

_{w}–axis,

As revealed in Fig. 5(a), a single pixel ray tracing from *S* to *V* and finally to *C* is taken as an example to describe the ray-tracing procedure. For the sake of consistency, all the coordinates are uniformly described in MCS because the light reflection is relatively straightforward to be expressed in this coordinate system.

**STEP 1.** Expressing the screen pixel *S* in MCS.

For a screen pixel *S*, its original pixel coordinate is ${\left[\begin{array}{cc}{u}_{s}& {v}_{s}\end{array}\right]}^{\text{T}}$. The transformations to SCS, WCS and MCS are

*z*coordinate of the screen pixel

*z*is set zero because the screen is located on the ${X}_{s}{O}_{s}{Y}_{s}$–plane. In the matrix,

_{s}*m*and

*n*are the pixel width and height (in mm), while the origin

*O*corresponds to the screen pixel ${\left[\begin{array}{cc}{d}_{x}& {d}_{y}\end{array}\right]}^{\text{T}}$. The detailed definitions of the screen intrinsic parameters are shown in Fig. 6(a).

_{s}In Eq. (5), the geometric transformation$\left[{R}_{s2w},{T}_{s2w}\right]$ is obtained by Eq. (3). As shown in Fig. 6(b), the geometric transformation $\left[{R}_{s2c},{T}_{s2c}\right]$ is expressed using the virtual image 1 associated with pose 1, whose extrinsic parameters $\left[{R}_{v2c1},{T}_{v2c1}\right]$ are obtained by

*d*is the

*z*coordinate of origin

*O*expressed in MCS at pose 1 $d=\u3008{e}^{\text{T}},{R}_{w2m1}{T}_{c2w}\u3009-h$, with $\u3008\cdot ,\cdot \u3009$ denoting the inner product operator. ${I}_{3}$ is a 3 × 3 identity matrix, and vector $e={\left[\begin{array}{ccc}0& 0& 1\end{array}\right]}^{\text{T}}$.

_{c}**STEP 2.** Expressing the virtual image pixel *V* in MCS.

The coordinates of the pixels *S* and *V* are described in WCS and MCS simultaneously, but their geometric relationship can be expressed from the perspective of MCS,

**STEP 3.** Re-projection to the camera pixel *C.*

The virtual image pixel *V* can also be expressed in WCS and CCS,

According to the imaging principle of the pin-hole camera, a point in CCS corresponds to a pixel on the camera sensor,

This model yields the camera pixel coordinates ${\left[\begin{array}{cc}{u}_{c}& {v}_{c}\end{array}\right]}^{\text{T}}$ associated with each screen pixel by ray-tracing at pose 1 and then converted into pose 0 using a geometric constraint that the relative difference between these two poses is a rotation about the *Z*_{w} axis, while the actual camera pixel coordinates are obtained from the images captured at pose 0. The relative deviation between them reflects the reliability of the geometric calibration, i.e. the re-projection error reaches minimum when the reference mirror’s pose is specified correctly.

#### 3.3 Iterative optimization of system parameters

Numerical optimization can be conducted to adjust the geometric parameters, so that the traced pixel coordinates are in accordance with the actual ones. There are totally *l* screen-to-camera pixel pairs to be traced, and a least-squares cost function is constructed as

*i*-th camera pixel at pose 0, and ${\left[\begin{array}{cc}{u}_{c}& {v}_{c}\end{array}\right]}_{i}{}^{\text{T}}$are the re-projected camera coordinates at pose 0, which depend on the variable

**x**and the input screen pixel ${\left[\begin{array}{cc}{u}_{s}& {v}_{s}\end{array}\right]}_{i}{}^{\text{T}}$.

Equation (10) is a non-linear least-square problem, and the Levenberg-Marquardt algorithm is applied to solve it iteratively. Appropriate setting of the initial solution is critical to the convergence rate and computational accuracy. Here the initial values of the variables $\left[{R}_{w2c},{T}_{w2c}\right]$, $\left[{R}_{v2c0},{T}_{v2c0}\right]$ and $\left[{R}_{v2c1},{T}_{v2c1}\right]$ are obtained from the calibrated camera. In addition, the initial $in{s}_{c}$ is obtained using a method presented in the camera calibration toolbox, and the initial $in{s}_{s}$is supplied by the manufacturer of the projecting screen. The rotation matrix ${R}_{w2m0}$ denotes the pose parameter of the reference mirror, and it is initially estimated as an identity matrix because MCS has a slight difference with WCS. The angle of rotation *ε* is obtained from the reading of the rotary stage and the thickness of the mirror *h* is measured by vernier calipers.

In order to improve the numerical stability of the multi-dimensional optimization, the variable **x** is separated into three groups,

*Groups A*,

*B*and

*C*are the screen intrinsic parameters, the camera intrinsic parameters, and the geometrical parameters of the reference mirror, respectively. The variables in

*Group C*are roughly estimated or simply measured, thus they are less reliable compared to those in

*Groups A*and

*B*. As a result special attention needs to be paid on the optimization of ${R}_{w2m0},\epsilon $ and

*h*. The alternative direction method (ADM) is used to conduct optimization successively for

*Groups A*,

*B*and

*C*, until the convergence is achieved. The complete self-calibration procedure is shown in Fig. 7.

## 4. Experimental verification

In order to verify the proposed method, a deflectometric measurement system is established, as shown in Figs. 8(a)-(c). A calibrated JAI camera SP-2000C-PMCL with a 50 mm lens and of resolution 5120 × 3840 is used. An iPad mini2 of resolution 2048 × 1536 is adopted as a projecting screen. A reference mirror with 130 mm diameter and flatness error less than *λ*/10 is placed on the rotary stage, making the camera able to capture the whole screen’s projection pattern.

Owing to the minor non-parallelism between the upper surface of the rotary stage and the reflecting surface of the mirror, the pictures captured before and after rotation have a slight difference, as illustrated in Fig. 9, where the magenta dots denote pose 0 and the cyan dots denote pose 1.

Then optimization is employed after initial calibration, and the root mean square (RMS) value of re-projection errors is employed as a cost function, which is decreased progressively until approaching a stable value. Different initial guesses of ${R}_{w2m0},\epsilon $ and *h* are tested and it is found that consistent convergence results can always be achieved. The convergence domain of the optimization solver is large enough to cover the errors of the initial parameters. The optimized results are shown in Figs. 10(a)-(c) and the detailed optimization results are listed in Table 2. The average projection error is reduced from (1.5154, 6.8170) down to (1.7274e-04, 0.0010), and the standard deviation of the re-projection errors is reduced from 0.2455 pixel down to 0.1200 pixel. It shows that not only the systematic error caused by the false geometric parameters can be eliminated, but also the consistency between these projection points is improved. The RMS value of re-projection errors is reduced down to 0.1199 pixel, which greatly improves the final calibration accuracy. Then the optimized positions of the camera, the rotary stage, and the screen are used in the following deflectometric measurement.

The sensitivity of optimized parameters is also analyzed by comparing the impact of the variable groups on the optimization results. We conduct optimization in three cases, solely *Group C*, combining *Group A* and *C*, and combining *Group B* and *C*, respectively. As shown in Figs. 11(a)-(c), the variables of *Group C* are the most sensitive in the three groups, and the detailed optimization results are listed in Table 3. It means that the pose of the reference mirror is the crucial factor for the overall optimization, and the geometric parameters of the camera and screen are relative less significant to improve the calibration accuracy.

Using the self-calibration result, a spherical concave surface with an aperture 63.85 mm is measured, as illustrated in Figs. 12(a) and 12(b). Figures 13(a) and 13(c) show the measurement results which are reconstructed by the modal reconstruction method [20] before and after applying the proposed calibration, respectively. In order to verify the results, the workpiece is also measured by a 4D Dynamic Interferometer PhaseCam 6000. The relative difference between the measured results of deflectometry and interferometry are shown in Fig. 13(b) and 13(d), and the detailed specifications of the measurement deviations are listed in Table 4. The experimental results demonstrate that this self-calibration method can significantly improve the accuracy of the deflectometric measurement by adjusting the positions of the camera and the screen by re-projection and numerical optimization.

Existing calibration methods normally use multiple independent poses of the reference mirror to eliminate the ambiguity of the reference mirror’s pose to identify the positions of the screen and the camera. While in this self-calibration method, only two poses of the reference mirror are needed, with the assistance of the geometric constraints on these two poses, thus only four images are required. Two images and one pose are used to calibrate WCS and the rest are used to optimize the reference mirror’s pose, thereby greatly improving the measurement efficiency.

## 5. Conclusions

This paper develops an accurate self-calibration method for monoscopic deflectometry used in the in situ measurement, and a re-projection model is proposed to improve the calibration accuracy of the geometrical positions of the screen and camera. The rotary spindle of the SPDT machine is fully utilized, which is combined with the reference mirror to simplify the calibration work, and only four images and two poses of the reference mirror are required to obtain the geometric parameters. In the numerical optimization, the alternative direction method is applied to improve the numerical stability, making the calibration result more reliable. The experimental results demonstrate the measuring accuracy of specular surfaces is comparable to high-precision interferometry, thus it is of significance for the in situ measurement in high-precision optical manufacturing.

## Funding

National Natural Science Foundation of China (51875107); Science Challenging Program (JCKY2016212A506-0106); National Key Research and Development Program of China (2017YFB1104700); Fudan University-CIOMP Joint Fund and European Horizon 2020 EMPIR project (15SIB01 FreeFORM).

## References

**1. **M. C. Knauer, J. Kaminski, and G. Hausler, “Phase Measuring Deflectometry: a new approach to measure specular free-form surfaces,” Proc. SPIE **5457**, 366–377 (2004). [CrossRef]

**2. **Y. Xu, F. Gao, and X. Jiang, “Enhancement of measurement accuracy of optical stereo deflectometry based on imaging model analysis,” Opt. Lasers Eng. **111**, 1–7 (2018). [CrossRef]

**3. **L. Huang, J. Xue, B. Gao, and M. Idir, “Three-dimensional shape measurement with modal phase measuring deflectometry,” Proc. SPIE **10449**, 1044909 (2017). [CrossRef]

**4. **Z. Niu, N. Gao, Z. Zhang, F. Gao, and X. Jiang, “3D shape measurement of discontinuous specular objects based on advanced PMD with bi-telecentric lens,” Opt. Express **26**(2), 1615–1632 (2018). [CrossRef] [PubMed]

**5. **Y. L. Xiao, X. Su, W. Chen, and Y. Liu, “Three-dimensional shape measurement of aspheric mirrors with fringe reflection photogrammetry,” Appl. Opt. **51**(4), 457–464 (2012). [CrossRef] [PubMed]

**6. **H. Guo, P. Feng, and T. Tao, “Specular surface measurement by using least squares light tracking technique,” Opt. Lasers Eng. **48**(2), 166–171 (2010). [CrossRef]

**7. **W. Zhao, “Testing an aspheric mirror based on phase measuring deflectometry,” Opt. Eng. **48**(10), 103603 (2009). [CrossRef]

**8. **Y. Liu, S. Huang, Z. Zhang, N. Gao, F. Gao, and X. Jiang, “Full-field 3D shape measurement of discontinuous specular objects by direct phase measuring deflectometry,” Sci. Rep. **7**(1), 10293 (2017). [CrossRef] [PubMed]

**9. **P. Su, M. A. H. Khreishi, T. Su, R. Huang, M. Z. Dominguez, A. Maldonado, G. Butel, Y. Wang, R. E. Parks, and J. H. Burge, “Aspheric and freeform surfaces metrology with software configurable optical test system: a computerized reverse Hartmann test,” Opt. Eng. **53**(3), 031305 (2013). [CrossRef]

**10. **P. Su, Y. Wang, J. H. Burge, K. Kaznatcheev, and M. Idir, “Non-null full field X-ray mirror metrology using SCOTS: a reflection deflectometry approach,” Opt. Express **20**(11), 12393–12406 (2012). [CrossRef] [PubMed]

**11. **J. H. Burge, P. Su, G. Butel, R. Huang, A. Maldonado, and T. Su, “Measuring large mirrors using SCOTS: the Software Configurable Optical Test System,” Proc. SPIE **9151**, 91510Z (2014). [CrossRef]

**12. **L. Huang, C. S. Ng, and A. K. Asundi, “Dynamic three-dimensional sensing for specular surface with monoscopic fringe reflectometry,” Opt. Express **19**(13), 12809–12814 (2011). [CrossRef] [PubMed]

**13. **R. Huang, P. Su, J. H. Burge, L. Huang, and M. Idir, “High-accuracy aspheric x-ray mirror metrology using Software Configurable Optical Test System/deflectometry,” Opt. Eng. **54**(8), 084103 (2015). [CrossRef]

**14. **H. Ren, F. Gao, and X. Jiang, “Iterative optimization calibration method for stereo deflectometry,” Opt. Express **23**(17), 22060–22068 (2015). [CrossRef] [PubMed]

**15. **T. Yuan, F. Zhang, X. Tao, X. Zhang, and R. Zhou, “Flexible geometrical calibration for fringe-reflection optical three-dimensional shape measurement,” Appl. Opt. **54**(31), 9102–9107 (2015). [CrossRef] [PubMed]

**16. **Y. L. Xiao, X. Su, and Z. You, “Pose transfer geometrical calibration for fringe-reflection optical three-dimensional measurement,” Opt. Commun. **305**, 143–146 (2013). [CrossRef]

**17. **Y. L. Xiao, X. Su, and W. Chen, “Flexible geometrical calibration for fringe-reflection 3D measurement,” Opt. Lett. **37**(4), 620–622 (2012). [CrossRef] [PubMed]

**18. **L. Huang, J. Xue, B. Gao, C. McPherson, J. Beverage, and M. Idir, “Modal phase measuring deflectometry,” Opt. Express **24**(21), 24649–24664 (2016). [CrossRef] [PubMed]

**19. **Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. **22**(11), 1330–1334 (2000). [CrossRef]

**20. **I. Mochi and K. A. Goldberg, “Modal wavefront reconstruction from its gradient,” Appl. Opt. **54**(12), 3780–3785 (2015). [CrossRef]