1. Introduction

Phase-measuring profilometry (PMP) is a powerful tool that allows the reconstruction of the high-resolution, whole-field three-dimensional (3D) shape of objects in a non-contact manner at video frame rates [1]. To record the 3D information of an object with PMP, fixed-pitch fringe patterns (called carrier fringes) are firstly projected onto the tested surface, following which the carrier fringes are modulated by the object surfaces to yield deformed fringe images. With some fringe analysis methods [2–5], a two-dimensional (2D) absolute phase map is extracted from the deformed fringe images. Finally, the 3D shape can be reconstructed from the retrieved phase once the PMP system is calibrated.

Since the surface coordinates are obtained from the phase, the phase quality is an important factor determining the measurement accuracy: any noise or distortion in the retrieved phase will be reflected in the final 3D measurement result. In practice, the distortions of lenses of the projector and camera are unavoidable owing to manufacturing and assembly errors. In this scenario, projector distortion might create variable period carrier fringes on a plane perpendicular to the optical axis of the projector lens. In comparison with fixed-pitch fringe projection, the measurement of an object surface with variable period fringe patterns will introduce an additional bending carrier phase into the retrieved phase, which in turn leads to measurement error. On the other hand, camera distortion would cause the mapping relationship between the 3D points and 2D image points to be nonlinear. Thus, we cannot directly use the pinhole model to calculate the transverse coordinates of the tested surface. These facts indicate that an accurate 3D shape cannot be reconstructed with PMP unless all the lens distortions are completely corrected.

One promising approach to address these problems is to (1) calibrate the projector and camera and (2) precode the projected fringe pattern and undistort the captured image according to the calibration results [6]. The influence of the camera distortion can be effectively eliminated once the captured images are undistorted according to the camera calibration results. In the past decades, camera calibration has been extensively studied in the fields of computer vision and photogrammetry [7, 8], and an accurate calibration result can be achieved through the bundle adjustment (BA) strategy even with an imperfect and not-measured calibration target [9]. However, accurate projector calibration remains difficult [10]. This is mainly because, unlike a camera, a projector cannot directly capture images. To overcome this limitation, Zhang et al [11] and Li [12] et al presented phase-aided methods to enable a projector to capture images indirectly. In these methods, two sets of orthogonal sinusoidal fringe patterns are sequentially projected onto a calibrated target (such as a 2D planar board), following which the correspondence relationship between the camera pixels and projector pixels can be established with the extracted phase pair. By using this relationship, feature points on the calibrated plate can be imaged on the projection plane. Accordingly, a projector can be calibrated with a pinhole model based on the 2D image points and 3D feature points.

Nevertheless, several factors affect the calibrated result, such as the symmetry of the calibration plate with respect to the optical axis, the extraction accuracy of feature points from the calibrated plate [13], and the number of view orientations required to capture an image. To ensure that the calibrated plate is symmetrically placed along the axes of the projector and camera during calibration, a coaxial structure of a projector and camera was presented in [14]. The challenge lies in aligning the optical axes of the projector and camera, which requires much skill and patience. In addition, to calibrate all the projector’s parameters, it needs to capture the calibrated target in at least two different view orientations. At each view orientation, two sets of orthogonal phase-shifting sinusoidal fringe patterns are required for determining the imaging points on the projection plane [15]. On the other hand, the uncertainty of the calibration parameters decreases with the number of view orientations [7]. Hence, it would prefer to capture the calibrated plate in multiple (usually large than three) different view orientations, and it takes a long time to calibrate a projector. Moreover, existing projector calibration methods always assume that the re-projection errors on the projection plane are normally distributed, and they utilize the maximum likelihood estimation (MLE) to calculate the projector’s parameters. Nonetheless, it is impossible to obtain the statistical characteristics of re-projection errors in the projection image because the relation between the projector image point and the camera measurement point is a nonlinear function [16]. Therefore, the use of the MLE to calibrate a projector always leads to biased parameter estimates and lower precision results. In the light of the above analysis, we know that the projector distortion may decrease the measurement accuracy, and it is still challenging and time consuming to calibrate a projector with high accuracy.

To circumvent the problems mentioned above, this paper introduces the concept of adaptive fringe projection for suppressing the projector distortion in PMP. In contrast to the existing techniques, the projector is not calibrated in this method. Instead, the proposed method modifies the carrier phase in the projected fringe patterns according to the projector distortion based on the bending carrier phase. Consequently, unevenly spaced fringe patterns (also called adaptive fringe patterns) are generated on the projection plane. When projecting these adaptive fringe patterns, the bending carrier phase is offset; thus, evenly spaced fringes are restored on a plane perpendicular to the optical axis of projector lens. Several experiments are conducted to validate the proposed method. Since the bending carrier phase is obtained by measuring a standard plate placed at any one position within the measurement volume, the proposed method is more convenient and rapid.

It should also be emphasized here that we are not the first to apply the concept of non-uniform fringe projection. In [17,18], Zhang et al used the uneven fringe projection technique to simplify the relationship between phase and depth. Nevertheless, in [17], the angle between the optical axes of projector and camera must be exactly determined in advance. In addition, the method relies on a particular measurement setup; for example, the optical axis of the camera lens should be perpendicular to the reference plane, and the line joining the optical centers of the projector and camera must be parallel to the reference plane. Furthermore, in order to eliminate the influence of the projector distortion, the projector should be calibrated before generating the non-uniform fringe patterns. In this work, however, not only the component locations but also the projection and capture directions can be arbitrarily arranged. More importantly, the adaptive fringe patterns are generated without calibrating the projector, thus it can avoid the calibration error of projector and enable us to improve the measurement precision.

2. Principle and implementation

2.1. Bending carrier phase induced by projector distortion

Knowledge of the bending carrier phase is the prerequisite for generating the adaptive fringe patterns. To prevent confusion, we define the carrier phase component induced by the crossed-optical-axes geometry of the PMP system as the principal carrier phase. Generally, the bending carrier phase and principal carrier phase are mixed together. To determine the bending carrier phase, the principal carrier phase must be characterized in advance.

Since PMP is a triangle-based technique, the optical axes of the camera and projector often lie in the same plane and intersect at a point near the center of the tested object. Under this setting, a variable period fringe is created across a plane perpendicular to the optical axis of the camera lens. Thereby, the carrier phase contains nonlinearity in its profile owing to the restrictions of system schemes. When the projector and CCD camera are set arbitrarily, the distribution of the principal carrier phase becomes more complex. Even so, the principal carrier phase for an arbitrary PMP system, according to the geometric analysis in [19], can be represented as

Recall that Eq. (1) was derived without considering the lenses distortions of the projector and camera. For a real PMP system, lenses distortions are unavoidable owing to the manufacturing and assembly errors. Thus, the measured phase of a standard plane mainly consists of three parts: the principal carrier phase, the additional bending carrier phase induced by the projector distortion, and the phase error induced by the camera distortion. To eliminate the last component, we need to undistort the captured pattern images before retrieving the phase. Subsequently, the measured carrier phase Φ_m(i, j) can be extracted from the undistorted pattern images. For a commercial projector, it is not easy to observe with naked eye that the projected fringe pattern is deformed by the projector distortion; thus, the bending carrier phase is much lower than the principal carrier phase. Consequently, Φ_c (i, j) can be acquired by directly fitting the measured carrier phase with Eq. (1). Finally, the bending carrier phase ΔΦ_a(i, j) induced by the projector distortion can be obtained by subtracting the principal carrier phase from the measured carrier phase as follows:

If the projector distortion is suppressed, the bending carrier phase calculated from Eq. (2) is close to zero. From the opposite perspective, if we can reduce the bending carrier phase by as much as possible, the influence of the projector distortion can be eliminated. Here, we define a fictitious plane G that is perpendicular to the optical axis of the projector lens. When the fringes projected on G are kept parallel to each other and are equally spaced with a fixed pitch, the bending carrier phase induced by the projector distortion will disappear. This paper introduces the idea of adaptive fringe projection to realize this objective. In the following subsections, we shall briefly review the principle of the adaptive fringe projection technique.

2.2. Principle of the adaptive fringe projection technique

In conventional PMP, fringe patterns with a constant period (e.g., sinusoidal fringe patterns) are projected onto an object surface to record the deformed fringe images for 3D reconstruction. In contrast, the ideal of adaptive fringe projection technique [20,21] is to invert the entire process, as shown in Fig. 1. We generate non-uniform fringe patterns on the projection plane, which are adapted to the object surface [22]. After projecting these patterns onto the object surface as shown in Fig. 1(a), the phase related to the object surface is offset. Consequently, we obtain evenly spaced fringe patterns on the imaging plane. On separating the carrier phase from the retrieved phase, we obtain a uniform phase map, as shown in Fig. 1(b). If the object surface has some deformations, the captured fringe will be distorted in that area. Therefore, it is easy to locate and evaluate the deformation [23]. In addition, it has been demonstrated that the adaptive fringe projection technique has the potential to increase the dynamic measurement range of PMP [24], avoid the image saturation [25], and handle varying surface reflectivity [26]. Here, we explore the adaptive fringe projection technique to offset the bending carrier phase induced by the projector distortion.

 

Fig. 1 Schematic of the adaptive fringe projection technique.

Download Full Size | PDF

With the concept of adaptive fringe pattern projection, it is possible to generate unevenly spaced fringe patterns on the projection plane that are adapted to the projector distortion. After projecting these adaptive fringe patterns, evenly spaced fringe patterns are restored on plane G. In other words, the bending carrier phase is offset. Therefore, the main problem involved is to generate the adaptive fringe patterns based on the bending carrier phase.

2.3. Algorithm for generating the adaptive fringe patterns

The key issue for generating the adaptive fringe pattern is to establish the correspondence between the input plane and output plane. For the adaptive fringe projection, the projection plane is set as the input plane, and the output plane is usually defined as the plane on which we wish to project a fringe pattern with the appropriate period. Since we wish to create an evenly spaced fringe pattern on G, plane G should be set as the output plane here. Therefore, we need to establish the mapping relation between the projection plane and pane G in advance. With the correspondence relationship, the adaptive fringe pattern can be generated from the phase distribution of the desired fringe pattern defined on G. The flowchart for generating the adaptive fringe pattern is shown in Fig. 2. Its detailed explanation is given below.

 

Fig. 2 Flowchart for generating the adaptive fringe patterns.

Download Full Size | PDF

Step 1. Acquisition of two absolute phase maps of a standard plate, Φ_h and Φ_v.

With PMP, the absolute phase map can be recovered from phase-shifting patterns. Recall that the kth frame of these fringe patterns is usually represented by

By following the procedure mentioned above, the two absolute phase maps of a standard plate, Φ_h (i, j) and Φ_v (i, j), can be obtained by projecting two sets of horizontal and vertical fixed-pitch fringe patterns onto the standard plate, respectively. With the two absolute phase maps, each pixel point (i, j) on the imaging plane can be associated with a phase value pair, Φ_h (i, j) and Φ_v (i, j). In addition, we can locate a unique point (s₁, t₁) on the projection plane that has the same phase value pair. Thus, we can establish a point-to-point mapping between the imaging plane and projection plane. With the two absolute phase values, the pixel coordinates of point (s₁, t₁) can be calculated as follows:

 

Fig. 3 Schematic diagram of the mapping results (the projector pixels are drawn in filled circles). (a) Mapping the camera pixels onto the projection plane (the correspondence points of the camera pixels are drawn with unfilled circles), (b) mapping G onto the projection plane (the correspondence points of plane G are drawn with filled stars).

Download Full Size | PDF

Step 2. Calculate the additional bending carrier phase.

Based on the two absolute phase maps, Φ_h (i, j) and Φ_v (i, j), the principal carrier phase for the PMP system can be acquired by fitting the measured phase data with Eq. (1). Once we obtain the two fitting results, the additional bending carrier phases, ΔΦ_a,h (i, j) and ΔΦ_a,v (i, j), can be computed using Eq. (2). Similar to Step 1, each pixel point (i, j) on the imaging plane can be associated with a bending carrier phase pair, ΔΦ_a,h (i, j) and ΔΦ_a,v (i, j); with its correspondence points (s₁, t₁) on the projection plane, we can build two sets of 3D points with coordinates (s₁, t₁, ΔΦ_a,h (i, j)) and (s₁, t₁, ΔΦ_a,v (i, j)), respectively. Because Zernike polynomials are widely used to represent the aberrations or figure error owing to its orthogonality, the two sets of 3D points were fitted with Zernike polynomials. After obtaining the fitting coefficients, the bending carrier phase values of the projector pixels, $\mathrm{\Delta }{\mathrm{\Phi }}_{a,h}^p(s,t)$ and $\mathrm{\Delta }{\mathrm{\Phi }}_{a,v}^p(s,t)$, can be calculated by substituting the pixel coordinate (s, t) into the Zernike polynomials.

Step 3. Establish the correspondence between the projection plane and plane G.

To build the mapping relationship between the projection plane and G, we need to acquire the absolute phase of G. Since plane G is perpendicular to the optical axis of the projector lens, if the projection lens is free of distortion, the fringes projected on G are parallel to each other and equally spaced with a fixed pitch. Under this condition, the absolute phase on G is the same as that on the projection plane. However, in a real PMP system, lens distortions are unavoidable. Thus, the absolute phase of G is the sum of the carrier phase defined on the projection plane and the bending carrier phase induced by the projector distortion. Based on the above analysis, each point on G can be mapped onto the projection plane as follows:

Step 4. Construct the phase distribution for the adaptive fringe pattern.

According to the principle of adaptive fringe projection, we need to define a desired fringe pattern on the output plane in advance and then construct the phase map of the adaptive fringe pattern based on the desired fringe pattern. To offset the bending carrier phase, the desired fringes defined on G should be kept parallel to each other and equally spaced with a fixed pitch. With the correspondence relation obtained in Step 3, the phase values of points (s₂, t₂) are mapped from the phase distribution of the desired fringe pattern defined on G. Once the phase values of points (s₂, t₂) are known, the phase distribution of the adaptive fringe pattern, ${\mathrm{\Phi }}_p^A(s,t)$, can be acquired by using the scattered points interpolation method [22].

Step 5. Generate the adaptive fringe patterns.

An adaptive fringe pattern is generated from its phase distribution, ${\mathrm{\Phi }}_p^A$, by applying a sinusoidal modulation,

2.4. Depth calibration when projecting adaptive fringe patterns

To reconstruct the depth map of the tested object from retrieved phase, we should obtain the relationship between the phase and depth in advance. In the literature, the phase-depth relationship has been extensively established based on the system arrangement. When the projector and camera are arbitrarily arranged, the mapping relationship between the phase and depth becomes more complex. Even so, the mapping relationship for each pixel, according to the geometry analysis in [29], can be expressed as follows:

Note that Eq. (8) was derived without considering the distortions of lenses of the projector and camera. When the projector distortion is suppressed using the proposed method and the camera distortion is corrected using the calibrated result, the depth map can be accurately reconstructed from the retrieved phase by using Eq. (8).

3. Experimental results and discussion

Now, we build an experimental system, as shown in Fig. 4, to demonstrate the proposed method. This system mainly consists of a digital-light-process (DLP) projector (DELL, M110, 1280 × 800 pixels) to project the fringe patterns, a black and white (B/W) CMOS camera (DH, MER-130-30UM, 1280 × 1024 pixels) to capture the distorted pattern images, and a computer for data processing. The focus length of the imaging lens (PENTAX, C1614-M) is 16mm. To retrieve the absolute phase map accurately, three groups of phase-shifting sinusoidal fringe patterns with pitches of 13, 14, and 15 pixels were sequentially projected onto the tested surface.

 

Fig. 4 Schematic of experimental setup.

Download Full Size | PDF

We began the experiment by generating the adaptive fringe patterns from the bending carrier phase. To obtain the bending carrier phase induced by the projector distortion, a ceramic plate (400mm × 300mm) with a flatness less than 0.01mm was randomly placed in the measurement volume. Two sets of horizontal and vertical sinusoidal fringe patterns were firstly projected onto the ceramic plate, following which the deformed fringe pattern images were sequentially captured. To remove the phase error caused by the camera distortion, Zhang’s camera calibration procedure was implemented in advance [7]. Subsequently, the captured images were undistorted according to the calibrated results. Later, the two absolute phase maps were extracted from the undistorted pattern images. To separate the principal carrier phase from the overall retrieved phase, we fitted the absolute phase data by using Eq. (1). Subtracting the fitting results from the measured phase data yields the additional bending carrier phase, the distributions of which are illustrated in Figs. 5(a) – 5(b). Note that the field of view (FOV) of the camera is less than that of the projector in our system. In addition, the projector and camera are arbitrarily located. Hence, the distribution of the bending carrier phases shown in Figs. 5(a) – 5(b) are asymmetric. For comparison, the captured images were not undistorted before extracting phase, and the fringe analysis method mentioned above was used again to remove the principal carrier phase. The residual phase maps are shown in Figs. 5(c) – 5(d), which shows that the distortions of lenses of the projector and camera introduce phase errors in the carrier phase. Although the phase error induced by the camera distortion will be eliminated once the captured images are undistorted, the bending carrier phase induced by the projector distortion remains, and it couples with the phase related to the object surface. To achieve highly accurate results with PMP, the projector distortion should be corrected before converting the retrieved phase to depth. By using the proposed method introduced in Section 2.3, the adaptive fringe pattern can be generated from the bending carrier phase. After projecting the adaptive fringe patterns, the bending carrier phase caused by the projector distortion can be effectively eliminated in the PMP. Several experiments are demonstrated in the next subsections to verify the proposed method.

 

Fig. 5 Residual phase error after removing the principal carrier phase. (a–b) The captured pattern images that were undistorted based on the camera calibration results; (c–d) the captured pattern images that were not undistorted.

Download Full Size | PDF

3.1. Measurement of a standard plate

To evaluate the performance of the proposed method, adaptive fringe patterns were projected onto the same ceramic plate. Similar fringe analysis methods were used again to obtain the absolute phase map. After removing the principal carrier phase from the retrieved phase by using Eqs. (1) – (2), the residual phase map is given in Fig. 6(a). In comparison with Fig. 5(a), it can be seen that the additional bending carrier phase was completely eliminated by using the proposed method. Next, we converted the phase to depth by using Eq. (8), and the 3D shape of the ceramic plate is reconstructed as shown in Fig. 6(b). In this case, a fitting process was performed to obtain the discrepancy associated with each pixel as the difference between the fitting result and reconstruction result. Figure 6(c) shows the discrepancy of the measured ceramic plate. For comparison, the ceramic plate was also measured by projecting the fixed-pitch sinusoidal fringe patterns. The discrepancy is shown in Fig. 6(d) by using a method similar to the one mentioned above. From these figures, it can be seen that the projector distortion will affect the distribution of the carrier fringe and, thereby, influence the 3D reconstruction result. With the proposed method, the bending carrier phase induced by the projector distortion can be effectively removed to reconstruct the 3D shape of the tested object accurately.

 

Fig. 6 Reconstruction of the ceramic plate. (a) Residual phase map when projecting the adaptive fringe patterns and removing the principle carrier phase; (b) 3D result of the ceramic plate with the proposed method (See Visualization 1) ; (c) discrepancy distribution obtained with the proposed method; (d) discrepancy distribution obtained by projecting the fixed-pitch fringe pattern.

Download Full Size | PDF

Meanwhile, we calibrated the projector with the phase-aided method mentioned in [12], where the BA strategy was used to optimize the calibrated parameters. The re-projection errors of the camera and projector are shown in Fig. 7. It can be seen that the fluctuation range of the re-projection error of the projector is greater than that of the camera. These calibration results agree well with the results of the theoretical analysis in the introduction. Based on these calibration results, the stereo-vision-based model, in which homologous points are established with the aid of the absolute phase map, was used to reconstruct the 3D shape [15]. Figure 8 shows the discrepancy distribution of the same ceramic plate, where the peak-to-valley (PV) and root-mean-square (RMS) values are 0.3780mm and 0.0355mm, respectively. Compared with Fig. 6(c), it is not difficult to find that the reconstruction error is significantly increased at the boundary area. This is mainly because there are some errors in the calibration results of projector, such as errors in the distortion coefficients. Under this condition, if the nonlinear distortion model is used to calculate the ideal imaging points from real imaging points, the calculation error increases with the distance between the imaging point and the center of the projector plane. Consequently, the reconstruction error in the boundary area is greater than that in the center area.

 

Fig. 7 Re-projection errors of camera (a) and projector (b). (Units: Pixels)

Download Full Size | PDF

 

Fig. 8 Depth discrepancy of the ceramic plate when calibrating the projector and reconstructing the shape with the stereo-vision-based model.

Download Full Size | PDF

To investigate the robustness of the proposed method further, the ceramic plate was randomly placed at 11 different positions. At each position, adaptive fringe patterns were projected onto the ceramic plate again to acquire the absolute phase. By using the method introduced in Section 2, we can reconstruct the 3D shape of the ceramic plate. The reader can refer to Visualization 1 for detailed information. Table 1 lists the PV and RMS values of all discrepancies. These results demonstrate that the proposed method can achieve a satisfactory accuracy in the measurement volume. Another evaluation experiment was performed to examine the uncertainty of the proposed method. The same plate was placed on an accurate translating stage with a resolution of 1.25μm. At each position, the absolute phase map of the plate was calculated from the captured pattern images. Then, the phase was converted to depth by using Eq. (8). The profiles along one row near the center area are plotted in Fig. 9. It can be seen that the measurement error fluctuates by approximately ±0.1mm, which might be caused by environmental noise such as vibration and sensor noise of the camera and projector.

Table 1. Statistical result of the discrepancy when the ceramic plate was measured at 11 different positions (Unit: mm).

View Table | View all tables in this article

Std: standard deviation.

 

Fig. 9 Measurement distance along the middle-row direction for four positions: (a) 9.84mm, (b) 19.68mm, (c) 29.52mm, and (d) 54.12mm. The horizontal axis represents the pixel position along the row direction, and the vertical axis represents the measurement distance with respect to the reference plane.

Download Full Size | PDF

3.2. Measurement of a standard sphere

To evaluate the accuracy of the proposed method, a ceramic sphere was measured using the proposed method. The radius of the ceramic sphere is 19.0469mm. After projecting the adaptive fringe pattern onto the spherical surface as shown in Fig. 10(a), we can reconstruct the 3D shape as shown in Fig. 10(b). The least-squares (LS) radius of the measured result is calculated as 19.0328mm, and the radial deviation with respect to the LS fitting center is given in Fig. 10(c). For comparison, the fixed-pitch sinusoidal fringe patterns were also projected to reconstruct the 3D shape of the ceramic sphere, and the LS radius is calculated as 18.8793mm. In addition, the radial deviation map acquired by projecting the fixed fringe patterns is given in Fig. 10(d). These results indicate that the projector distortion will decrease the measurement accuracy, and we cannot obtain the true 3D shape for the ceramic sphere if the fixed-pitch fringe patterns are used. While projecting the adaptive fringe patterns generated by the proposed method, the bending carrier phase can be completely offset and, thereby, the 3D shape of the ceramic sphere can be accurately reconstructed. To check the robustness of the proposed method, the ceramic sphere was measured 11 times at different positions. At each position, the adaptive fringe patterns were projected to acquire the 3D shape of the ceramic sphere. Table 2 lists the statistical results of the LS radius. From this table, it can be concluded that with the proposed method, we can eliminate the influence of projector distortion and obtain a group of constant radial results.

 

Fig. 10 Measurement result for a standard sphere. (a) A frame of the captured pattern image, (b) 3D result of the ceramic sphere, (c) radial deviation obtained by projecting adaptive fringe patterns, and (d) radial deviation obtained by projecting fixed-pitch fringe patterns.

Download Full Size | PDF

Table 2. Statistical results of the LS radius when the ceramic sphere was measured at different positions (Unit: mm).

View Table | View all tables in this article

Std: standard deviation.

4. Conclusion

In summary, we have analyzed the phase and measurement errors induced by the projector distortion in PMP, and we proposed an adaptive fringe projection technique to suppress the projector distortion. With the proposed method, the carrier phase in the projected fringe patterns is modified according to the projector distortion. Consequently, unevenly spaced fringe patterns are generated on the projection plane. After projecting these adaptive fringe patterns, the bending carrier phase is offset. Then, evenly spaced fringe patterns are yielded on a plane perpendicular to the optical axis of projector lens, whereby the influence of the projector distortion is eliminated. Experimental results demonstrate that this method has several advantages over others. Firstly, as it does not require the calibration of the projector, it has low computational complexity and can improve the measurement precision. Secondly, the proposed method does not need to place and capture the standard plate at multiple positions; instead, it only requires to place a standard plate at any one position within the measurement volume. Consequently, the preparation time of the proposed method is far less than that of projector calibration. Thirdly, the proposed method does not rely on a particular measurement setup in which the projector and camera must be carefully adjusted in advance. Thus, the proposed method enables us to suppress the projector distortion in an arbitrarily arranged PMP system.