1. Introduction

The dynamic measurement of three-dimensional (3-D) scenes becomes more and more important in the fields of industrial quality control and human-machine interaction [1]. Due to its non-contact, full-field, and high-resolution nature, fringe projection profilometry (FPP) has proven to be one of the most promising techniques for measuring the motion or deformation of dynamic objects. Recently, numerous dynamic 3-D measurement techniques have been developed based on fringe projection [2–4]. Different from those techniques for applications with static objects, the primary problem in this field is to reliably recover the 3-D information of moving objects or dynamic scenes with minimized total acquisition time for reducing potential motion artifacts. Generally, this problem can be solved in two directions. The first one is to increase the speed of pattern projection and image acquisition, which could be realized by the using the digital-light-processing (DLP) technology and high-frame-rate hardware, as demonstrated by [5–7]. The other direction focuses on minimizing the number of projected patterns. However, the unambiguous measurement of 3-D dynamic scenes in the presence of large discontinuities and spatially isolated surfaces with full sensor resolution and minimal projected patterns is still a challenge.

To deal with this problem, some researchers resorted to Fourier transform profilometry (FTP) [8, 9], by which only a single fringe pattern is sufficient to retrieve the phase. However, due to the global character of the Fourier transform, the phase calculated at an arbitrary pixel depends on the whole recorded fringe pattern, making FTP methods suffer from frequency band overlapping problem caused by variations in object texture and excessive object slopes [10]. Although this problem could be remedied by the π phase-shifting FTP [11], the demand for an additional fringe pattern would influence the instantaneous feature of the FTP to some extent. Besides, the measurement accuracy of FTP is seriously limited by information about the noise and bandwidth of the modulating signals of the pattern to be analyzed.

Another well-known method used in 3-D measurement is phase-shifting profilometry (PSP) [12]. Compared to FTP, PSP has the advantages of higher accuracy, larger resolution, and greater insensitivity to ambient light [13, 14]. Theoretically, at least three fringe patterns are required to get the wrapped phase in PSP. Rahman et al. [15] utilized spatial phase constraint to unwrap the phase and only three patterns were employed. This method bases on an assumption that the height jumps of the measured object are less than a fringe period, so this method is not qualified for measuring discontinuous or step-like surfaces with step height more than 2π which is common in practical measurement. In fact, the extra information or constraint is necessary to judge the mentioned irregular jump in period order. In [16, 17], at least three more fringe patterns are projected to get extra phase information which is used to help recover the absolute phase of an arbitrary surface. However, these two methods do not make full use of the redundancy in PSP, with which the additional fringe patterns could reduce to less than three. Liu et al. [6] and Zuo et al. [7] presented the dual-frequency pattern scheme and 2+2 phase shifting method respectively where only five or four fringe patterns are adequate to retrieve a complex phase map with the presumption that the average intensity and even the modulation keep constant within a certain period of time. Nevertheless, the scheme with five or four fringe patterns is not the most efficient way to unwrap the phase. Moreover, the additional patterns are not desirable for high-speed measurement. To further reduce the sensitivity to dynamic scenes, several methods were presented which can calculate the absolute phase of a complex surface using only three fringe patterns. These techniques can be divided into two categories. The first one is the pattern-embedded method, in which the special signal such as speckle [18] or triangular wave [19] is embedded into original fringe patterns, and in this way extra information is obtained for phase unwrapping. However, the algorithm in speckle-embedded scheme [18] is based on image correlation which is time-consuming, so this limits the measurement speed in time-critical conditions. For the case where the encoding signal is a triangular wave [19], the related method is sensitive to noise, which means the measurement precision cannot be guaranteed. In the second technique, a multi-camera system is employed to eliminate the ambiguity of period order with geometry constraint [20–23]. Generally, only the rough period orders are available using this geometry constraint, so an optimization algorithm or an accurate depth constraint is necessary to refine the results in this method. In [20], graph cut and loopy belief propagation are used as an optimization but it results in very high computation costs. Following this work, Burchardt et al. [22] and Li et al. [23] utilized measurement volume to restrict the correspondence searching range on the epipolar line. Without the complex optimization, a high-speed measurement rate still can be obtained with a priori of measurement volume. However, to keep the robustness of choosing the correct correspondence, the measurement volume must be set small enough when using dense fringes.

In this work, a multi-view phase unwrapping method using three composite fringes is proposed. The composite patterns are generated by embedding the triangular wave in the original phase-shifting fringes under the guidance of the number theory to ensure the uniqueness for each point. Benefited from the geometry constraint, each pixel can explore its corresponding points independently through the wrapped phase map, and in this process, by analysing the 3-D position of each corresponding point, the correspondence whose 3-D position exceeds the boundaries of the geometry constraint is ruled out. Meanwhile, the correspondence having a considerable difference in the wrapped phase of the sinusoidal wave is also eliminated. After these steps, few candidates remain, then we just select the one with the closest value in the triangular wave as the optimal correspondence. However, in multi-view system, not all points can be captured by two cameras due to the existence of shaded areas, which means some points have no correct correspondence. As the final period order of these points in the first camera and that of their corresponding points in the second camera always have different values, left-right consistency check is qualified to eliminate the period order in this case [24]. Compared with traditional techniques, several merits exist in this system: (1) the sensitivity to movement decreases because no additional patterns are required; (2) the robustness of phase unwrapping improves and this character is useful to high-precision measurement; (3) there are no complex algorithm, which ensures the high speed in our system. These features make our method suitable for high-speed 3-D measurement with high precision. To accelerate the calculation, all these steps are implemented on GPU and the rate of 3-D reconstruction reaches 120 fps with a resolution of 644 × 484.

2. Principle

2.1. Three-step phase-shifting algorithm

Phase-shift profilometry (PSP) is a well-known fringe projection method for the retrieval of the 3-D information. A set of phase-shifted sinusoidal patterns are projected, and the phase is calculated at each point. The minimum number of images is three, but more images will improve the accuracy of the reconstructed phase. However, considering sensitivity to dynamic scenes, we use three images, elsewhere this allows the images to be projected at high speed using the modified DLP. The intensities for each pixel of the three images can be described by the following formulas assuming a linear projector, a linear camera, constant lighting, and a static object during the recording interval [7, 12, 25]

In our system, we use a shift offset of θ = 2π/3 which gives the following final expressions for phase ϕ′

The accurate I_e could not be derived from the Eq. (7), but considering that I_am1 as well as I_am2 is much smaller than I_e, and I_dc is an constant value, a substitute value $I_e^{\prime }$ for I_e is utilized in our paper. Meanwhile, in order to avoid the extra calculation and eliminate the effect of reflectivity, the $I_e^{\prime }$ can be computed as

The intensity of embedded signal will be replaced with $I_e^{\prime }$ in the following sections.

2.2. Choice of encoding signals

Either the speckle signal or the triangular wave is the main-stream auxiliary information in 3-D measurement [18, 19, 28]. In this section, the performance of these two signals is reevaluated combining with the multi-view system to explain which one is suitable in our system.

Image distortion, including stretch, skew and rotation, caused by perspective transformation is a common phenomenon in the multi-camera system and its impacts on embedded signal must be taken into account. As shown in Fig. 1, p, p_corr are a pair of corresponding points in the left image and the right image, but the same-size sub-images shown in Figs. 1(c) and 1(d) centering on these two points contain some difference, which exerts negative effect on correlation algorithm in Eq. (9)

 

Fig. 1 Illustration of distortion and deviation in multi-view system. (a) Speckle pattern derived from the composite patterns in the first camera; (b) Speckle pattern derived from the composite patterns in the second camera; (c) Subimage around point p in (a); (d)-(e) Subimages around corresponding points of p.

Download Full Size | PDF

Being different from the random property of the speckle pattern, the character of the triangular wave is periodicity, so if encoding signal becomes the triangular wave an extra task is to select the appropriate period numbers to guarantee a unique information for each point. The basic principle of number-theoretical phase unwrapping [29–31] can act as the guideline to design the composite pattern embedded with the triangular wave. In this composite signal, the main high frequency phase in Eq. (7) and the phase of the triangular wave should meet the following relations

Combining Eqs. (10) and (11) yields

The key destination for composite pattern design is to ensure the uniqueness of the $(n{\varphi }_e^{\prime }-n_e{\varphi }^{\prime })$ for each pixel in the phase map. Generally, the number of fringes should be integers, so the right hand side of Eq. (12) is an integer. Therefore, the left-hand side must also be the same integer. The values of the left-hand side are used as the entries to determine these two integers uniquely if n and n_e are prime to each other [26, 31–33]. For simplicity of explanation, we consider an example with n = 5 and n_e = 3. Figures 2(a) and 2(c) show changes of two wrapped phase maps along the absolute phase axis where the first wrapped phase map has n = 5 and the second has n_e = 3. The value of $(5{\varphi }_e^{\prime }-3{\varphi }^{\prime })/2\pi$ are plotted in Fig. 2(e) from which it is easily observed that there is a one-to-one relationship between the (k, k_e) and the value of $(5{\varphi }_e^{\prime }-3{\varphi }^{\prime })/2\pi$. Theoretically, a absolute phase map could be derived from these composite patterns without any other assistance, but from [31] we know if the maximal phase error is larger than π/(n + n_e), mistakes will occur in determining the fringe orders. So it must be guaranteed that the relationship n + n_e < π/Δϕ_max can be established. In this composite pattern scheme, the effect of ambient light is unavoidable, which undoubtedly increases the value of Δϕ_max and require the π/(n + n_e) to be small enough for robust phase unwrapping. Paradoxically, to guarantee the high precision, n should be as large as possible. In this case, the relationship n + n_e < π/Δϕ_max is no longer valid as well as the one-to-one relationship between the (k, k_e) and the value of $(5{\varphi }_e^{\prime }-3{\varphi }^{\prime })/2\pi$. As is shown in Figs. 2(b), 2(d), and 2(f), the larger n + n_e is, the more errors in $(n{\varphi }_e^{\prime }-n_e{\varphi }^{\prime })/2\pi$ when Δϕ_max break through the boundary. This contradiction can be alleviated from two perspectives in traditional methods: the first one is reducing Δϕ_max by projecting the triangular wave separately [31], and the other one is limiting the number of fringes to decrease n + n_e [19]. However, these methods are not satisfied ways because of sacrificing the anti-motion ability or high-precision property in measurement. Note that small changes in $(n{\varphi }_e^{\prime }-n_e{\varphi }^{\prime })/2\pi$ (e.g.±1) typically result in large changes in the fringe order k [26], that is, the ambiguities are very likely to emerge in non-adjacent period. Given this point, the geometry constraint is an effective instrument in eliminating ambiguities of period order when n + n_e < π/Δϕ_max is not established since this constraint can easily reject those period orders away from the correct one. In this paper, a multi-view system is introduced to address the paradox without the defects in traditional methods. Several more reasons are taken into account for this scheme to explain the feasibility: (1) the composite patterns designed with number theory ensure the uniqueness in $({\varphi }^{\prime },{\varphi }_e^{\prime })$ theoretically; (2) assuming the wave length of embedded signal is 100 pixel, then the rate of phase change is only 0.06 rad/pixel. Compared to the effect of noise, the influence resulted from deviation or distortion in multi-view system is very marginal; (3) only simple computation is involved in this scheme to guarantee the speed of phase unwrapping.

 

Fig. 2 Illustration of the influence of noise in composite patterns embedded with the triangular wave.

Download Full Size | PDF

2.3. Excluding false period orders using geometry constraint and the difference of the wrapped phase

As mentioned above only the wrapped phase ϕ′ can be calculated, and Eq. (5) shows that for each pixel N possibilities exist. This means that the recorded phase can originate from exactly N different positions in the projector image (see Eq. (2)), and thus N possible 3-D positions.

For each possible period k, the 3-D position is calculated using calibration parameters between the first camera and the projector. From Fig. 3, it can be found that only the area insides the red dotted lines can be captured by both two cameras, which means a possible 3-D position beyond this area is not reasonable and the corresponding period k is not true. From [23], we know that the smaller disparity range can get a shorter search line, and in that work, to keep the robustness of choosing the correct correspondence, the measurement volume must be set small enough when using dense fringes. In this paper, a much more free disparity range is allowed as long as the fringe modulation of the captured images to be within an usable range. With this depth constraint, some of the candidates are rejected in this step without any other computation. Note that both projector and cameras are calibrated internally as well as externally, so the resulting 3-D point between the red solid lines in Fig. 3 is projected into the second camera, which is shown in Fig. 4(b). Then the range of candidate period is limited to a small interval before next optimization. For an efficient implementation on the GPU, several LUTs are employed to calculate the 3-D positions and the corresponding projection points in the second camera.

 

Fig. 3 Illustration of the geometry constraint.

Download Full Size | PDF

 

Fig. 4 Information of an arbitrary point and its corresponding points in another camera. (a) A point in the first camera; (b) The corresponding projection points in the second camera; (c) Wrapped phase of sinusoidal signal and the intensity of the triangular wave of any point in the first camera; (d)-(i) Wrapped phase of sinusoidal signal and the intensity of the triangular wave of projection points in period 10, 11, 12, 13, 14, 15.

Download Full Size | PDF

It is well known that the wrapped phase ϕ′ in Eq. (7) of p ought to have a high degree of similarity to the correct matched point in the second camera. Theoretically, the correspondence with the minimal difference of the wrapped phase is the correct one, but noise as well as system error makes this hypothesis unreliable so all the candidates perform similarly in the wrapped phase should be reserved for prudential reasons. In other words, only if the difference between p and its corresponding point in the wrapped phase exceeds a threshold, the related candidate will be ruled out. The value of the threshold depends on the maximum offset caused by system error in the wrapped phase of the sinusoidal signal, and in this paper it is 0.6 rad. Besides, considering the phase jumps in discontinuous region of the wrapped phase, another threshold 5.7 rad is set to avoid mistakes in this case. Finally, the candidates satisfy the following formula

2.4. Confirming final period order with the triangular wave

Because of the system error, the p′s projection point in second camera has a minor deviation compared to the ideal one. This deviation only affects slightly in the last step since the setting of suitable thresholds gives some margin for this system error. However, the unique correspondence with the minimum difference in the triangular-wave map should be determined, so it is necessary to correct the position of the rest candidates and this process is operated in the wrapped phase of the sinusoidal wave. Referring to Eq. (13), searching the wrapped phase closest to ϕ′ through the 5 × 5-sized neighborhood for each candidate is not difficult. Once the closest wrapped phase is obtained, the point corresponding to this phase will take place of the original candidate in the following steps.

Supposing p_corr (k)′ is the point after correction in the second camera, and $I_e^{\prime }(p_{corr}{(k)}^{\prime })$ the corresponding intensity of the triangular wave, then on account of the similar principle with previous statement, Eq. (13) can be rewritten as

As a result of previous optimization, a few candidates remain in this step, which increases the possibility to select the correct k value when the relationship n + n_e < π/Δϕ_max in section 2.2 is not established. Certainly, in order to get a more stable result, using the mean value in a 5 × 10-sized neighborhood s to replace the center point may depress the noise to some extent. Then Eq. (14) will update as

However, considering the symmetry of the triangular wave, when p and p_corr (k)′ are located at symmetric position of the triangular wave, Eq. (15) is found unable to distinguish these two points. The first method occurs to us is computing the gradient map to indicate direction for each point to remove the ambiguity in a symmetric position. Unfortunately, the noise makes the gradient map unreliable. In this method, the neighborhood in Eq. (15) is split into left and right component, and the formula is refined as

As shown in Fig. 5, it is nearly impossible for traditional method to distinguish the points in three vertical lines from each other because of the ambiguity resulted from similar wrapped phase in these positions. On the contrary, confirming the correct period order is completely feasible through several steps in the proposed method, and this is shown in Figs. 4(c)–4(i).

 

Fig. 5 Wrapped phase of sinusoidal signal and the intensity of triangular wave in line 330 of the first camera.

Download Full Size | PDF

All of the previous discussion is based on a general assumption that the point p can be captured by both the two cameras. However, in multi-view system, not all points can be captured by two cameras owing to the existence of shaded areas, which means some points have no correct correspondence, so period order error emerges in these areas just displaying in a single camera. As the final period order of these points in the first camera and the one of their corresponding points in the second camera always have different values, left-right consistency check is qualified to eliminate the period order in this case [24].

3. Experiments

From the analysis of the correspondence method, the procedures required to determine the period order uniquely and accurately is summarized. It mainly includes three steps:

Step 2 and Step 3 are detailed in Fig. 6.

 

Fig. 6 The process of confirming period order for point p before left-right consistency.

Download Full Size | PDF

An experimental system is developed to validate the actual performance of the proposed strategy. This system includes a DLP projector (Light Crafter 4500) and two digital CMOS cameras (AVT GigE Mako G-030B), and the two cameras are synchronized with the DLP projector. The camera resolution is 644 × 484 with a maximum frame rate of 309 fps, and the projector has a resolution of 912 × 1140.

Firstly, a ceramic plate was measured to present the performance of Step 2, Step 3, and validate the precision by the way. Then accuracy was analyzed with several static objects including isolated objects and complex surfaces. Finally, deformation objects were measured to evaluate the validation of our method in real-time 3-D measurements. Note that the measurement range in multi-view system is not infinite, so in these experiments, a depth threshold is set as −200mm and 200mm to guarantee the fringe modulation of the captured images to be within an usable range.

3.1. Precision analysis

A ceramic plate was measured, and the performance of the related step is shown in Fig. 7. Although geometry constraint removes quite a lot of erroneous correspondence, as shown in Figs. 7(a) and 7(c), there is still a great possibility that a false value is selected according to the wrapped phase in Step 2. In Step 3, another optimization basing on encoding signal is employed to determine the final period order, and Figs. 7(b) and 7(d) present the final result. The blue region in Fig. 7(b) is the area that cannot be captured by both cameras, so the period orders in this region are unreliable and can be removed by left-right consistency check. The same ceramic plate is also used to validate the precision of the proposed method, Fig. 8 is the measurement result. The MSE between the acquired points and approximating points are largely due to the random noise of the system, which measured approximately 0.0683 mm.

 

Fig. 7 Performance of Step 2 and Step 3. (a) The period order after Step 2; (b) The period order after Step 3; (c) Line 333 of (a); (d) Line 333 of (b).

Download Full Size | PDF

 

Fig. 8 Measurement result of the ceramic plate. (a) 3-D reconstruction; (b) Height in 300 column.

Download Full Size | PDF

3.2. Static scenes measurement

The further experiments were carried out to verify the accuracy of our system with arbitrary shape. A number of different objects were measured including a desk fan, the tooth model, the Doraemon model and the statue of David. The 3-D results obtained by our method are shown in Fig. 9. The fan blades are separated from each other, so this object is suitable for testing if a discontinuous surfaces could be reconstructed correctly, and Fig. 9(a) shows the 3-D reconstruction. In order to prove that our method can also be applied for complex shape, the statue of David was adopted, and the result is shown in Fig. 9(b). The tooth model and the Doraemon model shown in Figs. 9(c) and 9(d) were employed to enhance the credibility of accuracy analysis. Quantitative analysis results about accuracy of the these objects are shown in Table 1. The number of points is the sum of points before left-right consistency check. To ensure objectivity of accuracy analysis, these objects were also measured by traditional multi-frequency PSP with the same high-frequency fringe in our method. Then the ratio of correctness and error is easily obtained by making a comparison between them after all optimization including left-right consistency check. The missing ratio means the proportion of points which perform badly in left-right consistency check, and most of them are captured by a single camera so that no sufficient information can be used to make a accurate judgement. From Table 1, it can be seen that the error ratio in our method is lower than 0.3%. When the measured surface is flat such as tablet, the miss ratio is also at a low level. As the surface becomes more complex, there are more shaded areas, and the missing ratio becomes larger as expected. Although shaded areas also make trouble in traditional method, they will be expanded when another camera is introduced in our technique. Fortunately, the mistakes in these areas can be removed by left-right consistency check in multi-view system. These results indicate that the correspondence method can measure arbitrary shape surfaces such as discontinuous surfaces and complex surfaces with high correctness ratio.

 

Fig. 9 Measurement results of several objects. (a) Desk fan; (b) Statue of David; (c) Tooth model; (d) Doraemon model.

Download Full Size | PDF

Table 1. Accuracy results of the proposed method

View Table

3.3. Real-time measurement

After precision and accuracy test, the next two experiments were carried out to verify the real-time performance of our system. Three phase-shift composite fringes are fused into a 24-bit pattern stored in Light Crafter 4500 where the R, G, and B digital channel represent a 8-bit composite fringe and are projected within 8500 us respectively. Meanwhile, the two cameras synchronized by the trigger signals from Light Crafter4500 capture each projection pattern with an exposure of 9000 us. Then the obtained images are processed according to the mentioned steps which are implemented on GPU. The computer used in this system is a DIY computer with an Intel i7-4790k CPU and a NVIDIA GeForce GTX970 graphics card. The graphics card has 1664 CUDA cores and 4 GB memory with a maximum bandwidth of 224 GB/sec.

We first measured a moving palm to show the 400mm measurement depth in our system and the less sensitivity to motion. In order to display the 3-D reconstruction better, the point cloud is translated into surface structure assisted with lighting and rendering. Fig. 10(a) shows several results of the palm and the more details are presented in Visualization 1 and Visualization 2. Then the facial expressions were tested to verify the performance of our system when measuring a complex surface in dynamic condition, the related 3-D reconstruction results are shown in Fig. 10(b) as well as Visualization 3 and Visualization 4. These two experiments testify the ability of our system to realize fast 3-D acquisition for dynamic scenes.

 

Fig. 10 Dynamic measurement results (associated Visualization 1, Visualization 2, Visualization 3, and Visualization 4). (a) Palm measurement; (b) Facial expressions.

Download Full Size | PDF

4. Conclusion

In this work, a novel high-speed 3-D measurement technique for recovering absolute phase and measuring spatially isolated objects with full revolution under dynamic conditions has been presented. This paper combines composite phase-shift fringe and multi-view system to strengthen the robustness of phase unwrapping. Compared to conventional temporal phase unwrapping method, only three composite fringe patterns are sufficient to obtain the absolute phase of complex surface, which makes our method less sensitive to dynamic scenes. On the other hand, being different from traditional composite-pattern scheme, the multi-view system is introduced to address the contradiction between precision and accuracy in high-speed 3-D measurement. Meanwhile, no strict depth constraint or high computation-cost algorithm is required with the assistance of composite patterns. Experimental results have demonstrated the success of our proposed method in its ability to produce fast, accurate, and precise depth information for complex scenes.

Finally, it should be also mentioned that several defects exist in this method. One example is that the amplitude of the high frequency fringe needs to be reduced to embed the extra signal, so the SNR of phase shifting is lowered. In the process of embedding the triangular wave into original fringe, only coprime requirements between two signals is analyzed but the optimal scheme is not presented in this paper. Besides, the discussion about embedded signal is limited to speckle and the triangular wave, which is not reasonable and this is anther defect. In the future, we will explore some other schemes to design a better composite pattern for multi-view system.

Funding

National Natural Science Fund of China (NSFC) (11574152, 61505081, 61377003); ‘Six Talent Peaks’ project (2015-DZXX-009); ‘333 Engineering’ research project (BRA2015294); Fundamental Research Funds for the Central Universities (30915011318); Open Research Fund of Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense (3092014012200417).