1. Introduction

In recent years, high-speed 3D shape measurement techniques are widely used in various fields such as biomechanics, intelligent monitoring, robot navigation, industrial quality control, and human-computer interaction. Among plenty of state-of-the-art methods, fringe projection profilometry (FPP) [1–4], which is based on the principle of structured light and triangulation, has been proven to be one of the most promising techniques due to its inherent advantages of non-contactness, high accuracy, high efficiency, and low cost. The mainstream FPP techniques can be generally divided into two groups: Fourier transform profilometry (FTP) [5, 6] and Phase-shifting profilometry (PSP) [7–10]. FTP is highly suited for dynamic 3D acquisition and can provide the phase map using a single fringe pattern. But it suffers from the spectrum overlapping problem which limits its measurement quality and precludes the recovery of the fine details of complex surfaces. In contrast, PSP is quite robust to ambient illumination and varying surface properties and can achieve pixel-wise phase measurement results with higher resolution and accuracy, but it generally requires the multiple fringe patterns to reconstruct the 3D shape of the object. When measuring dynamic scenes, the motion will lead to phase distortion artifacts, especially when the object motion during the interframe time gap is non-negligible [11]. With the rapid advances of the high-frame-rate camera, high-speed digital light processing (DLP) projection technique, and high-performance processors, numerous PSP methods have been demonstrated in high-speed 3D shape measurement of dynamic scenes such as fast moving objects, rapidly rotating, or vibrating non-rigid bodies [12–19]. Furthermore, both PSP and FTP adopt the arctangent function ranging between $-\pi$ and π for phase calculation, which results in phase ambiguities in the wrapped phase map with $2\pi$ phase jumps. Therefore, it is necessary to carry out the phase unwrapping to eliminate the phase ambiguity and convert the wrapped phase maps into the absolute ones.

In general, phase unwrapping algorithms can be classified into three main categories: spatial phase unwrapping [20–23], temporal phase unwrapping (TPU) [24–26], and stereo phase unwrapping [27–30]. By assuming the phase continuity, spatial phase unwrapping generally uses the relationship between the phase information of the spatial neighboring pixels to remove the phase ambiguities in a wrapped phase map [20]. However, the assumption will be easily violated when measuring isolated objects or surfaces with abrupt depth discontinuities. In order to solve the common limitation in spatial phase unwrapping, TPU approaches have to be used to realize pixel-wise phase unwrapping with the aid of additional wrapped phase maps with different fringe periods [24]. Obviously, the increased number of required patterns is undesirable, which limits TPU’s application for high-speed measurements. To address this issue, several composite phase-shifting schemes (e.g., dual-frequency PSP [31], bi-frequency PSP [32], and 2+2 PSP [33]) have been proposed, which can solve the phase ambiguity problem without significantly increasing the number of projected patterns [8]. On a different note, stereo phase unwrapping approaches have been proposed to overcome the limitations of spatial phase unwrapping and TPU. Compared with TPU, stereo phase unwrapping can retrieve the absolute phase using two or more wrapped phase maps obtained from two perspectives based on geometry constraint without projecting any additional patterns [27, 28]. Several methods based on this principle has been presented in the literature and achieved remarkable success [27–30]. However, in order to ensure the stability of phase unwrapping, these methods have to compromise the fringe frequency or the depth of field, leading to limited 3D measurement accuracy or measurement volume [8].

Geometry constraint has been widely used in stereo vision because it greatly facilitates the establishment of binocular correspondence of the two cameras in different perspectives. Weise et al. [34] introduced geometric constraint into PSP using only three fringe patterns for a single 3D reconstruction. In addition, some complicated post-process techniques, such as graph cut and loopy belief propagation, are used to get a global absolute phase map but it leads to massive computing overhead [35–37]. It is easy to understand that using more cameras will increase the likelihood that more constraints can be exploited to completely eliminate the phase ambiguities in the multi-view system. Following this idea, Tao et al. [30] set up a position-optimized quad-camera system to guarantee the reliability of phase unwrapping. The phase differences from different perspectives were applied to eliminate phase ambiguities. Besides, the weighted phase difference scheme was introduced to realize stereo phase unwrapping of high-frequency fringes. However the valid measurement areas, defined as the common regions of the cameras and projector, were significantly reduced. On the other hand, the phase-coding schemes, in which the additional signal is embedded into original fringe patterns, have been proposed for a multi-view system, which provide auxiliary information to assist the absolute phase recovery. As the special signal speckle or triangular wave were chosen, both methods need to perform time-consuming image correlation for each valid pixel which limits the measurement speed [38–40]. In addition, these methods have to reduce the amplitude of the fringe signal, resulting in the reduction of the phase measurement accuracy. To maximize the fringe signal to noise ratio (SNR), Zhang et al. [36] fully exploited the limited redundancies in three phase-shifting fringe patterns, but the speckle pattern is essentially susceptible to ambient illumination which affects the correlation results, so it still needs complicated post-processing algorithms to produce a more reliable phase unwrapping result. In order to overcome the problems mentioned above, Wang et al. [41] developed a period-coded phase-shifting (PCPS) algorithm by embedding additional slope signals into standard 4-step phase-shifting patterns. However, the coding patterns become no longer smooth due to the added signals, which are susceptible to lens defocusing and leads to the limited accuracy of 3D measurement. It is obvious that the slope-embedded patterns are not suitable to be applied to the high-speed binary defocusing projection system which has a certain amount of de-focus.

In this work, a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system is proposed. Firstly, we review and reproduce the design method of the composite patterns via skillfully embedding speckles without compromising the fringe modulation and the phase measurement accuracy. Then, the optimized speckle pattern is mathematically generated based on the system configuration in the experiment to ensure the period-wise uniqueness for each point instead of the conventional global uniqueness. In addition, we present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. Benefited from the optimized design for fringe patterns, the periodic ambiguities can be ruled out effectively based on geometry constraint and the proposed adaptive window image correlation. Though our method can provide superior stereo phase unwrapping performance even under noisy conditions, fringe order errors are still inevitable especially around dark regions and object edges where the fringe quality is low. Furthermore, the mismatched regions will be substantially resolved using the proposed regional diffusion compensation technique (RDC). In general, considering some wrong regions caused by mismatched or shaded effects in multi-view system, left-right consistency check, as the common technique in stereo vision, will be able to eliminate wrong points with fringe order errors. But according to some previous methods [30, 38], the mismatched points can be neither completely removed nor correctly compensated using the check operation, which leads to the final 3D measurement results with a few outliers and holes. Different from the check operation, the proposed compensation technique is a relatively low-cost method that only requires the absolute phase map obtained from main camera and can not only detect but also correct the error points. Therefore, these proposal techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.

2. Principle

2.1. Speckle embedded phase-shifting algorithm

In this subsection, we will discuss in detail how to embed the auxiliary information in the phase-shifting patterns. Firstly, the standard N-step phase-shifting fringe patterns can be described as

Where $A\left(x,y\right)$ is the DC intensity, $B\left(x,y\right)$ is the fringe signal amplitude, $\mathrm{\Phi }\left(x,y\right)$ is the phase information, and $2\pi n/N$ is the shift offset. To maximize the signal to noise ratio (SNR) and normalize the intensity pattern $I_n\left(x,y\right)$, $A\left(x,y\right)$ and $B\left(x,y\right)$ are set as 0.5 so that the value of $I_n\left(x,y\right)$ ranges from 0 to 1. In general, researchers embed the additional information into fringe patterns by modifying $A\left(x,y\right)$ or $B\left(x,y\right)$ [38–40, 42]. For example, Lohry et al. [39] insert binary speckle signals into three patterns by adjusting $B\left(x,y\right)$ to 0.25 or 0.5. Tao et al. [38] choose the triangular wave as the coded signal and modify $A\left(x,y\right)$ to encode fringe patterns. However, in order to ensure that the value of $I_n\left(x,y\right)$ does not exceed the range $\left[0,1\right]$, these methods have to reduce the amplitude of $B\left(x,y\right)$, which results in a poor quality of the phase map. To embed the speckle signal without reducing the amplitude of fringe, Zhang et al. [36] fully exploited the limited redundancies in three phase-shifting fringe patterns as shown in Fig. 1. For Phase-shifting profilometry, the speckle-embedded method applies the theoretical minimum of three images into the absolute phase recovery, which greatly improves the efficiency of 3D measurement. However, the embedded speckle pattern is low quality and essentially susceptible to ambient illumination which affects the correlation results, so it still needs complicated post-processing algorithms to produce a more reliable unwrapping result. To solve this issue, Wang et al. [41] developed a period-coded phase-shifting (PCPS) algorithm by embedding additional slope signals into standard 4-step phase-shifting patterns. The intensities of the PCPS fringe images in the projector space are:

To guarantee that the retrieved phase is unaffected by the embedded signals, the coding functions should satisfy:

And the value of $I_n^p\left(x,y\right)\left(n=0,1,2,3\right)$ also should ranges from 0 to 1. So we have:

Combining Eqs. (4) and (5) yields:

And the embedded signal $D\left(x,y\right)$ is also demodulated from Eq. (2) given by:

So combining Eqs. (6) and (7) yields:

According to Eq. (8), we can find the value of $D\left(x,y\right)$ along the x axis with the dynamic range of $\left[-0.5,0.5\right]$ as shown in Fig. 2(a). Notably, the quality of the coding patterns is improved substantially because the relative amplitude of $D\left(x,y\right)$ is 1. However, the designed coding signals $C_0\left(x,y\right)$ and $C_1\left(x,y\right)$ also become no longer smooth which is sensitive to lens defocusing as shown in Fig. 2(b). In order to ensure the robust phase unwrapping, the period of fringe is generally restricted to a very small range ($\le 20$), because of the slope signal which is susceptible to noise, which results in the limited accuracy of 3D measurement. Given this, Wang’s method is not suitable to be applied to the high-speed binary defocusing projection system which has a certain amount of de-focus and is much more difficult to generate desired low-frequency fringe patterns than the high-frequency ones. Consequently, the type of the auxiliary signals should be chosen carefully.

 

Fig. 1 The principle of Zhang’s method [36]. (a) The three-step phase-shifting patterns. (b) The speckle pattern. (c) The intensity of the speckle signal in conventional three-step phase-shifting patterns. (d) The three-step speckle-embedded phase-shifting patterns. (e) The embedded speckle pattern.

Download Full Size | PDF

 

Fig. 2 (a) The slope signal $D\left(x,y\right)$ is designed within the range of $\left[-0.5,0.5\right]$. (b) The designed coding signals $C_0\left(x,y\right)$ and $C_1\left(x,y\right)$, where $C_0\left(x,y\right)=C_2\left(x,y\right)$, and $C_1\left(x,y\right)=C_3\left(x,y\right)$.

Download Full Size | PDF

In order to overcome the problems mentioned above, we chose the speckles as our auxiliary information instead of the slope patterns. Compared with the original one-dimensional slope signal, the two-dimensional speckle signal is less sensitive to the lens defocusing and suitable for the binary defocusing systems. And the final composite fringe patterns and the embedded speckle pattern generated using the proposed speckle embedded algorithm are shown in Fig. 3.

 

Fig. 3 The proposed speckle-embedded phase-shifting algorithm. (a) The four-step speckle-embedded phase-shifting patterns. (b) The embedded speckle pattern.

Download Full Size | PDF

2.2. Optimized design of the speckle pattern for phase unwrapping

Then, considering the characteristics of speckle, we mathematically developed an optimized design method of the random pattern. By skillfully embedding speckles without sacrificing the phase measurement accuracy, we can realize phase unwrapping with high-frequency fringes. The key challenge of the proposed design method is to achieve global phase unwrapping in combination with speckle patterns and multi-view constraints. By introducing depth constraint into the digital speckle image correlation for phase unwrapping, the only thing we need to do is to eliminate the periodic ambiguity within the pre-defined local range rather than the traditional global range, which means that our method just requires the speckle patterns with period-wise uniqueness instead of the conventional global uniqueness. This idea gives a guideline for the design approach and greatly simplifies the requirements of speckle patterns compared to the traditional speckle-based methods [36, 39, 43, 44]. At the same time, to avoid a universal phenomenon in the multi-view system, such as image distortion caused by perspective transformation (including stretching, skewing and rotation), we are obliged to reduce the baseline between two cameras to make the quality of stereo images better, which can improve the correlation accuracy to realize robust and efficient phase unwrapping. Besides, taking the multi-view system into consideration, some parameters of the speckle pattern should be closely related to the system parameters [43]:

Step 1: A large number of consecutive sub-windows are generated to fill the projection pattern as shown in Fig. 4, and the size is $M_r\times N_r$ where N_r is its horizontal resolution. For vertical fringes, it is appropriate to set the wavelength of fringes as N_r since the speckle is only used to distinguish periodic candidate points. In order to perform the region match efficiently, it is necessary that the value of M_r should be selected suitably to satisfy $S\left(2L+1\right)\le M_r$, and L is the window radius of image correlation on the camera plane.

Step 2: For any sub-window, we randomly fill the speckle with a size of $M_s\times M_s$. Theoretically, each sub-window can contains N_s speckles ($N_s=\left(M_r\times N_r\right)/M_s^2$). In order to enhance the uniqueness of the speckle, we determine the range ($35\%N_s\sim 45\%N_s$) for the number of speckles in each sub-window based on the matching quality metric after an exhaustive empirical search.

 

Fig. 4 The diagram of the optimized design method of the speckle pattern.

Download Full Size | PDF

The above steps have generated a speckle pattern I as shown in Fig. 5(a). Next, we test the pattern with a simple and effective evaluation standard to judge the quality of the speckle pattern. For any point $P_r\left(x_r,y_r\right)$ in I_r, we can also find the candidate target point $P_t\left(x_t,y_t\right)$ in I_t using the formula as follows:

 

Fig. 5 (a) A speckle pattern is designed using the proposed method. (b) The period order map based on (a) is obtained using the proposed test method. (c) The error rate for phase unwrapping on the designed speckle patterns with different speckle size.

Download Full Size | PDF

2.3. Computational framework for phase unwrapping using geometry constraint and speckle pattern

In the first two subsections, we have presented that how to design an optimized speckle pattern for phase unwrapping and how to encode the speckles into standard 4-step phase-shifting fringe patterns without compromising the phase measurement accuracy. So the composite fringe images captured by the camera can be described as

After the wrapped phases ${\varphi }^c\left(x,y\right)$ from different views are obtained according to Eq. (12), since the captured images contain the invalid points (the background) and the valid points (the tested object), the value of $B^c\left(x,y\right)$ for each pixel can be used to suppress the background information by the following formula:

2.3.1. Eliminating the periodic ambiguities based on geometry constraint and the adaptive window image correlation

After the removal operation for invalid points is performed, for the wrapped phases ${\varphi }^c\left(x,y\right)$ with $N-$period fringe patterns, the fringe order of each valid pixel exists N possibilities. Taking each possible order $k\left(x,y\right)$ into consideration, the corresponding depth value $Z_k\left(x,y\right)$ is calculated using calibration parameters between the main camera and the projector. It is easy to find that some possible depth values $Z_k\left(x,y\right)$ are beyond the pre-defined depth range based on depth constraint and the corresponding orders $k\left(x,y\right)$ can be excluded from the candidates as shown in Fig. 6. For the remaining ones, the region match based on speckle patterns will be implemented according to Eq. (10) to further remove wrong candidates. To avoid the effect of reflectivity and enhance the quality of the speckle pattern, the speckle pattern $I_{spk}^c\left(x,y\right)$ are obtained from the composite fringe images using the formula as follows:

In the digital speckle image correlation, how to select the appropriate window radius L is our main concern. It is obvious that the smaller L can speed up the calculation efficiency but just provide a coarse correlation result. On the contrary, the bigger L output reliable corresponding points at the expense of the computational cost. Given this, the smaller L firstly should be applied to obtain a coarse order map in a short time, then a few points with low reliability (such as$corr\left(x,y\right)<0.8$) can be matched correctly using the larger window radius. The value of L depends on the quality of the speckle patterns from different views caused by the design method of the speckle pattern and the baseline between two cameras. Since the optimized design method is proposed and the baseline is reduced, the adaptive window radius L is set as 5 pixels and 10 pixels adequately in this paper. The operation process of confirming the period order map based on geometry constraint and the adaptive window image correlation is shown in Fig. 7.

 

Fig. 6 Illustration of an arbitrary point p in the camera and its corresponding points in 3D space and the projector using depth constraint.

Download Full Size | PDF

Though geometry constraint and the adaptive window image correlation using the optimized speckle patterns can provide superior stereo phase unwrapping performance (obtaining the absolute phase with 64-period) even under noisy conditions, fringe order errors are still inevitable especially around dark regions and object edges where the fringe quality is low. Considering the common cases which are mismatched or shaded effects in stereo vision, left-right consistency check is usually used to eliminate wrong points with fringe order errors. But according to some previous methods [30, 38], error points can be neither completely removed nor correctly compensated using the check operation, which gives rise to 3D measurement results with a few outliers and holes. Therefore, the further compensation step is aimed at identifying and adjusting the residual mistakes automatically.

 

Fig. 7 The operation process of confirming period order based on geometry constraint and the adaptive window image correlation. (a) The wrapped phase obtained from main camera. (b) The speckle pattern obtained from main camera. (c) The wrapped phase obtained from auxiliary camera. (d) The speckle pattern obtained from auxiliary camera. (e) The correlation result obtained using the smaller L (5 pixels). (f) The period order map obtained using the smaller L (5 pixels). (g) The correlation result obtained using the larger L (10 pixels). (h) The period order map obtained using the larger L (10 pixels). (i) The enlarged detail of the region in (f). (j) The enlarged detail of the region in (h).

Download Full Size | PDF

2.3.2. Regional diffusion compensation (RDC) of fringe order errors

To void this issue mentioned above and acquire 3D reconstruction results with high completeness, a simplified compensation technique based on spatial phase unwrapping is introduced into the traditional stereo phase unwrapping approach. In spatial phase unwrapping, reliability-guided phase unwrapping is one of the most effective methods using its neighboring pixels to obtain the relative fringe order [19, 20]. In this work, we propose the regional diffusion compensation algorithm (RDC) for detecting and correcting those fringe order errors by exploiting additional information in spatial domain. There does exist some cases that false matching points are still reserved after left-right consistency check, but these cases are very rare after RDC. To clearly indicate the compensation procedure of RDC, a flowchart is plotted as shown in Fig. 8.

After image correlation, the elementary absolute phase map ${\mathrm{\Phi }}^c\left(x,y\right)$ can be obtained in which the values of most of valid points are correct and reliable to assist the compensation procedure, despite the fact that there are a small number of wrong points in some areas of high reflectivity, low contrast, occlusion, and edges. Before the implementation of RDC, a binary mask $Mas{k}_c\left(x,y\right)$ is built, which denotes the valid points to be computed, and the initial value of $Mas{k}_c\left(x,y\right)$ is set as $Mas{k}_v^c\left(x,y\right)$. For every valid pixel in the absolute phase map ${\mathrm{\Phi }}^c\left(x,y\right)$, if the absolute phase difference between adjacent pixels is less than the threshold, the corresponding adjacent pixels will be grouped into a class.

 

Fig. 8 The flowchart of the compensation procedure of RDC.

Download Full Size | PDF

In the traditional reliability-guided phase unwrapping method, the input is the wrapped phase map, and the quality map is used to find the optimal phase unwrapping path, which starts from the point with higher quality value to ensure the accuracy rate of phase unwrapping. Different from the spatial phase unwrapping method, the input of RDC is the absolute phase map after geometry constraint and the adaptive window image correlation, where most of the points have been correctly unwrapped, i.e., the points with high-quality value have been unwrapped rightly. Therefore, the quality map no longer plays a significant role for phase unwrapping. And then the number of points in the group (where the absolute phase difference of the adjacent pixels satisfys Eq. (16)) is equivalent to the overall quality of the group, so the phase correction operation is performed firstly from the group with the largest number of points in $Mas{k}_c\left(x,y\right)$ to its adjacent points. Since our previous algorithm implements a stable stereo phase unwrapping, it can be reasonably derived that groups with more than $Are{a}_r$ points are reliable groups, and they cannot be compensated but are used as a reference. The value of $Are{a}_r$ depends on the resolution of the cameras, so $Are{a}_r=300$ is acceptable for the 640 × 440 resolution. If its adjacent points belong to different groups, all points of the smaller class will be corrected according to the following formula:

Therefore, the two classes merge into one class, and the region of the group with the largest number of points will be further expanded in $Mas{k}_c\left(x,y\right)$, which is similar to the regional growth. In addition, owing to the vertical fringes areused, the absolute phase is limited by the monotonicity constraint in the horizontal direction, so the points in the horizontal direction should take precedence over its in the vertical direction as a reference. After all the neighboring points have been adjusted properly, the processed group will be removed from $Mas{k}_c\left(x,y\right)$, and the phase correction operation is performed again from the extant group with the largest number of points in the updated $Mas{k}_c\left(x,y\right)$. This step is repeated until the mask $Mas{k}_c\left(x,y\right)$ is empty, which means that all the valid points have been computed and the compensation procedure is completed. Eventually, the phase errors can be effectively compensated, and the final absolute phase map ${\mathrm{\Phi }}_{final}^c\left(x,y\right)$ is obtained. In addition, there is an optional removal operation for isolated points that the processed group with the total number of points less than $Are{a}_i$ will be intentionally excluded from the compensation results. In most cases, $Are{a}_i$ is set as 200 pixels properly. In general, the left-right consistency check is time-consuming since it needs two absolute phase maps acquired by the image correlation from two views independently. Compared with the check operation, the proposed compensation technique is a relatively low-cost method that only requires the absolute phase map from main camera and can not only detect but also correct the error points as shown in Fig. 9.

 

Fig. 9 (a) The elementary absolute phase map obtained from auxiliary camera. (b) The final absolute phase map obtained using the left-right consistency check. (c) The 3D reconstruction result obtained using (b). (d) The enlarged detail of the region in (c). (e) The elementary absolute phase map obtained from main camera. (f) The final absolute phase map obtained using RDC. (g) The 3D reconstruction result obtained using (f). (h) The enlarged detail of the region in (g).

Download Full Size | PDF

3. Experiments

In the experimental section, a high-speed FPP system is built to verify the actual performance of the proposed method. This system includes two high-speed complementary metal oxide semiconductor (CMOS) cameras (Vision Research Phantom V611) and a high-speed DLP projection system. The high-speed projection system consists of a DLP development kit (Texas Instruments DLP Discovery 4100), a digital micro-mirror device (DMD) with the XGA resolution (1024 × 768), and a custom-designed optics module [19]. For the optimized speckle patterns in this system, the designed parameters are set to include N_r = 16 pixels (i.e., the wavelength of fringes is 16 pixels), M_r = 32 pixels, M_s = 3 pixels, and the speckle area of each sub-window is at about 40%. Benefited from the binary defocusing fringe projection technique, the optimized composite fringe patterns generated by the binary dithering algorithm are projected into the measured object by the DMD at the speed of20,000 Hz [45, 46]. In this work, two cameras are operated at a reduced image resolution (640 × 440) to match the speed of the projection system with an exposure time of $46\mu s$. Since our approach only uses 4 composite structured-light patterns to recover one absolute phase map, our system can achieve a high-speed, dense, and accurate 3D shape measurement with 64-period fringe patterns at 5000 frames per second.

Then, to acquire 3D results with high completeness, it should also be emphasized here that the relative spatial positions between the cameras and the projector are carefully arranged in our multi-view system. In the conventional multi-view system [27, 28], the 3D measurement is realized by using the matching result of the dual-camera as shown in Fig. 10(a). Since the projector is usually placed between two cameras, the measurement accuracy can be improved by increasing the baseline of the dual-camera. But, in our system, the brightness of the binary fringe patterns, which is projected using the high-speed binary defocusing projection technique with limited exposure time, is much less than that of the 8-bit patterns [19, 47]. In order to ensure the measurement quality and void the influence of the ambient light, the measured area (i.e., the defocusing range of the projector) should be closer to the projector to enhance the pattern intensity. In fact, due to the large size of the DLP projector, it results in a reduction in the common area of the dual cameras located on both sides of the projector, which is not suitable for measuring objects with large cross-sections. To solve this problem,we try to narrow the baseline of the dual-camera and place the dual-camera on the same side of the projector, and then obtain the absolute phase map to achieve 3D measurement based on the main camera and the projector. Besides, in order to improve the accuracy of the measurement by increasing the baseline of the main camera and the projector, the auxiliary camera should be brought closer to the projector than the main camera as shown in Fig. 10(b). Meanwhile, as is mentioned in Section 2.2, reducing the baseline between two cameras can enhance the quality of stereo images, which can improve the correlation accuracy to realize robust and efficient phase unwrapping. The high-speed FPP system is finally shown in Fig. 11.

 

Fig. 10 (a) The diagram of the traditional multi-view system. (b) The diagram of our multi-view system.

Download Full Size | PDF

 

Fig. 11 This system includes two high-speed CMOS cameras and a DLP projection system.

Download Full Size | PDF

3.1. Precision analysis

To quantitatively analyze the precision of 3D measurement our system can achieve, we measured a standard ceramic plate with the assistance of the proposed method. The tested object and the corresponding one of the four-step phase-shifting patterns from main camera are shown in Figs. 12(a)–12(b). After the wrapped phase and the demodulated speckle pattern are obtained using Eqs. (12) and (15) as shown in Figs. 12(c)–12(d), geometry constraint is firstly executed to preclude some periodic candidate points, because its corresponding depth value is beyond the pre-defined depth range. However, there are still a lot of wrong candidate points for the fringe patterns with 64-period as shown in Fig. 12(e). Then the adaptive window image correlation can efficiently remove a lot of the periodic ambiguities and output the reliable periodic order map in Fig. 12(f). As a common method in stereo vision, left-right consistency check is carried out to eliminate wrong points with fringe order errors, which will results in the periodic order map and the absolute phase map with a few holes as shown in Figs. 12(g) and 12(k). Instead of the check operation, the period orders in the wrong regions can be reasonably compensated using RDC, and then the periodic order map, the absolute phase map, and the final 3D measurement results with higher completeness are presented in Figs. 12(h), 12(l), and 13(a). To obtain the measured errors, we perform the plane fitting using the 3D data in Fig. 13(a), then the fitted plane is set as the ground truth. The differences between the measured data and the ground truth are shown in Fig. 13(b), and we display the quantitative histograms of the differences as shown in Fig. 13(c). It can be easily found that the major measured errors are less than 100 $\mu m$ with the rms of 61.73 $\mu m$, respectively. This result verifies that the proposed method can significantly increase the measurement accuracy due to the use of high-frequency fringe patterns and high SNR of the recovered phase map.

 

Fig. 12 The phase measurement results of a standard ceramic plate. (a) The ceramic plate to be measured. (b) The 2D camera image (one of the four-step phase-shifting patterns) from main camera. (c) The wrapped phase is obtained using Eq. (12). (d) The demodulated speckle pattern is obtained using Eq. (15). (e) The phase order map based on geometry constraint. (f) The phase order map based on geometry constraint and the adaptive window image correlation. (g) The phase order map using left-right consistency check. (h) The phase order map using RDC. (i)-(l) The corresponding absolute phase map of (e)-(h).

Download Full Size | PDF

 

Fig. 13 The 3D measurement results of a standard ceramic plate. (a) The 3D reconstruction results of the plate. (b) The distribution of the errors of (a). (c) The histogram of (b).

Download Full Size | PDF

3.2. Static scenes measurement

In FFP, in order to acquire the 3D reconstructed results with high-quality, high-frequency fringe patterns are usually projected to restrain phase error and improve the precision of 3D measurement, but at the expense of the robustness of phase unwrapping. After confirming the 3D measurement precision of our system, as 64-period fringe patterns are used in our system, the further experiments need to be carried out to demonstrate the reliability of our method by measuring multiples objects with complex shapes. Some different objects were measured in Figs. 14(a)–14(c) including a girl statue with a ceramic plate, a Voltaire model, a statue of the Goddess, and the statue of Skadi, and the corresponding 3D results are shown in Figs. 14(d)–14(f). And then the corresponding enlarged details are presented to illustrate the reliability of our method which can achieve robust 3D shape measurement for objects with complex surfaces and geometric discontinuities in Figs. 14(g)–14(i).

Besides, in order to quantitatively analyze the robustness of phase unwrapping our method can achieve, the accuracy results of the absolute phase for different measured objects are calculated and shown in Table 1. The number of points is the sum of all valid points in $Mas{k}_v^c\left(x,y\right)$ from main camera. To realize the reliability analysis of the accuracy results, the absolute phase map obtained using traditional multi-frequency PSP with the same high-frequency fringe can be served as the reference phase. The left-right consistency check (L-R check) and the regional diffusion compensation algorithm (RDC) are performed as two separate groups. Then the correct matching rate and the error matching rate are easily calculated by making a comparison of the absolute phase for each valid point. The missing matching rate means the ratio of points which is excluded due to poor performance or is invalid in the reference phase. From the comparison results of Cmr and Mmr in Table 1, we caninfer that there exist some mismatched regions in the elementary absolute phase map ${\mathrm{\Phi }}^c\left(x,y\right)$ before L-R check or RDC, and L-R check can remove a number of error points resulting in a relative change between Emr and Mmr. In contrast, the proposed RDC can substantially correct error points which can always provide the better Cmr and the lower Mmr. However, for edge regions and abrupt depth discontinuities of complex objects, the reference phase obtained using traditional multi-frequency PSP is slightly different to ${\mathrm{\Phi }}^c\left(x,y\right)$. In this case, these phases can be removed in L-R check, but for RGC, it can be adjusted to new phases using Eq. (17) which may be different from the corresponding ones of the reference phase. As a result, RDC has the slightly higher Emr compared with L-R check. Furthermore, since RDC can detect and correct the error points, the Cmr still keep at a high level even for measuring these challenging shapes, which demonstrates the reliability of our method for phase unwrapping.

 

Fig. 14 The measurement results of several objects. (a) A girl statue with a ceramic plate. (b) A Voltaire model and a statue of the Goddess with discontinuous surfaces. (c) The statue of Skadi. (d)-(f) The corresponding 3D reconstruction results of (a)-(c). (g)-(i) The corresponding enlarged detail of the region in (d)-(f).

Download Full Size | PDF

Table 1. Accuracy results of the proposed method

View Table

3.3. Dynamic scenes measurement

Next, our system is applied to record a one-time transient event for high-speed 3D shape measurement: the statue of Voltaire and a free-falling balloon filled with water as shown in Fig. 15(a). Figures 15(b)–(f) show the color-coded 3D reconstruction results at different time points. During the whole dynamic measurement, the Voltaire model is placed alone on one side of the scene. On the other side, the balloon with water is artificially released in the air. Due to the acceleration of freely falling bodies, the balloon arrives quickly and happens to deform by the reaction of the ground. And then, the balloon has undergone severe deformation, losing its original smooth shape. Finally, the balloon gradually returns to its former appearance and bounces back into the air again. The whole 3D measurement results can be referred to in Visualization 1. In the whole measuring procedures, the surfaces of the Voltaire statue and the balloon are correctly reconstructed with high-quality, verifying the reliability of the proposed method to perform the absolute 3D shape measurement with high completeness at high speed.

 

Fig. 15 The 3D reconstruction results for a one-time transient event: the statue of Voltaire and a free-falling balloon filled with water ( Visualization 1).

Download Full Size | PDF

In addition, our system is applied to imaging high-speed 3D shape measurement of a dynamic object: the rip of paper, and the corresponding color-coded 3-D reconstructions at different time points are shown in Fig. 16. Even if the objects are isolated in motion, moreover, have noticeable changes in surface conditions (such as convex and concave curvatures), no irregularities will be recognizable in the final result ( Visualization 2), which confirms that the proposed method can successfully measure multiple moving objects simultaneously.

 

Fig. 16 The high-speed 3D reconstruction results for the rip of paper ( Visualization 2).

Download Full Size | PDF

4. Conclusion

In conclusion, we propose a high-speed 3D shape measurement technique based on composite structured-light patterns and a multi-view system. For the 3D measurement precision, by skillfully embedding the speckle pattern into the original phase-shifting fringes without decreasing the fringe amplitude, we realize unambiguous 3D shape measurement with 64-period fringe patterns, achieving a measurement precision of 61.73 $\mu m$. For the reliability for phase unwrapping, the optimized speckle pattern is mathematically generated based on our system configuration to ensure the period-wise uniqueness for each point. A simple and effective test method of the designed speckle pattern is presented to provide the better match results for phase unwrapping. Benefited from the optimized design method, the periodic ambiguities can be effectively eliminated using geometry constraint and the adaptive window image correlation. Furthermore, the mismatched regions will be substantially resolved using the regional diffusion compensation technique (RDC) based on spatial phase unwrapping algorithm. Finally, the absolute phase map with high completeness can be achieved by a whole computational framework above. For the speed of 3D measurement, our method can realize a high-speed, dense, and accurate 3D shape measurement at 5000 frames per second using only 4 composite patterns based on the binary defocusing fringe projection system. This high-performance method greatly improves the measurement speed and accuracy which can provide a powerful technical guide for some high-speed profilometry in engineering applications to a large extent.

There are several aspects that need to be further improved in the proposed method, which we will leave for future consideration. First, there are several parameters in the proposed approach, which should be properly selected for different kinds of objects. Currently, we empirically set these parameters according to the overall property of the test scene. It will be better if these parameters can be automatically determined and adaptively updated. Second, in RDC the smaller group is corrected according to the adjacent larger group. When the measured object contains small isolated areas (pixel number smaller than $Are{a}_r$), their fringe orders are not easy to be determined by the spatial correlation and are also quite error-prone in the RDC process. How to handle this situation is another interesting direction for further investigation.