1. Introduction

Structured light-based technique plays an important role in fast, flexible and accurate three-dimensional (3-D) measurement [1–3], which reconstructs 3-D shape of the measured object by the triangulation method [4]. When the image correspondence between the camera and the projector is established, the 3-D surface shape can be reconstructed by combining with system parameters [4]. This image correspondence is usually established by following feature [1,5,6] or phase [4,7–9] consistency. Both the feature and phase do not change during the projection and acquisition processes, which can be easily encoded in the projected light and decoded from the captured light. The feature-based method reconstructs the 3-D shape with relatively low resolution [10,11]. The phase-based method is preferable for high-speed and high-resolution 3-D measurement [4].

Fringe projection profilometry (FPP), a typical phase-based method, can establish pixel-level image correspondence and sense 3-D data with thousands of frames per second [4,8]. In FPP, the system point spread function (PSF) becomes non-ideal with a finite size because of the lens aberration and the diffraction limit of the camera. As we know, the captured intensity is the integral of the intensity inside the PSF of each pixel, which will deviate from its actual intensity in surface areas containing discontinuous reflectivity [12]. For example, the captured intensity of the low reflectivity area will be averaged towards that of the high reflectivity area, which will generate error around discontinuous areas in the calculated phase. The phase error will result in pixel correspondence establishment error, thus generating non-ignorable discontinuous reflectivity (DR) error.

FPP can reduce the DR error by first estimating the PSF distribution and then compensating the error area by its nearest correct area [13–17]. This PSF-estimation method reduces the root mean square height error to one-third of the original, but it is difficult to determine the correct area for objects containing sharply depth change. A deconvolution-based error reduction method has been recently proposed, which first calibrates the camera PSF and then retrieves the desired patterns for the phase retrieval [12]. The PSF-estimation method reduces the RMS height error by up to four times, but the ringing effect of its deconvolution introduces ripple errors in areas including feature details and sharp depth change [18].

Generally, the phase error could be less than 0.3 rad when the PSF standard deviation is less than 8 [12]. If the fringe frequency is larger than 0.05, the error of pixel correspondence is usually less than one pixel in the measurement range [19]. The DR error is proportional to the size of the camera PSF, which can be reduced by alleviating the defocus level or increasing the acquisition resolution of the scene [20]. Thus, we can directly reduce the DR error by selecting a high-resolution camera but heavily increase the system cost [21]. In addition, the desired patterns can be retrieved if we can normalize the discontinuous reflectivity of adjacent pixels before the scene is modulated by the projected patterns, but it is difficult to obtain the scene reflectivity for FPP.

Different from feature- and phase-based 3-D reconstruction techniques, single-pixel imaging metrology (SIM) establishes the image correspondence by following the scene intensity consistency between the camera and the projector [22,23]. By treating each camera pixel as a single-pixel detector, SIM projects patterns with full image spatial frequency and acquires spectrum components with full spatial frequency in transformation domain, which can reconstruct the scene reflectivity through reverse transform [24–33]. The reconstructed scene is the convolution of the camera PSF and the reflected light and the desired pixel coordinate is the grayscale center of the reconstructed scene. When the camera PSF contains discontinuous reflectivity, the grayscale center will be biased towards the area with high reflectivity. Thus image correspondence error is introduced and causes DR error.

As aforementioned, the DR error can be alleviated by normalizing the discontinuous reflectivity of the reconstructed scene before calculating the corresponding pixel. The sum of the reconstructed scene reflectivity of each camera pixel is the modulated projector light intensity, which is proportional to the scene reflectivity. Thus, the captured scene reflectivity of each camera pixel can be normalized to alleviate the DR error. However, traditional SIM is very time-consuming caused by the large number of encoding patterns, i.e., thousands of frames.

In summary, DR error is introduced at discontinuous reflectivity edges for structured light-based 3-D reconstruction methods. Traditional FPP is high-speed but the DR error reduction methods need to calibrate scene-dependent parameters, which is inflexible and has limitations in measurement scenes. While SIM shows potential for accurate 3-D measurement for objects with discontinuous reflectivity due to its scene reconstruction ability, but suffers from long data acquisition time.

In this paper, we first construct the model of DR error of both FPP and SIM. From which, we conclude that the DR error of SIM can be compensated through reflectivity normalization. Based on the conclusion drawn above, a reflectivity-based accurate 3-D reconstruction method under discontinuous reflectivity is proposed to alleviate the DR error. Furthermore, due to the extremely long acquisition time of SI, we adopt parallel single-pixel imaging technique [22,23], and FPP is used for the pixel-level correspondence establishment to accelerate the data acquisition. Both the error model and the experiment results verify that the proposed method can reconstruct accurate 3-D shape with discontinuous reflectivity.

The rest of the paper is organized as follows. Section 2 constructs the pixel correspondence error model of FPP and SIM. Section 3 presents the principle of the proposed reflectivity-based accurate 3-D reconstruction method under discontinuous reflectivity. Section 4 exhibits and discusses the experimental results. Section 5 concludes this paper.

2. Analysis of DR error

As aforementioned, the measurement error is induced by the spatial averaging effect caused by the camera PSF at discontinuous reflectivity area. The model of DR error of both FPP and SIM are constructed for the error compensation. From the point view of establishment error model, we analyze the pixel correspondence error and conclude that the DR error of SIM can be compensated. The constructed error model in FPP and SIM are detailed as follows.

2.1 Error analysis of FPP

In FPP, the projector projects a set of phase-shifted sinusoidal fringe patterns onto the object, and the camera captures these patterns reflected from the object’s surface [27]. Each pixel of the camera will uniquely correspond to one projector pixel with the same phase. These patterns are designed as

We assume that when the light emits from the projector pixel $(u,v)$ and reaches the camera pixel $(x,y)$, the scene reflectivity captured by one camera pixel can be expressed by $r(u,v;x,y)$. Then, the image captured by the camera can be expressed as

For clarity, we illustrate the light transmission process in Fig. 1. As shown in Fig. 1 (a), when the system PSF is ideal, the camera pixel $(x,y)$ receives the light reflected from the object surface which is illuminated by the projector pixel $(u,v)$. But when the system PSF is non-ideal, the camera pixel $(x,y)$ receives the light reflected from the object’s surface which is illuminated by several adjacent projector pixels, e.g., $(u_0,v_0)$, $(u_1,v_1)$ and $(u_2,v_2)$ have discontinuous surface reflectivity $r$, $r(u_1,v_1)$ and $r(u_2,v_2)$, respectively.

 

Fig. 1. Light transmission of 3-D shape measurement system for object with discontinuous reflectivity under conditions that (a) the system PSF is ideal and (b) the system PSF is non-ideal.

Download Full Size | PDF

For any projector-camera system with an ideal PSF, a pixel of the camera only receives the light emitted from one single pixel of the projector. In that situation, the wrapped phase can be expressed as

Due to the diffraction limit of the camera and the projector, the non-ideal PSF generates the blur effect. Each pixel of the camera receives light originated from a small area of the projector sensor, and the wrapped phase becomes

The phase error caused by the diffraction limit of the projector-camera system can be obtained by

The absolute phase can be transferred to the corresponding projector pixel through [34]

Thus, the pixel correspondence error of FPP can be expressed as

From Eq. (7), we can conclude the following points:

In summary, in FPP, the DR error is related to the system PSF, fringe frequency and the difference between two different reflectivity. These influencing factors are difficult to be changed, thus DR error is hard to be alleviated in FPP.

2.2 Error analysis of SIM

In SIM, a series of basis patterns are projected onto the measured scene. The captured intensity can be expressed as [27]

Here, HSI reconstructs the captured scene by [26]

When the object contains uniform surface reflectivity, the correspondence of $(x,y)$ and $(u,v)$ can be established by calculating the gray center of the captured reflectivity [22]. Taking the $u$ axis as an example, we have the coordinate

The grayscale center at the edge of discontinuous reflectivity becomes

The pixel correspondence error can be obtained by

From Eq. (12), we can conclude the following points:

The measured scene is illuminated by the projector and captured by the camera. The captured reflectivity of each camera pixel ${{r}_{c}}(u,v;x,y)$ is calculated by SI. ${{r}_{c}}(u,v;x,y)$ can be transformed to the DMD plane and can be regarded as the scene reflectivity "captured" by the projector ${{r}_{p}}(u,v)$. By dividing $r({{u}_{1}},{{v}_{1}})$ and $r({{u}_{2}},{{v}_{2}})$ by ${r}_{p}$, i.e., normalizing the captured reflectivity, $r({{u}_{1}},{{v}_{1}})$ and $r({{u}_{2}},{{v}_{2}})$ becomes $1$ in Eq. (12) and $\Delta u^\prime$ becomes $0$ if $G$ is an ideal PSF.

In summary, in SIM, the DR error is related to the system PSF, reflectivity distribution and the difference between two different reflectivity. The former two factors cannot be changed in SIM. Thus, the DR error of SIM can be alleviated through normalizing the captured reflectivity.

3. Principle of reflectivity-based accurate 3-D reconstruction method under discontinuous reflectivity

As aforementioned, SIM can get accurate 3-D shape through normalizing the captured reflectivity. The error model of SIM with normalized reflectivity is analyzed, which is opposite to the correspondence calculated by the original reflectivity. From which, a reflectivity-based accurate 3-D reconstruction method is proposed, it first acquires coarse pixel correspondence by FPP, and then uses SIM to achieve accurate pixel correspondence. From the point view of establishment error model, we analyze the pixel correspondence error and verify the feasibility of the proposed method. Both the constructed model and the proposed method are detailed as follows.

3.1 DR error removal based on reflectivity normalization

In SIM, the scene reflectivity captured by each camera pixel is calculated. From that, the modulated projector light intensity can be calculated to normalize the captured reflectivity and compensate the pixel correspondence error.

The modulated projector light intensity can be calculated by [23]

To eliminate the DR error, the captured reflectivity is normalized by ${{r}_{c}}/{{r}_{p}}$, and the pixel correspondence is changed into

As mentioned above, ${u_s}^{\prime \prime }=u_0$ if the system PSF $G$ is ideal. However, if $G$ is not an ideal PSF, the calculated corresponding projector pixel ${u_s}^{\prime \prime }$ can be expressed as

Thus, the pixel correspondence error can be expressed by

It can be concluded that by normalizing $r_c$ with $r_p$, pixel correspondence with an error opposite to the result of Eq. (12) is obtained, thus enables error compensation for pixel correspondence.

Thus, the accurate corresponding projector pixel coordinate can be calculated by

The pixel correspondence error after compensation is

Considering $abs(\Delta r)\le 1/2$, it can be concluded that $abs(\Delta u)\le 1/6$.

Because of the extremely long data acquisition time of SI, we adopt parallel single-pixel imaging technique [23], and FPP is used for the pixel-level correspondence establishment. Based on the above, we propose a reflectivity-based accurate 3-D reconstruction method.

3.2 Procedure of the reflectivity-based accurate 3-D reconstruction method

According to the important conclusion drawn above that the pixel correspondence error obtained by the captured scene reflectivity and the normalized scene reflectivity are opposite, the DR error can be removed through reflectivity normalization. Combining the high speed of FPP and the high accuracy of SIM, we propose a reflectivity-based accurate 3-D reconstruction method. The schematic of the proposed reflectivity-based accurate 3-D reconstruction method is illustrated in Fig. 2.

 

Fig. 2. Schematic of the proposed reflectivity-based accurate 3-D reconstruction method.

Download Full Size | PDF

A scene point $(X,Y)$ is illuminated by a projector pixel $(u_0,v_0)$ and captured by the camera pixel $(x,y)$. The intensity of the captured patterns for camera pixel $(x,y)$ is

The wrapped phase $\phi$ of camera pixel $(x,y)$ can be resolved by

Because phase error is introduced at the edge of discontinuous reflectivity, the coarse projector pixel $(u_c,v_c)$ corresponds to $(x,y)$ can be established by

Then, the periodical extension pattern of SIM is used for the scene reflectivity reconstruction. Specifically, Hadamard single-pixel imaging (HSI) is used due to its high speed and noise resistance characteristics. The Hadamard basis pattern $P_H (u,v)$ can be obtained by applying an inverse Hadamard transform to a delta function $\delta _H (m,n)$

Specially, Hadamard matrix with the size of $16\times 16$ pixels is chosen to cover the visible region of each camera pixel. Thus, a series of periodical extension patterns with a period of $16$ pixels is used to accelerate the data acquisition [22], and the number of Hadamard basis patterns required for the measurement is $512$ frames. The periodical extension patterns are designed by

Then, by applying inverse Hadamard transform in Eq. (9), the scene reflectivity captured by one camera pixel $r_c (u,v;x,y)$ is obtained. And the subpixel coordinate $({u_s}^{\prime },{v_s}^{\prime })$ with error caused by discontinuous reflectivity is calculated by grayscale centroid method.

Then, the object reflectivity captured by the projector $r_p (u,v)$ is calculated by Eq. (13), and the subpixel coordinate $({u_s}^{\prime \prime },{v_s}^{\prime \prime })$, whose subpixel error is opposite to $({u_s}^{\prime },{v_s}^{\prime })$ is calculated. Thus, the accurate subpixel correspondence is established.

To sum up, the detailed reflectivity-based accurate 3-D reconstruction method is given as follows:

4. Experiment

We further test the performance of the proposed reflectivity-based accurate 3-D reconstruction method through the following experiments. A calibration board and a complex object with discontinuous reflectivity are first measured to evaluate the measurement accuracy. Then a bottle with discontinuous reflectivity and a colored toy are measured to verify the performance on complex surfaces.

The experimental configuration is illustrated in Fig. 3. The 3-D measurement system mainly includes a DLP6500 projector with a resolution of $1920\times 1080$, a Basler acA800-510um CMOS camera with a resolution of $800\times 600$, and a suited camera lens with a 16 mm focal length. The measured object is placed around 0.5 m away from the measurement system. For FPP, according to Eq. (1), $25$ phase-shifted fringe patterns are generated by selecting $f=1/20$, $N=25$, and takes 2 seconds for each measurement. For SIM, Fourier slice patterns and periodical extension patterns with period of 16 pixels are used, $1536$ frames of Fourier slice patterns and $512$ frames of Hadamard basis patterns are projected and takes $102$ seconds for each measurement. For the proposed method, the fringe patterns of FPP and periodical extension patterns with period of 16 pixels are projected and takes $8$ seconds.

 

Fig. 3. Experimental setup. The digital projector projects patterns onto the measured object, and the camera captures light reflected by the object’s surface.

Download Full Size | PDF

4.1 Accuracy evaluation

We first verify the 3-D measurement accuracy of the proposed reflectivity-based accurate 3-D reconstruction method. The plane fitting error of the measured calibration board and the measurement error of a complex surface are both used to evaluate the measurement accuracy.

4.1.1 Accuracy evaluation by a calibration board

We first verify the influence of image resolution and fringe frequency to 3-D reconstruction accuracy in FPP by measuring a dot calibration board. The details of the reconstructed 3-D with different resolution is compared and the plane fitting error is compared for the 3-D shape reconstructed by using fringe with four different frequencies. The experimental results are shown in Fig. 4.

 

Fig. 4. The 3-D reconstruction result with different captured image resolution and different fringe frequencies. (a) the measured calibration board, (b)-(d) the calibration board reconstructed with the captured image resolution of $200\times 200$ pixels, $520\times 520$ pixels, and $1800\times 1800$ pixels, (e)-(h) the plane fitting error distribution of 3-D shape measured by fringe with frequency of $f=1/20$, $f=1/30$, $f=1/40$ and $f=1/60$.

Download Full Size | PDF

In Fig. 4 (b), the 3-D shape reconstructed with low resolution fringe image has obvious errors around the discontinuous reflectivity edges. In Fig. 4 (c) and (d), these errors become smaller with the increasing of fringe image resolution. From Fig. 4 (e)- (f), it can be concluded that with the increasing of fringe frequency, the plane fitting error decreases. However, the error at discontinuous reflectivity edges cannot be eliminated by reducing the fringe frequency. This conclusion is consistent with the conclusion drawn from Eq. (7).

Then, the measurement accuracy of the proposed method is verified. Firstly, sinusoidal fringe patterns are projected to calculate the coarse pixel correspondence, the vertical and horizontal coarse pixel correspondence are shown in Fig. 5 (a) and (b), respectively. Figure 5 (c) and (d) show the errors of the vertical and horizontal coarse pixel correspondence, respectively. It can be concluded that the pixel correspondence error is less than 1, which makes an accurate coarse pixel correspondence. Then, periodical extended Hadamard basis patterns are projected to calculate the captured reflectivity of each camera pixel. The calculated captured reflectivity is periodically distributed, as shown in Fig. 5 (e). With the help of coarse pixel correspondence established by FPP, the needed reflectivity can be extracted, which is inside the yellow block in Fig. 5 (e). Then, the subpixel correspondence is established as shown in Fig. 5 (f). From the calculated reflectivity, the scene reflectivity is calculated, as shown in Fig. 5 (g). The scene reflectivity is used to normalize the calculated reflectivity, and the DR can be compensated through the normalized reflectivity. The pixel correspondence after DR error compensation is shown in Fig. 5 (h).

 

Fig. 5. Subpixel correspondence establishment result for the measured calibration board. (a) the vertical coarse pixel correspondence established by FPP, (b) the horizontal coarse pixel correspondence established by FPP, (c) error distribution of (a), (d) error distribution of (b), (e) the captured reflectivity calculated by HSI, (f) the calculated subpixel correspondence, (g) the scene reflectivity map, and (h) the accurate subpixel correspondence after DR error removal.

Download Full Size | PDF

The error distribution of the 3-D shape reconstructed by the three methods are shown in Fig. 6. As shown in Fig. 6 (d) and (e), the plane fitting error of the 3-D shape reconstructed by FPP, i.e., 0.071 mm, is higher than that of SIM, i.e., 0.043 mm. The possible reason is that, SIM performs better for the low reflectivity regions. Besides, there exists significant errors at the discontinuous reflectivity edges. While as shown in Fig. 6 (f), plane fitting error at the discontinuous reflectivity edges are reduced significantly. The plane fitting error is reduced to 0.033 mm for the 3-D shape reconstructed by the proposed reflectivity-based accurate 3-D reconstruction method. The last row of Fig. d shows the plane fitting error of the blue line drawn in the second row. It is clear that the plane fitting error of 3-D shape reconstructed by the proposed method is uniformly distributed, while that of 3-D shape reconstructed by traditional FPP and SIM has significant errors at discontinuous reflectivity edges.

 

Fig. 6. Accuracy evaluation. The 3-D shape reconstructed by (a) FPP, (b) traditional SIM and (c) the proposed reflectivity-based accurate 3-D reconstruction method, the error distribution of 3-D shape reconstructed by (d) FPP, (e) traditional SIM and (f) the reflectivity-based accurate 3-D reconstruction method.

Download Full Size | PDF

4.1.2 Accuracy evaluation of a complex surface

Then, a complex object with discontinuous reflectivity is measured to compare the reconstruction accuracy and efficiency of traditional FPP, SIM and the proposed method. To verify the measurement accuracy, a white object is used in this experiment, as shown in Fig. 7 (b). The white object is first measured by traditional FPP to get the ground truth of the object’s 3-D shape, as shown in Fig. 7 (a). Then the white object was painted by black paint to get discontinuous reflectivity, as shown in Fig. 7 (c). The object with discontinuous reflectivity was measured by traditional FPP, SIM and the proposed method. The experimental results are shown in Fig. 7.

 

Fig. 7. Measurement result of a white object painted by black paint. (a) 3-D shape of the white object, (b) captured image of the white object, (c) captured image of the painted object, (d)-(f) 3-D shape of the painted object measured by FPP, traditional SIM, and the proposed method, (h) measurement error of (d), (i) measurement error of (e) and (j) measurement error of (f).

Download Full Size | PDF

Figure 7 (d) and (h) shows the measurement result and the measurement error of traditional FPP. It can be concluded that discontinuous reflectivity causes measurement errors at the discontinuous reflectivity edges. Figure 7 (e) and (i) shows the measurement result and the measurement error of SIM. It can be concluded that the measurement result is similar to that of FPP, which has significant errors at discontinuous reflectivity edges. Figure 7 (f) and (j) shows the measurement result and the measurement error of the proposed method. Compared with the other two methods, the measurement error was significantly reduced by the proposed method.

The measurement error and the measurement time of traditional FPP, SIM and the proposed method are shown in Table 1. The measurement error is calculated from the painted area, which is inside the blue block in Fig. 7 (c). From Table 1, it can be concluded that FPP has high speed but has significant error for object with discontinuous error, traditional SIM can slightly reduce the measurement error, but suffers from very long data acquisition time. While the proposed method can achieve high accuracy measurement for object with discontinuous reflectivity and takes the time much less than that of the traditional SIM.

Table 1. Comparison of time and accuracy of the traditional FPP, SIM and the proposed method.

View Table

4.2 3-D measurement of object with complex surface

A black bottle with white words on it and a colored toy with details are measured to verify the performance of the proposed reflectivity-based accurate 3-D reconstruction method. Traditional FPP is also used to measure the objects to compare with the proposed method. The experimental results are exhibited in Fig. 8.

 

Fig. 8. 3-D shape of an object with complex surface. (a) captured scene, (b) the 3-D shape reconstructed by FPP and (c) the 3-D shape reconstructed by the proposed reflectivity-based accurate 3-D reconstruction method.

Download Full Size | PDF

In the first row of Fig. 8, a black bottle with white word is measured. In the red block, it is clear that at the border of black and white, there exists error in 3-D shape reconstructed by traditional FPP while the error is smaller in 3-D reconstructed by the proposed method. In the second row of Fig. 8, a colored toy is measured. In the orange block, at the place of black line, there exists concave in 3-D shape reconstructed by FPP, while the 3-D shape reconstructed by the proposed method is smooth. In the yellow block, the black area is raised in 3-D shape reconstructed by FPP while the 3-D shape reconstructed by the proposed method is smooth.

5. Conclusion

In this paper, the error model of FPP and SIM are constructed. From the constructed error model, it can be concluded that both DR error in FPP and SIM are influenced by system PSF and the scene reflectivity. Because of the unknown scene reflectivity, it is hard to alleviate the DR error in FPP. Meanwhile, the scene reflectivity can be reconstructed by SIM. As analyzed in the error model of SIM, the pixel correspondence calculated by the normalized reflectivity has error opposite to the original one. Based on this, a DR error removal method based on reflectivity normalization is proposed, which first reconstructs the scene reflectivity by traditional SIM, and then normalizes the captured reflectivity for error compensation. To accelerate the data acquisition of SIM, FPP is introduced to get the coarse pixel correspondence. As shown in the experiments, compared to the traditional SIM and FPP, the measurement error has reduced to $60\%$ and $46\%$, respectively, and the measurement time is reduced to $8\%$ of the traditional SIM.