High-accuracy 3D surface measurement using hybrid multi-frequency composite-pattern temporal phase unwrapping

Yingying Wan; Yingying Wan; Yiping Cao; Yiping Cao; Jonathan Kofman; Jonathan Kofman

doi:10.1364/OE.410690

1. Introduction

Three-dimensional (3D) surface measurement is used in a wide range of applications such as reverse engineering, industrial design and inspection, virtual reality, medical engineering, and cultural heritage preservation [1–4]. Fringe projection phase-shifting profilometry (PSP) is one of the most commonly used methods because it can achieve non-contacting full-field surface measurement with high resolution and accuracy [5,6]. Generally, PSP requires at least three phase-shifted fringe patterns, projected onto the object surface and corresponding images captured by camera, to retrieve the phase information required for 3D surface geometry reconstruction. PSP generates phase values in the range –π to π, and a phase unwrapping procedure is required to obtain continuous phase, from which the surface height can be computed. Spatial phase unwrapping (SPU) [6,7] can be used to eliminate the 2π phase ambiguity; however, the method uses information of the phase of neighboring pixels, making measurement of objects with surface discontinuities and multiple isolated objects error prone [7,8]. To overcome this problem, phase unwrapping methods based on geometric constraints [9–12] have been developed to eliminate the phase ambiguity at each pixel, independent of neighbouring pixels. However, these geometry-constraint based methods usually limit the measurement depth range or the frequency of the fringe patterns to maintain high reliability in determining the correct correspondences between projector and multiple-camera images. Temporal phase unwrapping (TPU) is an alternative pixel-wise approach, without any geometric constraint, that can be used for objects with surface discontinuities or multiple isolated objects [13,14]. TPU methods determine fringe orders using information from additional coded patterns or phase maps with different frequencies, generated from additional sets of phase-shifted images captured over time. TPU methods thus generally require a large number of fringe patterns, which contributes to a slower measurement process.

To reduce the number of patterns required in TPU methods, several strategies have been proposed. One approach [15] uses unit-frequency phase-shifted patterns and orthogonal carrier fringe patterns with different frequencies, combined into a single composite pattern. Although this method does not need phase unwrapping, the use of bandpass and low-pass filtering limits the depth reconstruction [15]. Additional period information has been embedded into fringe patterns in conventional four-step PSP to assist in phase unwrapping. However, the embedded binary De Bruijn sequence reduces the amplitude of the sinusoidal fringe [16], or the frequency of the fringe patterns is restricted to be low due to the limited dynamic range of the embedded signal [17]. To increase the frequency of the fringe patterns for high-accuracy measurement, the fringe order has been encoded into one additional fringe pattern and decoded for phase unwrapping of the conventional three-step PSP [18]. Phase coding methods embed codewords into the phase domain of phase-shifted fringe patterns, and only require three additional images to determine more fringe orders compared to the conventional gray coding method [19]. However, all of the above phase-shifting methods with period information encoded into one or more fringe patterns are sensitive to lens defocus since the encoding causes patterns to have abrupt edges at the period boundaries.

TPU algorithms that utilize additional wrapped phase maps from phase-shifted fringe patterns with different frequencies are not sensitive to the defocus blur, and therefore have been widely used [20]. For the conventional dual-frequency TPU approaches in PSP [21,22], the minimum number of the required patterns is six (two sets of 3-step phase-shifted patterns with different frequencies). A unit-frequency (single-period) phase map can be used as a reference to determine the fringe order of the high-frequency phase map, which is then used for 3D surface reconstruction [22]. Some dual-frequency schemes have been used to reduce the minimum number of patterns. Two sinusoidal components with different frequencies are combined into a phase-shifted fringe pattern, and 5-step phase-shifted patterns are used for recovering two phase maps [23]. One method that uses a total of five fringe patterns, employs three phase-shifted fringe patterns with high frequency (3-step) and two unit-frequency (single period) fringe patterns (2-step, shifted by π/2), to compute two phase maps [24], where the two groups of fringe patterns share the same average intensity. To further reduce the number of the patterns to four, one algorithm uses two π-shift fringe patterns and two linearly increasing/decreasing slope intensities to obtain two phase maps, which share both the same average intensity and the same intensity modulation [25]. However, in dual-frequency approaches, to ensure reliability of the phase unwrapping, the frequency of the higher frequency patterns is still limited. Since high frequency fringe patterns contribute to achieve a high measurement accuracy, a third set of fringe patterns with higher frequency is generally employed for measurement of multiple isolated objects and objects with surface discontinuities. The minimum number of patterns commonly used in conventional multi-frequency TPU is nine, including three unit-frequency, three low-frequency and three high-frequency fringe patterns.

This paper presents a new hybrid multi-frequency composite-pattern temporal phase unwrapping method that uses fewer patterns than conventional multi-frequency TPU for high-accuracy 3D surface measurement of multiple isolated objects and objects with surface discontinuities. The new hybrid multi-frequency composite-pattern method combines a unit-frequency ramp pattern with three low-frequency 3-step phase-shifted fringe patterns to form three composite phase-shifted patterns. These composite patterns are used together with three high-frequency 3-step phase-shifted fringe patterns, to generate a high-accuracy phase map for recovering the object shape. The low-frequency and unit-frequency information extracted from the composite fringe patterns is used to assist the phase unwrapping. The optimal high frequency to achieve high accuracy and reliability of the phase unwrapping is estimated by analyzing the effect of temporal intensity noise on the new method. Compared to the conventional multi-frequency TPU which uses nine patterns, the number of required images is reduced to six using the new method with grayscale patterns. The number of patterns is further reduced by half to only three patterns using color patterns with grayscale fringe patterns encoded into the red and blue channels. The grayscale and color hybrid multi-frequency composite-pattern TPU methods require fewer images than conventional multi-frequency TPU methods for high-accuracy measurement of objects with surface discontinuities.

2. Principle and method

2.1 Phase-shifting profilometry using conventional multi-frequency temporal phase unwrapping

In N-step phase-shifting profilometry (PSP), the intensities of the projected fringe patterns can be written as:

(1)$$I_n^p({x^p},{y^p}) = {a^p} + {b^p}\cos (2\pi {f^p}{x^p} - {{2\pi n} / N}),{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} n = 0,1,\ldots N - 1, $$

where n represents the phase-shift index, ${a^p}$ background, ${b^p}$ amplitude, $({x^p},{y^p})$ projector pixel coordinates, and ${a^p}\textrm{ = }{b^p} = 0.5$. ${f^p}$ is the fringe frequency, ${f^p}\textrm{ = }{F / W}$, where F is the total number of fringe periods, and W the horizontal resolution of the projected pattern. The fringe patterns are sequentially projected onto an object surface, and the deformed fringe patterns are captured by camera and can be expressed as:

(2)$${I_n}(x,y) = A(x,y) + B(x,y)\cos [{\varphi (x,y) - {{2\pi n} / N}} ],{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} n = 0,1,\ldots N - 1, $$

where $(x,y)$ are the captured image coordinates, and $A(x,y)$ and $B(x,y)$ are the average intensity and intensity modulation, respectively. $\varphi (x,y)$ is the phase containing depth information and can be retrieved as follows [26]:

(3)$$\varphi (x,y) = {\tan ^{ - 1}}\frac{{\sum\nolimits_0^{N - 1} {{I_n}(x,y)\sin ({{2\pi n} / N})} }}{{\sum\nolimits_0^{N - 1} {{I_n}(x,y)\cos ({{2\pi n} / N})} }}.$$

At least three images ($N \ge 3$) are required to compute $\varphi (x,y)$, the wrapped phase, which must be unwrapped to eliminate the 2π phase ambiguity in order to reconstruct the 3D object shape:

(4)$$\Phi (x,y)\textrm{ = }\varphi (x,y)\textrm{ + 2}\pi k(x,y), $$

where $\Phi (x,y)$ is the unwrapped phase, $\varphi (x,y)$ the wrapped phase retrieved from Eq. (3), and $k(x,y)$ the integer phase order. The key to phase unwrapping is the determination of the correct phase order $k(x,y)$ for each pixel. The basic approach of multi-frequency temporal phase unwrapping is to unwrap the phase with the aid of two additional phase maps with different fringe frequencies. The wrapped phase maps are obtained as ${\varphi _u}(x,y)$, ${\varphi _l}(x,y)$, and ${\varphi _h}(x,y)$ with unit, low, and high fringe frequency, respectively. The fringe orders ${k_l}(x,y)$ and ${k_h}(x,y)$ for the low-frequency and high-frequency phase maps, respectively, can be determined as:

(5)$$\left\{ \begin{array}{l} {k_l}(x,y)\textrm{ = Round}\left[ {\frac{{{{({F_l}} / {{F_u}}}){\Phi _u}(x,y) - {\varphi_l}(x,y)}}{{2\pi }}} \right]\textrm{ = Round}\left[ {\frac{{{\Delta _l}(x,y)}}{{2\pi }}} \right]\\ {k_h}(x,y)\textrm{ = Round}\left[ {\frac{{{{({F_h}} / {{F_l}}}){\Phi _l}(x,y) - {\varphi_h}(x,y)}}{{2\pi }}} \right]\textrm{ = Round}\left[ {\frac{{{\Delta _h}(x,y)}}{{2\pi }}} \right] \end{array} \right., $$

where Round[] is the rounding function to obtain the nearest integer value; ${F_u}$, ${F_l}$ and ${F_h}$ represent the total number of the periods of the fringes for the unit-frequency, low-frequency and high-frequency fringe patterns, respectively, and ${F_u}\textrm{ = }1$. ${\Phi _u}(x,y)$ is the unwrapped phase of the unit-frequency patterns, which contains only one fringe and requires no phase unwrapping; that is ${\Phi _u}(x,y) = {\varphi _u}(x,y)$; ${\Phi _l}(x,y)$ is the low-frequency unwrapped phase; ${\Delta _l}(x,y)$ and ${\Delta _h}(x,y)$ are the corresponding weighted phase differences. The high-frequency unwrapped phase ${\Phi _h}(x,y)$ can be obtained using Eq. (4) and is used for the final 3D shape reconstruction.

2.2 New hybrid multi-frequency composite-pattern temporal phase unwrapping

The minimum number of the patterns required in the conventional multi-frequency TPU using the 3-step phase-shifting algorithm is nine. To reduce the number of the patterns while maintaining high measurement accuracy, a new hybrid multi-frequency composite-pattern temporal phase unwrapping method was developed as follows. To achieve high-accuracy measurement, the high-frequency fringe patterns use the same 3-step phase-shifted fringe patterns as the conventional multi-frequency TPU:

(6)$$I_i^p({x^p},{y^p}) = {a^p} + {b^p}\cos (2\pi f_h^p{x^p} - {{2\pi n} / 3}), $$

where i is the image index, and $i = 0,1,2$ corresponds to the phase-shift index $n = 0,1,2$, respectively; $f_h^p$ is the frequency of the high-frequency fringe patterns. To use fewer patterns than the conventional multi-frequency TPU, composite 3-step phase-shifted fringe patterns are formed by combining a low-frequency pattern and a unit-frequency ramp pattern, encoded as:

(7)$$I_i^p({x^p},{y^p}) = {a^p} + (1 - M){b^p}\cos (2\pi f_l^p{x^p} - {{2\pi n} / 3}) + M{b^p}({{2{x^p}} / W} - 1), $$

where image index $i = 3,4,5$ corresponds to the phase-shift index $n = 0,1,2$, respectively; $f_l^p$ is the frequency of the low-frequency fringe patterns, and M is the factor determining the relative amplitude of the ramp intensity; here M = 0.5. Figure 1 shows an example of one of the projected high-frequency fringe patterns and one of the projected composite fringe patterns.

Fig. 1. Example of projected patterns of the hybrid multi-frequency composite-pattern TPU method: (a) one high-frequency fringe pattern, (b) one composite fringe pattern, (c) and (d): corresponding intensities to (a) and (b).

Download Full Size | PDF

The two groups of patterns (Eqs. (6) and (7)) are sequentially projected onto the object surface, and captured by camera. The captured fringe patterns are expressed as:

(8)$${I_i}(x,y) = {A_1}(x,y) + {B_h}(x,y)\cos [{{\varphi_h}(x,y) - {{2\pi n} / 3}} ],{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} i = 0,1,2, $$

(9)$${I_i}(x,y) = {A_1}(x,y)\textrm{ + }0.5{B_l}(x,y)\cos [{{\varphi_l}(x,y) - {{2\pi n} / 3}} ]\textrm{ + }0.5{B_l}(x,y){\varphi _b}(x,y),{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} i = 3,4,5, $$

where ${A_1}(x,y)$ is the average intensity of the first group of patterns, ${B_h}(x,y)$ and ${B_l}(x,y)$ are the intensity modulations of the first and second groups of patterns, respectively; ${\varphi _b}(x,y)$ is the base phase without any wrapping and range $( - 1,1]$, which can be scaled to unit-frequency phase with range $( - \pi ,\pi ]$, and ${\varphi _h}(x,y)$ and ${\varphi _l}(x,y)$ are the high-frequency and low-frequency phase maps, respectively. For brevity, $(x,y)$ is omitted in the remainder of this paper. ${\varphi _h}$ and ${\varphi _l}$ can be calculated using the standard 3-step phase-shifting algorithm as:

(10)$${\varphi _h} = {\tan ^{ - 1}}\frac{{\sqrt 3 ({I_1} - {I_2})}}{{2{I_0} - {I_1} - {I_2}}},$$

(11)$${\varphi _l} = {\tan ^{ - 1}}\frac{{\sqrt 3 ({I_4} - {I_5})}}{{2{I_3} - {I_4} - {I_5}}}.$$

To recover the unit-frequency phase map, Eq. (9) is rewritten as:

(12)$${I_i}\textrm{ = }{A_2}\textrm{ + 0}\textrm{.5}{B_l}\cos [{{\varphi_l} - {{2\pi n} / 3}} ],{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} i = 3,4,5$$

where ${A_2}$ is the average intensity of the second group of patterns:

(13)$${A_2} = {A_1}\textrm{ + }0.5{B_l}\varphi_b.$$

The base phase ${\varphi _b}$ can be calculated by:

(14)$${\varphi _b} = \frac{{{A_2} - {A_1}}}{{0.5{B_l}}}.$$

The average intensity of the first and second group ${A_1}$, ${A_2}$, respectively, and the intensity modulation ${B_l}$ can be determined as:

(15)$${A_1} = \frac{{{I_0} + {I_1} + {I_2}}}{3}, $$

(16)$${A_2} = \frac{{{I_3} + {I_4} + {I_5}}}{3}, $$

(17)$${B_l} = \frac{2}{3}\sqrt {3{{({I_4} - {I_5})}^2} + {{(2{I_3} - {I_4} - {I_5})}^2}}.$$

The unit-frequency phase map ${\varphi _u}$ with dynamic range $( - \pi ,\pi ]$ can be obtained by multiplying the base phase ${\varphi _b}$ by π:

(18)$${\varphi _u} = \pi {\varphi _b}.$$

Then, the high-frequency phase map ${\varphi _h}$ can be unwrapped, using the low-frequency and unit-frequency phase maps ${\varphi _l}$ and ${\varphi _u}$ in multi-frequency TPU (Eqs. (4) and (5)). Although the amplitudes of the low-frequency and unit-frequency fringe patterns are reduced, the high-frequency phase map can still have high accuracy for 3D reconstruction if it can be successfully unwrapped. The number of the fringe patterns is reduced from nine grayscale images for the conventional multi-frequency TPU to six grayscale images using the new hybrid multi-frequency composite-pattern temporal phase unwrapping method. The complete process of the hybrid multi-frequency composite-pattern temporal phase unwrapping method is summarized in the flowchart in Fig. 2.

Fig. 2. Flowchart of the hybrid multi-frequency composite-pattern temporal phase unwrapping method.

Download Full Size | PDF

The number of the projected fringe patterns can be further reduced to three by utilizing the red, green, and blue (RGB) space of a color image. The three images of the first group (Eq. (6)) are encoded into the R channel of three color images, respectively, and the three images of the second group (Eq. (7)) are encoded into the B channel of the same color images, respectively. The intensities in the G channel are set to 0 in the three color images to eliminate color crosstalk.

2.3 Noise analysis and optimal frequency selection

In the hybrid multi-frequency composite-pattern temporal phase unwrapping method, the amplitude of the second group of fringe patterns, formed by combining the low-frequency and unit-frequency ramp patterns (Eq. (7)) and captured as Eq. (9), is allocated to the unit-frequency and low-frequency phase components. The computation of the two phase maps could be sensitive to intensity noise because of the reduced amplitude, and this sensitivity could further influence the measurement accuracy. In this section, the optimal frequency for high-accuracy measurement is determined based on analysis of the temporal intensity noise.

The temporal intensity noise is assumed to be zero-mean random additive Gaussian system noise with variance ${\sigma ^2}$. The captured images with noise can be written as:

(19)$${I_i}^\varepsilon = {I_i} + {\varepsilon _i},{\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} i = 0,1,2,3,4,5, $$

where $I_i^\varepsilon $ and ${I_i}$ are the images with and without noise, respectively, and ${\varepsilon _i}$ is the random noise, ${\varepsilon _i} \sim N(0,{\sigma ^2})$. Generally, the noise is much smaller than the fringe intensity signal, and the variance of the phase error using the 3-step phase-shifting algorithm can be approximated as [27]:

(20)$$\sigma _{\Delta \varphi }^2 = \frac{{2{\sigma ^2}}}{{3{B^2}}}, $$

where B is the captured fringe-pattern intensity modulation. When the phase is mapped to the depth, the error variance of the depth $\sigma _{\Delta \textrm{H}}^2$ can be further reduced by a factor ${F^2}$ [28]:

(21)$$\sigma _{\Delta \textrm{H}}^2 = \frac{{2{\sigma ^2}}}{{3{B^2}{F^2}}}, $$

where F is the number of periods of the fringes, representing the fringe frequency. According to Eq. (21), the reconstruction error depends on parameters $\sigma $, B and F. The variance of the system noise and the intensity modulation are dependent on the system setup. Generally, increasing the fringe frequency is the most practical approach to improve measurement accuracy. However, using higher frequency fringes causes more phase ambiguities in phase unwrapping. The optimal high frequency of the hybrid multi-frequency composite-pattern TPU needs to be determined to ensure reliability of the phase unwrapping, considering the effect of the intensity noise.

Using the new hybrid multi-frequency composite-pattern TPU method, the high-frequency and low-frequency phase maps are calculated using the 3-step phase-shifting algorithm, and the error variances can be expressed in the form of Eq. (20):

(22)$$\sigma _{\Delta {\varphi _h}}^2 = \frac{{2{\sigma ^2}}}{{3B_h^2}}, $$

(23)$$\sigma _{\Delta {\varphi _l}}^2 = \frac{{2{\sigma ^2}}}{{3{{(0.5{B_l})}^2}}}\textrm{ = }\frac{{8{\sigma ^2}}}{{3B_l^2}}.$$

According to Eqs. (14)–(19), the unit-frequency phase with noise can be expressed as:

(24)$$\varphi _u^\varepsilon = \frac{{2\pi }}{{3{B_l}}}({I_3} + {I_4} + {I_5} - {I_0} - {I_1} - {I_2} + \varepsilon ),$$

where $\varepsilon = {\varepsilon _3} + {\varepsilon _4} + {\varepsilon _5} - {\varepsilon _0} - {\varepsilon _1} - {\varepsilon _2}$, and auxiliary variable $\varepsilon$ is also Gaussian distributed, i.e. $\varepsilon \sim N(0,6{\sigma ^2})$. The phase error induced by the noise can be computed by the difference of the measured phase $\varphi _u^\varepsilon $ and actual phase ${\varphi _u}$:

(25)$$\Delta {\varphi _u} = \varphi _u^\varepsilon - {\varphi _u} = \frac{{2\pi }}{{3{B_l}}}\varepsilon.$$

The error variance of the unit-frequency phase can be represented as:

(26)$$\sigma _{\Delta {\varphi _u}}^2 = {\left( {\frac{{2\pi }}{{3{B_l}}}} \right)^2}6{\sigma ^2} = \frac{{8{\sigma ^2}{\pi ^2}}}{{3B_l^2}}.$$

To accurately unwrap the wrapped phase in multi-frequency TPU, the weighted phase differences ${\Delta _m}{\kern 1pt} (m = l,h)$ in Eq. (5) should not exceed $\pi $. The variance of ${\Delta _m}$ can be expressed as:

(27)$$\sigma _{{\Delta _l}}^2 = {\left( {\frac{{{F_l}}}{{{F_u}}}} \right)^2}\sigma _{\Delta {\Phi _u}}^2 + \sigma _{\Delta {\varphi _l}}^2,$$

(28)$$\sigma _{{\Delta _h}}^2 = {\left( {\frac{{{F_h}}}{{{F_l}}}} \right)^2}\sigma _{\Delta {\Phi _l}}^2 + \sigma _{\Delta {\varphi _h}}^2, $$

where $\sigma _{\Delta {\Phi _u}}^2$ and $\sigma _{\Delta {\Phi _l}}^2$ are the error variances of unit-frequency and low-frequency unwrapped phases, respectively, which are equal to the error variances of the corresponding wrapped phases [22], i.e. $\sigma _{\Delta {\Phi _u}}^2\textrm{ = }\sigma _{\Delta {\varphi _u}}^2$and $\sigma _{\Delta {\Phi _l}}^2\textrm{ = }\sigma _{\Delta {\varphi _l}}^2$. The three-sigma rule, commonly used in statistics, indicates that nearly all data are within three standard deviations of the mean. Here, the stricter 4.5-sigma limit is adopted [22] as:

(29)$$4.5\sigma _{{\Delta _m}}^2 \le {\pi ^2}, $$

Substituting Eqs. (27) and (28) into the above equation, the range of the fringe frequency (number of periods) for low-frequency and high-frequency fringe patterns can be estimated as:

(30)$${F_l} \le \frac{1}{\pi }{\left( {\frac{{{\pi^2}}}{{{{4.5}^2}}}\frac{{3B_l^2}}{{8{\sigma^2}}} - 1} \right)^{{1 / 2}}}, $$

(31)$${F_h} \le {\left( {\frac{{{\pi^2}}}{{{{4.5}^2}}}\frac{{3B_l^2}}{{8{\sigma^2}}} - \frac{{B_l^2}}{{4B_h^2}}} \right)^{{1 / 2}}}{F_l}. $$

The variance of the system noise ${\sigma ^2}$ can be determined before measurement [27]. The intensity modulation can be considered as constant when the fringe frequency is low, and ${B_l}$ can be estimated before measurement as well [27]. Although intensity modulation will have a marked decline when the fringe frequency is high because of projector defocus, the upper limit of the intensity-modulation ratio, ${{({B_l}} / {{B_h}}}{)_{\max }}$, can be obtained by calibration before measurement [28]. Therefore, Eq. (31) can be rewritten, and the range of fringe frequency for high frequency fringe patterns can be estimated as:

(32)$${F_h} \le {\left( {\frac{{{\pi^2}}}{{{{4.5}^2}}}\frac{{3B_l^2}}{{8{\sigma^2}}} - \frac{1}{4}\left( {\frac{{{B_l}}}{{{B_h}}}} \right)_{\max }^2} \right)^{{1 / 2}}}{F_l}.$$

To guarantee reliable phase unwrapping, the low frequency should meet the constraint in Eq. (30) when using the unit-frequency phase for unwrapping, and the high frequency should meet the constraint in Eq. (32) when using the low-frequency phase for unwrapping. From Eq. (21), the final measurement accuracy is dependent on the product ${B^2}{F^2}$ of the high-frequency fringe patterns. However, the measurement accuracy cannot be improved by arbitrarily increasing fringe frequency, due to the effect of projector defocus on the intensity modulation, as determined in Ref. [28]. Experiments can be performed to determine a reference fringe-frequency range, for which the product ${B^2}{F^2}$ is at least 90% of its maximum value. The final optimal high frequency can be selected in the reference fringe-frequency range while still satisfying Eq. (32).

3. Experiments and results

To verify the new hybrid multi-frequency composite-pattern TPU methods, a 3D shape measurement system was set up with a digital-light-processing (DLP) projector (Wintech PRO4500) with 912×1140 resolution, and one 3-CMOS color camera (JAI AP-1600 T) with 1456×1088 resolution and 12 mm focal length lens. While most grayscale fringe-projection methods are sensitive to gamma nonlinearity if not corrected for, the projector had no gamma nonlinearity, and no correction was needed. The projector-camera system was calibrated based on a stereovision model, and 3D surface reconstruction was performed using stereovision techniques [29,30].

3.1 Determination of optimal frequency

To determine the optimal fringe frequency, experiments were performed to determine the intensity modulation sensitivity to fringe frequency. Grayscale 3-step phase-shifted fringe patterns with fringe frequency 1, 5, 10, 20, 30, 40, 60, 80, 100, 120, 140, 160, 180, 200 and 220 periods/frame were projected onto a white flat plate at close, medium, and far distances (approximately 700 mm, 900 mm and 1100 mm, respectively) between the measurement object and the projector-camera system, and the fringe intensity modulation of each frequency was computed. The experimental results are shown in Fig. 3. The measured normalized intensity modulations $\bar{B}$ with respect to fringe frequency F at different distances are illustrated in Fig. 3(a). The intensity modulation decreased with increasing frequency at all three distances, and the upper limit of the normalized intensity modulation ratio was found to be: ${{{{\bar{B}}_l}} / {{{\bar{B}}_h}}} \le 5.4$, i.e. ${{({B_l}} / {{B_h}}}{)_{\max }} = 5.4$ in Eq. (32). The product ${\bar{B}^2}{F^2}$ monotonically increased over the full range of frequencies at the medium object-system distance, which was close to the plane of focus. ${\bar{B}^2}{F^2}$ reached a maximum at middle frequencies 126 and 158 periods/frame, at close and far distances, respectively, above which frequency the projector defocus seems to have effect. In order to determine a common optimal frequency for different object-system distances, fringe frequency ranges, for which the product ${\bar{B}^2}{F^2}$ is at least 90% of its maximum value for both close and far distances, are selected as reference fringe-frequency ranges. The reference fringe-frequency ranges for close and far distances are [99, 156] and [123, 197], respectively. The optimal high frequency can be selected in the common reference fringe-frequency range [123, 156] to achieve high-accuracy measurement.

Fig. 3. Intensity modulation tested at different distances: (a) normalized modulation $\bar{B}$, and (b) product of normalized modulation squared and frequency squared, ${\bar{B}^2}{F^2}$.

Download Full Size | PDF

In the new hybrid multi-frequency composite-pattern TPU methods, the range of the low and high frequency fringes to guarantee reliable phase unwrapping can be estimated using the parameters of system noise variance ${\sigma ^2}$ and intensity modulation ${B_l}$. ${\sigma ^2}$ was determined to be 2.62 by averaging the temporal fluctuation of each pixel. ${B_l}$ can be considered as constant when the frequency is low. ${B_l}$ was estimated to be uniform ${B_l} \approx 88.85$ using 3-step phase-shifting measurement on a white flat plate. Substituting the parameters, ${\sigma ^2}$, ${B_l}$ and ${{({B_l}} / {{B_h}}}{)_{\max }}$, into Eqs. (30) and (32), the maximum number of periods (fringe frequency) of low and high frequency to guarantee reliable phase unwrapping was estimated to be 7 and 163. To improve the reliability of the phase unwrapping using the limited intensity of the unit-frequency patterns, the low frequency was selected to be 6, and the corresponding maximum high frequency 140. The maximum high frequency, 140, is within the reference range [123, 156], and can be selected as the final optimal frequency. Based on the system noise analysis, the new hybrid multi-frequency composite-pattern TPU methods should still achieve high-accuracy measurement, even if some intensity amplitude is sacrificed. An experiment was conducted to demonstrate the system measurement accuracy with the optimal frequency 140 periods/frame in comparison to other frequencies. Measurements on a white flat plate were performed using the unit frequency and low frequency with 1 and 6 periods/frame, respectively, and high frequencies 90, 115, 140, 163 periods/frame, respectively. The mean absolute error (MAE) and root mean square (RMS) error are compared in Table 1 for the different high frequencies. The high frequency with 140 periods/frame had the smallest MAE and RMS error. In the following section, object-surface measurement experiments were performed using the new methods, with the frequencies of the unit-frequency, low-frequency and high-frequency fringe patterns: 1, 6, and 140 periods/frame, respectively.

Table 1. Measurement accuracy results for different high frequencies.

View Table | View all tables in this article

3.2 Measurement accuracy evaluation

In order to evaluate the performance of the new grayscale and color hybrid multi-frequency composite-pattern TPU methods, experiments were performed on a double-hemisphere object (Fig. 4(a)), which had true radii 50.800 ± 0.015 mm and distance between centers 120.000 ± 0.005 mm. For the grayscale hybrid multi-frequency composite-pattern TPU method, which will be called grayscale hybrid method for brevity in the remainder of this paper, six grayscale patterns, designed using Eqs. (6) and (7), were projected onto the object sequentially. One of the captured high-frequency fringe patterns and one of the composite fringe patterns are shown in Figs. 4(b) and (c), respectively. The unit-frequency phase map, and low-frequency and high-frequency wrapped phase maps were computed using Eqs. (10)–(18), and are shown in Figs. 5(a)–(c), respectively. The unwrapped phase map of the high-frequency fringe patterns was obtained using multi-frequency TPU (Eqs. (4) and (5)), and is shown in Fig. 5(d).

Fig. 4. Captured images of double-hemisphere using grayscale hybrid method: (a) texture image, (b) one captured high-frequency fringe pattern, and (c) one captured composite fringe pattern.

Download Full Size | PDF

Fig. 5. Phase maps for grayscale hybrid method: (a) unit-frequency phase, (b) low-frequency wrapped phase, (c) high-frequency wrapped phase, and (d) high-frequency unwrapped phase.

Download Full Size | PDF

For the color hybrid multi-frequency composite-pattern TPU method, which will be called color hybrid method for brevity in the remainder of this paper, three color encoded patterns (Eq. (6) encoded in the R channel, Eq. (7) encoded in the B channel) were sequentially projected and captured. One of the captured color images is shown in Fig. 6(a). It should be noted that there is no spectral overlap between the red and blue channels (as specified in the camera datasheet [31]), and the color imbalance of the red and blue channels can be reduced by setting a suitable current for the red and blue LEDs of the DLP projector in advance of measurements. The encoded fringe patterns can be spectrally separated and extracted from the RB channels of the captured color images, without requiring calibration for color crosstalk. The extracted fringe patterns were directly used for phase computation. The phase maps of unit-frequency phase, low-frequency and high-frequency wrapped phase, and high-frequency unwrapped phase were computed similarly to the grayscale hybrid method, and are shown in Figs. 6(b)–(e), respectively. The high-frequency unwrapped phase maps using both grayscale and color hybrid methods have some errors at high gradient regions close to the edges of the hemisphere, where the low intensity modulation makes the phase maps more sensitive to system noise, as commonly found [32]. For the color hybrid method, the residual color imbalance between the red and blue channels may cause additional phase error.

Fig. 6. Measurement results for double-hemisphere using color hybrid method: (a) one captured color image; and phase maps: (b) unit-frequency phase, (c) low-frequency wrapped phase, (d) high-frequency wrapped phase, and (e) high-frequency unwrapped phase.

Download Full Size | PDF

To evaluate the measurement accuracy of the new grayscale and color hybrid multi-frequency composite-pattern TPU methods, the 3D shape was reconstructed using the unwrapped phase maps. For comparison, measurements of the object were also performed using the conventional multi-frequency TPU (9 frames) and conventional dual-frequency TPU (6 frames) using the same experimental setup. The frequencies of the unit-frequency, low-frequency, and high-frequency fringe patterns for the conventional multi-frequency method were 1, 6, and 140 periods/frame, respectively; and the frequencies of unit-frequency and high-frequency fringe patterns for the conventional dual-frequency method were 1 and 140 periods/frame, respectively.

The 3D shape reconstructions of the four methods are shown in Figs. 7(a)–(d), respectively. Reconstructions for the new grayscale hybrid, new color hybrid, and conventional multi-frequency TPU methods were successful Figs. 7(a)–(c); however, reconstruction using the conventional dual-frequency method failed (Fig. 7(d)), because of an inability unwrap the phase map with a frequency as high as 140 periods/frame. The measurement accuracy of the new grayscale hybrid, new color hybrid, and conventional multi-frequency TPU methods was computed by least-square fitting of a sphere to each hemisphere point cloud. The corresponding sphere-fitting residual errors are shown in Figs. 7(e)–(g), respectively. The residual errors of the new grayscale and color hybrid methods were as low as the conventional multi-frequency TPU method, mostly under 0.1 mm. The computed radii of the two hemispheres, distance between hemisphere centers, MAE, RMS error, and sphere-fitting standard deviation (SD) are compared in Table 2 for the three measurement methods. The new grayscale hybrid method performed as well as the conventional multi-frequency TPU method. The new color hybrid method had slightly higher errors than the other two methods, mainly due to the color imbalance between the red and blue channels. The new grayscale and color hybrid multi-frequency TPU methods both have high measurement accuracy with the advantage of using fewer fringe patterns (6 and 3, respectively) compared to the conventional multi-frequency TPU method, which uses 9 patterns.

Fig. 7. Measurement results for the double-hemisphere: 3D shape reconstruction using (a) grayscale hybrid method, (b) color hybrid method, (c) conventional multi-frequency TPU method and (d) conventional dual-frequency TPU method; (e)–(g) corresponding sphere-fitting residual errors for (a)–(c), respectively.

Download Full Size | PDF

Table 2. Comparison of experimental results for the new grayscale hybrid method, new color hybrid method and conventional multi-frequency TPU method.

View Table | View all tables in this article

3.3 Multiple isolated-object measurement

Fig. 8. Measurement results of two isolated objects: (a) texture images of a cylinder and stepped block, (b) one captured high-frequency fringe image using grayscale hybrid method, (c) one composite fringe image using grayscale hybrid method, (d) one captured color image using color hybrid method; and 3D shape reconstruction using: (e) grayscale hybrid method, and (f) color hybrid method.

Download Full Size | PDF

To verify the ability of the new grayscale and color hybrid multi-frequency TPU methods to measure spatially isolated discontinuous surfaces, measurements were performed simultaneously on two spatially isolated objects, a cylinder and stepped block as shown in Fig. 8(a), using the two methods, respectively. For the grayscale hybrid method, one of the captured high-frequency fringe images is shown in Fig. 8(b) and one of the composite fringe images is shown in Fig. 8(c); one of the captured color images using the color hybrid method is shown in Fig. 8(d). The 3D shape reconstructions using the new grayscale and color hybrid methods are illustrated in Figs. 8(e) and (f), respectively. The few white points on the reconstructed surface of the stepped block in Fig. 8(f) were invalid (outlier) points, possibly due to color imbalance. The experimental results show that the new grayscale and color hybrid multi-frequency composite-pattern TPU methods permit measurement of discontinuous surfaces of isolated objects. The grayscale hybrid method performed slightly better than the color hybrid method.

3.4 Complex object measurement

To demonstrate the ability of the two new methods to measure complex geometry, measurements were conducted on a manikin head, as shown in Fig. 9(a). One of the captured high-frequency fringe images and one of the composite fringe images using the grayscale hybrid method are shown in Figs. 9(b) and (c), respectively. One of the captured color images using the color hybrid method is shown in Fig. 9(d). The 3D reconstructions using the grayscale and color hybrid methods are shown in Figs. 9(e) and (f), respectively. The 3D shape was reconstructed successfully using the grayscale hybrid method, with invalid points due to occlusion seen in the white region near the ear. Using the color hybrid method, some low-intensity areas failed to reconstruct, which is consistent with the results in Sec. 3.3.

Fig. 9. Measurement results of a complex object: (a) texture image of a manikin head, (b) one captured high-frequency fringe image using grayscale hybrid method, (c) one composite fringe image using grayscale hybrid method, (d) one captured color image using the color hybrid method; and 3D shape reconstruction using: (e) grayscale hybrid method, and (f) color hybrid method.

Download Full Size | PDF

4. Conclusion

A new hybrid multi-frequency composite-pattern TPU method was developed for high-accuracy 3D shape measurement of objects with surface discontinuities and multiple isolated objects, using fewer patterns than conventional TPU methods. The number of fringe patterns required for 3D shape measurement is reduced to six using the new grayscale hybrid multi-frequency TPU method, compared to nine patterns using conventional TPU. By using color fringe patterns in the color hybrid multi-frequency TPU method, only three patterns are required for measurement compared to nine patterns using conventional TPU. The optimal high frequency was determined for high measurement accuracy and reliable phase unwrapping, based on analysis of the effect of temporal intensity noise on the phase error. Experimental results demonstrated that high measurement accuracy could be achieved using both the new grayscale and color hybrid multi-frequency composite-pattern TPU methods. Compared to the conventional multi-frequency TPU method, the new grayscale hybrid method has similar measurement accuracy, but with the advantage of fewer patterns required for measurement using the new method. The new color hybrid method had slightly lower measurement accuracy but with a greater advantage of requiring only three patterns compared to nine. The color hybrid method may have some limitation in measuring colored surfaces, since the surface reflectivity will affect the captured intensity in the red and blue channels. The use of color images for measuring colored surfaces would need further investigation.

Funding

Natural Sciences and Engineering Research Council of Canada; Special Grand National Project of China (2009ZX02204-008); China Scholarship Council.

Disclosures

The authors declare no conflict of interest.

References

1. D. Bak, “Rapid prototyping or rapid production? 3D printing processes move industry towards the latter,” Assembly Autom. 23(4), 340–345 (2003). [CrossRef]

2. P. Garbat and M. Kujawińska, “Combining fringe projection method of 3D object monitoring with virtual reality environment: concept and initial results,” in Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission (IEEE, 2002), pp. 504–508.

3. C. P. Honrado and W. F. Larrabee, “Update in three-dimensional imaging in facial plastic surgery,” Curr. Opin. Otolaryngol. 12(4), 327–331 (2004). [CrossRef]

4. G. Sansoni, M. Trebeschi, and F. Docchio, “State-of-the-art and applications of 3D imaging sensors in industry, cultural heritage, medicine, and criminal investigation,” Sensors 9(1), 568–601 (2009). [CrossRef]

5. S. S. Gorthi and P. Rastogi, “Fringe projection techniques: whither we are?” Opt. Lasers Eng. 48(2), 133–140 (2010). [CrossRef]

6. C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Opt. Lasers Eng. 109, 23–59 (2018). [CrossRef]

7. X. Su and W. Chen, “Reliability-guided phase unwrapping algorithm: a review,” Opt. Lasers Eng. 42(3), 245–261 (2004). [CrossRef]

8. M. Zhao, L. Huang, Q. Zhang, X. Su, A. Asundi, and K. Qian, “Quality-guided phase unwrapping technique: comparison of quality maps and guiding strategies,” Appl. Opt. 50(33), 6214–6224 (2011). [CrossRef]

9. Z. Li, K. Zhong, Y. F. Li, X. Zhou, and Y. Shi, “Multiview phase shifting: a full-resolution and high-speed 3D measurement framework for arbitrary shape dynamic objects,” Opt. Lett. 38(9), 1389–1391 (2013). [CrossRef]

10. Y. An, J. S. Hyun, and S. Zhang, “Pixel-wise absolute phase unwrapping using geometric constraints of structured light system,” Opt. Express 24(16), 18445–18459 (2016). [CrossRef]

11. X. Liu and J. Kofman, “High-frequency background modulation fringe patterns based on a fringe-wavelength geometry-constraint model for 3D surface-shape measurement,” Opt. Express 25(14), 16618–16628 (2017). [CrossRef]

12. Y. Wan, Y. Cao, X. Liu, T. Tao, and J. Kofman, “High-frequency color-encoded fringe-projection profilometry based on geometry constraint for large depth range,” Opt. Express 28(9), 13043–13058 (2020). [CrossRef]

13. X. He, D. Zheng, K. Qian, and G. Christopoulos, “Quaternary gray-code phase unwrapping for binary fringe projection profilometry,” Opt. Lasers Eng. 121, 358–368 (2019). [CrossRef]

14. S. Zhang, “Absolute phase retrieval methods for digital fringe projection profilometry: A review,” Opt. Lasers Eng. 107, 28–37 (2018). [CrossRef]

15. C. Guan, L. G. Hassebrook, and D. L. Lau, “Composite structured light pattern for three-dimensional video,” Opt. Express 11(5), 406–417 (2003). [CrossRef]

16. P. Wissmann, R. Schmitt, and F. Forster, “Fast and accurate 3D scanning using coded phase shifting and high speed pattern projection,” In Proceedings of IEEE Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission (IEEE, 2011), pp. 108–115.

17. Y. Wang, K. Liu, Q. Hao, D. L. Lau, and L. G. Hassebrook, “Period coded phase shifting strategy for real-time 3-D structured light illumination,” IEEE Trans. Image Process. 20(11), 3001–3013 (2011). [CrossRef]

18. S. Yang, G. Wu, Y. Wu, J. Yan, H. Luo, Y. Zhang, and F. Liu, “High-accuracy high-speed unconstrained fringe projection profilometry of 3D measurement,” Opt. Laser Technol. 125, 106063 (2020). [CrossRef]

19. Y. Wang and S. Zhang, “Novel phase-coding method for absolute phase retrieval,” Opt. Lett. 37(11), 2067–2069 (2012). [CrossRef]

20. J. Tian, X. Peng, and X. Zhao, “A generalized temporal phase unwrapping algorithm for three-dimensional profilometry,” Opt. Lasers Eng. 46(4), 336–342 (2008). [CrossRef]

21. H. Zhao, W. Chen, and Y. Tan, “Phase-unwrapping algorithm for the measurement of three-dimensional object shapes,” Appl. Opt. 33(20), 4497–4500 (1994). [CrossRef]

22. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]

23. K. Liu, Y. Wang, D. L. Lau, Q. Hao, and L. G. Hassebrook, “Dual-frequency pattern scheme for high-speed 3-D shape measurement,” Opt. Express 18(5), 5229–5244 (2010). [CrossRef]

24. C. Zuo, Q. Chen, G. Gu, S. Feng, F. Feng, R. Li, and G. Shen, “High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection,” Opt. Lasers Eng. 51(8), 953–960 (2013). [CrossRef]

25. C. Zuo, Q. Chen, G. Gu, S. Feng, and F. Feng, “High-speed three-dimensional profilometry for multiple objects with complex shapes,” Opt. Express 20(17), 19493–19510 (2012). [CrossRef]

26. V. Srinivasan, H. C. Liu, and M. Halioua, “Automated phase-measuring profilometry of 3-D diffuse objects,” Appl. Opt. 23(18), 3105–3108 (1984). [CrossRef]

27. J. Li, L. G. Hassebrook, and C. Guan, “Optimized two-frequency phase-measuring-profilometry light-sensor temporal-noise sensitivity,” J. Opt. Soc. Am. A 20(1), 106–115 (2003). [CrossRef]

28. M. Zhang, Q. Chen, T. Tao, S. Feng, Y. Hu, H. Li, and C. Zuo, “Robust and efficient multi-frequency temporal phase unwrapping: optimal fringe frequency and pattern sequence selection,” Opt. Express 25(17), 20381–20400 (2017). [CrossRef]

29. Z. Li, Y. Shi, C. Wang, and Y. Wang, “Accurate calibration method for a structured light system,” Opt. Eng. 47(5), 053604 (2008). [CrossRef]

30. Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Machine Intell. 22(11), 1330–1334 (2000). [CrossRef]

31. https://www.1stvision.com/cameras/JAI/dataman/Datasheet_AP-1600T-USB.pdf

32. C. Zhang, H. Zhao, and L. Zhang, “Fringe order error in multifrequency fringe projection phase unwrapping: reason and correction,” Appl. Opt. 54(32), 9390–9399 (2015). [CrossRef]

High frequency (periods/frame)	90	115	140	163
MAE (mm)	0.029	0.024	0.022	0.054
RMS error (mm)	0.036	0.031	0.028	0.064

	New grayscale hybrid method	New color hybrid method	Conventional multi-frequency TPU method
Radii of two spheres (mm)	[50.823, 50.811]	[50.828, 50.842]	[50.822, 50.813]
Distance between hemisphere centers (mm)	120.114	120.196	120.115
MAE (mm)	[0.029, 0.023]	[0.038, 0.046]	[0.029, 0.024]
RMS error (mm)	[0.036, 0.029]	[0.046, 0.055]	[0.036, 0.030]
Sphere-fitting SD (mm)	[0.017, 0.016]	[0.023, 0.021]	[0.017, 0.016]
Number of images	6	3	9

High frequency (periods/frame)	90	115	140	163
MAE (mm)	0.029	0.024	0.022	0.054
RMS error (mm)	0.036	0.031	0.028	0.064

	New grayscale hybrid method	New color hybrid method	Conventional multi-frequency TPU method
Radii of two spheres (mm)	[50.823, 50.811]	[50.828, 50.842]	[50.822, 50.813]
Distance between hemisphere centers (mm)	120.114	120.196	120.115
MAE (mm)	[0.029, 0.023]	[0.038, 0.046]	[0.029, 0.024]
RMS error (mm)	[0.036, 0.029]	[0.046, 0.055]	[0.036, 0.030]
Sphere-fitting SD (mm)	[0.017, 0.016]	[0.023, 0.021]	[0.017, 0.016]
Number of images	6	3	9

High-accuracy 3D surface measurement using hybrid multi-frequency composite-pattern temporal phase unwrapping

Abstract

1. Introduction

2. Principle and method

2.1 Phase-shifting profilometry using conventional multi-frequency temporal phase unwrapping

2.2 New hybrid multi-frequency composite-pattern temporal phase unwrapping

2.3 Noise analysis and optimal frequency selection

3. Experiments and results

3.1 Determination of optimal frequency

3.2 Measurement accuracy evaluation

3.3 Multiple isolated-object measurement

3.4 Complex object measurement

4. Conclusion

Funding

Disclosures

References

Cited By

Figures (9)

Tables (2)

Equations (32)

Optics Express