Open Access
Add to BibSonomy
Abstract
Fringe projection profilometry (FPP) has become increasingly important in dynamic 3-D shape measurement. In FPP, it is necessary to retrieve the phase of the measured object before shape profiling. However, traditional phase retrieval techniques often require a large number of fringes, which may generate motion-induced error for dynamic objects. In this paper, a novel phase retrieval technique based on deep learning is proposed, which uses an end-to-end deep convolution neural network to transform a single or two fringes into the phase retrieval required fringes. When the object’s surface is located in a restricted depth, the presented network only requires a single fringe as the input, which otherwise requires two fringes in an unrestricted depth. The proposed phase retrieval technique is first theoretically analyzed, and then numerically and experimentally verified on its applicability for dynamic 3-D measurement.
© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Dynamic three-dimensional (3-D) has been widely used in applications, such as bio-medicine [1], reverse engineering [2], and face recognition [3], etc. A typical structured light technique, fringe projection profilometry (FPP) is often used because of its attributes of high-resolution, high-speed, and high-resolution [4], etc.
FPP first calculates the desired phase by using a phase-shifting algorithm [5–7] or transform-based algorithm [8–10]. The phase-shifting algorithm requires at least three phase-shifted sinusoidal fringes, which may generate motion-induced error [11,12] for dynamic objects. The transform-based algorithm can use a single sinusoidal fringe to calculate the desired phase, but it is difficult to preserve the object’s edges [13]. Deep learning has been recently introduced to calculate the desired phase, which can directly transform a single fringe into the desired phase, and simultaneously preserve the object’s edges [14,15].
The calculated phase requires to be unwrapped to obtain the absolute phase, because it is always wrapped in a range of $(-\pi ,\pi ]$ [16]. Therefore, a phase unwrapping process is necessary for FPP, which can be classified into two categories of spatial [17] and temporal phase unwrapping [18]. The former often fails for a complex surface because of local error propagation [19]. The latter is commonly used in real measurement, but requires a large number of fringes in forms, such as binary-coded [20], ternary-coded [21], phase-coded [22], or multi-frequency [23], etc.
The multi-frequency method was originated from laser interferometry [18], which can be directly applied to FPP by using two or multiple sets of phase-shifted sinusoidal fringes [19]. In a noise-free system, the way using two sets works well. But in a real noise system, multiple sets are required [24]. Deep learning has been introduced to reduce the required fringes [14,25], but still requires at least three fringes for successful phase unwrapping. In addition, deep learning has also been introduced to transform a single fringe into 3-D shapes, but generates a non-ignorable measurement error up to 2 mm [26]. Therefore, it has great importance to reduce the fringes, but simultaneously preserve the accuracy, especially for dynamic 3-D measurement.
Deep learning can be used in image transformation tasks [27] with pixel-level and even sub-pixel-level, such as segmentation [28], super-resolution [29], and inpainting [30], etc. In this paper, deep learning is introduced to design a fringe pattern transformation network (FPTNet), which can be trained to transform a single or two fringes into the phase retrieval required fringes, and enables a novel phase retrieval technique detailed as follows:
- (i) To calculate the desired phase, FPTNet is trained to transform a single sinusoidal fringe into one set of phase-shifted sinusoidal fringes with the same fringe frequency.
- (ii) To obtain the absolute phase, FPTNet is trained to transform a single or two sinusoidal fringes into multiple sets of phase-shifted sinusoidal fringes with different frequencies.
It should be noted that, FPTNet requires a single fringe as the input when the object’s surface is located in a restricted depth (i.e., never introducing a phase shift more than one fringe period), which otherwise requires two fringes in an unrestricted depth, as explained in Section 3.2.
Both simulations and experiments verify that: the proposed phase retrieval technique can calculate the desired phase accurately, and obtain the absolute phase correctly, which shows great potential for dynamic 3-D measurement.
The rest of the paper is organized as follows. Section 2 illustrates the traditional phase retrieval technique. Section 3 introduces the proposed phase retrieval technique using deep learning. Section 4 gives experiments. Section 5 concludes this paper.
2. Traditional phase retrieval technique
To calculate the desired phase, FPP using the phase-shifting algorithm requires one set of phase-shifted sinusoidal fringes as [31]
where $N$ denotes the number of phase steps, $f$ denotes the fringe frequency, and $I_n^f$ denotes the n-th fringe in the set of $I^f$,which can be described by where $(x,y)$ denotes the image coordinate, $a(x,y)$ denotes the background, $b(x,y)$ denotes the modulation, $\delta _n=2\pi \times (n-1)/N$ denotes the amount of phase shift, and $\varphi ^f(x,y)$ denotes the desired phase, which can be calculated by using a least-squares algorithm [32]As we know, FPP often uses sinusoidal fringes with a high frequency to achieve a high measurement accuracy [34], and thus $\Phi ^{f_s}$ is selected for the 3-D reconstruction by combining with system calibrated parameters [35]. In real measurement, seven or eight sets of sinusoidal fringes are often required to satisfy the demand of the projector’s resolution [23].
3. The proposed phase retrieval technique based on deep learning
3.1 Principle
The proposed phase retrieval technique using deep learning, first transforms a single or two sinusoidal fringes into the phase retrieval required sinusoidal fringes, and then retrieves the phase from these transformed fringes.
A fringe pattern transformation network (FPTNet) is designed for the fringe-to-fringe transformation task [36], which includes two sub-networks, fringe pattern transformation network for the phase calculation (i.e., FPTNet-C), and fringe pattern transformation network for the phase unwrapping (i.e., FPTNet-U).
It should be noted that we divide FPTNet into two sub-networks, which aim to explain it from the two processes of phase calculation and unwrapping clearly. Actually, only a network is required to achieve phase retrieval.
FPTNet-C is trained to transform a single sinusoidal fringe into one set of phase-shifted sinusoidal fringes with the same fringe frequency. In FPTNet-U, when the object’s surface does not introduce a phase shift more than one fringe period, FPTNet-U I is trained to transform one sinusoidal fringe into multiple sets of sinusoidal fringes with different frequencies. Otherwise, FPTNet-U II is trained to transform two sinusoidal fringes into multiple sets of sinusoidal fringes with different frequencies.
FPTNet includes two steps of training and testing. In the training step, FPTNet is trained to learn the transformation from the input to the output by minimizing the difference between the outputted fringes and the actual fringes captured by the camera. This difference can be described by a loss function [37], and the actual fringes are the ground-truth of FPTNet. In the testing step, the trained FPTNet directly outputs the desired fringes from the input [38].
Taking FPP using $s$ sets as an example, we detail the proposed phase retrieval technique as follows. FPTNet outputs $s$ sets of phase-shifted sinusoidal fringes with the input of a single or two fringes selected from $I^{f_i},i=1,2,\ldots ,s$. For convenience, the frequency of the outputted fringes is denoted by $f^{out}$, the frequency of the output of the i-th set can be denoted by $f_i^{out}$. Therefore, the outputted $s$ sets of fringes can be denoted by $I^{f_i^{out}},i=1,2,\ldots ,s$, where the i-th set can be described by $I^{f_i^{out}}=I_n^{f_i^{out}},n=1,2,\ldots ,N$.
For clarity, the diagram of the proposed phase retrieval technique is provided in Fig. 1. FPTNet-C outputs $I_n^{f_i^{out}},n=1,2,\ldots ,N$ with the input of $I_1^{f_s}$. By substituting $I_n^{f_s}$ with $I_n^{f_s^{out}}$ in Eq. (3), the desired phase of $\varphi ^{f_s^{out}}$ can be calculated.
FPTNet-U I outputs $I^{f_i^{out}},i=1,2,\ldots ,s-1$ with the input of $I_1^{f_s}$, and FPTNet-U II outputs $I^{f_i^{out}},i=1,2,\ldots ,s-1$ with the input of $I_1^{f_s}$ and $I_1^{f_l}$ (Note: $f_l < f_s$). By reusing Eq. (3) $s-1$ times, $\varphi ^{f_i^{out}},i=1,2,\ldots ,s-1$ can be calculated. Therefore, $\Phi ^{f_i^{out}},i=1,2,\ldots ,s$ can be obtained by using Eq. (5). It should be noted that, $I^{f_i^{out}}$ may has a learning difference from $I^{f_i}$. This difference can be described as $\Delta I^{f_i^{out}}=I^{f_i^{out}}-I^{f_i}$, and results in the phase error as $\Delta \varphi ^{f_i^{out}}=\varphi ^{f_i^{out}}-\varphi ^{f_i}$ , which may generate wrong fringe order as $\Delta K^{f_i^{out}}=K^{f_i^{out}}-K^{f_i}$. According to Eq. (5), $\Delta K^{f_i^{out}}$ can be described by [39]
Table 1. The layer disposal of FPTNet-C. Output sizes are given for an example input of 512 $\times$ 256
It should be noted that FPTNet-U has the same setting as FPTNet-C. In FPTNet, we take the outputted fringes with the frequency of $f_i^{out}$ as an example, the loss function can be described as
where $\theta _1$ denotes the parameter space in the network, $m$ denotes the number of pixels and $\left \|\cdot \right \|^2$ is 2-norm.3.2 Type selection strategy for FPTNet-U
In real, FPTNet-U I and FPTNet-U II can be flexibly selected according to the depth of the object’s surface, as illustrated in Fig. 2. Figure 2(a) gives the FPP system, where $E_{p1}-E_{p2}$ and $E_{c1}-E_{c2}$ represent the optical axis of a projector lens and a camera lens, respectively;, $(x,y,z)$ denotes the world coordinate system. $d$ is the distance between the projector system and the camera system, $L$ is the measurement range, $l$ is the distance between the camera system and the reference plane, and the depth of $z=0$ corresponds to the reference plane [9].
Fig. 2. The diagram of type selection strategy for FPTNet-U. (a) The FPP system with three object’s surface points of A, B and C. (b) The wrapped phase and absolute phase corresponding to the points of A, B and C.
In the measurement range, five planes are selected with depths of $z=h_0,p_1,h_1,p_2,h_2$, respectively;, which introduce phase shifts of $\Delta \varphi ,2\pi ,2\pi +\Delta \varphi ,4\pi ,4\pi +\Delta \varphi$ to the reference plane, respectively. The object’s surface may be located in any plane due to the object’s dynamic movement. Along the camera optical axis, three object’s surface points of $A$, $B$ and $C$ are selected from the above three planes with depths of $z=h_0,h_1,h_2$, respectively. It can be assumed that the selected three surface reflect the projected light to the same camera image pixel of $(x_0^c,y_0^c)$.
As shown in Fig. 2(b), we have $\varphi _A\left (x_0^c,y_0^c \right )=\varphi _B\left (x_0^c,y_0^c \right )=\varphi _C\left (x_0^c,y_0^c \right )$ and $\Phi _A\left (x_0^c,y_0^c \right )=\Phi _B\left (x_0^c,y_0^c \right )-2\pi =\Phi _C\left (x_0^c,y_0^c \right )-4\pi$. It is apparent that: (i) when the object’s surface is located in the depth range of $(0,p_1)$, the calculated phase of each pixel corresponds to a unique absolute phase; (ii) when the object’s surface is located in the depth range of $(p_1,L)$, the calculated phase of any pixel may correspond to multiple absolute phases. When one pixel corresponds to multiple absolute phases, FPTNet-U I cannot differentiate them, and may lead to a wrong fringe transformation, where FPTNet-U II is required.
As aforementioned, the selection strategy for FPTNet-U is given as follows:
- (i) When the object’s surface does not introduce a phase shift more than $2\pi$ (i.e., one fringe period), FPTNet-U I is selected with the input of $I_1^{f_s}$.
- (ii) When the object’s surface introduces a phase shift more than $2\pi$, FPTNet-U II is selected with the input of $I_1^{f_s}$ and $I_1^{f_l}$.
The highest frequency of $f_s$ can be selected according to the projector resolution [42]. According to the following simulation, and experiments provided in Section 4, the lower frequency of $f_l$ can be selected as follows:
- (i) $f_l$ can be selected as any frequency that does not a common divisor of $f_s$, e.g., $f_s=64,f_l=45$.
- (ii) $f_l$ can also be selected as a common divisor of $f_s$, when the object’s surface does not introduce a phase shift more than $2\pi$ for the fringes with the frequency of $f_l$, e.g., $f_s=64,f_l=16$.
It should be noted that, when vertical fringe patterns are projected, the restricted depth range of $z_{th}$ can be determined by [43]
where $c_1$ is the projected length of the projector in the horizontal direction.A simulation is provided to verify the above selection strategy by using two simulated datasets. For simplicity, both the object and the reference plane are assumed to have the same reflectivity. The object’s surface is generated as follows: first, a random matrix is generated by randomly selecting a amplitude from the range of (5, 55); second, a Gaussian filter with the standard deviation range of (2, 125) is used to filter the generated random matrix; third, the filtered result is scaled and moved to generate the desired object’s surface of $z(x,y)$ ; finally, according to Eq. (2), one set of fringes of $I_n^f$ with a frequency of $f$ can be generated by [43,44]
where $N$ denotes the number of phase steps, and $c$ is a constant determined by the measurement system.Dataset I and dataset II are generated for FPTNet-U I and FPTNet-U II, respectively;, which are deformed by objects’ surfaces introducing phase shifts less and more than $2\pi$, respectively. According to Eq. (10), we select $c=1/35$ and $f_s=35$, and thus we have $z_{th}=6.28$. Therefore, dataset I and dataset II are constructed when objects’ surfaces are located in ranges of (0, 6.28) and (0, 25), respectively. FPTNet-U I selects the input with the frequency of $f_s=35$, and FPTNet-U II selects the input with two frequencies of $f_s=35$ and $f_l=30$. Both FPTNet-U I and FPTNet-U II output fringes with the frequency of $f^{out}=25$. The twelve-step phase-shifting algorithm is selected to calculate the desired phase.
For each dataset, the training set, validation set and testing set are generated from 500 surfaces, 100 surfaces and 50 surfaces, respectively. The resolution of these generated fringes is selected as $300\times 300$. For each surface, two sets with two frequencies of 35 and 30 are generated, respectively;, and another set with the frequency of 25 is also generated as the ground-truth. Therefore, the training set, validation set and testing set include 18000, 3600 and 1800 fringes, respectively. It should be noted that the validation set and testing set are not used in training step. The validation set is aimed to monitor during training the accuracy of the neural network on the data [14], and all the results are obtained in the testing set in this paper.
The results of FPTNet-U I are shown in Fig. 3. One surface as shown in Fig. 3(a) is selected. The corresponding input and output are shown in Fig. 3(b) and 3(c), respectively. The actual phase and the desired phase are calculated by using Eq. (3) twice. By subtracting the actual phase from the desired phase, the phase error is obtained and shown in Fig. 3(d). The phase error is generally less than 0.01 rad, which is much less than the threshold of $\pi$ given in Eq. (7). FPTNet-U I can obviously obtain the desired phase for a correct phase unwrapping.
Fig. 3. The results of FPTNet-U I verified on dataset I . (a) The surface. (b) The input. (c) The output. (d) The phase error.
To verify FPTNet-U II, one surface as shown in Fig. 4(a) is selected, and two fringes with two frequencies of 35 and 30 are selected as the input and shown in Fig. 4(b). FPTNet-U I is also tested on the same surface for the comparison, but only selects the single fringe with the frequency of 35 as the input. The results of FPTNet-U I and FPTNet-U II are provided in Fig. 4(c) and 4(d), respectively, where the left, middle, and right show the outputted fringes, the grayscale error of the 12-th outputted fringe, and the phase error, respectively. As we can see, FPTNet-U I outputs fringes with a large grayscale error, and results in a calculated phase with a phase error larger than $\pi$, which will lead to a wrong phase unwrapping according to Eq. (7). FPTNet-U II outputs fringes with a small grayscale error, and also results in a phase with a small phase error less than 0.02 rad, which will enable a correct phase unwrapping. Both FPTNet-U I and FPTNet- U II are also tested on datasets with other frequencies, and similar results can also be obtained.
Fig. 4. The results of FPTNet-U verified on dataset II. (a) The surface. (b) The input. (c) The results of FPTNet-U I. (d) The results of FPTNet-U I. For (c) and (d), each column shows the outputted fringe, the grayscale error, and the phase error, respectively.
4. Experiment
We first verify the proposed phase retrieval technique, and then provide real dynamic 3-D measurements to evaluate its performance. The experiment system includes a DLP6500 projector with a resolution of $1920\times 1080$, a Basler acA800-510um CMOS camera with a resolution of $496\times 496$, and a Computar lens of 12mm focal length. The distance between the camera system and the reference plane, the projected length of the projector in the horizontal direction, the distance between the projector system and the camera system are about $l=1000$ mm, $c_1=280$ mm, and $d=60$ mm, respectively. The above parameters can be adjusted according to different FPP systems.
To verify the proposed phase retrieval technique, a twelve-step ($N=12$) phase-shifting algorithm is also used to calculate the desired phase, and a multi-frequency ($s=7, f_i=2^{i-1},i=1,2,\ldots ,7$) phase unwrapping method is used to obtain the absolute phase. Therefore, for each frequency, the outputted set should be $I^{f_i^{out}}=I_n^{f_i^{out}},n=1,2,\ldots ,12$. For comparison, another three sets are also outputted with three frequencies of 45, 90 and 120, respectively.
Two datasets of I and II with real scenes are constructed in a restricted and an unrestricted depths to verify the accuracy, respectively. In our system, the restricted depth of $z_{th}$ and the measurement range are about 73 mm and 250 mm, respectively. Therefore, dataset I is constructed when the measured object is located in the range of (0, 73 mm). The training set, validation set and testing set of dataset I contain 50 scenes, 20 scenes and 20 scenes, respectively. Dataset II is constructed when the measured object is located in the range of (0, 250 mm). The training set, validation set and testing set of dataset II contain 150 scenes, 25 scenes and 25 scenes, respectively. For each scene, ten sets are captured as the ground-truth.
4.1 Experiments for FPTNet-C
On both dataset I and dataset II, FPTNet-C outputs $I^{f_7^{out}}$ with the input of $I_1^{f_7}$. By using Eq. (3), the desired phase of $\varphi ^{f_7^{out}}$ and the actual phase of $\varphi ^{f_7}$ can be calculated from $I^{f_7^{out}}$ and $I^{f_7}$, respectively.
The experimental results are provided in Fig. 5 and Fig. 6. In Fig. 5, the left, middle and right box show the input, the output and the desired phase, respectively. By subtracting the actual phase from the desired phase, the phase error of FPTNet-C is obtained and shown in Fig. 6(a). The mean value of the phase error is 0.05 rad, which is comparable with 0.02 rad of the random phase error obtained by using the traditional phase-shifting algorithm [45]. In Fig. 6(a), two areas containing complex and smooth textures are enlarged and reshown in Fig. 6(b) and 6(c), respectively. It is apparent that, the phase error can be up to 0.6 rad in the complex surface, but it is generally less than 0.07 rad in the smooth area.
Fig. 5. The results of FPTNet-C, and the box of each column shows the input, the output and the desired phase, respectively.
Fig. 6. (a) The phase error of FPTNet-C. (b) The corresponding enlarged detail of the red box in (a). (c) The corresponding enlarged detail of the green box in (a).
FPTNet-C is also verified on other different frequencies selected from the range of [4, 120]. The mean values of the phase error range from 0.04 rad to 0.06 rad, and FPTNet-C performs constantly well according to different frequencies, as shown in Fig. 7. For clarity, the results of the three frequencies of 4, 32 and 120 are provided in Fig. 7(a), 7(b) and 7(c), respectively;, and each column shows the input, the output, the desired phase, and the phase error.
Fig. 7. (a) The results with the frequency of 4. (b) The results with the frequency of 32. (c) The results with the frequency of 120. Each column shows the input, the output, the desired phase and the phase error, respectively.
FPTNet-C is then verified on phase-shifting algorithms with different phase steps selected from the range of [3, 12], and the fringe of $I_1^{f_7}$ is also selected as the input. The mean value of the phase error is nearly constant at around 0.04 rad. FPTNet-C also performs constantly well for phase-shifting algorithms with different phase steps. Because the accuracy of dataset I is similar to dataset II, the number of scenes is sufficient to verify FPTNet.
4.2 Experiments for FPTNet-U
FPTNet-U I is verified on dataset I, which outputs $I^{f_i^{out}},i=1,2,\ldots ,6$ with the input of $I_1^{f_7}$. Therefore, six desired phases of $\varphi ^{f_i^{out}},i=1,2,\ldots ,6$ are calculated from these outputted six sets of fringes. Seven absolute phases of $\Phi ^{f_i^{out}},i=1,2,\ldots ,7$ can be obtained from $\varphi ^{f_i^{out}},i=1,2,\ldots ,6$ and the above $\varphi ^{f_7^{out}}$ obtained by FPTNet-C. For comparison, $\Phi ^{f_i},i=1,2,\ldots ,7$ can be obtained from the actual phase of $\varphi ^{f_i},i=1,2,\ldots ,7$.
The results of FPTNet-U I are provided in Fig. 8, where the left, middle and right box show the input, the output and the absolute phase, respectively. The desired phases with different frequencies can be calculated by the same way as FPTNet-C, which also have similar accuracy as FPTNet-C. Therefore, FPTNet-U I can obtain the same correct fringe order for the phase unwrapping as the traditional multi-frequency method, which will enable a correct phase unwrapping.
Fig. 8. The results of FPTNet-U I verified on dataset I, and the box of each column shows the input, the output and the absolute phase, respectively.
FPTNet-U I is also verified on dataset II, as shown in Fig. 9. We take the outputted fringes with the frequency of $f_5^{out}=32$ as an example. For comparison, $I_3^{f_5}$ is selected as shown in Fig. 9(a), and the corresponding outputted $I_3^{f_5^{out}}$ is shown Fig. 9(b). The grayscale error between $I_3^{f_5}$ and $I_3^{f_5^{out}}$ is shown in Fig. 9(c), which generates a phase error larger than $\pi$ as shown in Fig. 9(d). According to Eq. (7), the calculated phase will lead a wrong phase unwrapping.
Fig. 9. The results for the object located in an unrestricted depth. (a) The ground-truth. (b) The output of FPTNet-U I. (c) The grayscale error. (d) The phase error.
In this condition, FPTNet-U II is necessary for dateset II. FPTNet-U II has the same output as FPTNet-U I, but selects the input of $I_1^{f_7}$ and $I_1^{f_5}$. The results of FPTNet-U II are provided in Fig. 10, where the left, middle and right box show the input, the output and the absolute phase, respectively. The desired phases calculated from the fringes transformed by FPTNet-U II also have similar accuracy as FPTNet-C, which can also enable a correct phase unwrapping.
Fig. 10. The results of FPTNet-U II verified on datasets II, and the box of each column shows the input, the output and the absolute phase, respectively.
4.3 Real dynamic experiments
In a restricted depth, an electric fan in rotation is measured. FPTNet outputs $I^{f_i^{out}},i=1,2,\ldots ,7$ with the input of $I_1^{f_7}$. Therefore, seven desired phases of $\varphi ^{f_i^{out}},i=1,2,\ldots ,7$ are calculated, and $\Phi ^{f_7^{out}}$ can be obtained for the 3-D reconstruction.
The measurement results are provided in Fig. 11. Four inputs with each including a single fringe of $I_1^{f_7}$ are selected, and their corresponding 3-D shapes are shown in Fig. 11(a) and 11(b), respectively. For clarity, the reconstructed 3-D shapes of the dynamic fan are provided in Visualization 1. The dynamic 3-D object located in a restricted depth can be measured without motion-induced errors [46].
Fig. 11. The results for a rotating electric fan located in a restricted depth. (a) The input. (b) The corresponding 3-D shapes (Visualization 1).
In an unrestricted depth, a toy with dynamic movement is measured. The measurement results are provided in Fig. 12. Four inputs with each including a single fringe of $I_1^{f_7}$ are selected, and their corresponding 3-D shapes are shown in Fig. 12(a) and 12(c), respectively. Another four inputs are selected with each including two fringes of $I_1^{f_7}$ and $I_1^{f_5}$, which are shown in Fig. 12(a) and 12(b), respectively. Their corresponding 3-D shapes are shown in Fig. 12(d). For clarity, the reconstructed 3-D shapes of the moving toy are provided in Visualization 2, where results I and results II represent the results reconstructed from a single or two fringes as the input, respectively. The dynamic 3-D object located in an unrestricted depth can also be measured.
Fig. 12. The results for a toy moving in an unrestricted depth. (a) The input of $I_1^{f_7}$. (b) The input of $I_1^{f_5}$ . (c) The corresponding 3-D shapes with a single fringe as the input. (d) The corresponding 3-D shapes with two fringes as the input (Visualization 2).
As above illustrated, FPTNet can calculate accurate wrapped phase and obtain correct absolute phase by using less fringes (i.e., a single or two) compared with the previous works [14,25], which is significant for dynamic 3-D measurement. All the experimental results are obtained from real scenes. Although different scenes may have slightly varying illumination [47,48], FPTNet still can achieve reliable dynamic 3-D measurement. It should be noted that, objects with ultra-fast movement may generate motion-induced errors, which can be resolved by increasing the fringe acquisition speed [49].
5. Conclusion
In this paper, a fringe pattern transformation network (FPTNet) is designed for the fringe-to-fringe transformation task using deep learning, which can be trained to transform a single or two sinusoidal fringes into multiple sets of phase-shifted sinusoidal fringes. A single or two fringes can be flexibly selected as the input of FPTNet according to the object located depth. Based on the FPTNet transformed fringes, the desired phase can be accurately calculated, and the absolute phase can be correctly obtained, which can reconstruct 3-D shapes without motion-induced errors for dynamic objects.
Funding
National Natural Science Foundation of China (61727802, 61971227); Key Research and Development programs in Jiangsu China (BE2018126).
Disclosures
The authors declare no conflicts of interest.
References
1. Y. Wang, J. I. Laughner, I. R. Efimov, and S. Zhang, “3d absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique,” Opt. Express 21(5), 5822–5832 (2013). [CrossRef]
2. B. Li and S. Zhang, “Novel method for measuring a dense 3d strain map of robotic flapping wings,” Meas. Sci. Technol. 29(4), 045402 (2018). [CrossRef]
3. J. Xue, Q. Zhang, C. Li, W. Lang, M. Wang, and Y. Hu, “3d face profilometry based on galvanometer scanner with infrared fringe projection in high speed,” Appl. Sci. 9(7), 1458 (2019). [CrossRef]
4. Z. Wang, D. A. Nguyen, and J. C. Barnes, “Some practical considerations in fringe projection profilometry,” Opt. Lasers Eng. 48(2), 218–225 (2010). [CrossRef]
5. S. Zhang and S.-T. Yau, “High-resolution, real-time 3d absolute coordinate measurement based on a phase-shifting method,” Opt. Express 14(7), 2644–2649 (2006). [CrossRef]
6. T. Tao, Q. Chen, J. Da, S. Feng, Y. Hu, and C. Zuo, “Real-time 3-d shape measurement with composite phase-shifting fringes and multi-view system,” Opt. Express 24(18), 20253–20269 (2016). [CrossRef]
7. B. Pan, Q. Kemao, L. Huang, and A. Asundi, “Phase error analysis and compensation for nonsinusoidal waveforms in phase-shifting digital fringe projection profilometry,” Opt. Lett. 34(4), 416–418 (2009). [CrossRef]
8. Q. Kemao, “Windowed fourier transform for fringe pattern analysis,” Appl. Opt. 43(13), 2695–2702 (2004). [CrossRef]
9. X. Su and Q. Zhang, “Dynamic 3-d shape measurement method: A review,” Opt. Lasers Eng. 48(2), 191–204 (2010). [CrossRef]
10. J. Zhong and J. Weng, “Spatial carrier-fringe pattern analysis by means of wavelet transform: wavelet transform profilometry,” Appl. Opt. 43(26), 4993–4998 (2004). [CrossRef]
11. L. Lu, Z. Jia, Y. Luan, and J. Xi, “Reconstruction of isolated moving objects with high 3d frame rate based on phase shifting profilometry,” Opt. Commun. 438, 61–66 (2019). [CrossRef]
12. S. Feng, C. Zuo, T. Tao, Y. Hu, M. Zhang, Q. Chen, and G. Gu, “Robust dynamic 3-d measurements with motion-compensated phase-shifting profilometry,” Opt. Lasers Eng. 103, 127–138 (2018). [CrossRef]
13. Q. Kemao, “Two-dimensional windowed fourier transform for fringe pattern analysis: Principles, applications and implementations,” Opt. Lasers Eng. 45(2), 304–317 (2007). [CrossRef]
14. S. Feng, C. Zuo, W. Yin, G. Gu, and Q. Chen, “Micro deep learning profilometry for high-speed 3d surface imaging,” Opt. Lasers Eng. 121, 416–427 (2019). [CrossRef]
15. S. Feng, Q. Chen, G. Gu, T. Tao, L. Zhang, Y. Hu, W. Yin, and C. Zuo, “Fringe pattern analysis using deep learning,” Adv. Photonics 1(02), 1–7 (2019). [CrossRef]
16. W. Yin, C. Zuo, S. Feng, T. Tao, Y. Hu, L. Huang, J. Ma, and Q. Chen, “High-speed three-dimensional shape measurement using geometry-constraint-based number-theoretical phase unwrapping,” Opt. Lasers Eng. 115, 21–31 (2019). [CrossRef]
17. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-d object shapes,” Appl. Opt. 22(24), 3977–3982 (1983). [CrossRef]
18. S. Zhang, “Absolute phase retrieval methods for digital fringe projection profilometry: A review,” Opt. Lasers Eng. 107, 28–37 (2018). [CrossRef]
19. C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Opt. Lasers Eng. 109, 23–59 (2018). [CrossRef]
20. G. Sansoni, M. Carocci, and R. Rodella, “Three-dimensional vision based on a combination of gray-code and phase-shift light projection: analysis and compensation of the systematic errors,” Appl. Opt. 38(31), 6565–6573 (1999). [CrossRef]
21. D. Zheng, Q. Kemao, F. Da, and H. S. Seah, “Ternary gray code-based phase unwrapping for 3d measurement using binary patterns with projector defocusing,” Appl. Opt. 56(13), 3660–3665 (2017). [CrossRef]
22. Y. Wang and S. Zhang, “Novel phase-coding method for absolute phase retrieval,” Opt. Lett. 37(11), 2067–2069 (2012). [CrossRef]
23. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]
24. J. M. Huntley and H. O. Saldner, “Shape measurement by temporal phase unwrapping: comparison of unwrapping algorithms,” Meas. Sci. Technol. 8(9), 986–992 (1997). [CrossRef]
25. W. Yin, Q. Chen, S. Feng, T. Tao, L. Huang, M. Trusiak, A. Asundi, and C. Zuo, “Temporal phase unwrapping using deep learning,” Sci. Rep. 9(1), 20175 (2019). [CrossRef]
26. H. Nguyen, H. Li, Q. Qiu, Y. Wang, and Z. Wang, “Single-shot 3d shape reconstruction using deep convolutional neural networks,” arXiv preprint arXiv:1909.07766 (2019).
27. C. Wang, C. Xu, C. Wang, and D. Tao, “Perceptual adversarial networks for image-to-image transformation,” IEEE Trans. on Image Process. 27(8), 4066–4079 (2018). [CrossRef]
28. E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell. 39(4), 640–651 (2017). [CrossRef]
29. A. Brifman, Y. Romano, and M. Elad, “Unified single-image and video super-resolution via denoising algorithms,” IEEE Trans. on Image Process. 28(12), 6063–6076 (2019). [CrossRef]
30. W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo, “Foreground-aware image inpainting,” in Proceedings of the IEEE conference on computer vision and pattern recognition (2019), pp. 5840–5848.
31. S. Zhang, “Recent progresses on real-time 3d shape measurement using digital fringe projection techniques,” Opt. Lasers Eng. 48(2), 149–158 (2010). [CrossRef]
32. D. Zheng, F. Da, Q. Kemao, and H. S. Seah, “Phase-shifting profilometry combined with gray-code patterns projection: unwrapping error removal by an adaptive median filter,” Opt. Express 25(5), 4700–4713 (2017). [CrossRef]
33. S. Zhang, “Phase unwrapping error reduction framework for a multiple-wavelength phase-shifting algorithm,” Opt. Eng. 48(10), 105601 (2009). [CrossRef]
34. D. Zheng and F. Da, “Phase coding method for absolute phase retrieval with a large number of codewords,” Opt. Express 20(22), 24139–24150 (2012). [CrossRef]
35. K. Liu, Y. Wang, D. L. Lau, Q. Hao, and L. G. Hassebrook, “Dual-frequency pattern scheme for high-speed 3-d shape measurement,” Opt. Express 18(5), 5229–5244 (2010). [CrossRef]
36. C. Wang, C. Xu, C. Wang, and D. Tao, “Perceptual adversarial networks for image-to-image transformation,” IEEE Trans. on Image Process. 27(8), 4066–4079 (2018). [CrossRef]
37. K. Yan, Y. Yu, C. Huang, L. Sui, K. Qian, and A. Asundi, “Fringe pattern denoising based on deep learning,” Opt. Commun. 437, 148–152 (2019). [CrossRef]
38. J. Shi, X. Zhu, H. Wang, L. Song, and Q. Guo, “Label enhanced and patch based deep learning for phase retrieval from single frame fringe pattern in fringe projection 3d measurement,” Opt. Express 27(20), 28929–28943 (2019). [CrossRef]
39. C. Zhang, H. Zhao, and K. Jiang, “Fringe-period selection for a multifrequency fringe-projection phase unwrapping method,” Meas. Sci. Technol. 27(8), 085204 (2016). [CrossRef]
40. E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo, “Efficient convnet for real-time semantic segmentation,” in 2017 IEEE Intelligent Vehicles Symposium (IV) (2017), pp. 1789–1794.
41. E. Romera, J. M. Alvarez, L. M. Bergasa, and R. Arroyo, “Erfnet: Efficient residual factorized convnet for real-time semantic segmentation,” IEEE Trans. Intell. Transport. Syst. 19(1), 263–272 (2018). [CrossRef]
42. C. Zuo, T. Tao, S. Feng, L. Huang, A. Asundi, and Q. Chen, “Micro fourier transform profilometry (μftp): 3d shape measurement at 10,000 frames per second,” Opt. Lasers Eng. 102, 70–91 (2018). [CrossRef]
43. Y. An, J.-S. Hyun, and S. Zhang, “Pixel-wise absolute phase unwrapping using geometric constraints of structured light system,” Opt. Express 24(16), 18445–18459 (2016). [CrossRef]
44. X. Su, W. Chen, Q. Zhang, and Y. Chao, “Dynamic 3-d shape measurement method based on ftp,” Opt. Lasers Eng. 36(1), 49–64 (2001). [CrossRef]
45. D. Zheng, F. Da, Q. Kemao, and H. S. Seah, “Phase error analysis and compensation for phase shifting profilometry with projector defocusing,” Appl. Opt. 55(21), 5721–5728 (2016). [CrossRef]
46. X. Liu, T. Tao, Y. Wan, and J. Kofman, “Real-time motion-induced-error compensation in 3d surface-shape measurement,” Opt. Express 27(18), 25265–25279 (2019). [CrossRef]
47. C. Chen, Y. Wan, and Y. Cao, “Instability of projection light source and real-time phase error correction method for phase-shifting profilometry,” Opt. Express 26(4), 4258–4270 (2018). [CrossRef]
48. Z. Zhang and E. Zhang, “Fluctuation elimination of fringe pattern by using empirical mode decomposition,” in 2013 International Conference on Optical Instruments and Technology: Optoelectronic Measurement Technology and Systems, vol. 9046 (International Society for Optics and Photonics, 2013), p. 90460.
49. S. Zhang, “High-speed 3d shape measurement with structured light methods: A review,” Opt. Lasers Eng. 106, 119–131 (2018). [CrossRef]
