Open Access
Add to BibSonomy
Abstract
Due to the nonlinearity in phase shifting profilometry (PSP) system, the captured images are often distorted with fringe harmonics, resulting in inaccurate phase map and measurement. Considering the fact that the phase error can be significantly reduced by modeling high-order fringe harmonics, this work formulates the phase extraction problem — with different frequency images and high-order fringe harmonic model — as a maximum likelihood estimation (MLE). To optimize it efficiently, we thus propose a combined-frequency phase extraction (CFPE) solution by introducing a latent phase map and incorporating the famous expectation-maximization (EM) framework. As a result, our CFPE method only needs ∼5% execution time of a high-order baseline, whilst keeps the high-order accuracy. Tested on synthetic images as well as practical measurements, our CFPE method demonstrated its performance improvement of efficiency and accuracy. In addition, our detailed implementation with experimental arrangement is also provided for interested researchers.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Phase shifting profilometry (PSP) is a popular technique for three-dimensional (3D) object shape measurement by projecting predefined sinusoidal fringe patterns onto to the surface [1–4]. As a widely-recognized cause of PSP error, the luminance nonlinearity of a typical PSP system must be addressed to achieve accurate measurement [5–8]. However, the nonlinearity could occur everywhere along signal transmission path, including sinusoidal pattern generation (quantization), projector nonlinear response and defocusing, reflection of surface, and photoelectric conversion in camera, etc. Obviously, the nonlinearity includes static component and dynamic component, i.e., the nonlinearity could be varied with different objects under different lighting conditions.
To reduce the nonlinear error of PSP, previous researchers have made a lot of additional efforts from system calibration to phase correction, as illustrated in Fig. 1. One intuitive class is to employ photometric calibration to identify the nonlinear relationship between ideal sinusoid fringe pattern $U$ and recorded images $I$, $I=M(U)$, where the nonlinear intensity modulation $M(\cdot )$ is often modeled as a power function, polynomial, piece-wise linear function, or spline curve [9,10]. Given the nonlinear modulation function $M(\cdot )$, two policies could be used — pre-encoding the fringe patterns with the inverse nonlinear function $\hat {U} = M^{-1}(U)$ [5,6,11] or image pre-processing with inverse modulation $\hat {I} = M^{-1}(I)$. Similarly, the measured phase and phase-error $(\phi, \Delta \phi )$ also can be calibrated, and thus enable phase error compensation with corresponding look-up table (LUT) [12]. To deal with the case-varying dynamic nonlinearity, a online calibration method is proposed with a modified multi-frequency configuration [13]. Totally speaking, the calibration based methods could significantly reduce the phase error from static nonlinearity, but they are vulnerable for dynamic nonlinearity. That says, for some applications, the case-varying nonlinear curve might be difficult to obtain or even not possible. Without calibration information, the contrast enhancement techniques (blind inverse gamma correction [14,15], histogram equalization [16]) have also been adopted to fringe images or phase maps. As additive remedies, those phase error reduction approaches (photometric calibration, image pre-processing, phase correction) could improve the PSP accuracy more or less.
Fig. 2. From top to bottom: the fringe images with harmonics, wrapped phase $(\psi )$, unwrapped phase $(\phi )$ of standard PE method, and the result of our CFPE method.
Without extra additive operation for error reduction, robust phase extraction methods target accurate phase map from fringe images by taking the high-order harmonic model into consideration. The behind motivation is that the $s-$step phase-shifting scheme can be employed to retrieve phase accurately from fringe images with nonlinear harmonics up to $(s-2)th$ order [5,17]. That says, with more available fringe images, the standard phase extraction method as well as other variants [18,19] can improve the accuracy significantly. However, the only drawback of large phase-shifting steps is the increasing time consumption for capturing more fringe images. A deep learning framework (Tree-Net) constructs six step phase-shifting patterns from three-step, and reduces phase error by about 90% [20]. Meanwhile, multi-frequency technique [17,21] —used to unwrap the phases (determine the absolute fringe orders) — provides extra ready fringe images.
Without surprise, an accurate phase extraction method considering multi-frequency images is implemented with linearization and least-square optimization [22], and termed as LLS for convenience. The LLS method is demonstrated with an impressive accuracy improvement due to the explicit employment of high-order harmonic model. And the authors found under-determined equation causes several pitfall points, and addressed it with interpolation-base post-processing. Reported $\sim 10s$ for $590\times 510$ pixels, the computational cost might be a practical obstacle for time-efficient applications.
Inspired by the success of LLS method, we propose a combined-frequency phase extraction (CFPE) method for efficient computing. Specifically, our CFPE rigorously formulates the phase extraction problem as a maximum likelihood estimation (MLE), which infers high-order nonlinear harmonic by using all available fringe images. To address the optimization challenge, an efficient expectation-maximization (EM) algorithm is derived by introducing a latent phase map. The contributions of this work:
- 1. Considering high-order harmonics, phase extraction of multi-frequency PSP is formulated as a MLE problem. And it provides a new regression perspective to understand the pitfall points phenomena as overfitting.
- 2. With the help of latent phase map, our CFPE solution — an expectation-maximization method— provides a fast iterative optimization.
- 3. Verified on a variety of synthetic nonlinear images and four practical PSP measurements. Compared to LLS, our CFPE method only needs $\sim \,5\%$ execution time while maintains the high-order accuracy of LLS.
The rest of this paper is arranged as follows. Related works about harmonics model and phase extraction preliminaries are given in Section 2. Section 3 details the proposed CFPE method from problem formulation to algorithm implementation. Section 4 comprehensively evaluates our algorithm on both synthetic images and practical measurement cases. Finally, Section 5 concludes this work with a few remarks.
2. Related works
2.1 Harmonics model
We begin with the theoretical sinusoidal fringe patterns $I^t_{i,j}(\boldsymbol {x})$ in multi-frequency PSP [3],
To depict the nonlinearity, the standard harmonics model is employed [5],
Because it is a pixel-wise model, Eq. (2) can be simplified by omitting $\boldsymbol {x}$, and redefine it as
2.2 Phase extraction and unwrapping
Given the captured images $I_{i,j}$, standard phase extraction becomes an optimization problem [5]. For the images with a same frequency,
3. Combined-frequency phase extraction method
To accurately estimate phase map, the phase extraction with combined-frequency images is firstly formulated as a regression problem that considers the nonlinear high-order harmonic. As this regression problem is equal to a standard maximum likelihood estimation (MLE), an iterative solution is proposed by incorporating latent variables (latent phase map) and expectation–aximization framework, resulting in our efficient combined-frequency phase extraction (CFPE) algorithm. The regression perspective also gains the insight of pitfall points issue as overfitting. Besides, the convergence of CFPE is also discussed in this part.
3.1 Problem formulation
The standard phase extraction (Eq. (4)) can not directly extend to high order harmonic model $(r=s-1)$, because it becomes an ill-posed problem ($r+2$ unknown variables, $r+1$ equations for a pixel) for individual frequency. Observed that different frequencies share the same variables $\{b_k\}_{k=0}^r$, we thus enables high order harmonic model by jointly optimizing multi-frequency phases. Besides, considering the phase relationship between frequencies [22], we thus formulate the combined-frequency phase extraction as a constrained optimization problem with a sum of squared difference objective,
Viewed in regression perspective, this objective takes more data (observed images) to fit a more accurate model— high order harmonic function $(h_{i,j}(\cdot ))$. Because the period ratio $\alpha _i \overset {def}= {T_1}/{T_i}={\phi _i}/{\phi _1}$ is a constant value related to the PSP configuration, the constrained optimization (Eq. (6)) can be turned into a unconstrained problem with objective $\mathcal {J}(\cdot )$,
Note that the idea of combined-frequency phase extraction is initially proposed by Jiang [22], and the optimization formulation is rigorously provided here. Due to this explicit formulation, we can adopt more efficient algorithm to obtain the solution, as detailed in the next.
3.2 Optimization via expectation maximization
For the simplicity and consistency of notation [25], let $\boldsymbol {\theta }=\{\phi _1, \theta \}$ and $\boldsymbol {X}=\{I_{i,j}\}$, the original problem (Eq. (8)) is equivalent to a maximum likelihood estimation in Bayesian probability,
with the likelihood function $p(\boldsymbol {X}|\boldsymbol {\theta }) = e^{-\mathcal {J}(\boldsymbol {\theta })}$. And $\ln (\cdot )$ denotes the natural logarithm. According to probability theory, the distribution can be treated as a marginalized distribution with arbitrary latent variable $z$, i.e., $p(\boldsymbol {X}|\boldsymbol {\theta }) = \int p(\boldsymbol {X},z|\boldsymbol {\theta })dz= \int p(\boldsymbol {X}|z,\boldsymbol {\theta })p(z|\boldsymbol {\theta })dz$. Our insight is that a deliberately constructed joint distribution —agrees with likelihood function $p(\boldsymbol {X}|\boldsymbol {\theta })$ — could significantly simplify the optimization procedure (the M-Step). Our deliberately constructed joint distribution is,Due to $\tilde {h}_{i,j}(\phi _1,\phi _1,\theta )=h_{i,j}(\phi _1,\theta )$, the original MLE problem (Eq. (9)) equivalently becomes $\boldsymbol {\theta }^* = \arg \max \ln \int p(\boldsymbol {X}|z,\boldsymbol {\theta })p(z|\boldsymbol {\theta })dz$ with latent phase map $z$. Recall a high order harmonic model $(r=s-1)$ is considered here, the $b_r$ is treated separately for later optimization.
To optimize the MLE problem with latent variables, we follow the standard EM procedure [25,26], and obtain the complete-data log likelihood,
Because the cross entropy between different delta distributions—$[\int p(z|\boldsymbol {\theta }^{old}) \ln p(z|\boldsymbol {\theta })dz]$—can be treated as a constant, the M-step (Eq. (15)) can be simplified by omitting it.
Similar to [5], we let $\frac {\partial \mathcal {L}}{\partial A_1}=0$, $\frac {\partial \mathcal {L}}{\partial B_1}=0$, $\frac {\partial \mathcal {L}}{\partial b_r}=0$, then obtain a closed-form solution, $\phi _1^{new}-z = -\tan ^{-1}(\frac {A_1}{B_1})$. That says,
Combine the E-step (Eq. (14)) and M-step (Eq. (17)), we finally obtain a concise iterative solution,
As an EM algorithm, this solution converges quickly, and the maximum number of iteration $(MaxIter)$ is set to $6$. For each iteration, only the desired phase map $\phi _1$ is updated with efficient closed-form equation (Eq. (19)), without considering other parameters $\{b_k\}_{k=0}^r$. To this end, we can summarize the proposed CFPE method in Algorithm 1, Fig. 3.
3.3 Pitfall points
The LLS [22] reports pitfall points problem when solving the high-order harmonic model with multi-frequency fringe images, and it happens with under-determined equation. Our CFPE also has the pitfall point phenomena near some special points where image data correlated, i.e., the images from different frequencies provide same observations $I_{1,j_1}=I_{2,j_2}=I_{3,j_3}$. Revisiting the objective (Eq. (7)), the phase extraction thus can be viewed as a special regression problem (curve fitting for each pixel). The correlated images cannot provide enough effective data to obtain a unique regression solution, resulting in overfitting [27]. The pitfall point problem of multi-frequency PSP [22] is thus recognized as overfitting, and can be addressed by reducing the model complexity with different regularization (lasso and ridge regression [28]). Our insight is that the overfitting condition (correlated images) can be avoided by configuring proper periods, for instance, the $3-$frequency $3-$step PSP with periods $(42,45,48)$ will not produce any pitfall points in the interval from $0$ to $1680$ which is large enough for a practical measurement.
3.4 Analysis of convergence
The energy function (Eq. (8)) with sinusoidal periodicity is not convex, any algorithm might fall into local minimum points. Because the objective function is convex in the region around the global minimum, our strategy is to initialize $\phi _1$ with a reliable method (standard multi-frequency heterodyne technique, Eq. (5)). Therefore, as the iteration proceeds, the global minimum can be reached.
4. Experiments
4.1 Settings
To fairly understand the performance of our CFPE method, two representative methods (PE [17] with temporal phase unwrapping [23], LLS [22]) are adopted as baselines. We also test multi-frequency phase extraction (MPE) method: an ablation method of CFPE without considering high order harmonics, i.e., $r<s-1$. That says, the Eq. (19) could be simplified due to $c_2=0,c_3=0$, resulting in a concise MPE update,
To fully evaluate the CFPE method, both synthetic images with specified nonlinearity and real captured PSP images are considered. Four specified types of synthetic image (noise free, gamma, harmonics (r=2), and harmonics (r=3)) are respectively generated by,
Fig. 5. Left: a lab-made PSP setup with an industrial camera (HKvision, 180 $fps$) and a home projector (XGIMI, 24 $fps$). Right: Four test objects for evaluation.
In addition to a subjective visual judgement, three objective metrics are also utilized to quantify the performance: 1) the execution time for different image size, 2) the root-mean-square-error (RMSE) of phase [29,30], and 3) the root-mean-square-error of point vector.
4.2 Results on synthetic images
At first, the phase error along a $x$- scanline is investigated in Fig. 6. For the standard PE method, error distribution satisfies the theory analysis [32,33], i.e., $\Delta \phi \propto \sin (3\phi )$, and error amplitude could reach $\sim 0.075 rad$. The MPE method does help, while the unstable improvement depends on phase/position $(\phi )$. It means that simply employing more images is not enough for accuracy improvement. As expected, the LLS and our CFPE method achieve similar error reduction due to the shared high-order objective. As discussed in Section 3.3, the pitfall points $(x$ near $0)$ can be simply addressed by discarding them.
Fig. 6. The phase error for gamma distorted images ($\gamma =1.4$), zoom-in plots along the blue line of Fig. 4. Note that the LLS and CFPE almost share a same curve.
To study the performance with more complex nonlinearities, Fig. 7 tells the RMSE of phase for three types of nonlinearities. It’s clear that LLS and our CFPE are robust to both gamma nonlinearity and high-order harmonic, resulting in accurate phase extraction. However, for a higher-order nonlinear harmonic, high-order methods (LLS and CFPE) failed. Fortunately, the PSP system nonlinearity is often dominated by low-order harmonics instead of higher-order harmonic. As a result, the LLS and CFPE methods are recommended for accurate phase extraction.
To demonstrate the efficiency of proposed CFPE, the PE, MPE, LLS and CFPE methods were tested using Python 3.7 on a 2.70GHz i5-11400H laptop computer (HP OMEN 16) with RAM 16.00GB. Tested over $10$ runs, the execution results (Table 1) with varied image size demonstrated that the CFPE is much faster than the high-order baseline LLS. For a $1024\times 1024$ case, our CFPE needs execution time less than $1.0$ second. Recall that only LLS and CFPE achieve accurate phase extraction by employing a high-order harmonic.
Table 1. The running time ($seconds$) (averaged over 10 runs, with standard deviations).
4.3 Results on captured images
As introduced in section 4.1, four objects are measured, and Figs. 8 and 9 display the phase map and corresponding 3D points (in mesh, outliers are rejected). Due to the nonlinear PSP system, the ripple-like artifacts [34] of standard PE method can be clearly identified in phase maps, and resulting in unsatisfactory 3D point results. These artifacts are suppressed for MPE, LLS and CFD methods, both LLS and CFPE achieved the best visual performance across different materials and shapes. Note that the pitfall point problem is not observed in these results, which verified our overfitting analysis of pitfall points.
To quantify the performance, Table 2 provide the RMSE comparison about phase map and 3D points. Actually, the RMSE results agree well with the visual evaluation. Benefiting from more available data of different frequencies, MPE performs much better than standard PE methods. Because the high-order harmonic model is employed, LLS and CFPE methods achieve the best accuracy with the smallest RMSE values. From phase map to point vector, the results of CFPE are very close to that of LLS, and the small difference probably stems from the numerical computation. Note that the high-order phase extraction methods (LLS and CFPE) can significantly reduce the nonlinear effect, but they cannot kill it completely due to other higher order harmonics noise in a complex PSP system.
Table 2. The accuracy measured by $RMSE_1\,(rad)$ and $RMSE_2\,(mm)$. Best in bold.
5. Conclusion
In conclusion, this work proposed a combined-frequency phase extraction (CFPE) method to directly cope with the nonlinearity of PSP measurement system. The behind motivation — solving a high-order harmonics nonlinear model by considering other frequency’s images — is formulated as a MLE regression problem with latent variables. And an expectation–maximization algorithm provides an efficient iterative solution. From regression perspective, the pitfall point is recognized as an overfitting phenomena, and thus can be avoided by setting proper fringe periods. We have also demonstrated the characteristic of proposed CFPE on both synthetic and captured PSP images. Our CFPE can estimate the phase accurately even when the nonlinearity is severe $(\gamma =1.4$ or $C=5)$, and it only needs $\sim 5\%$ execution time of state-of-the-art LLS method. These advantages are also verified through four different materials in practical measurements. Meanwhile, the CFPE has the limitation to process the nonlinearity of higher-order harmonics, which will be investigated in our future work.
Funding
Fundamental Research Funds for the Central Universities (WUT:2022IVA027).
Acknowledgments
The authors appreciate the anonymous reviewers for their comments and suggestions.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. S. Zhang, “Recent progresses on real-time 3d shape measurement using digital fringe projection techniques,” Opt. Lasers Eng. 48(2), 149–158 (2010). [CrossRef]
2. M. Servin, J. A. Quiroga, and M. Padilla, Fringe pattern analysis for optical metrology: theory, algorithms, and applications (John Wiley & Sons, 2014).
3. C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Opt. Lasers Eng. 109, 23–59 (2018). [CrossRef]
4. C. Lin, D. Zheng, Q. Kemao, J. Han, and L. Bai, “Spatial pattern-shifting method for complete two-wavelength fringe projection profilometry,” Opt. Lett. 45(11), 3115–3118 (2020). [CrossRef]
5. T. Hoang, B. Pan, D. Nguyen, and Z. Wang, “Generic gamma correction for accuracy enhancement in fringe-projection profilometry,” Opt. Lett. 35(12), 1992–1994 (2010). [CrossRef]
6. Z. Li and Y. Li, “Gamma-distorted fringe image modeling and accurate gamma correction for fast phase measuring profilometry,” Opt. Lett. 36(2), 154–156 (2011). [CrossRef]
7. S. Xing and H. Guo, “Correction of projector nonlinearity in multi-frequency phase-shifting fringe projection profilometry,” Opt. Express 26(13), 16277–16291 (2018). [CrossRef]
8. S. Lei and S. Zhang, “Flexible 3-d shape measurement using projector defocusing,” Opt. Lett. 34(20), 3080–3082 (2009). [CrossRef]
9. S. Ma, C. Quan, R. Zhu, and C. Tay, “Investigation of phase error correction for digital sinusoidal phase-shifting fringe projection profilometry,” Opt. Lasers Eng. 50(8), 1107–1118 (2012). [CrossRef]
10. S. Feng, C. Zuo, L. Zhang, T. Tao, Y. Hu, W. Yin, J. Qian, and Q. Chen, “Calibration of fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 143, 106622 (2021). [CrossRef]
11. X. Yu, Y. Liu, N. Liu, M. Fan, and X. Su, “Flexible gamma calculation algorithm based on probability distribution function in digital fringe projection system,” Opt. Express 27(22), 32047–32057 (2019). [CrossRef]
12. S. Zhang and P. S. Huang, “Phase error compensation for a 3-d shape measurement system based on the phase-shifting method,” Opt. Eng. 46(6), 063601 (2007). [CrossRef]
13. Z. Zheng, J. Gao, J. Mo, L. Zhang, and Q. Zhang, “A fast self-correction method for nonlinear sinusoidal fringe images in 3-d measurement,” IEEE Trans. Instrum. Meas. 70, 1–11 (2021). [CrossRef]
14. A. Kamagara, X. Wang, and S. Li, “Nonlinear gamma correction via normed bicoherence minimization in optical fringe projection metrology,” Opt. Eng. 57(03), 1 (2018). [CrossRef]
15. Y. Lee, S. Zhang, M. Li, and X. He, “Blind inverse gamma correction with maximized differential entropy,” Signal Processing 193, 108427 (2022). [CrossRef]
16. Y. Wang, J. Cai, D. Zhang, X. Chen, and Y. Wang, “Nonlinear correction for fringe projection profilometry with shifted-phase histogram equalization,” IEEE Trans. Instrum. Meas. 71, 1–16 (2022). [CrossRef]
17. C. Zuo, L. Huang, M. Zhang, Q. Chen, and A. Asundi, “Temporal phase unwrapping algorithms for fringe projection profilometry: A comparative review,” Opt. Lasers Eng. 85, 84–103 (2016). [CrossRef]
18. P. S. Huang, Q. J. Hu, and F.-P. Chiang, “Double three-step phase-shifting algorithm,” Appl. Opt. 41(22), 4503–4509 (2002). [CrossRef]
19. Z. Lei, C. Wang, and C. Zhou, “Multi-frequency inverse-phase fringe projection profilometry for nonlinear phase error compensation,” Opt. Lasers Eng. 66, 249–257 (2015). [CrossRef]
20. Y. Yang, Q. Hou, Y. Li, Z. Cai, X. Liu, J. Xi, and X. Peng, “Phase error compensation based on tree-net using deep learning,” Opt. Lasers Eng. 143, 106628 (2021). [CrossRef]
21. D. C. Ghiglia and M. D. Pritt, “Two-dimensional phase unwrapping: theory, algorithms, and software,” A Wiley Interscience Publication (1998).
22. C. Jiang, S. Xing, and H. Guo, “Fringe harmonics elimination in multi-frequency phase-shifting fringe projection profilometry,” Opt. Express 28(3), 2838–2856 (2020). [CrossRef]
23. J.-S. Hyun and S. Zhang, “Enhanced two-frequency phase-shifting method,” Appl. Opt. 55(16), 4395–4401 (2016). [CrossRef]
24. S. Zhang, “Absolute phase retrieval methods for digital fringe projection profilometry: A review,” Opt. Lasers Eng. 107, 28–37 (2018). [CrossRef]
25. C. M. Bishop and N. M. Nasrabadi, Pattern recognition and machine learning, vol. 4 (Springer, 2006).
26. Y. Lee, H. Yang, and Z. Yin, “Outlier detection for particle image velocimetry data using a locally estimated noise variance,” Meas. Sci. Technol. 28(3), 035301 (2017). [CrossRef]
27. C. M. Hurvich and C.-L. Tsai, “Regression and time series model selection in small samples,” Biometrika 76(2), 297–307 (1989). [CrossRef]
28. W. J. Fu, “Penalized regressions: the bridge versus the lasso,” Journal of Computational and Graphical Statistics 7, 397–416 (1998).
29. S. Zhang, “Comparative study on passive and active projector nonlinear gamma calibration,” Appl. Opt. 54(13), 3834–3841 (2015). [CrossRef]
30. C. Chen, N. Gao, X. Wang, Z. Zhang, F. Gao, and X. Jiang, “Generic exponential fringe model for alleviating phase error in phase measuring profilometry,” Opt. Lasers Eng. 110, 179–185 (2018). [CrossRef]
31. X. Zhao, R. Mo, Z. Chang, and J. Lu, “A gamma correction method based on constant-intensity images in phase-measuring profilometry,” Insight-Non-Destructive Testing and Condition Monitoring 62(5), 256–263 (2020). [CrossRef]
32. Y. Liu, X. Yu, J. Xue, Q. Zhang, and X. Su, “A flexible phase error compensation method based on probability distribution functions in phase measuring profilometry,” Opt. Laser Technol. 129, 106267 (2020). [CrossRef]
33. X. Yu, S. Lai, Y. Liu, W. Chen, J. Xue, and Q. Zhang, “Generic nonlinear error compensation algorithm for phase measuring profilometry,” Chin. Opt. Lett. 19(10), 101201 (2021). [CrossRef]
34. S. Xing and H. Guo, “Directly recognizing and removing the projector nonlinearity errors from a phase map in phase-shifting fringe projection profilometry,” Opt. Commun. 435, 212–220 (2019). [CrossRef]
