Open Access
Add to BibSonomy
Abstract
In microscopic fringe projection profilometry (MFPP), the traditional absolute phase retrieval method using composite frequency fringe has the shortcomings of low accuracy and poor robustness due to mutual crosstalk of harmonic from the different channels of frequency-division multiplexing. In this study, an absolute phase retrieval method that avoids the inter-channel crosstalk is proposed. By introducing guard bands to accommodate the frequency channels corresponding to the second harmonic that dominate the high order harmonics, the aliasing between the second harmonic and the fundamental is eliminated. Consequently, phase maps without crosstalk can be demodulated using appropriate phase-shifting algorithms. The proposed method is well-suited for high-precision three-dimensional shape measurement scenarios in many fields such as integrated circuit manufacturing process control and micro-electro-mechanical system quality inspection. The experiment results demonstrate that the anti-crosstalk method is effective and can realize three-dimensional reconstruction for discontinuous planar surface and spherical surface.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Microscopic fringe projection profilometry (MFPP) is an extension of general fringe projection profilometry (FPP) aiming at achieving greater sensitivity and accuracy [1]. In contrast to FPP, MFPP is well-suited for high-precision micro-scale measurement which is widely used in three-dimensional (3D) data acquisition in the fields of electronic production [2], chip packaging [3,4], wafer metrology [5], MEMS quality inspection [6] and so on.
The sensitivity and uncertainty of the absolute phase are crucial in MFPP when performing 3D reconstruction. Introducing the two-wavelength fringe projection method is one effective approach to enhance sensitivity and reduce uncertainty [7]. Servin et al. provided mathematical evidence that using the two-wavelength fringe for absolute phase retrieval has considerable potential for enhancing sensitivity and improving the signal-to-noise ratio gain [8]. To obtain phase maps corresponding to fringe signal with various spatial frequency, two sets of fringe patterns can be projected separately. Based on the research by Liu et al. [9], these two sets of single frequency fringe patterns can be fused into one set of composite frequency fringe patterns containing two frequency components. Subsequently, two phase maps can be generated by applying a phase extraction algorithm that matches with the composite frequency fringe signal. The composite fringe signal inherits the advantages of the two-wavelength fringe projection method and significantly reduces the number of required patterns [10]. This makes it a promising strategy that trade-off between measurement accuracy and speed.
The idea of this method may be analogous to the principle of frequency-division multiplexing (FDM) in communication technology. If the composite frequency fringe signal has sufficient bandwidth, it is possible to modulate various phase information into specific sub-bands by carriers. If the phase information of the carriers can be extracted without distortion, it is possible to achieve frequency division multiplexing transmission of phase information with diverse meanings. More specifically, if one sub-band transmits a height-dependent wrapped phase map and the other sub-band transmits a non-wrapped phase map for phase unwrapping, an absolute phase map can be recovered via one set of fringe patterns. It is worth noting that the carriers for implementing FDM can be either spatial or temporal carriers, thus FDM can be classified into spatial-based FDM and temporal-based FDM. Spatial-based FDM relies on Fourier transform profilometry. The composite fringe comprises of at least two spatial carriers. The phase maps from different spatial carriers are separated by configuring several spatial filters [11–13]. This method can achieve absolute phase retrieval with one-shot pattern, but it also inherits the limitations associated with Fourier transform profilometry [14]. In contrast, temporal-based FDM is based on phase-shifting profilometry (PSP). In this method, the composite fringe typically comprises of two temporal carriers, then a temporal filter is utilized instead of a spatial filter to demodulate the phase maps [15,9]. The temporal-based FDM method inherits the high-precision measurement potential of phase-shifting profilometry [10] since the modulation of phase by depth information is independent of time. However, this method inherits the problem that harmonic components in the fringe pattern (so-called non-sinusoidal fringe) can disturb the phase maps.
The nonlinear effects of digital micromirror devices (DMD) are a primary source of harmonics, and static calibration methods cannot completely eliminate these effects due to the inherent nature of commercial DLP [16]. In typical phase demodulation using a least-squares phase-shifting algorithms (LS-PSA), the phase distortion mechanism is attributed to the spectral response zeros of the LS-PSAs failing to cover all higher order harmonics [17]. To reduce the phase error caused by this mechanism, one can increase the number of phase shift steps [18]. Alternatively, passive compensation can be achieved through the relationship between the gamma model and the phase contribution of harmonics [19,20]. These methods effectively address harmonic distortion in single frequency fringe signal. However, when dealing with composite frequency fringe signal using a temporal-based FDM, the elimination of phase distortion through these methods may face challenges. Because the phase distortion arises from another mechanism that is undefined (as far as we know) in optical fringe analysis. For the composite fringe signal, due to the influence of this mechanism, the phase of the fundamental (first harmonic) is disturbed by phase of the harmonics from the other frequency component. We term this phase distortion mechanism “crosstalk” because it similar to telephone crosstalk observed in FDM communication systems with sub-bands aliasing. In optical fringe pattern analysis, crosstalk has the potential to introduce significant errors into the phase maps.
In this study, a systematic theoretical framework for effectively defining the crosstalk mechanism in optical fringe pattern analysis is developed. The framework combines frequency transfer function (FTF) theory with FDM theory. Based on these theoretical studies, a temporal carrier configuration method is proposed to avoid crosstalk form second-order harmonic, then the corresponding absolute phase retrieval algorithm is given. This method incorporates guard bands, a concept from communication engineering, into the design of composite frequency fringe patterns. Simulation and experimental results proved that the proposed method effectively eliminates phase distortion caused by the crosstalk mechanism, in contrast to conventional phase recovery using the dual-frequency pattern. Furthermore, to illustrate the practical applicability of the proposed method, absolute phase measurements for micro-scale discontinuous surfaces are provided using the proposed algorithm.
2. Theoretical framework
While the absolute phase retrieval method using composite fringe theoretically has obvious advantages in measurement accuracy, the harmonic induced from the nonlinearity of the DMD reduce the robustness of this method, which significantly constrains its practicality. Despite Liu et al. proved the effectiveness of this method initially through experiments [9], they did not systematically analyze the principle of this phase retrieval method, so it is difficult to determine the crosstalk mechanism and solve the phase distortion problem in the stage of the signal modulation. In order to systematically analyze the modulation and demodulation process in the method, it may be necessary to utilize relevant theories in the fields of communication technology and electrical engineering. Servin et al. introduced a theoretical framework based on the Fourier description of phase-shifting interferometry utilizing the FTF [17,21,22]. This framework is highly effective and systematic in modeling fringe signal and the response of the PSAs. Therefore, we choose this theoretical framework as the fundamental theory for analyzing the frequency response of LS-PSAs to fringe signal. Furthermore, to analyze the frequency shift behavior in composite fringe signal and illustrate the anti-crosstalk principle of the proposed method, we apply the theory of temporal-based FDM in communication technology to the optical fringe analysis. In this section, we provide a brief overview of the FTF theory for LS-PSAs, followed by an analysis of the general principle of absolute phase retrieval method using temporal-based FDM.
2.1 Frequency transfer function for LS-PSAs
Firstly, a temporal carrier term needs to be introduced for the fringe patterns to enable the subsequent temporal Fourier analysis. The fringe pattern required for N-Step LS-PSAs is
The corresponding complex representation of the cosine function is given by
The positive component $(b/2)\textrm{exp} (i\varphi )\textrm{exp} (i{\omega _0}t)$ is the so-called analytic signal.
Secondly, since PSA is a synchronization sampling system, according to linear system theory, the impulse response function of the system is given by
where ${k_n}$ is the constant parameters that specify the system. The TFT is defined as the temporal Fourier transform of the impact response function $h(t)$, we haveThe analytic signal in Eq. (5) can be obtained by convolution operation, we have
Combining all the constant coefficients, Eq. (8) can be rewritten as
where the constant coefficient ${A_0} = (\textrm{b}/2)H({\mathrm{\omega }_0})\textrm{exp} (i{\mathrm{\omega }_0}t)$.The wrapped phase $\varphi $ can be determined byThirdly, to determine the general formula of the TFT for N-Step LS-PSA, the generalized N-Step LS-PSA formula is given by
The analytic signal $\sum\nolimits_{n = 0}^{N - 1} {\textrm{exp} (i{\omega _0}t){I_n}}$ of the N-Step LS-PSA follows to the convolution relationship in Eq. (9), we have
Combining Eqs. (9–12), we have
Then the impulse response function and TFT for N-Step LS-PSA are given by
Figure 1 illustrates the principle of analytic signal demodulation using LS-PSAs. The blue curve is the TFT for LS-PSAs, and the orange arrow indicates the discrete spectrum resulting from the temporal Fourier transform of $I(t )$. When dealing with an ideal sinusoidal signal, the N-Step LS-PSAs (only the 3-Step and 5-Step LS-PSA in Fig. 1) can demodulate the analytic signal $\textrm{exp} (i\varphi )$ while effectively removing the direct current (DC) component and the conjugate component $\textrm{exp} ( - i\varphi )$ (indicated by arrows with a cross mark). Thus, the LS-PSAs is essentially a bandpass filter that frequency response fulfills the conditions $H({\omega _0}) \ne 0$, $H( - {\omega _0}) = 0$, $H(0) = 0$. In other words, the FTF has first-order zeros at $\omega = 0$ and $\omega ={-} {\omega _0}$. Note that the FTF for LS-PSA has the following properties:(1) More phase-shifting steps, bring in the more first-order zeros. (2) The first-order zeros are located at certain frequencies that are integer multiples of the angular frequency ${\omega _0}$.
2.2 Temporal-based FDM in composite frequency fringe analysis
We assume that the composite frequency fringe pattern involves two sinusoidal components with different spatial frequency. According to Eq. (4), the composite frequency fringe is simplified to be a time-dependent signal with two temporal carriers:
where M is the coefficient of frequency multiplication, which determines the integer ratio of frequencies at which the two channels are located. The subscript H indicates that the variable belongs to the fringe component with higher spatial frequency and the subscript L indicates that the variable belongs to the fringe component with lower spatial frequency. The temporal Fourier transform of Eq. (16) is given byFigure 2 illustrates that the composite fringe can simultaneously contain two phase maps with distinct spatial frequency. The phase maps derived from the analytic signals $\textrm{exp} (i{\varphi _H})$ and $\textrm{exp} (i{\varphi _L})$. To independently demodulate analytic signals, it is necessary to use two bandpass filters with center frequency ${\omega _0}$ and $M{\omega _0}$. Assuming that the bandpass filters are ideal filters, the relationship between the FTF for the filter and $I(\omega )$ as shown in Fig. 3, where the red and blue dashed line represent the FTF for filters, with functions as bandpass filters for $\textrm{exp} (i{\varphi _H})$ and $\textrm{exp} (i{\varphi _L})$, respectively.
Fig. 2. Two phase maps contained in the composite fringe ${I_n}(x,y)$. (a) ${\varphi _H}(x,y)$ is the phase map of the higher spatial frequency fringe, (b) ${\varphi _L}(x,y)$ is the phase map of the lower spatial frequency fringe
Fig. 3. Frequency-division multiplexing of phase maps, ${\varphi _H}$ occupies ${\pm} {\omega _0}$, ${\varphi _L}$ occupies ${\pm} M{\omega _0}$
According to Eq. (15), the LS-PSAs can theoretically achieve the same effect as the ideal bandpass filter because the spectrum of the composite frequency signal is discrete. To achieve this, the FTF for the LS-PSAs used to demodulate $\textrm{exp} (i{\varphi _H})$ must satisfy the following constraints
The TFT for the LS-PSA used to demodulate $\textrm{exp} ({\textrm{i}{\varphi_L}} )$ must satisfy the following constraints
Ideally, the composite frequency fringe pattern is a dual-frequency sinusoidal pattern without higher order harmonic, Eq. (18) and Eq. (19) can be met by configuring M = 2 and employing a 5-Step LS-PSA. This condition is utilized in the method proposed by Liu et al [9].
2.3 Discussion
According to the above analysis, the idea of the absolute phase retrieval method for composite frequency fringe pattern is similar to that of FDM in communication technology. Specifically, as shown in Fig. 3, when the maximum bandwidth within temporal carriers is sufficient, multiple phase maps with arbitrary spatial distribution can be modulated into independent sub-bands, and then these phase maps can be demodulated by using well-matched band-pass filters. However, in practical scenarios, due to the complexity of the spectrum of fringe signal, phase crosstalk can occur if there is no guard band separating the sub-bands. This crosstalk is manifested in fringe analysis by the fact that the phase of high-frequency fringe and low-frequency fringe interfere with each other, resulting in the deterioration of both the measurement accuracy and the robustness of the phase unwrapping in the MFPP system.
3. Issues with phase crosstalk due to the nonlinearity of DMD
3.1 Second harmonic caused by nonlinearity of DMD
In practice, the nonlinear response of DMD is difficult to completely eliminate by static calibration [16]. Although the typical use of gamma calibration method [23] for the DMDs, including TI-DLP4500 and TI-DLP670s, nonlinearity still exists in the response tests for DMDs. However, this nonlinearity is different from the typical gamma effect. The test results for response of TI-DLP670s are shown in Fig. 4(a)-(c). Even an ideally linear image is used as input, the DMD's output response exhibits local nonlinear that is systematic rather than random. The cross-sectional data demonstrate the validity of gamma calibration and the linearity of the DMD's response across the entire dynamic range (0-255). However, if the dynamic range is subdivided into eight intervals $[(n - 1) \cdot {2^5},n \cdot {2^5}],n \in (1,2, \ldots ,8)$, it is not difficult to find that there is a grayscale bias between adjacent intervals, and a noticeable grayscale discontinuity at the interval boundaries as shown in Fig. 4(c). These issues are likely at the chip level and result from the segmentation of registers used in constructing pulse width modulation (PWM) sequence signals for intensity control [24]. Figure 4(d)-(f) shows sinusoidal fringe pattern projected by the gamma-calibrated DMD, obviously the pattern exhibiting non-sinusoidal fringe due to gray-scale discontinuity and bias. Figure 5 shows the spectrum of the non-sinusoidal fringe (the DC component is omitted), where the grayscale discontinuity and bias contribute second harmonic in the spectrum, which are difficult to be perfectly eliminated by static gamma calibration. Furthermore, in our experiments, we observed that the magnitude of second harmonic increases as the fringe pitch decreases.
Fig. 4. Nonlinear effect of DMD after gamma correction. (a) Linear intensity image that is input to the DMD, (b) Image acquired from CMOS, (c) Cross section of (b), (d) Quasi-sinusoidal fringe image acquired from CMOS, (e) Zoomed image of (d), (f) Cross section of (e).
3.2 Phase crosstalk caused by the second harmonic
According to Eq. (16), the composite frequency fringe containing second harmonic is given by
In the method proposed by Liu et al. M = 2, the Fourier transform of Eq. (20) is given by:
Figure 6(a) illustrates the spectrum associated with Eq. (21), the analytic signals $\textrm{exp} (i{\varphi _{H\_1st}})$ and $\textrm{exp} ( - i{\varphi _{L\_2nd}})$ exhibit aliasing due to the temporal carrier has a frequency shifting to the second order harmonic as well, so phase ${\varphi _{H\_1st}}$ is disturbed by ${\varphi _{L\_2nd}}$. As shown in Fig. 6(b), when demodulation is performed using the 5-Step LS-PSA, the FTF exhibits a bandpass effect for both $\textrm{exp} (i{\varphi _{L\_1st}})$ and $\textrm{exp} ( - i{\varphi _{H\_2nd}})$ (as indicated by arrows without a cross mark). This results in crosstalk between ${\varphi _{L\_1st}}$ and ${\varphi _{H\_2nd}}$. Similarly, the FTF for 5-Step LS-PSA with a carrier of $2{\omega _0}t$ and spectrum are shown in Fig. 6(c), the crosstalk occurring between ${\varphi _{H\_1st}}$ and ${\varphi _{L\_2nd}}$.
Fig. 6. Phase crosstalk due to second harmonic. (a) Spectrum of composite fringe containing second harmonic, (b) bandpass filter fails to out the $\textrm{exp} ( - i{\varphi _{L\_2nd}})$, (c) The $\textrm{exp} (i{\varphi _{H\_1st}})$ and $\textrm{exp} ( - i{\varphi _{L\_2nd}})$ are aliased in same frequency.
The characteristics of the crosstalk mechanism are as follows:
- a) Crosstalk does not occur when the fringe signal contains only two fundamental components.
- b) Crosstalk is induced by the non-ideal filtering effect of the LS-PSA due to limited sampling. When the FTF of the LS-PSA fails to satisfy Eq. (18) and (19), harmonics from one frequency component are responded simultaneously with the fundamental from the other, leading to phase distortion. Increasing the number of phase shift steps can alleviate this distortion.
- c) Improper temporal carrier configuration can cause phase crosstalk. Due to the frequency shift effect of the temporal carrier, when the harmonic from one of the frequency components and the fundamental from another frequency component appear to aliasing, the phase of the fundamental will be distorted. This phase distortion cannot be eliminated by increasing the number of phase shift steps.
- d) Distortion intensity relies on the energy levels of both the harmonic and the fundamental. Severe phase distortion of the fundamental occurs when the energy of the harmonics closely matches that of the fundamental.
4. Proposed anti-crosstalk absolute phase retrieval method
As previously analyzed, the issue arises from the aliasing of the analytic signals, leading to phase crosstalk. To address this issue, we can insert a guard band between the sub-bands containing the analytic signal of first harmonic and then configure appropriate band-pass filter to effectively suppress the second harmonic signal component. In this section, the principle of the proposed method is explained following the design process.
4.1 Usage of guard band
First, the guard band for accommodating the second harmonic is configured. To prevent aliasing between the second harmonic and the fundamental frequency, the ratio of the angular frequency of the two temporal carriers should be not less than three (M ≥ 3). In addition, to ensure that the bandwidth of the guard band is not redundant, the M should be equal to three.
Secondly, the temporal carrier of the high-frequency fringe component should be far away from the baseband (narrow band near the DC component). Since in practice, precision measurement usually requires more phase-shifting steps. As the steps increases, the angular frequency ${\omega _0}$ of the analytic signal gradually shift to the baseband. Due to the non-ideal transient response of the DMD [25] and the instability of the light source [26], the background signal (DC component) will fluctuate with time. This behavior is equivalent to broadening the baseband in the frequency domain. Phase error can be contributed by illumination fluctuation when the sub-band of the analytic signal overlaps the baseband. Since the phase used for phase-to-height conversion is more sensitive to error than the phase used for phase unwrapping, to maintain the phase measurement robustness of the proposed method, the angular frequency of the temporal carrier of the high-frequency fringe component should be higher than that of the low-frequency component.
The composite fringe signal meeting the above criteria is given by
4.2 Phase demodulation
According to Eq. (22), the composite fringe signal containing second harmonic is given by
The corresponding temporal Fourier transform is given by
Figure 7(a) shows the spectrum corresponding to Eq. (24) and illustrates that with the addition of the guard band, the first order and second harmonic components are positioned at distinct frequency, Thus, aliasing is eliminated. To achieve bandpass for the first harmonic components, the first order zeros of the FTF for the LS-PSA should be situated at $\omega = (0, - {\omega _0}, \pm 2{\omega _0}, \pm 3{\omega _0}, \pm 4{\omega _0})$. Therefore, the 8-Step LS-PSA is employed. According to Eq. (15), To extract $\textrm{exp} (i{\varphi _{H\_1st}})$, the corresponding FTF for 8-Step LSPSA is given by
Fig. 7. (a) No aliasing due to the usage of guard band, (b) bandpass only for $\textrm{exp} (i{\varphi _{L\_1st}})$, (c) bandpass only for $\textrm{exp} (i{\varphi _{H\_1st}})$.
Figure 7(b) shows the spectrum of the Eq. (24) and FTF for Eq. (25). The analytic signals are filtered except $\textrm{exp} (i{\varphi _{H\_1st}})$. This means that the desired phase ${\varphi _{H\_1st}}$ is not disturbed by the phase from other analytic signals. Similarly, Fig. 7(c) illustrates that ${\varphi _{L\_1st}}$ is also free from crosstalk.
In summary, the composite fringe signal with anti-crosstalk capability should be consistent with Eq. (22). For minimize phase-shifting steps, the angular frequency of the time-carrier ${\omega _0} = 2\pi /8$ and the sampling fulfills $t\textrm{ } = \textrm{ \{ }0,\textrm{ }1,\textrm{ }\ldots ,\textrm{ }7\}$. The corresponding equations for solving the ${\phi _H}$ and ${\phi _L}$ are given by
4.3 Phase unwrapping
After the phase maps are demodulated, the absolute phase map can be calculated using the two-frequency hierarchical algorithm [27].
where $\mathrm{\Phi }(x,y)$ is the absolute phase and $k({x,y} )$ is the fringe order for each pixel, $k(x,y)$ is determined by5. Simulation
The simulation used a composite frequency fringe with the second harmonic for both phase demodulation and 3D reconstruction of spherical and planar structures. The amplitude ratio of the high-frequency to low-frequency components in the composite frequency fringe signal is 8:2, while the amplitude ratio of the first harmonic to the second harmonic is 10:1. The temporal carrier configuration method adopts the method proposed by Liu et al. and the method proposed by us, respectively. The results of phase demodulation and 3D reconstruction with different number of phase steps are shown in Fig. 8.
Fig. 8. Simulation results of phase map demodulation and 3D reconstruction. (a) Simulation results of Liu et al.'s method. (b) Simulation results of the proposed method.
Figure 8(a) shows the simulation results using the method proposed by Liu et al. Due to the absence of guard bands in the temporal carrier configuration, the low-frequency phase experiences significant periodic distortion caused by the crosstalk mechanism. Phase distortion in the low-frequency component introduces error in the codewords utilized for phase unwrapping. Consequently, this leads to obviously periodic error in the 3D reconstruction results. It is essential to emphasize that increasing the number of phase steps does not eliminate the crosstalk-induced distortion. Despite increasing the number of phase steps from 5 to 11, the 3D reconstruction result remains uncorrected.
Figure 8(b) shows the simulation results obtained using the proposed method. In contrast to the method suggested by Liu et al., our proposed method effectively eliminates second harmonic crosstalk by introducing guard bands matched with the second harmonic. Consequently, the phase map contains only the phase of the analytic signal of the first harmonic, resulting in 3D reconstruction that are consistent with the ground truth. In addition, an increase in the number of phase steps does not impact the crosstalk mechanism. When the number of phase steps is raised from 8 to 14, the 3D reconstruction results remain consistent.
In summary, the simulation verify that the proposed method effectively avoids phase distortion induced by the crosstalk mechanism, irrespective of whether the measured surface is planar or spherical. Additionally, the simulations demonstrate that increasing the number of phase steps cannot eliminate phase distortion to one of the carriers. This behavior differs from the distortion caused by gamma effect in a conventional single-frequency fringe pattern. This behavior also suggests that passive phase error compensation methods based on the relationship between harmonic energy and phase contribution cannot be employed to eliminate the phase distortion caused by crosstalk. Since passive compensation presupposes obtaining phase contribution from only the higher order harmonics, which assumes that the higher order harmonics are separated from the first harmonic in frequency domain. This assumption is naturally met in the case of single-frequency fringe signal containing harmonics. However, for composite frequency fringed signal, the second harmonic are aliasing with the first harmonic due to the improper configuration of the temporal carrier (as shown in Fig. 6(a)), and thus it may not be possible to obtain the independent phase contribution of the second harmonic using LS-PSAs.
6. Experiment
The main components of the experiment setup include a projection module consisting of a DMD (Texas Instruments DLP670s) and a telecentric lens (Edmund 62#902), and an imaging module consisting of an industrial camera (IDS U3-3800SE-M-GL Rev. 1.2) and a telecentric lens (Edmund 15#872). The altitude angle of the optical axes of the two modules is 43° and the azimuth angle is 180° relative to the normal direction of the surface under test. A low coherence LED with a central wavelength of 465 nm was used as the light source, and the response of the gamma calibrated DMD is consistent with Fig. 4(c). To suppress the higher order harmonics generated by the discrete pixel effect of the DMD [28], the f-number of the telecentric lens in the projection module is set to be 12 approximatively. It should be noted that the frequency of the higher order harmonics caused by the pixel effect is much higher than that of the second harmonic of the fringe. Therefore, the spatial filtering effect by changing the f-number will not affect the second harmonic of the fringe. In addition, to maintain a consistent focus over the field of view, the positions of the DMD and CMOS satisfy the Scheimpflug principle with respect to the reference plane.
6.1 Validation of anti-crosstalk capability
To verify the anti-crosstalk capability of the proposed method, fringe patterns are projected onto a flat board (polished wafer). Firstly, two sets of experiments are conducted to illustrate the issue of crosstalk in traditional phase retrieval method using the dual-frequency pattern. In this experiments, the fringe pattern is generated by the method proposed by Liu et al [9]. Since this method has no anti-crosstalk properties. The composite fringe is consistent with Eq. (22), where $a = 255/2$, ${b_H} = 0.8a$, ${b_L} = 0.2a$, ${f_H} = 1/25$, ${f_L} = 1/100$ and the coefficient of frequency multiplication for temporal carriers is set to 2 (M = 2). In the first set of experiments, the number of fringe patterns is set to 5 (N = 5), which is the minimum number of patterns required by the algorithm [10]. Since another study by Liu et al. indicated that harmonic effects are negligible when N exceeds 5 [19], the number of fringe patterns was set to 8 in a subsequent set of experiments. We expect that when N = 8, the crosstalk mechanism will entirely dominate phase distortion.
Figure 9 shows the results of two sets of experiments. Both experiments exhibit nearly identical harmonic components within their fringe signals. The consistency of harmonic components ensures comparability of the experiments. The phase maps of the low-frequency fringe components in both sets of experiments exhibit significant distortion, as illustrated in Fig. 9(d1) and (d2). The distortion in the corresponding cross section curve is periodic and results from the crosstalk of the second-order harmonic signals. As shown in Fig. 9(c1) and 1(c2), despite the harmonic signals from the low-frequency components having significantly lower energy compared to the fundamental signal (${f_{L\_2nd}} = \textrm{ }{f_{L\_1st}}$), the phase maps of the high-frequency components remain distorted. In the subpanels of Fig. 9(e1) and (e2), the residual curves from data in the red region also exhibit periodicity. It is worth noting that the root means square error (RMSE) of the residual at N = 8 is slightly lower than that at N = 5, because the phase distortion in the high-frequency component results from the inadequate filtering capacity of the LS-PSA. Based on the conclusions in chapter 3, increasing the number of LS-PSA samples can suppress the distortion. However, a comparison of Fig. 9(f1) and Fig. 9(f2) reveals that the phase distortion in the low-frequency components is basically identical. This suggests that the crosstalk caused by spectral aliasing cannot be removed by increasing the number of fringe patterns. In practice, the substantial distortion in the low-frequency phase map utilized for phase unwrapping prevents obtaining an accurate encoding of the fringe order, leading to the failure of absolute phase retrieval. Furthermore, distortion in the high-frequency phase map used for 3D reconstruction adversely impacts measurement accuracy.
Fig. 9. The demodulation results of the composite frequency fringe patterns, using the method proposed by Liu et al. (a1) Phase-shifted fringe patterns, (b1) Fourier transform of the fringe signal, (c1) phase map of the high-frequency fringe component, (d1) phase map of the low-frequency fringe component, (e1) cross section phase of (c1), (f1) cross section phase of (d1). (a2) – (f2) have the same meaning as (a1) – (f1) but with the difference that N = 8.
In conclusion, the first experiment proves the existence of the crosstalk mechanism during the demodulation of composite frequency fringes and its substantial negative impact on the phase demodulation in the MFPP system.
Secondly, we measured the phase of the wafer surface using the proposed phase retrieval method. The harmonics of fringe signal consistent with those in the previous experiment, as shown in Fig. 10(e). A comparison of the cross sections of the phase maps in Fig. 10(f) and Fig. 10(g) with those from the previous experiments reveals the clear elimination of periodic phase distortion in the phase maps of both high-frequency and low-frequency components. Due to the accurate encoding of fringe order in the low-frequency phase maps, phase unwrapping can be achieved with Eq. (29) and Eq. (30). By utilizing the two phase maps with different frequency from Fig. 10(b) and Fig. 10(c), we can achieve phase unwrapping. The absolute phase map and cross section data in Figs. 10(d) and 11(h), respectively.
Fig. 10. Absolute phase retrieval result using the proposed method. (a) Phase-shifted fringe patterns, (b) phase map of the high-frequency fringe component, (c) phase map of the low-frequency fringe component, (d) absolute phase map, (e) Fourier transform of the fringe signal, (f) cross section phase of (b), (g) cross section phase of (c), and (h) cross section phase of (d).
The experiments above illustrate the effectiveness of the proposed method in avoiding crosstalk effects resulting from an incorrect temporal carrier configuration. While our proposed method demands the use of eight fringe patterns, it can extract two accurate phase maps from the composite frequency signal. Based on this, the absolute phase map can be obtained using a two-frequency hierarchical algorithm.
6.2 3D reconstruction for a discontinuous planar surface
To verify the effectiveness of the proposed method for measuring a discontinuous surface, a 3D checkerboard target referencing the VLSI step height standard [29] is fabricated using the photolithography process. The substrate of the target is a polished wafer, the length of the square is 500µm, and the step-height of the neighboring squares is 20µm. Figure 11(a)-(c) shows the 3D structure of the target using scanning electron microscope (SEM). To ensure that the measurement system has proper sensitivity, the pitch of the high-frequency fringe component occupies 10 DMD pixels. The parameters of the fringe is: $a = 255/2$, ${b_H} = 0.8a$, ${b_L} = 0.2a$, ${f_H} = 1/10$, ${f_L} = 1/160$. Figure 12 shows the region of interest (14.8 mm × 8.7 mm) used for measurement, which is slightly smaller than the maximum FOV of the imaging module.
Fig. 12. 3D checkerboard target measurement process. (a) ROI for measurement, (b) target under uniform illumination, (c) one of the fringe patterns from the measurement process, (d) enlarged view of (c).
The phase recovery results are shown in Fig. 13, when no post-processing is applied to the phase data, including filtering and invalid point cloud removal. Figure 13(a) shows the phase map within the entire region of interest (ROI) (the polished wafer plane is used as the reference plane), the contour of the phase is essentially the same as the target. Figure 13(c) and (d) are the cross-section data corresponding to the red and green dash lines in Fig. 13(b). The cross section data illustrate that except for the phase error caused by edge effects (this could be attributed to shadow areas or multi-path [30]), the other flat areas are basically consistent with the true contour of the target.
Fig. 13. 3D reconstruction of checkerboard target. (a) 3D reconstruction in ROI, (b) enlarged view of the red rectangle in (a), (c) cross section of the red dash line in (b), (d) cross section of the green dash line in (b)
6.3 3D reconstruction for a spherical surface
To demonstrate the effectiveness of our proposed method in reconstructing non-planar surfaces, we prepared a semi-ellipsoidal solder ball using the surface tension of melted solder material. However, preparing microscale hemispheres with Lambertian surfaces is very difficult. To demonstrate the 3D reconstruction using our proposed method without introducing additional complex techniques to enhance the dynamic range, we sputter a developer solution primarily composed of calcium carbonate to the solder ball. This transformed the smooth surface into an approximate Lambertian surface. Because of the random pits on the surface of solidified solder balls and the non-uniform spraying of the developer solution, optical signals in certain small areas become saturated or obscured. Consequently, this can result in 3D reconstruction that may not accurately represent the real surface profile in those regions.
In this experiment, the parameters of the fringe is: $a = 255/2$, ${b_H} = 0.8a$, ${b_L} = 0.2a$, ${f_H} = 1/25$, ${f_L} = 1/100$. Figure 14 shows the 3D reconstruction results of the solder ball within the ROI (without significant defocus area). The point cloud data remain unprocessed by spatial filtering. As shown in Fig. 14(c), the 3D reconstruction generally corresponds to a semi-ellipsoidal surface, with certain spikes potentially arising from signal saturation or local occlusion. As shown in Fig. 14(d), the curves in the side regions appear smoother compared to those in the central region, possibly due to defocusing.
Fig. 14. 3D reconstruction of solder ball. (a)ROI for measurement, (b)one of the composite frequency fringe patterns, (c) 3D reconstruction result, (d) 312th column cross section of the 3D reconstruction result
7. Conclusion
In this study, an anti-crosstalk absolute phase retrieval method for microscopic fringe projection profilometry is proposed which retrieves the absolute phase without crosstalk-induced phase distortion by introducing the concept of guard band from the FDM technique in the temporal carrier design. In contrast to the traditional dual-frequency fringe projection method, the proposed method can extract two phase maps without crosstalk distortion when the fringe signal contains second harmonics. Since the unexpected second harmonic are located on the guard band to which the LS-PSA does not response, the aliasing of the frequency-shifted second harmonic with the first harmonic is avoided, eliminating crosstalk between two phase maps. It is worth noting that the proposed method can be flexibly optimized using the FTF theory employed in this paper. For example, the crosstalk of higher order harmonic can be eliminated by broadening the guard band while the phase errors caused by lighting fluctuation can be minimized with an additional frequency shift. The flexibility of the proposed method makes it suitable for trade-off of accuracy, field of view, and speed in MFPP systems.
Funding
National Key Research and Development Program of China (2022YFF0705700).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. Y. Hu, Q. Chen, S. Feng, et al., “Microscopic fringe projection profilometry: A review,” Optics and Lasers in Engineering 135, 106192 (2020). [CrossRef]
2. Y. Liu, Y. Fu, Y. Zhuan, et al., “Large depth-of-field 3D measurement with a microscopic structured-light system,” Opt. Commun. 481, 126540 (2021). [CrossRef]
3. T. Cheng, X. Liu, L. Qin, et al., “A Practical Micro Fringe Projection Profilometry for 3-D Automated Optical Inspection,” IEEE Trans. Instrum. Meas. 71, 1–13 (2022). [CrossRef]
4. Z. Li, Y. C. Kang, J.-H. Moon, et al., “The optimum projection angle of fringe projection for ball grid array inspection based on reflectance analysis,” Int J Adv Manuf Technol 67(5-8), 1597–1607 (2013). [CrossRef]
5. L.-C. Chen, D.-H. Duong, and C.-S. Chen, “Optical 3-D Profilometry for Measuring Semiconductor Wafer Surfaces with Extremely Variant Reflectivities,” Appl. Sci. 9(10), 2060 (2019). [CrossRef]
6. C. Quan, C. J. Tay, X. Y. He, et al., “Microscopic surface contouring by fringe projection method,” Opt. Laser Technol. 34(7), 547–552 (2002). [CrossRef]
7. M. Servin, G. Paez, M. Padilla, et al., “Simultaneous phase and amplitude modulation for dual-sensitivity profilometry of discontinuous objects,” J. Opt. Soc. Am. A 39(10), 1874 (2022). [CrossRef]
8. M. Servin, M. Padilla, and G. Garnica, “Super-sensitive two-wavelength fringe projection profilometry with 2-sensitivities temporal unwrapping,” Optics and Lasers in Engineering 106, 68–74 (2018). [CrossRef]
9. K. Liu, Y. Wang, D. L. Lau, et al., “Dual-frequency pattern scheme for high-speed 3-D shape measurement,” Opt. Express 18(5), 5229 (2010). [CrossRef]
10. C. Zuo, S. Feng, L. Huang, et al., “Phase shifting algorithms for fringe projection profilometry: A review,” Optics and Lasers in Engineering 109, 23–59 (2018). [CrossRef]
11. L. Yang, F. Li, G. Shi, et al., “Frequency-division-multiplexing based period-coded fringe pattern for reliable depth sensing,” J. Eur. Opt. Soc.-Rapid Publ. 14(1), 2 (2018). [CrossRef]
12. H. An, Y. Cao, H. Wu, et al., “The spatial phase-shifting measuring profilometry based on dual-frequency grating,” Optics and Lasers in Engineering 143, 106638 (2021). [CrossRef]
13. C. A. García-Isáis and N. Alcalá Ochoa, “One shot profilometry using a composite fringe pattern,” Optics and Lasers in Engineering 53, 25–30 (2014). [CrossRef]
14. Y. Li, J. Qian, S. Feng, et al., “Composite fringe projection deep learning profilometry for single-shot absolute 3D shape measurement,” Opt. Express 30(3), 3424–3442 (2022). [CrossRef]
15. J.-L. Li, H.-J. Su, and X.-Y. Su, “Two-frequency grating used in phase-measuring profilometry,” Appl. Opt. 36(1), 277 (1997). [CrossRef]
16. A. Kamagara, X. Wang, and S. Li, “Towards gamma-effect elimination in phase measurement profilometry,” Optik 172, 1089–1099 (2018). [CrossRef]
17. M. Servin, J. A. Quiroga, and M. Padilla, Fringe Pattern Analysis for Optical Metrology: Theory, Algorithms, and Applications (John Wiley & Sons, 2014).
18. X.-Y. Su, W.-S. Zhou, G. von Bally, et al., “Automated phase-measuring profilometry using defocused projection of a Ronchi grating,” Opt. Commun. 94(6), 561–573 (1992). [CrossRef]
19. K. Liu, Y. Wang, D. L. Lau, et al., “Gamma model and its analysis for phase measuring profilometry,” J. Opt. Soc. Am. A 27(3), 553 (2010). [CrossRef]
20. S. Lin, H. Zhu, and H. Guo, “Harmonics elimination in phase-shifting fringe projection profilometry by use of a non-filtering algorithm in frequency domain,” Opt. Express 31(16), 25490–25506 (2023). [CrossRef]
21. M. Servin, J. C. Estrada, and J. A. Quiroga, “The general theory of phase shifting algorithms,” Opt. Express 17(24), 21867 (2009). [CrossRef]
22. M. Padilla, M. Servin, and G. Garnica, “Profilometry with digital fringe-projection at the spatial and temporal Nyquist frequencies,” Opt. Express 25(19), 22292 (2017). [CrossRef]
23. S. Zhang and S.-T. Yau, “Generic nonsinusoidal phase error correction for three-dimensional shape measurement using a digital video projector,” Appl. Opt. 46(1), 36–43 (2007). [CrossRef]
24. D. Doherty and G. Hewlett, “10.4: Phased Reset Timing for Improved Digital Micromirror DeviceTM (DMDTM) Brightness,” Dig. Tech. Pap. - Soc. Inf. Disp. Int. Symp. 29(1), 125–128 (1998). [CrossRef]
25. Y. Wang, B. Bhattacharya, E. H. Winer, et al., “Digital micromirror transient response influence on superfast 3D shape measurement,” Optics and Lasers in Engineering 58, 19–26 (2014). [CrossRef]
26. C. Chen, Y. Wan, and Y. Cao, “Instability of projection light source and real-time phase error correction method for phase-shifting profilometry,” Opt. Express 26(4), 4258–4270 (2018). [CrossRef]
27. J. M. Huntley and H. Saldner, “Temporal phase-unwrapping algorithm for automated interferogram analysis,” Appl. Opt. 32(17), 3047–3052 (1993). [CrossRef]
28. B. Li and S. Zhang, “Microscopic structured light 3D profilometry: Binary defocusing technique vs. sinusoidal fringe projection,” Optics and Lasers in Engineering 96, 117–123 (2017). [CrossRef]
29. P. de Groot and D. Fitzgerald, “Measurement, certification and use of step-height calibration specimens in optical metrology,” in Optical Measurement Systems for Industrial Inspection X (SPIE, 2017), Vol. 10329, pp. 328–336.
30. Y. Zhang, D. L. Lau, and Y. Yu, “Causes and Corrections for Bimodal Multi-Path Scanning With Structured Light,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2019), pp. 4426–4434.
