Expand this Topic clickable element to expand a topic
Skip to content
Optica Publishing Group

Divide and conquer: high-accuracy and real-time 3D reconstruction of static objects using multiple-phase-shifted structured light illumination

Open Access Open Access

Abstract

Multiple-phase-shifted structured light illumination achieves high-accuracy 3D reconstructions of static objects, while typically it can’t achieve real-time phase computation. In this paper, we propose to compute modulations and phases of multiple scans in real time by using divide-and-conquer solutions. First, we categorize total N = KM images into M groups and each group contains K phase equally shifted images; second, we compute the phase of each group; and finally, we obtain the final phase by averaging all the separately computed phases. When K = 3, 4 or 6, we can use integer-valued intensities of images as inputs and build one or M look-up tables storing real-valued phases computed by using arctangent function. Thus, with addition and/or subtraction operations computing indices of the tables, we can directly access the pre-computed phases and avoid time-consuming arctangent computation. Compared with K-step phase measuring profilometry repeated for M times, the proposed is robust to nonlinear distortion of structured light systems. Experiments show that, first, the proposed is of the same accuracy level as the traditional algorithm, and secondly, with employing one core of a central processing unit, compared with the classical 12-step phase measuring profilometry algorithm, for K = 4 and M = 3, the proposed improves phase computation by a factor of 6 ×.

© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

1. Introduction

For 3D scans by means of multiple-phase-shifted structured light illuminations (SLIs), the more patterns are projected, the less accurate the 3D surfaces of moving objects are reconstructed, while the more accurate the surfaces of static objects are [13]. In the past decades, researchers have been attempting to reduce the number of patterns in order to achieve both real-time and accurate 3D reconstructions for moving objects, e.g. [4,5] ; however, static objects also need to be scanned in real time and in high accuracy. In industrial applications, e.g. pipeline analyses [6], railway inspections [7], welding robots [8], etc., users do need such requirements on both speed and accuracy. For scanning static objects by using multiple patterns, the increased scanning time can be decreased by employing high-speed scanning hardware [914], and meanwhile, the increased computation time may be decreased by employing high-performance but expensive computation units, e.g., graphics processing units (GPUs) [1517].

On real-time computations, we hold on that, if we can mathematically develop more efficient algorithms [1820] for classical equations, of course we can achieve much better performance once the advanced algorithms are implemented over special-purpose hardware including GPUs, field-programmable gate arrays (FPGAs) [21,22], digital signal processors (DSPs) [23,24], etc.

With this idea in mind, in this paper, we propose to reconstruct 3D surfaces in real time by using divide-and-conquer solutions for static objects. First, we categorize total $N=KM$ phase-shifted images into $M$ groups and each group contains $K$ phase equally shifted images; second, we compute the modulation and phase in each group; and finally, we obtain final modulation and phase by averaging all separately computed modulations and phases. When $K=3$, $4$ or $6$, we can use integer-valued intensities of images as inputs, and build one or $M$ look-up tables (LUTs), commonly used in engineering to speed up computations [2527], storing real-valued modulations and phases computed by using square root and arctangent functions. Thus only with addition and/or subtraction operations computing indices of the LUTs, we can directly access the pre-computed modulations and phases and, thus avoid time-consuming square root and arctangent computations. Compared with $K$-step phase measuring profilometry (PMP) repeated by $M$ times, the proposed is robust to nonlinear distortion of structured light systems. Experiments show that, compared with the traditional 12-step PMP algorithm, the proposed can improve phase computation by a factor of 6$\times$ when $K=4$ and $M=3$ through single-thread C++ program running over one core of a central processing unit (CPU).

2. Methods

After sinusoid patterns are projected onto an object and captured by a camera, at a camera coordinate $\left (x,y\right )$, the image intensity $I_n$ with the pattern information is modeled as

$$I_n=A+B\cos\left(\phi-\frac{2\pi n}{N}\right),$$
where $N$ $(\geq 3)$ is the total phase shifts and $n$ $(=0,1,\ldots ,N-1)$ is the index of a phase shift. The direct component $A$ is calculated by averaging all $I_n$, and $B$ and $\phi$ are the modulation and phase components and are traditionally computed by
$$\left\{ \begin{array}{l} B = \frac{2}{N}\sqrt{S^2 + C^2}\\ \phi = \tan^{{-}1}\left(\frac{S}{C}\right) \end{array} \right. \mathrm{where} \left\{ \begin{array}{l} S = \sum_{n=0}^{N-1}I_n\sin\left(\frac{2\pi n}{N}\right) \\ C = \sum_{n=0}^{N-1}I_n\cos\left(\frac{2\pi n}{N}\right) \end{array} \right..$$
In practice, the arctangent function $\tan ^{-1}(\cdot )$ is implemented by using $\mathrm {atan2}(\cdot ,\cdot )$ with output range of $[0,2\pi )$. All of $I_n$, $A$, $B$, and $\phi$ are functions of $\left (x,y\right )$ which is ignored in this paper for concisely expressing.

If $N=KM$, we can divide all $I_n$ into $M$ groups and each group contains $K$ images, and $\{I_n\}$ of $m$-th group are categorized as

$$G_m = \left\{I_{n}~\left|~n=m+kM~~\mathrm{and}~~k\in\left[0, K-1\right] \right.\right\},$$
where $m\in \left [0, M-1\right ]$. Then the modulation and phase over $G_m$ are consequently computed by
$$\left\{ \begin{array}{l} B_m = \frac{2}{N}\sqrt{S_m^2 + C_m^2} \\ \phi_m = \left[ \tan^{{-}1}\left( \frac{S_m}{C_m} \right) + \frac{2\pi m}{KM} \right]~\mathrm{mod}~2\pi \end{array} \right. \mathrm{where} \left\{ \begin{array}{l} S_m = \sum_{k=0}^{K-1}I_{m+kM}\sin\left(\frac{2\pi k}{K}\right) \\ C_m = \sum_{k=0}^{K-1}I_{m+kM}\cos\left(\frac{2\pi k}{K}\right) \end{array} \right.$$
and $\mathrm {mod}$ is the modulo operation. We have final modulation and phase by averaging $B_m$ and $\phi _m$ as
$$\left\{ \begin{array}{l} \bar{B} = \frac{1}{M}\sum_{m=0}^{M-1}B_m \\ \bar{\phi} = \frac{1}{M}\sum_{m=0}^{M-1}\phi_m \end{array} \right.,$$
respectively. Note that, when $M\geq 2$, the averaging procedure of Eq. (5) automatically suppress harmonics caused by nonlinear distortion of optical systems [28], while, if we repeat $K$-step PMP for $M$ times, the averaged modulation and phase are still significantly distorted by nonlinearity.

In this paper, in order to clearly and easily describe pattern strategies, we introduce a triple notation

$$\mathrm{PMP}\;\langle K,M,R \rangle,$$
as illustrated in Fig. 1, to indicate: 1) performing $KM$-step PMP scanning; 2) categorizing $KM$ patterned images into $M$ groups guided by Eq. (3); 3) employing Eq. (4) to decode $B_m$ and $\phi _m$ for each group; 4) employing Eq. (5) to obtain final $\bar {B}$ and $\bar {\phi }$; 5) repeating 1)-4) for $R$ times. Additionally, when $M=1$, the proposed Eq. (5) is the same as the traditional Eq. (2).

 figure: Fig. 1.

Fig. 1. The pipeline for the defination of PMP$\langle K, M, R \rangle$.

Download Full Size | PDF

In practice, there always exists error sources, e.g., noise, nonlinearity, etc., causing inaccurate computations of modulation and phase. If $I_{m+kM}$ is polluted by independent additive Gaussian noise with zero mean. By using the noise model presented in [29], we can easily prove the variance of the phase error of Eq. (4) is equivalent to that of Eq. (2). However, noise will cause a wrap issue at $0$ or $2\pi$ for high-frequency patterns and we can use the phase computed from $m=0$ as a reference to correct wrap errors as

$$\phi_m = \left\{ \begin{array}{l} \phi_m + 2\pi ~~\mathrm{if}~~\phi_0 - \phi_m\;>\;\pi\\ \phi_m - 2\pi ~~\mathrm{if}~~\phi_0 - \phi_m\;<\;{-}\pi \end{array} \right.$$
where $m>0$, to remove outliers.

Then, we employ the gamma model proposed in [28] to analyze error caused by nonlinearity or gamma. As shown in Fig. 2, when we simulate $\mathrm {PMP}\;\langle 3,4,1 \rangle$ with $\gamma =2.2$, there are such phase shifting errors for $\phi _1$ and $\phi _3$. By deriving the absolute error of $\phi _m$ and calibrating the harmonic energies $\hat {B}_{h}$ with large number of patterns, e.g. 120 or 240 [28], we obtain compensators for the phase shifting errors as

$$\Delta\phi_m\left(K \right) = \tan^{{-}1}\left[ \frac{ \sum_{h=1}^{\infty}\left( \hat{B}_{hK+1} - \hat{B}_{hK-1} \right) \sin\left( h\frac{2\pi m}{M} \right) }{ \hat{B}_1 + \sum_{h=1}^{\infty}\left( \hat{B}_{hK+1} + \hat{B}_{hK-1} \right) \cos\left( h\frac{2\pi m}{M} \right)} \right],$$
from which we know $\Delta \phi _m\left (K\right )=0$ if $m=0$ or $M/2$ for even $M$. So the phase shifting errors can be corrected by
$$\phi_m = \left[ \tan^{{-}1}\left( \frac{S_m}{C_m} \right) + \frac{2\pi m}{KM} + \Delta\phi_m\left(K\right) \right]~~\mathrm{mod}~~2\pi,$$
as shown in Fig. 3. Note that $\mathrm {PMP}\;\langle K,2M,R \rangle$ isn’t equivalent to $\mathrm {PMP}\;\langle 2K,M,R \rangle$ because $2K$-step PMP is of much better capability of anti-nonlinearity than that of $K$-step PMP [28].

 figure: Fig. 2.

Fig. 2. With $\gamma =2.2$ and $\mathrm {PMP}\;\langle 3,4,1 \rangle$, nonlinearly distorted phases of each group: (a) $\phi _0$, (b) $\phi _1$, (c) $\phi _2$, and (d) $\phi _3$. Besides nonlinear distortion, there are phase shifting errors in (b) and (d).

Download Full Size | PDF

 figure: Fig. 3.

Fig. 3. Compensated (a) $\phi _1$ and (b) $\phi 3$ for those in Figs. 2(b) and (d).

Download Full Size | PDF

In order to compute $B_m$ and $\phi _m$ in real time, for $M\geq 2$ and $K=3$, $4$ or $6$ [18], we can further derive Eq. (4) into LUT forms. There are 1 modulation LUT and $M$ phase LUTs or one phase LUT for each case of $K$. For $K=3$, with defining the integer-valued indices of LUTs as $X_{3}=2I_{m} - I_{m+M} - I_{m+2M}$ and $Y_{3}=I_{m+M} - I_{m+2M}$ where $X_{3}\in \left [-510,510\right ]$ and $Y_{3}\in \left [-255,255\right ]$ for 8-bit grayscale, the modulation LUT, denoted as $\mathrm {MLUT}$, and phase LUTs, denoted as $\mathrm {PLUT}$, are built as

$$\left\{ \begin{array}{l} \mathrm{MLUT}_{3}\left(Y_{3},X_{3}\right) = \frac{1}{3}\sqrt{ 3Y_{3}^2 + X_{3}^2}\\ \mathrm{PLUT}_{3}^{\left(0\right)}\left(Y_{3},X_{3}\right) = \tan^{{-}1}\left( \frac{\sqrt{3}Y_{3}}{X_{3}} \right)\\ \mathrm{PLUT}_{3}^{\left(m\right)}\left(Y_{3},X_{3}\right) = \left[ \mathrm{PLUT}_{3}^{\left(0\right)}\left(Y_{3},X_{3}\right) + \mathrm{CLUT}_3\left( m \right) \right]~~\mathrm{mod}~~2\pi \end{array} \right.,$$
where $m\in \left [1,M-1\right ]$, and $\mathrm {CLUT}_3(m) = 2\pi m/(3M) + \Delta \phi _m(3)$ is a 1D LUT with length of $M-1$ and is used to compensate the phase shifting errors as previously described. The size of each 2D LUT and the number of LUTs are listed in Table 1, and $\mathrm {MLUT}_3$ and $\mathrm {PLUT}_3^{(0)}$ are visualized in Fig. 4. We intentionally differ $\mathrm {PLUT}_3^{(0)}$ from $\mathrm {PLUT}_3^{(m)}$ $(m>0)$ in order to indicate that we propose two LUT strategies in this paper. The first strategy is constructing $M$ PLUTs to expect more quickly looking up at cost of consuming more memory space. The second one is to let $M-1$ groups share $\mathrm {PLUT}_3^{(0)}$ with additional addition and modulo operations to save memory space when $M$ is larger. And the reasons are the same for $K=4$ and $K=6$.

 figure: Fig. 4.

Fig. 4. (a) $\mathrm {MLUT}_{3}$ and (b) $\mathrm {PLUT}_{3}^{\left (0\right )}$

Download Full Size | PDF

Tables Icon

Table 1. The size and number of LUTs in each case.

In Table 2, we compare the computation complexity of $\mathrm {PMP}\;\langle 3,M,1 \rangle $ (i.e. Eq. (5)) with that of $\mathrm {PMP}\;\langle 3M,1,1 \rangle $ (i.e. Eq. (2)). For $M$-LUT $\mathrm {PMP}\;\langle 3,M,1 \rangle $, the addition or subtraction operation is from computing the indices of LUTs and from Eq. (5) computing the final modulation and phase, and the “Compare” operation is from Eq. (7) to cancel outliers when high-frequency patterns are employed and it will slow down processing speed with increasing $M$. The multiplication operation for $2I_{m}$ is treated as $I_{m} + I_{m}$ and the division operation in Eq. (5) can be pre-processed in $\mathrm {MLUT}_{3}$ and $\mathrm {PLUT}_{3}^{\left (m\right )}$ in engineering. For one-LUT $\mathrm {PMP}\;\langle 3,M,1 \rangle $, the modulo and additional looking up operations relate to Eq. (10). For $\mathrm {PMP}\;\langle 3M,1,1 \rangle $, the looking up operation is from two 1D LUTs built for $\sin$ and $\cos$ functions in Eq. (2), and the $\tan ^{-1}$ and square root functions and multiplication operations are the dominant slowing-down factors. The computation complexities for $K=4$ and $K=6$ can be similarly analyzed.

Tables Icon

Table 2. Comparison of the computation complexities.

For $K=4$, with defining the integer-valued indices of LUTs as $X_{4}=I_{m} - I_{m+2M}$ and $Y_{4}=I_{m+M} - I_{m+3M}$ where $X_{4}\in \left [-255,255\right ]$ and $Y_{4}\in \left [-255,255\right ]$, the modulation LUT and phase LUTs are built as

$$\left\{ \begin{array}{l} \mathrm{MLUT}_{4}\left(Y_{4},X_{4}\right) = \frac{1}{2}\sqrt{ Y_{4}^2 + X_{4}^2 }\\ \mathrm{PLUT}_{4}^{\left(0\right)}\left(Y_{4},X_{4}\right) = \tan^{{-}1}\left( \frac{Y_{4}}{X_{4}} \right)\\ \mathrm{PLUT}_{4}^{\left(m\right)}\left(Y_{4},X_{4}\right) = \left[ \mathrm{PLUT}_{4}^{\left(0\right)}\left(Y_{4},X_{4}\right) + \mathrm{CLUT}_4(m) \right]~~\mathrm{mod}~~2\pi \end{array} \right.,$$
where $\mathrm {CLUT}_4(m) = \pi m/(2M) + \Delta \phi _m\left (4 \right )$. The size of each 2D LUT and the number of LUTs are listed in Table 1, and $\mathrm {MLUT}_4$ and $\mathrm {PLUT}_4^{(0)}$ are visualized in Fig. 5. With similar analysis on computation complexity, we know $\mathrm {PMP}\;\langle 4,M,1 \rangle $ achieves fastest speed.

 figure: Fig. 5.

Fig. 5. (a) $\mathrm {MLUT}_{4}$ and (b) $\mathrm {PLUT}_{4}^{\left (0\right )}$

Download Full Size | PDF

For $K=6$, with defining the integer-valued indices as $X_{6}=2I_{m} + I_{m+M} - I_{m+2M} - 2I_{m+3M} - I_{m+4M} + I_{m+5M}$ and $Y_{6}=I_{m+M} + I_{m+2M} - I_{m+4M} - I_{m+5M}$ where $X_{6}\in \left [-1020,1020\right ]$ and $Y_{6}\in \left [-510,510\right ]$, the modulation LUT and phase LUTs are built as

$$\left\{ \begin{array}{l} \mathrm{MLUT}_{6}\left(Y_{6},X_{6}\right) = \frac{1}{6}\sqrt{ 3Y_{6}^2 + X_{6}^2}\\ \mathrm{PLUT}_{6}^{\left(0\right)}\left(Y_{6},X_{6}\right) = \tan^{{-}1}\left( \frac{\sqrt{3}Y_{6}}{X_{6}} \right)\\ \mathrm{PLUT}_{6}^{\left(m\right)}\left(Y_{6},X_{6}\right) = \left[ \mathrm{PLUT}_{6}^{\left(0\right)}\left(Y_{6},X_{6}\right) + \mathrm{CLUT}_6(m) \right]~~\mathrm{mod}~~2\pi \end{array} \right.,$$
where $\mathrm {CLUT}_6(m) = \pi m/(3M) + \Delta \phi _m\left (6 \right )$. The size of each 2D LUT and the number are listed in Table 1, and $\mathrm {MLUT}_6$ and $\mathrm {PLUT}_6^{(0)}$ are visualized in Fig. 6. Next we will evaluate the proposed algorithm on accuracy and speed through experiments.

 figure: Fig. 6.

Fig. 6. (a) $\mathrm {MLUT}_{6}$ and (b) $\mathrm {PLUT}_{6}^{\left (0\right )}$

Download Full Size | PDF

3. Results

As shown in Fig. 7, our experimental setup consists of an AVT Prosilica GC640C camera with pixel resolution of 640$\times$480 and a Casio XJ-M140 projector whose working pixel resolution is set to 800$\times$600. We synchronize the camera and projector by using the VSync signal extracted from the video graphics array (VGA) port in our controlling computer. There is an Intel i7-3770 8-core CPU running at 3.4GHz in the computer. By implementing the proposed algorithm in single-thread C++ programming, we conduct three groups of experiments to evaluate (1) the accuracy and (2) speed of the proposed algorithm, and (3) visualize the 3D reconstructions of the proposed. We don’t scan moving objects as we state that, suffering from motion noise [30], multiple-pattern strategies don’t work well for moving objects. Note that the speed mentioned in this work is on phase computation without counting the time for phase unwrapping [19].

 figure: Fig. 7.

Fig. 7. Experimental setup.

Download Full Size | PDF

In the first group of experiments, we scan a flat white wall to evaluate accuracy of the proposed. We treat the modulation and phase of $\mathrm {PMP}\;\langle 240,1,1 \rangle $ as the ground truths, and simultaneously have a series of $\hat {B}_h$ to compute phase compensators, i.e. $\Delta \phi _m\left (K\right )$, with Eq. (8). In order to be consistent with speed evaluations conducted later, we use the least common multiple of 3, 4 and 6, i.e. 12, as the total phase shifts. Then we scan the white wall with 12-step PMP, and compute modulations and phases by using $\mathrm {PMP}\;\langle 3,4,1 \rangle $, $\mathrm {PMP}\;\langle 4,3,1 \rangle $, $\mathrm {PMP}\;\langle 6,2,1 \rangle $, and $\mathrm {PMP}\;\langle 12,1,1 \rangle $, respectively, to compare accuracy on noise issue. We also scan the wall with $\mathrm {PMP}\;\langle 3,1,4 \rangle $, $\mathrm {PMP}\;\langle 4,1,3 \rangle $, and $\mathrm {PMP}\;\langle 6,1,2 \rangle $ to compare accuracy on nonlinearity issue. In Table 3, it shows the phase compensators for $\mathrm {PMP}\;\langle 3,4,1 \rangle $ and $\mathrm {PMP}\;\langle 4,3,1 \rangle $, and $\mathrm {PMP}\;\langle 6,2,1 \rangle $ doesn’t need to be compensated with $\Delta \phi _m(K)$ in Eq. (8).

Tables Icon

Table 3. Compensation factors

In Figs. 8(a) and (b), the cross-section at the 320th column of the modulation and phase errors of $\mathrm {PMP}\;\langle 3,4,1 \rangle $, $\mathrm {PMP}\;\langle 3,1,4 \rangle $, and $\mathrm {PMP}\;\langle 12,1,1 \rangle $ are shown. For $\mathrm {PMP}\;\langle 3,1,4 \rangle $, its errors on modulation and phase are obviously caused by nonlinearity. For $\mathrm {PMP}\;\langle 3,4,1 \rangle $, its modulation error is significantly offset from zero and the modulation difference between $\mathrm {PMP}\;\langle 3,4,1 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $ is shown in Fig. 8(c), and, as shown in Table 4, the mean and root mean square (RMS) values of the modulation error are $-3.19$ and $3.23$, respectively, while its phase error is nearly overlapped with the error of $\mathrm {PMP}\;\langle 12,1,1 \rangle $, so we further plot the phase difference between $\mathrm {PMP}\;\langle 3,4,1 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $ in Fig. 8(d). When we compute the variance or RMS of the phase errors, as listed in Table 4, we see that, in the same order of magnitude, the accuracy of $\mathrm {PMP}\;\langle 12,1,1 \rangle $ is little better than $\mathrm {PMP}\;\langle 3,4,1 \rangle $.

 figure: Fig. 8.

Fig. 8. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 3,4,1 \rangle $, $\mathrm {PMP}\;\langle 3,1,4 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $, and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 3,4,1 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $.

Download Full Size | PDF

Tables Icon

Table 4. Comparing the accuracy of the proposed with the traditional method.

In Figs. 9(a) and (b), the cross-section at the 320th column of the modulation and phase errors of $\mathrm {PMP}\;\langle 4,3,1\rangle $, $\mathrm {PMP}\;\langle 4,1,3\rangle $, and $\mathrm {PMP}\;\langle 12,1,1\rangle $ are shown. For $\mathrm {PMP}\;\langle 4,1,3\rangle $, its errors on modulation and phase are obviously caused by nonlinearity. For $\mathrm {PMP}\;\langle 4,3,1\rangle $, its modulation error is close to that of $\mathrm {PMP}\;\langle 12,1,1\rangle $ and the modulation difference between $\mathrm {PMP}\;\langle 4,3,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ is shown in Fig. 9(c), and, as shown in Table 4, the mean value of the modulation error of $\mathrm {PMP}\;\langle 4,3,1\rangle $ is higher than that of $\mathrm {PMP}\;\langle 12,1,1\rangle $ in a order of magnitude and the RMS value of the error is little higher than that of $\mathrm {PMP}\;\langle 12,1,1\rangle $. Its phase error is nearly overlapped with the error of $\mathrm {PMP}\;\langle 12,1,1\rangle $, so we further plot the phase difference between $\mathrm {PMP}\;\langle 4,3,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ in Fig. 9(d). When we compute the mean, variance and RMS of the phase errors, as listed in Table 4, we see that, in the same order of magnitude, the accuracy of $\mathrm {PMP}\;\langle 4,3,1\rangle $ is little better than $\mathrm {PMP}\;\langle 12,1,1\rangle $.

 figure: Fig. 9.

Fig. 9. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 4,3,1\rangle $, $\mathrm {PMP}\;\langle 4,1,3\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $, and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 4,3,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $.

Download Full Size | PDF

In Figs. 10(a) and (b), the cross-section at the 320th column of the modulation and phase errors of $\mathrm {PMP}\;\langle 6,2,1\rangle $, $\mathrm {PMP}\;\langle 6,1,2\rangle $, and $\mathrm {PMP}\;\langle 12,1,1\rangle $ are shown. For $\mathrm {PMP}\;\langle 6,1,2\rangle $, its errors on modulation and phase are obviously caused by nonlinearity. For $\mathrm {PMP}\;\langle 6,2,1\rangle $, its modulation error is nearly overlapped with that of $\mathrm {PMP}\;\langle 12,1,1\rangle $ and the modulation difference between $\mathrm {PMP}\;\langle 6,2,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ is shown in Fig. 10(c), and, as shown in Table 4, the mean value of the modulation error of $\mathrm {PMP}\;\langle 6,2,1\rangle $ is nearly twice of that of $\mathrm {PMP}\;\langle 12,1,1\rangle $ and the RMS value of the error is little higher than that of $\mathrm {PMP}\;\langle 12,1,1\rangle $. Its phase error is nearly overlapped with the error of $\mathrm {PMP}\;\langle 12,1,1\rangle $, so we further plot the phase difference between $\mathrm {PMP}\;\langle 6,2,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ in Fig. 10(d). When we compute the mean, variance and RMS of the phase errors, as listed in Table 4, we see that, in the same order of magnitude, the accuracy of $\mathrm {PMP}\;\langle 12,1,1\rangle $ is little better than $\mathrm {PMP}\;\langle 6,2,1\rangle $.

 figure: Fig. 10.

Fig. 10. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 6,2,1\rangle $, $\mathrm {PMP}\;\langle 6,1,2\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $, and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 6,2,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $.

Download Full Size | PDF

In the second group of experiments, we evaluate the speeds of the proposed LUT methods. After scanning the wall with 12-step PMP, by using CPU-based single-thread C++ programming, we process the 12 patterned images as $\mathrm {PMP}\;\langle 3,4,1\rangle $, $\mathrm {PMP}\;\langle 4,3,1\rangle $, $\mathrm {PMP}\;\langle 6,2,1\rangle $, and $\mathrm {PMP}\;\langle 12,1,1\rangle $, respectively. For $\mathrm {PMP}\;\langle 3,4,1\rangle $, $\mathrm {PMP}\;\langle 4,3,1\rangle $, and $\mathrm {PMP}\;\langle 6,2,1\rangle $, we compute modulations and phases with the proposed 2D $M$-LUT and one-LUT algorithms, respectively, while, for $\mathrm {PMP}\;\langle 12,1,1\rangle $, we can only build two 1D LUTs storing $\sin (2\pi n / N)$ and $\cos (2\pi n / N)$ where $N=12$. We run the programs for $1000$ times, and finally obtain averaged frame rates on the modulation and phase computations, as shown in Tables 5 and 6 from which we can see that $\mathrm {PMP}\;\langle 4,3,1\rangle $ achieve highest frame rate and $M$-LUT-based processing isn’t as expected as faster than one-LUT-based method. Thus, especially from Table 6, we know that one-LUT method is of advantages on both speed and storage space, and, from Tables 4 and 6, we see one-LUT $\mathrm {PMP}\;\langle 4,3,1\rangle $ is of advantages on all accuracy, speed, and storage space in our cases.

Tables Icon

Table 5. Comparing the speeds of the proposed LUT-based methods with the traditional method on modulation computation.

Tables Icon

Table 6. Comparing the speeds of the proposed LUT-based methods with the traditional method on phase computation.

In order to further evaluate the speed performance of the proposed, we increase the number of patterns to $24$, $48$, $60$, $120$, and $240$. We select these numbers because they are both the integer divisors of $240$ and the integer multiples of $12$. And, as concluded as previously, we just perform one-LUT $\mathrm {PMP}\;\langle 4,M,1\rangle $, where $M=6$, $12$,$15$, $30$, and $60$, respectively, to compare with $\mathrm {PMP}\;\langle 4M,1,1\rangle $ on phase computation. The frame rates and corresponding improvements are listed in Table 7, and we see the frame rates of the proposed are decreased with increasing $M$, because the “Compare” operation in Eq. (7) is dominant when $M$ is large. Without using Eq. (7) to correct wrap errors for high-frequency patterns, there will be significant outliers, as shown in Fig. 11. Equation (7) may be disabled for unit-frequency patterns to further accelerate computation.

 figure: Fig. 11.

Fig. 11. The front and side views of the 3D point clouds of a plaster statue reconstructed by using $\mathrm {PMP}\;\langle 4,3,1\rangle $ without using Eq. (7) to correct wrap errors which cause lots of outliers.

Download Full Size | PDF

Tables Icon

Table 7. Comparing the speeds of the proposed One-LUT $\mathrm {PMP}\;\langle 4,M,1\rangle $ with the traditional $\mathrm {PMP}\;\langle 4M,1,1\rangle $ on phase computation.

In the third group of experiments, we visually demonstrate the 3D reconstructions of a plaster statue scanned with 12-step PMP. After the phases are computed with $\mathrm {PMP}\;\langle 3,4,1\rangle $, $\mathrm {PMP}\;\langle 4,3,1\rangle $, $\mathrm {PMP}\;\langle 6,2,1\rangle $, and $\mathrm {PMP}\;\langle 12,1,1\rangle $, respectively, the 3D point clouds, also computed with LUT algorithms [18,20], are shown in Figs. 12(a), (b), (c) and (d), respectively. Visually it is difficult for us to tell the difference among the four clouds, while, with previous analysis, we know $\mathrm {PMP}\;\langle 4,3,1\rangle $ achieves the best performance.

 figure: Fig. 12.

Fig. 12. The front and side views of the 3D point clouds of a plaster statue reconstructed by using (a) $\mathrm {PMP}\;\langle 3,4,1\rangle $, (b) $\mathrm {PMP}\;\langle 4,3,1\rangle $, (c) $\mathrm {PMP}\;\langle 6,2,1\rangle $, and (d) $\mathrm {PMP}\;\langle 12,1,1\rangle $, respectively.

Download Full Size | PDF

4. Conclusion and future work

In this paper, in order to achieve real-time phase computation for multiple-phase-shifted SLI, we propose a divide-and-conquer algorithm, i.e., (1) grouping patterned images, (2) computing the phase over the images in each group, and (3) averaging all the phases to have a final phase. When the number of images in each group is 3, 4, or 6, we can construct LUTs to directly access pre-computed modulation and phase values with looking up operations instead of square root and arctangent functions. We also propose two LUT strategies, and there are $M$-LUT and one-LUT methods. One-LUT strategy is of more advantages on computation speed and storage space. Additionally, if we let 4 images be categorized into each group, it achieves the best accuracy even compared with traditional method.

In the future, if we want to further speed up computation, we need to directly handle the wrap errors to avoid the “Compare” operation in Eq. (7). We are also curious that why the phase accuracy of $\mathrm {PMP}\;\langle 4,3,1\rangle $ is better than that of $\mathrm {PMP}\;\langle 12,1,1\rangle $ and why the modulation accuracy of $\mathrm {PMP}\;\langle 4, 1, R\rangle $ is better than that of $\mathrm {PMP}\;\langle 6,1,R\rangle $, although phase accuracy is typically more concerned.

Funding

National Natural Science Foundation of China (61473198); Sichuan Province Science and Technology Support Program (2018GZ0198); Chengdu Science and Technology Bureau (2018-YFYF-00029-GX).

Disclosures

The authors declare no conflicts of interest.

References

1. S. V. der Jeught and J. J. Dirckx, “Real-time structured light profilometry: A review,” Opt. Lasers Eng. 87, 18–31 (2016). [CrossRef]  

2. S. Zhang, “High-speed 3D shape measurement with structured light methods: A review,” Opt. Lasers Eng. 106, 119–131 (2018). [CrossRef]  

3. C. Zuo, S. Feng, L. Huang, T. Tao, W. Yin, and Q. Chen, “Phase shifting algorithms for fringe projection profilometry: A review,” Opt. Lasers Eng. 109, 23–59 (2018). [CrossRef]  

4. Y. Wang, K. Liu, Q. Hao, D. L. Lau, and L. G. Hassebrook, “Period coded phase shifting strategy for real-time 3-D structured light illumination,” IEEE Trans. on Image Process. 20(11), 3001–3013 (2011). [CrossRef]  

5. C. Zuo, Q. Chen, G. Gu, S. Feng, and F. Feng, “High-speed three-dimensional profilometry for multiple objects with complex shapes,” Opt. Express 20(17), 19493–19510 (2012). [CrossRef]  

6. T. Tahara, Y. Shimura, and M. Niimura, “3D measurement and FFS assessment for LTA in pressure equipment according to WES2820: 2015,” in ASME 2016 Pressure Vessels and Piping Conference, (American Society of Mechanical Engineers, 2016), pp. V007T07A008–V007T07A008.

7. Q. Mao, H. Cui, Q. Hu, and X. Ren, “A rigorous fastener inspection approach for high-speed railway from structured light sensors,” ISPRS J. Photogramm. Remote. Sens. 143, 249–267 (2018). [CrossRef]  

8. L. Yang, E. Li, T. Long, J. Fan, and Z. Liang, “A high-speed seam extraction method based on the novel structured-light sensor for arc welding robot: A review,” IEEE Sens. J. 18(21), 1 (2018). [CrossRef]  

9. Y. Wang, J. I. Laughner, I. R. Efimov, and S. Zhang, “3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique,” Opt. Express 21(5), 5822–5832 (2013). [CrossRef]  

10. B. Li and S. Zhang, “Superfast high-resolution absolute 3D recovery of a stabilized flapping flight process,” Opt. Express 25(22), 27270–27282 (2017). [CrossRef]  

11. J.-S. Hyun, G. T.-C. Chiu, and S. Zhang, “High-speed and high-accuracy 3D surface measurement using a mechanical projector,” Opt. Express 26(2), 1474–1487 (2018). [CrossRef]  

12. C. Zuo, T. Tao, S. Feng, L. Huang, A. Asundi, and Q. Chen, “Micro Fourier transform profilometry (μFTP): 3D shape measurement at 10,000 frames per second,” Opt. Lasers Eng. 102, 70–91 (2018). [CrossRef]  

13. W. Yin, S. Feng, T. Tao, L. Huang, M. Trusiak, Q. Chen, and C. Zuo, “High-speed 3D shape measurement using the optimized composite fringe patterns and stereo-assisted structured light system,” Opt. Express 27(3), 2411–2431 (2019). [CrossRef]  

14. Z. Wu, C. Zuo, W. Guo, T. Tao, and Q. Zhang, “High-speed three-dimensional shape measurement based on cyclic complementary Gray-code light,” Opt. Express 27(2), 1283–1297 (2019). [CrossRef]  

15. S. Feng, Q. Chen, and C. Zuo, “Graphics processing unit-assisted real-time three-dimensional measurement using speckle-embedded fringe,” Appl. Opt. 54(22), 6865–6873 (2015). [CrossRef]  

16. H. Nguyen, D. Nguyen, Z. Wang, H. Kieu, and M. Le, “Real-time, high-accuracy 3D imaging and shape measurement,” Appl. Opt. 54(1), A9–A17 (2015). [CrossRef]  

17. W. Bao, X. Xiao, Y. Xu, and X. Zhang, “Reference image based phase unwrapping framework for a structured light system,” Opt. Express 26(22), 29588–29599 (2018). [CrossRef]  

18. K. Liu, Y. Wang, D. L. Lau, Q. Hao, and L. G. Hassebrook, “Dual-frequency pattern scheme for high-speed 3-D shape measurement,” Opt. Express 18(5), 5229–5244 (2010). [CrossRef]  

19. J. Song, D. L. Lau, Y.-S. Ho, and K. Liu, “Automatic look-up table based real-time phase unwrapping for phase measuring profilometry and optimal reference frequency selection,” Opt. Express 27(9), 13357–13371 (2019). [CrossRef]  

20. K. Liu, J. Song, D. L. Lau, X. Zheng, C. Zhu, and X. Yang, “Reconstructing 3D point clouds in real time with look-up tables for structured light scanning along both horizontal and vertical directions,” Opt. Lett. 44(24), 6029–6032 (2019). [CrossRef]  

21. J. Wang, Z. Xiong, Z. Wang, Y. Zhang, and F. Wu, “FPGA design and implementation of Kinect-like depth sensing,” IEEE Trans. Circuits Syst. Video Technol. 26(6), 1175–1186 (2016). [CrossRef]  

22. G. Zhan, H. Tang, K. Zhong, Z. Li, Y. Shi, and C. Wang, “High-speed FPGA-based phase measuring profilometry architecture,” Opt. Express 25(9), 10553–10564 (2017). [CrossRef]  

23. M. Humenberger, C. Zinner, M. Weber, W. Kubinger, and M. Vincze, “A fast stereo matching algorithm suitable for embedded real-time systems,” Comput. Vis. Image Underst. 114(11), 1180–1202 (2010). [CrossRef]  

24. R. G. Lins, S. N. Givigi, and P. R. G. Kurka, “Vision-based measurement for localization of objects in 3-D for robotic applications,” IEEE Trans. Instrum. Meas. 64(11), 2950–2958 (2015). [CrossRef]  

25. S.-C. Kim and E.-S. Kim, “Effective generation of digital holograms of three–dimensional objects using a novel look-up table method,” Appl. Opt. 47(19), D55–D62 (2008). [CrossRef]  

26. S.-C. Kim, J.-M. Kim, and E.-S. Kim, “Effective memory reduction of the novel look-up table with one-dimensional sub-principle fringe patterns in computer-generated holograms,” Opt. Express 20(11), 12021–12034 (2012). [CrossRef]  

27. S. Jiao, Z. Zhuang, and W. Zou, “Fast computer generated hologram calculation with a mini look-up table incorporated with radial symmetric interpolation,” Opt. Express 25(1), 112–123 (2017). [CrossRef]  

28. K. Liu, Y. Wang, D. L. Lau, Q. Hao, and L. G. Hassebrook, “Gamma model and its analysis for phase measuring profilometry,” J. Opt. Soc. Am. A 27(3), 553–562 (2010). [CrossRef]  

29. Y. Wang, K. Liu, Q. Hao, X. Wang, D. L. Lau, and L. G. Hassebrook, “Robust active stereo vision using Kullback-Leibler divergence,” IEEE Transactions on Pattern Analysis Mach. Intell. 34(3), 548–563 (2012).

30. D. L. Lau, K. Liu, and L. G. Hassebrook, “Real-time three-dimensional shape measurement of moving objects without edge errors by time synchronized structured illumination,” Opt. Lett. 35(14), 2487–2489 (2010). [CrossRef]  

Cited By

Optica participates in Crossref's Cited-By Linking service. Citing articles from Optica Publishing Group journals and other participating publishers are listed here.

Alert me when this article is cited.


Figures (12)

Fig. 1.
Fig. 1. The pipeline for the defination of PMP $\langle K, M, R \rangle$ .
Fig. 2.
Fig. 2. With $\gamma =2.2$ and $\mathrm {PMP}\;\langle 3,4,1 \rangle$ , nonlinearly distorted phases of each group: (a) $\phi _0$ , (b) $\phi _1$ , (c) $\phi _2$ , and (d) $\phi _3$ . Besides nonlinear distortion, there are phase shifting errors in (b) and (d).
Fig. 3.
Fig. 3. Compensated (a) $\phi _1$ and (b) $\phi 3$ for those in Figs. 2(b) and (d).
Fig. 4.
Fig. 4. (a) $\mathrm {MLUT}_{3}$ and (b) $\mathrm {PLUT}_{3}^{\left (0\right )}$
Fig. 5.
Fig. 5. (a) $\mathrm {MLUT}_{4}$ and (b) $\mathrm {PLUT}_{4}^{\left (0\right )}$
Fig. 6.
Fig. 6. (a) $\mathrm {MLUT}_{6}$ and (b) $\mathrm {PLUT}_{6}^{\left (0\right )}$
Fig. 7.
Fig. 7. Experimental setup.
Fig. 8.
Fig. 8. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 3,4,1 \rangle $ , $\mathrm {PMP}\;\langle 3,1,4 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $ , and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 3,4,1 \rangle $ and $\mathrm {PMP}\;\langle 12,1,1 \rangle $ .
Fig. 9.
Fig. 9. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 4,3,1\rangle $ , $\mathrm {PMP}\;\langle 4,1,3\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ , and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 4,3,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ .
Fig. 10.
Fig. 10. (a) Modulation and (b) phase errors of $\mathrm {PMP}\;\langle 6,2,1\rangle $ , $\mathrm {PMP}\;\langle 6,1,2\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ , and (c) modulation and (d) phase difference between $\mathrm {PMP}\;\langle 6,2,1\rangle $ and $\mathrm {PMP}\;\langle 12,1,1\rangle $ .
Fig. 11.
Fig. 11. The front and side views of the 3D point clouds of a plaster statue reconstructed by using $\mathrm {PMP}\;\langle 4,3,1\rangle $ without using Eq. (7) to correct wrap errors which cause lots of outliers.
Fig. 12.
Fig. 12. The front and side views of the 3D point clouds of a plaster statue reconstructed by using (a) $\mathrm {PMP}\;\langle 3,4,1\rangle $ , (b) $\mathrm {PMP}\;\langle 4,3,1\rangle $ , (c) $\mathrm {PMP}\;\langle 6,2,1\rangle $ , and (d) $\mathrm {PMP}\;\langle 12,1,1\rangle $ , respectively.

Tables (7)

Tables Icon

Table 1. The size and number of LUTs in each case.

Tables Icon

Table 2. Comparison of the computation complexities.

Tables Icon

Table 3. Compensation factors

Tables Icon

Table 4. Comparing the accuracy of the proposed with the traditional method.

Tables Icon

Table 5. Comparing the speeds of the proposed LUT-based methods with the traditional method on modulation computation.

Tables Icon

Table 6. Comparing the speeds of the proposed LUT-based methods with the traditional method on phase computation.

Tables Icon

Table 7. Comparing the speeds of the proposed One-LUT P M P 4 , M , 1 with the traditional P M P 4 M , 1 , 1 on phase computation.

Equations (12)

Equations on this page are rendered with MathJax. Learn more.

I n = A + B cos ( ϕ 2 π n N ) ,
{ B = 2 N S 2 + C 2 ϕ = tan 1 ( S C ) w h e r e { S = n = 0 N 1 I n sin ( 2 π n N ) C = n = 0 N 1 I n cos ( 2 π n N ) .
G m = { I n   |   n = m + k M     a n d     k [ 0 , K 1 ] } ,
{ B m = 2 N S m 2 + C m 2 ϕ m = [ tan 1 ( S m C m ) + 2 π m K M ]   m o d   2 π w h e r e { S m = k = 0 K 1 I m + k M sin ( 2 π k K ) C m = k = 0 K 1 I m + k M cos ( 2 π k K )
{ B ¯ = 1 M m = 0 M 1 B m ϕ ¯ = 1 M m = 0 M 1 ϕ m ,
P M P K , M , R ,
ϕ m = { ϕ m + 2 π     i f     ϕ 0 ϕ m > π ϕ m 2 π     i f     ϕ 0 ϕ m < π
Δ ϕ m ( K ) = tan 1 [ h = 1 ( B ^ h K + 1 B ^ h K 1 ) sin ( h 2 π m M ) B ^ 1 + h = 1 ( B ^ h K + 1 + B ^ h K 1 ) cos ( h 2 π m M ) ] ,
ϕ m = [ tan 1 ( S m C m ) + 2 π m K M + Δ ϕ m ( K ) ]     m o d     2 π ,
{ M L U T 3 ( Y 3 , X 3 ) = 1 3 3 Y 3 2 + X 3 2 P L U T 3 ( 0 ) ( Y 3 , X 3 ) = tan 1 ( 3 Y 3 X 3 ) P L U T 3 ( m ) ( Y 3 , X 3 ) = [ P L U T 3 ( 0 ) ( Y 3 , X 3 ) + C L U T 3 ( m ) ]     m o d     2 π ,
{ M L U T 4 ( Y 4 , X 4 ) = 1 2 Y 4 2 + X 4 2 P L U T 4 ( 0 ) ( Y 4 , X 4 ) = tan 1 ( Y 4 X 4 ) P L U T 4 ( m ) ( Y 4 , X 4 ) = [ P L U T 4 ( 0 ) ( Y 4 , X 4 ) + C L U T 4 ( m ) ]     m o d     2 π ,
{ M L U T 6 ( Y 6 , X 6 ) = 1 6 3 Y 6 2 + X 6 2 P L U T 6 ( 0 ) ( Y 6 , X 6 ) = tan 1 ( 3 Y 6 X 6 ) P L U T 6 ( m ) ( Y 6 , X 6 ) = [ P L U T 6 ( 0 ) ( Y 6 , X 6 ) + C L U T 6 ( m ) ]     m o d     2 π ,
Select as filters


Select Topics Cancel
© Copyright 2024 | Optica Publishing Group. All rights reserved, including rights for text and data mining and training of artificial technologies or similar technologies.