Scaled SFS method for Lambertian surface 3D measurement under point source lighting

Long Ma; Yi Lyu; Xin Pei; Yan Min Hu; Feng Ming Sun

doi:10.1364/OE.26.014251

1. Introduction

SFS is a common Nondestructive Testing (NDT) technology which can recover the 3D shape from a single image by analyzing the light intensity distribution. Due to its less requirements on system complexity, it has becoming increasingly important and shown great potential on medical testing [1], endoscope [2], remote measurement [3,4] etc.

Lambertian surface is a general assumption in many practical applications like the ground surface [5], human face [6] and certain mechanical parts [7]. Under this assumption, many efforts have been made for the application of SFS on 3D measurements [8–10]. However, parallel light illumination assumption is used for most of SFS applications, which is difficult to be guaranteed and may bring errors to the result [11,12]. To solve this problem, some researches focus the methods based on the point light source model, for example, Huang et al. [13] developed a method employed point light source and showed several advantages over the methods using parallel source model. Ju et al. [14] used a point light source model to the minimization SFS method and testified its feasibility using the synthetic images. Unfortunately, these researches [13,14] together with the other related works [15–18] use too many simplifications and neglect the influences like the attenuation of the light flux and the anisotropy of the source, which may lead the methods be inaccurate. Moreover, the SFS by itself cannot derive the dimension information from the measurements [19,20]. Some methods were proposed by incorporating SFS with other optical methods [21,22]. However, these methods require additional equipment and more complex algorithm, which make them hard to be realized.

In this paper, a novel scaled SFS method under the illumination of a point light source is proposed combining a more accurate point light source model with the intrinsic matrix of the camera. This method can measure the shape of the surface with dimension using only one image. The experiments show that the precision of the method is about 1% using the synthetic images and 2% using the practical images.

2. Modeling of point light source

2.1. Point light source model

In SFS technology, light source model is very important for the measurement accuracy. Compared with these simple models for the light source [13–18], an overall model of the point light source is developed, which builds a detailed relationship among the point light source, Lambertian surface and grayscale of charge coupled device(CCD) pixels, and consider four important processes during the propagation of the light including the anisotropy of the source, the reflection on the Lambertian surface, the refraction in the lens of camera and the attenuation of the light flux during propagation.

The intensity I_s(ϕ) of a light ray from a source will depend on its direction of propagation due to the anisotropy of the source, which can be expressed as shown in Eq. (1),

I_{s} (ϕ) = I_{0} \cos^{μ} (ϕ)

where ϕ is the angle between the optical axis of the source and the vector of the light flux, I₀ is the intensity along the optical axis, and μ is the intensity distribution parameter of the source.

When the light beam reaches point M on the Lambertian surface, its intensity E(M) after reflection satisfies Eq. (2),

E (M) = \frac{ρ}{π} I_{s} (ϕ) \cos (θ)

where θ is the angle between the normal vector of point M and the vector of incoming ray, ρ describes the albedo of point M, which is treated as a constant.

Here notes pixel m is the projection of point M. Then, the light intensity ε(m) at m can be expressed as

ε (m) = E (M) \frac{π}{4} {(\frac{d}{f})}^{2} \cos^{4} (α)

where d and f are the diameter and focal length of the camera lens, α is the angle between the vector of the arriving light beam at pixel m and the optical axis of the camera.

Finally, we obtain the expression of the recorded grayscale R_org(m) of pixel m by adding the attenuation factor 1/r(M)² and the proportionality coefficient of CCD β, as shown in Eq. (4),

{\begin{matrix} R_{o r g} (m) = K \frac{1}{r {(M)}^{2}} \cos^{4} (α) \cos (θ) \cos^{μ} (ϕ) \\ K = β I_{0} \frac{ρ}{4} {(\frac{d}{f})}^{2} \end{matrix}

where r is defined by the distance between M and source.

2.2. Evaluation of model

The proposed model was evaluated by using a Lambertian plane. We use a single LED manufactured by CREE company as the point light source, the distribution of the luminous intensity was recorded by a CCD camera. The evaluation result is shown in Fig. 1. Figure 1(a) is the evaluation result of the point light source model given in section 2.1, where the colored point cloud refers to the actual distribution and the black ones indicate simulated intensity distribution according to the Eq. (4). Figure 1(b) shows the deviation between them, where the RMSE equals to 1.89. The accuracy of the model proposed in this work is verified.

Fig. 1 Evaluation of the point light model.

Download Full Size | PDF

3. Scaled SFS under point lighting

3.1. Combination with intrinsic matrix

For convenience, we firstly define the world coordinate frame F_W, the camera frame F_C and the image coordinate frame F_I. In this paper, F_W and F_C are set be coincident to simplify the problem. Supposing in F_C a 3D world point M = [X,Y,Z]^T is projected into the image domain at a 2D homogenous point m = [x,y,1]^T. The relationship between m and its counterpart point M is summarized as

s m = B M

where s = Z is called the depth, and B is the intrinsic matrix. Here note should be paid s is the only variable in Eq. (5) to be determined. Once knowing the intrinsic matrix, we can get the back projection for each pixel with the help of Eq. (5) as

M = s B^{- 1} m

Now, we can use the variable s to represent the terms of Eq. (4) by combining Eq. (6). We take L_p as the coordinates of the source in F_C, L_a as the vector of the source’s optical axis, L as the vector of the ray arriving at M, V as the normal vector at M, and f_p as the focal length with unit of pixel. Again, note that L_p and L_a are the parameters of point source and can be calculated through calibration of the system. Then we can easily get

Z (X, Y) = 300

Meanwhile, Tankus A has proven that ∂M/∂X × ∂M/∂Y and ∂M/∂x × ∂M/∂y are collinear [23], which means F_W and F_I are interchangeable during the computation. Constant parameter K in Eq. (4) and cos(α) are independent with the shape of the surface. Thus, we can further simplify the Eq. (4) by eliminating cos(α) and K through R(m) = R_org(m)/Kcos⁴(α). Combining the Eqs. (4) and (7) yields the expression of R(m), the proposed model can be expressed as

\begin{matrix} R (m) = \frac{{({(M - L_{p})}^{T} L_{a})}^{μ}}{{(L_{a}^{T} L_{a})}^{\frac{μ}{2}} {({(M - L_{p})}^{T} (M - L_{p}))}^{1 + \frac{μ}{2}}} \\ * \frac{{(\frac{\partial M}{\partial x} \times \frac{\partial M}{\partial y})}^{T} (M - L_{p})}{\sqrt{{(\frac{\partial M}{\partial x} \times \frac{\partial M}{\partial y})}^{T} (\frac{\partial M}{\partial x} \times \frac{\partial M}{\partial y})} \sqrt{{(M - L_{p})}^{T} (M - L_{p})}} \end{matrix}

3.2. Iteration of the depth map

We use three constraints including the brightness constraint C_L, the smoothness constraint C_S and the integrability constraint C_I to build the cost function, as shown by the following

{\begin{cases} C_{L} = \int_{Ω}^{} {(I (m) - R (m))}^{2} d Ω \\ C_{S} = \int_{Ω}^{} {(M_{x x}^{T} M_{x x} + M_{y y}^{T} M_{y y})}^{2} d Ω \\ C_{I} = \int_{Ω}^{} {(M_{x y} - M_{y x})}^{T} (M_{x y} - M_{y x}) d Ω \end{cases}

where Ω represents the image domain (x,y), M_xx = ∂(∂M/∂x)/∂x, M_yy = ∂(∂M/∂y)/∂y, M_xy = ∂(∂M/∂x)/∂y, M_yx = ∂(∂M/∂y)/∂x, R(m) is calculated from Eq. (8), and I(m) indicates the grayscale at m recorded by the camera. Here, the brightness constraint C_L is a vital constraint to guarantee the consistency between the simulated intensity map and that acquired by the CCD camera, C_S and C_I are the auxiliary constraints that ensure the final reconstruction should be smooth and valid. By adding the weighting coefficient, the final cost function is

\begin{matrix} C = λ_{L} C_{L} + λ_{S} C_{S} + λ_{I} C_{I} \\ = \int_{Ω}^{} f (s, s_{x}, s_{y}, s_{x x}, s_{y y}, s_{x y}, s_{y x}) d Ω \end{matrix}

Through the calculus of variations and gradient descent optimization, the iteration format of s can be derived as

s^{t + 1} = s^{t} - δ \frac{\partial C}{\partial s}

where δ is an augment factor and ∂C/∂s can be calculated through Eq. (12).

\begin{matrix} \frac{\partial C}{\partial s} = \frac{\partial f}{\partial s} - \frac{\partial}{\partial x} (\frac{\partial f}{\partial s_{x}}) - \frac{\partial}{\partial y} (\frac{\partial f}{\partial s_{y}}) + \frac{\partial^{2}}{\partial x^{2}} (\frac{\partial f}{\partial s_{x x}}) \\ + \frac{\partial^{2}}{\partial y^{2}} (\frac{\partial f}{\partial s_{y y}}) + \frac{\partial^{2}}{\partial x \partial y} (\frac{\partial f}{\partial s_{x y}}) + \frac{\partial^{2}}{\partial y \partial x} (\frac{\partial f}{\partial s_{y x}}) \end{matrix}

3.3. Simulation with the synthetic examples

To evaluate the proposed method, the synthetic images of three types of structure were created according to the Eq. (4). Table 1 shows the corresponding mathematical expression and the range of variables in X and Y axis. Figure 2 shows the result together with their RMSE by using our method, where the first column is the generated images, the corresponding depth maps are shown in the second column, the initial depth maps of iteration and the calculated results are shown in the third and final column.

Table 1. Formula and the range of variables (unit in mm).

View Table | View all tables in this article

Fig. 2 Evaluation of the point light model.

Download Full Size | PDF

From the result, one can find that the RMSE increases with the range of depth. We divide the RMSE by their range in Z direction and find that their percentage errors equal to 1.19%, 1.01% and 1.16%, which proves the precision of proposed method.

4. Experiment and analysis

The experiment was carried out by measuring a PVC tube to validate our proposal, the inner surface of PVC tube can be treated as a perfect Lambertian surface with the inner diameter of 100mm, as shown in Fig. 3. A LED was used as the point light source, the image was recorded by a CCD camera with resolution of 640*480.

Fig. 3 Tube used in the measurement.

Download Full Size | PDF

The result is shown in Fig. 4. Figure 4(a) is the 3D point cloud of the tube calculated by the method proposed, Fig. 4(b) shows the comparison between the measured value and truth value of the profiles. It is obvious that the measurement results are in accordance with the actual shape of the tube.

Fig. 4 Tube measurement result.

Download Full Size | PDF

The deviation between the result and the 3D model of the tube is also analyzed as shown in Fig. 5. The statistics of the deviation are shown in Table 2, where the third column is obtained through dividing the second column by the radius of the tube. One can see that the RMSE of our method is 1.97% which is slightly higher than the simulation. This result proves the applicability of our method in the 3D measurement of Lambertian surface.

Fig. 5 Tube measurement result.

Download Full Size | PDF

Table 2. Statistics of deviation.

View Table | View all tables in this article

5. Conclusion

This paper presented a scaled SFS method under the near point source lighting for the 3D measurement of Lambertian surface. In our method, a more practical near point light source model instead of the traditional parallel source model is studied. The new model takes into consideration of the attenuation of light flux, anisotropic of the source and the refraction of light flux at the camera lens, which is more suitable for the actual case. Coupled with the intrinsic matrix of camera, the relationship between the surface depth map and the recorded image grayscale is established, which enables the proposed method to measure a Lambertian surface with dimensions. Further analysis of the measurements using both the synthetic and practical images verified the feasibility and precision of the new method.

Funding

National Natural Science Foundation of China and the Civil Aviation Administration of China (CAAC) (U1633101, U1733119); The Fundamental Research Funds for the Central Universities of Civil Aviation University of China special funds (3122014H004).

References and links

1. M. T. E. Melegy, A. S. Abdelrahim, and A. A. Farag, “Better Shading for Better Shape Recovery,” In CVPR, 2307–2312 (2014)

2. D. Roxo, N. Gonçalves, and J. P. Barreto, “Perspective shape from shading for wide-fov near-lighting endoscopes,” in Iberian Conference on Pattern Recognition and Image Analysis, Springer, Berlin, Heidelberg (2013), pp. 21–30. [CrossRef]

3. H. Tang, L. Yan, and P. Gao, “A modified SFS algorithm based on stereo images for the three-dimension reconstruction of Urban buildings,” in Proceedings of IEEE Conference on Geoscience and Remote Sensing Symposium (IEEE, 2009), pp. IV-390. [CrossRef]

4. G. D. Martino, A. D. Simone, A. Iodice, D. Riccio, and G. Ruello, “SAR Shape from Shading in suburban areas,” In Urban Remote Sensing Event (JURSE) (2015), pp. 1–4.

5. L. Yang, Y. Xue, Y. Li, C. Li, J. Guang, X. He, J. Dong, and T. Hou, “Uncertainty from Lambertian surface assumption in satellite aerosol retrieval,” in Proceedings of IEEE Conference on Geoscience and Remote Sensing Symposium (IEEE, 2012), pp. 3662–3665. [CrossRef]

6. Y. Sun, J. Dong, M. Jian, and L. Qi, “Fast 3D face reconstruction based on uncalibrated photometric stereo,” Multimedia Tools Appl. 74(11), 3635–3650 (2015). [CrossRef]

7. L. Yang, E. Li, T. Long, J. Fan, Y. Mao, Z. Fang, and Z. Liang, “A welding quality detection method for arc welding robot based on 3D reconstruction with SFS algorithm,” Int. J. Adv. Manuf. Technol. 94(1–4), 1209–1220 (2018). [CrossRef]

8. J. Hou, “A novel cloud surface shape estimation method based on SFS,” in Proceedings of IEEE Conference on Computer Science and Network Technology (IEEE, 2016 5th), pp. 80–83. [CrossRef]

9. Y. Zhang and J. Peng, “Surface shape estimation of textureless area using shape from shading for Landsat imagery,” in Remote Sensing of the Atmosphere, Clouds, and Precipitation V (2014), Vol. 9259, p. 92591S.

10. L. Min, D. Li, and S. Dong, “3D Surface Roughness Measurement Based on SFS Method,” in Proceedings of IEEE Conference on Intelligent Human-Machine Systems and Cybernetics (IEEE, 2016 8th), Vol. 2, pp. 484–488. [CrossRef]

11. J. Wu, P. L. Rosin, X. Sun, and R. R. Martin, “Improving shape from shading with interactive tabu search,” J. Comput. Sci. Technol. 31(3), 450–462 (2016). [CrossRef]

12. H. Bingwei, C. Zhipeng, L. Dongyi, and Z. Xiaolong, “Research on reconstruction method for unknown objects through incorporating SFS algorithm and active vision technology,” PhChin. J. Sci. Instrum. 4, 002 (2012).

13. X. Huang, M. Walton, G. Bearman, and O. Cossairt, “Near light correction for image relighting and 3D shape recovery,” in Proceedings of IEEE Conference on Digital Heritage (IEEE, 2015),pp. 215–222. [CrossRef]

14. Y. C. Ju, A. Bruhn, and M. Breuß, “Variational perspective shape from shading,” in International Conference on Scale Space and Variational Methods in Computer Vision, Springer, Cham, Heidelberg (2015), pp. 538–550.

15. J. Ackermann, S. Fuhrmann, and M. Goesele, “Geometric Point Light Source Calibration,” in VMV (2013), pp. 161–168.

16. T. Aoto, T. Taketomi, T. Sato, Y. Mukaigawa, and N. Yokoya, “Position estimation of near point light sources using a clear hollow sphere,” in Proceedings of IEEE Conference on Digital Heritage (IEEE, 2012), Vol. 1, pp. 3721–3724.

17. A. Giachetti, C. Daffara, C. Reghelin, E. Gobbetti, and R. Pintus, “Light calibration and quality assessment methods for Reflectance Transformation Imaging applied to artworks’ analysis,” in Optics for Arts, Architecture, and Archaeology V (2015), Vol. 9527, p. 95270B.

18. H. L. Shen and Y. Cheng, “Calibrating light sources by using a planar mirror,” J. Electron. Imaging 20(1), 013002 (2011). [CrossRef]

19. J. Ma, P. Zhao, and B. Gong, “A shape-from-shading method based on surface reflectance component estimation,” in Proceedings of IEEE Conference on Fuzzy Systems and Knowledge Discovery (IEEE, 2012 9th), pp. 1690–1693. [CrossRef]

20. J. Wang, F. H. Wu, X. L. Li, and J. C. Wang, “Smoothing of SFS Reconstructed Surface Based on Genetic Algorithm,” Key Eng. Mater. 579, 877–884 (2014).

21. T. S. F. Haines and R. C. Wilson, “Combining shape-from-shading and stereo using Gaussian-Markov random fields,” in Proceedings of IEEE Conference on Pattern Recognition (IEEE, ICPR, 2008), pp. 1–4. [CrossRef]

22. M. G. H. Mostafa, S. M. Yamany, and A. A. Farag, “Integrating stereo and shape from shading,” in Proceedings of IEEE Conference on Image Processing (IEEE, 1999), Vol. 3, pp. 130–134.

23. A. Tankus, A. N. Sochen, and Y. Yeshurun, “Shape-from-shading under perspective projection,” Int. J. Comput. Vis. 60(1), 21–43 (2005).

	Mathematical expression	Range of X	Range of Y
1.	If $\sqrt{X^{2} + Y^{2}} \leq 50$ : $Z (X, Y) = 300 - \sqrt{2500 - X^{2} - Y^{2}}$ Else: $Z (X, Y) = 300$	(−97.85,81.786)	(98.14,79.34)
2.	$Z (X, Y) = 300 + X^{2} + Y^{2}$	(−101.31,101.63)	(−81.84,82.16)
3.	$Z (X, Y) = 300 + X^{2}$	(−37.11,37.25)	(−29.98,30.117)

Parameter	Value(mm)	Percentage(%)
Max positive deviation	3.3525	6.71
Max negative deviation	−2.6566	−5.31
Average positive deviation	0.8437	1.69
Average negative deviation	−0.7797	−1.56
Average deviation	−0.0126	−0.03
Standard Deviation	0.9859	1.97
RMSE	0.9860	1.97

	Mathematical expression	Range of X	Range of Y
1.	If $\sqrt{X^{2} + Y^{2}} \leq 50$ : $Z (X, Y) = 300 - \sqrt{2500 - X^{2} - Y^{2}}$ Else: $Z (X, Y) = 300$	(−97.85,81.786)	(98.14,79.34)
2.	$Z (X, Y) = 300 + X^{2} + Y^{2}$	(−101.31,101.63)	(−81.84,82.16)
3.	$Z (X, Y) = 300 + X^{2}$	(−37.11,37.25)	(−29.98,30.117)

Parameter	Value(mm)	Percentage(%)
Max positive deviation	3.3525	6.71
Max negative deviation	−2.6566	−5.31
Average positive deviation	0.8437	1.69
Average negative deviation	−0.7797	−1.56
Average deviation	−0.0126	−0.03
Standard Deviation	0.9859	1.97
RMSE	0.9860	1.97

Scaled SFS method for Lambertian surface 3D measurement under point source lighting

Abstract

1. Introduction

2. Modeling of point light source

2.1. Point light source model

2.2. Evaluation of model

3. Scaled SFS under point lighting

3.1. Combination with intrinsic matrix

3.2. Iteration of the depth map

3.3. Simulation with the synthetic examples

4. Experiment and analysis

5. Conclusion

Funding

References and links

Cited By

Figures (5)

Tables (2)

Equations (12)

Optics Express