Open Access
Add to BibSonomyAbstract
A new projection pattern control technique is presented in an attempt to solve the problem whereby an image having an ideal intensity distribution cannot be photographed when measurement conditions, such as object color or object surface reflection, change. The proposed technique can adjust the intensity distribution of a projection pattern automatically, according to changes in the measurement conditions. An image with an ideal intensity distribution can then be obtained in a short time, approximately three projections on average. Thus, the speed, robustness, and practicality of 3-D image measurement can be improved.
©2005 Optical Society of America
1. Introduction
Three-dimensional (3-D) image measurement techniques based on pattern projection can be divided roughly into binary methods and non-binary methods [1]. Binary methods use binary projection patterns and binary images. Techniques such as spot pattern projection, slit pattern projection, and coding spatial pattern (structured light) projection, are binary methods. Non-binary methods use non-binary projection patterns and monochrome or color images. Examples of non-binary methods are multi-intensity pattern projection, intensity-inclination pattern projection, intensity-modulation pattern projection, and color-modulation pattern projection [2–11].
In order to achieve high measurement accuracy by these methods, it is important to detect stripe order correctly. Binary methods have a high detection accuracy of stripe order, but to compute the stripe order, many projections are needed and the measurement takes time. Non-binary methods can obtain 3-D information of objects in a shorter time than binary methods, because a large amount of information is acquired in one projection. In order to detect the stripe order of a projection pattern correctly in a non-binary method, it is necessary that the observation pattern image being photographed has a large intensity distribution and not be saturated. However, measurement conditions, such as surface color or the surface reflection factor of the object, can change the ideal observation pattern image and an ideal intensity may not be obtained. As such, the detection accuracy of stripe order decreases or measurement becomes impossible.
Either the projected light automatic control method or the camera parameter automatic control method may be used to automatically obtain a reflection pattern image having an ideal intensity distribution [12, 13]. The automatic mode of the camera, which adjusts the shutter speed and iris diaphragm, for example, automatically, is considered. In automatic mode, the intensity value of one specific point (or two or more points) specified beforehand can be controlled, and the intensity distribution of the entire image can be distributed within ideal limits. However, since the camera cannot extract the measurement object automatically from the observation pattern image, the intensity of the stripes reflected by the measurement object cannot be distributed within required limits. As such, it is difficult to automatically obtain the ideal reflection pattern image for 3-D measurement.
The present paper proposes a technique for the automatic control of projection pattern intensity distribution in addition to camera parameter regulation. Using this technique, the intensity distribution of a projection pattern can be adjusted automatically according to surface color or reflection factor distribution, and the ideal projection pattern for 3-D measurement is generated automatically.
2. Measurement principle
The intensity distribution of an ideal observation pattern image is given in Fig. 1. In this figure, Imax is the maximum and Imin is the minimum intensity value of the image system. For such an intensity distribution, highly precise stripe order detection is possible, because most of the intensity information is used.
However, an image with an ideal histogram might not be obtained, due to the influence of the measurement environment, such as the color and surface reflection factor of the measurement object. Figure 2 is an example of this phenomenon. Figure 2(a) is the original image of two objects with different surface reflection factors. Figure 2(b) is the observation pattern image obtained when an intensity-modulation stripe pattern is projected on the objects. Figure 2(c) is the intensity distribution of the AA’ line on the left object, and Fig. 2(d) is the intensity distribution of the BB’ line on the right object. Figures 2(e) and 2(f) are histograms of the left and right objects, respectively. The intensity range of the stripes on the left object is from grade 30 to grade 200, which is ideal for measurement. In comparison, the intensity range of the right object is from grade 0 to grade 90, which is very narrow. Detection of the stripe order from the stripe intensity change in the narrow intensity range shown in Fig. 2(d) is very difficult. Therefore, in order to obtain the intensity distribution of the observation pattern image in the required range, it is necessary to adjust the intensity distribution of the projection pattern.
Fig. 2. Observation images and intensity distributions of two objects with different surface reflection factors: (a) Original image, (b) Observation image, (c) Intensity distribution of line AA’, (d) Intensity distribution of line BB’, (e) Histogram of left object, and (f) Histogram of right object.
In order to obtain an ideal image within the shortest time, an intensity control algorithm of the projection pattern, shown in Fig. 3, is proposed. The explanation of each process is as follows:
Step 1: The initial projection pattern is a uniform-intensity no-stripe pattern. The intensity takes the mean value of the projection intensity range (which is set at 128 in this paper).
Step 2: The initial projection pattern reflected by the object is photographed as an RGB color image. This is called the observation pattern image.
Step 3: The measurement object is extracted from the observation pattern image using the background comparison method.
Step 4: In every pixel of the extracted object image, the color and intensity distributions are detected [14]. The channel of the observation pattern image intensity maximum is chosen to be the measurement channel of the pixel by
where l is the measurement channel, l∊{R,G,B} ; (i, j) are the image coordinates of the current pixel; and IR, IG and IB are the intensities of the R, G and B channels, respectively, of the current pixel.
Since the frequent changes in the measurement channel reduces the measurement stability in a certain area, the choice of the measurement channel is corrected so that distribution of the measurement channel is changed as little as possible. For example, when l=R, the correction can be written as
where δ is the size of the area around the pixel (i, j), which influences the channel choice of the pixel (i, j), the value is settled by image size. For the case in which the image size is 1024×768 pixels, we generally set δ=3~7. λ is a coefficient used to compare the intensity value of the measurement channel to that of the other channel. For example, R channel was chosen as the measurement channel by formula (1) for pixel (i, j). However, most of the measurement channel of the pixels around pixel (i, j) are G, and the intensity value of the G channel of pixel (i, j) is not smaller than λ times the intensity value of the R channels. In this case, the measurement channel of the pixel (i, j) is changed into G from R. Since the value of λ is small, the stability of the choice of the measurement channel is higher. However, if the value of λ is too small, the intensity range of the measurement image will become small. We generally set λ=0.5~1.
Step 5: The purpose of camera parameter regulation is to roughly adjust the intensity range of the observation pattern image. Fine regulation is realized by intensity control of the projection pattern. The intensity distribution of the observation pattern image in the measurement channel is compared with an ideal intensity distribution, and the camera parameter is adjusted if the difference exceeds a certain value.
Step 6: In the present study, the iris diaphragm is fixed and the shutter speed is adjusted as follows:
where n is the number of projections (n=0 when projecting the initial pattern), Vn is the shutter speed when carrying out the n-th photograph, C is a constant, Sn is the stripe maximum intensity value of the observation pattern image for the n-th projection, and S is the desired value of the stripe maximum intensity in the observation pattern image. In the present paper, the position at which the area of the histogram reaches 2% from the right is defined as the stripe maximum intensity value for control. This is done in order to suppress the influence of noise or highlights.
When using a general-purpose camera, the camera parameter cannot be set to an arbitrary value. For this reason, a shutter speed value near the value calculated using formula (5) is used. The error produced by this approximation is corrected by the k2 clause of formula (6).
Step 7: The maximum intensity of stripes of the projection pattern is calculated as follows:
where k 1–k 4 are constants.
The maximum intensity of stripes of the 1st projection pattern is decided by the clauses of k1 and k2. The clause of k1 is a simple proportionality control component. The clause of k2 is a compensation component accompanying regulation of the shutter speed of the camera, the value of which is 0 when there is no change in the camera shutter speed. The maximum intensity of stripes of the 2nd projection is decided by the clause of k3, which is a simple proportionality control. In order to shorten the control time and minimize the regular error, the maximum intensity of stripes of the 3rd projection henceforth uses the original proportionality integral control, which is shown in the clause of k4. The technique of differential control is often employed when the goal is high-speed control, but since the influence on stability is large, this control method is not employed herein.
Steps 8–10: The processes from 8 to 10 of Fig. 3 are straightforward and therefore their explanation is not included herein.
Step 11: When the object has a nonuniform surface, the stripe intensity of the observation pattern images of the same order are not necessarily the same. In this case, it is necessary to use the technique of Ref. [9] to correct the intensity distribution of the compound image:
where Il(i, j) and I0(i, j) are the intensities of the measurement image and the initial observation pattern image, respectively, M(n) is the intensity modulation function of the projection pattern, O(x,y) is the reflection function of the object, and k and k’ are constants. The stripe intensity of the corrected image is dependent only on the intensity modulation function of the projection pattern.
Step 12: Finally, the 3-D representation can be obtained by the fringes intensity of corrected measurement image.
Fig. 4. Measurement results using the proposed technique: (a) Initial projection pattern, (b) Initial observation image, (c) Measurement channel image of (b), (d) Measurement image of (b), (e) Adjusted projection pattern, (f) Observation image by pattern (e), (g) Measurement channel image of (f), (h) Measurement image of (f), (i) Histogram of (d), (j) Histogram of (h), (k) Corrected measurement image, (l) Intensity distribution of line AA’, and (m) Intensity distribution of line BB’.
3. Experimental results and evaluation
Figure 4 shows an example of a measurement using the abovementioned intensity control technique. The measurement object is a wooden box with complicated surface color and discontinuous form. Experimental data are listed in Table 1. The projector is a liquid crystal projector having a spatial resolution of 1024×768 pixels and an intensity resolution of 8 bits. The camera is an 8-bit 1024×768 pixels 3-CCD color camera, and the photography speed is 7.5 frames per second. The personal computer used in the present study had a 2-GHz Athlon™ 64 CPU, 1 GB of RAM and the OS of Microsoft Windows XP Professional.
Table 1. Experimental Data for Intensity Control of Projection Pattern
Note: S=210
Figure 4(a) is the initial projection pattern, the intensity of which is 128. Figures 4(b), 4(c), and 4(d) are the initial observation pattern image, the measurement channel image and the single-channel measurement image, respectively. Figure 4(i) is the histogram of the measurement image. The maximum intensity value S0 (red line) of the object is approximately 141, and the difference between S0 and the ideal value S (green line, S=210) is fairly large. Figure 4 (e) is the projection pattern adjusted using the proposed technique. The projection intensity is from 40 to 205. Figure 4(f) is the observation pattern image measured using the projection pattern of Fig. 4 (e). Figure 4(g) is the measurement channel image, and Fig. 4(h) is the measurement image of Fig. 4(f). Figure 4(j) is the histogram of Fig. 4(h), and S2 (red line) and S (green line) are in approximate agreement. Figure 4(k) is the corrected measurement image, and Figs. 4(l) and (m) are the intensity distributions of line AA’ and line BB’, respectively, of Fig. 4(k). Both are approximate linearity distributions in the range from 50 to 210. By analyzing the above intensity distribution, the detection accuracy of the stripe order reaches 100%, and the 3-D representation can be obtained as shown in Fig. 5.
Fig. 6. Measurement results obtained by camera automatic-mode: (a) Observation image, (b) Measurement image, and (c) Histogram of (b)
For comparison, the experimental results obtained by camera automatic-mode are shown in Fig. 6. Figures 6(a) and 6(b) show the reflected pattern and measurement image, respectively. The measurement object is the same as the object shown in Fig. 4 and the projection pattern is as shown in Fig. 4(e). Figure 6(c) shows the histogram for Fig. 6(b). Since there is no portion with a large intensity value of the reflection pattern on the object, the range of the stripe intensity reflected from the object is insufficient.
In order to evaluate the validity of the proposed technique, a total of 12 objects of different materials, color distributions, textures, and surface reflectances were measured. The measurement objects are shown in Fig. 7, and the measurement results are listed in Table 2. The proposed technique was effective for all of the measurement objects. The average number of adjustments of the projection pattern intensity was two and the control error of the intensity was less than 2 (1%).
Table 2. Experimental Results of Color-Intensity Control of Projection Pattern
Note: P0=128, S=201.
The measurement time is shown in the last two columns of Table 2. The process time contains the times for multiple projection pattern generations, the times for multiple pattern projection and reflection image processing, and 3-D coordinate computation time. The total time is the total of the process time, the image capture time, and the image data transference time. The total time is the comprehensive measurement time from measurement object discovery until the three-dimensional information calculation. The measuring time is mainly dependent on the measurement algorithm, the camera, and the computer. Although the process time reflects the measurement speed of the proposed algorithm, it changes with the calculation speed of the computer and the response speed of the projector, for example. The image capture time and the image data transference time depend on the performance of the camera and computer. By using a high speed camera, the measurement time can be shortened further.
4. Conclusion
A technique for photographing an image that has an ideal intensity distribution using a small number of projections is greatly needed for 3-D optical measurement based on pattern projection. Therefore, a robust measurement technique that can automatically adjust the intensity distribution of the projection pattern according to changes in the surface color, surface reflective distribution of the measured object, etc., is required.
The stage formula projection pattern intensity control method was proposed herein. Using this method, observation pattern images with ideal intensity distributions can be obtained in a short time with an average of three projections on objects of various materials and colors. Using the proposed technique, a 3-D image measurement system can be built using standard equipment, such as a general-purpose liquid crystal projector and a CCD color camera.
High-accuracy detection of stripe order of projection patterns is an important technology for non-binary 3-D image measurement methods. The technique proposed in the present paper increases the detection accuracy of stripe order, particularly when used in conjunction with the previously proposed optimal combination technique of projection pattern stripe intensity [15].
Acknowledgments
This research was supported in part by a research cost subsidy from the Japanese Ministry of Education, Culture, Sports, Science and Technology.
References and links
1. J. Batlle, E. Mouaddib, and J. Salvi, “Recent progress in coded structured light as a technique to solve the correspondence problem: a survey,” Pattern Recogn. 31, 963–982 (1998). [CrossRef]
2. E. Horn and N. Kiryati, “Toward optimal structured light patterns,” Image Vision Comput. 17, 87–97 (1999). [CrossRef]
3. Y. C. Hsieh, “Decoding structured light patterns for three dimensional imaging systems,” Pattern Recogn. 34, 343–349 (2001). [CrossRef]
4. B. Carrihill and R. Hummel, “Experiments with the intensity ratio depth sensor,” Computer Vision, Graphics and Image Processing 32, 337–358(1985). [CrossRef]
5. M. Ito and A. Ishii, “A Three-Level Checkerboard Pattern (TCP) Projection Method for Curved Surface Measurement,” Pattern Recogn. 28, 27–40 (1995). [CrossRef]
6. K. Kalms, P. Jueptner, and W. Osten, “Automatic adaptation of projected fringe patterns using a programmable LCD-projector,” in Sensors, Sensor Systems, and Sensor Data Processing, O. Loffeld, ed., Proc. SPIE3100, 156–165(1997).
7. E. Schubert, “Fast 3-D object recognition using multiple color coded illumination,” in Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (Institute of Electrical and Electronics Engineers, New York, 1997), pp. 3057–3060.
8. D. Caspi, N. Kiryati, and J. Shamir, “Range imaging with adaptive color structured light,” IEEE Trans. Pattern Analysis and Machine Intelligence 20, 470–480 (1998). [CrossRef]
9. C. Lu and S. Inokuchi, “Intensity-Modulated Moiré Topography,” App. Opt. 38, 4019–4029(1999). [CrossRef]
10. H. Hioki, “Adaptive light projection and highlight analysis method for measuring three-dimensional scenes,” in Proceedings of IEEE International Conference on Image Processing (Institute of Electrical and Electronics Engineers, New York, 2000), 1, pp. 565–568.
11. G. H. Notni, P. Kühmstedt, M. Heinze, and G. Notni, “Method for Simultaneous Measurement of 3-D Shape and Color Information of Complex Objects,” in Proceedings of the International Symposium on Photonics in Measurement, T. Pfeifer and W. Holzapfel, eds. (Aachen, Germany, 2002), pp. 293–298.
12. G. Sansoni, L. Biancardi, U. Minoni, and F. Docchio, “A Novel, Adaptive System for 3-D Optical Profilometry Using a Liquid Crystal Light Projector,” IEEE Trans. Instrumentation and Measurement 43, 558–565 (1994). [CrossRef]
13. X. Zhao, T. Suzuki, and O. Sasaki, “Photothermal Phase-Modulating Laser Diode Interferometer with High-Speed Feedback Control,” Opt. Rev. 9, 13–17 (2002). [CrossRef]
14. C. Lu, G. Cho, and J. Zhao, “Practical 3-D Image Measurement System using Monochrome-Projection Color-Analysis Technique,” in Proceedings of the 7th IASTED International Conference on computer graphics and imaging, M. H. Hamza, ed. (Hawaii, USA, 2004), pp. 254–259.
15. C. Lu and L. Xiang, “Optimal Intensity-Modulation Projection Technique for Three-Dimensional Shape Measurement,” App. Optics-IP , 42, 4649–4657 (2003). [CrossRef]
