Open Access
Add to BibSonomyAbstract
Shadow is an inseparable aspect of all natural scenes. When there are multiple light sources or multiple reflections several different shadows may overlap at the same location and create complicated patterns. Shadows are a potentially good source of information about a scene if the shadow regions can be properly identified and segmented. However, shadow region identification and segmentation is a difficult task and improperly identified shadows often interfere with machine vision tasks like object recognition and tracking. We propose here a new shadow separation and contrast enhancement method based on the polarization of light. Polarization information of the scene captured by our polarization-sensitive camera is shown to separate shadows from different light sources effectively. Such shadow separation is almost impossible to realize with conventional, polarization-insensitive imaging.
©2006 Optical Society of America
1. Introduction
Shadows are formed whenever an occlusion partially blocks the illumination of a surface or object by a light source. With the exception of the ambient light, which is assumed to be omnidirectional, light sources illuminate surfaces from only one specific direction. In addition to classification by the source direction, shadows are further classified into “self” and “cast”. A “self” shadow refers to the regions of an object not directly illuminated by a light source due to its surface orientation, whereas “cast” shadow refers to a region not illuminated by a source due to occlusion by other objects. Shadowed regions usually appear darker than the lit regions and their color properties (e.g., hue and saturation) can also appear different than the directly illuminated regions. Such differences in intensity and color create patterns and boundaries/edges that often confuse human observers or machine vision algorithms which attempt to segment scenes and identify objects using these cues. For this reason many techniques have been developed to identify, segment, and remove shadows from an image or a video sequence[1–19]. However, all previously published methods use only two aspects of light, its intensity and/or spectral (“color”) distribution as information in shadow segmentation, though; in some cases these are combined with temporal and geometric information available. It appears that a third fundamental property of light, its polarization, has not heretofore been used for the purpose of shadow segmentation. Furthermore, most existing shadow segmentation algorithms assume a relatively simple shadow model: an area of a scene is classified either as shadow or non-shadow. In fact it is possible for a specific region of a scene to be both shadow for one source and illuminated simultaneously by another source or sources, as explained below, and polarization information can assist in “parsing” such complications in scene segmentation.
Polarization is an intrinsic property of light. Light from the dominant natural source, the sun, is not polarized but light scattered from small particles in the sky and most light reflected or scattered from object surfaces is partially polarized. The unaided human eye and most machine vision cameras are “blind” to polarization, but some animal species can detect and utilize polarization information and use it for a variety of purposes, including navigation and object recognition [20–23]. Inspired by biological polarization vision, our group has developed polarization sensitive cameras and processing methods for the detection of targets in scattering media, detecting latent fingerprints and enhancing surveillance [24–28]. We have also developed methods for displaying polarization information effectively to human observers [29,30]. It has been reported that polarization increase in dark surface area [31], and that polarization can be used to enhance details in shadow [32]. It has also been reported that polarization increases with increasing incident light angle [33]. In this investigation we show that complex overlapping cast shadows that are almost impossible to distinguish in images generated with only intensity and color information can be readily segmented from each other in images generated from the polarization parameters of a scene.
2. Polarization and Shadow
Fig. 1. (a) General macroscopic reflection model (b) Polarization of light resulting from specular reflection from a dielectric surface.
According to the generally accepted macroscopic description of the interaction of light with a surface, reflected light can be subdivided into specular and diffuse components [Fig. 1(a)]. The ratio of energy carried by the diffuse and specular components depends on the angle of incidence and the material properties of the surface. The diffusely reflected components often undergo multiple random reflections microscopically, so statistically they tend to be unpolarized. In contrast, the specularly reflected component is usually at least partially polarized, with the polarization direction (dominant axis of E-field oscillation) parallel to the local tangent plane of the surface [Fig. 1(b)]. These physical phenomena can be formalized through appropriate application of Fresnel’s analysis and Eq. [34].
In addition to the scattering by surfaces, another important natural source of polarization is the scattering of light by the atmosphere of the earth. The polarization of sun light by the air particles can be explained by the theory of Rayleigh scattering [34], which describes the particles as electric dipoles: because oscillating dipoles do not radiate in the direction of oscillation, a polarization-sensitive observer will see the dome of the sky to exhibits polarization pattern that depends on the location of the sun. Since pioneering investigations of von Frisch it has been well established that many insects can avail themselves of this polarization for navigation [20–23]. Such polarization has consequences for the segmentation of shadows: thus, as we will show below, an area that is inside a shadow cast by direct sunlight, but which is lit by the polarized ambient sky light will show a distinctive polarization, whereas an area that is inside both the shadow cast by sunlight and the shadow cast by skylight will show no polarization at all.
Because most imaging devices integrate light energy over a time epoch that is long relative to the oscillation period (fs), phase information is not recorded. With the phase information lost, when a linear polarization analyzer is placed in front of the camera, the measured intensity I at a specific image location or pixel, as a function of the angle of orientation φ of the polarization analyzer is given by
where θ is the orientation angle of the major axis of the polarization ellipse, IU is 50% of the total intensity at each pixel, and p ≡ IA/IU defines the degree of linear polarization at the pixel. The reference axis for the two angles φ and θ can be arbitrarily chosen, and complete information about the polarization state of the light can be obtained by capturing images with the polarizer oriented at three different angles, for example φ =0, 45 and 90 degrees [26,35] From these three images, one can recover IU, IA, and θ for each pixel of the image using the following expressions:
Here indices 0, 45, and 90 indicate the orientation of the polarizer in degrees when each specific image was taken. Because θ and θ+π are indistinguishable for phase-blind sensors, the meaningful range of θ is restricted to π, and θ ranges from 0 to π. In the work presented here we sample three angles (0, 45 and 90) by manually or mechanically rotating a single linear polarizer mounted in front of an intensity integrating camera. The camera used in our experiments is a calibrated Olympus E-10 digital camera with 4 Mega pixels and 10 bit pixel depth (we use the RAW mode).
3. Experiments
The first example is an outdoor scene of a walkway in front of a building with all-glass walls (Fig. 2 to Fig. 4, the glass-walled building is visible in Fig. 4). To help reader grasp the relationship between pictures in Fig. 2 to Fig. 4, we overlay a yellow circle over a sewer drainage cover that is visible in all pictures but Fig. 4 Left to call attention to readers that this is the exact same object in all the pictures. Then a green square is overlaid on the same glass door that is visible in both Fig. 4 Left and Fig. 4 Right. The sun illuminated the scene from the right hand side of the picture: shadows cast by trees are seen along the walkway, most clearly in the upper portion of the image. Most existing shadow handling algorithms would simply segment the dark areas as shadows, reducing or eliminating the contrast in brightness caused by the shadow. However there is a more complicated overlapping shadow pattern hidden inside the scene that is not detectable from analysis of the intensity distribution.
Fig. 2. Left: “Intensity-only” image of an outdoor scene with light and shadow. Right: “Degree-of-polarization” image of the same scene; this image plots the quantity p = IA/IU [see Eq. (2)], extracted for each image pixel. Hidden patterns of shadows within shadows are clearly visible in high contrast. The glass-walled building is shown in Fig. 4. The yellow circle points out the same sewage drain cover that is visible in all related pictures up to Fig. 4 except Fig. 4 Left to help orientation.
In this scene the glass-wall building to the left rear side of the scene reflected sunlight from its glass panels, but not from the thinner frames around each piece of glass. The reflected light was partially polarized, and the reflection pattern cast on the scene overlapped with the shadow pattern cast by the direct sunlight. The light reflected by the glass was weaker than the direct sunlight, and the pattern it creates is essentially invisible in the “intensity-only” image at left. However, when our polarization-sensitive camera was used to extract the “degree of polarization” image, the hidden pattern of overlapping shadow was revealed (Fig. 2, right panel). The area that was lit neither by direct sunlight, nor by the reflected light from the glass, is both dark and unpolarized, and thus appears dark in both images. These are the cast pattern of the glass frames of the glass-wall building to the left of the picture. The areas that were not lit by direct sunlight - and thus appear as shadows in the intensity-only image - but which were illuminated by the partially polarized reflected light from the glass-wall building, exhibit strong polarization. The degree-of-polarization image normalizes the polarization signal with respect to the total intensity [Eq. (2)], so these areas show up as a bright pattern in the degree-of polarization-image (Fig. 2, right). To establish that this pattern is unique to the polarization analysis, and not hidden in the intensity-only image due to low contrast in the shadow area, we performed linear contrast enhancement, followed by gamma correction of 0.5 to both images of Fig. 2: from the results (Fig. 3) it is clear that the shadow patterns are only revealed in the degree-of-polarization image. To further document the nature of the sunlight and glass wall sources to the shadows revealed by polarization analysis we provide images of the glass-wall and frames of the building (Fig. 4, left), and of the walkway when the bright direct sunlight is blocked (Fig. 4, right). Note that pictures in Fig. 4 are taken with the camera at about the same position and general view direction as when the pictures in Fig. 2 and Fig. 3 were taken. The only difference is that in Fig. 4 the camera zooms out and points more upward in order to put the tall glass-walled building into view. In sum, the patterns revealed in the degree-of-polarization image are indeed caused by shadows created by polarized light reflected from the glass source.
Fig. 3. Images in Fig. 2 are contrast-enhanced by linear intensity range stretch followed by gamma correction of 0.5 to show the details in the dark area. Left: intensity image. Right: degree of polarization image. It is clear that the pattern revealed in the polarization image is not present in the intensity image even after contrast enhancement. The yellow circle points out the same sewage drain cover that is visible in all related pictures up to Fig. 4 except Fig. 4 Left to help orientation.
Fig. 4. (Pictures shown in this figure are all regular intensity images with no polarization information) Left: the glass-wall building showing big glass rectangles and frames. Right: A picture of the same walk way as in Fig. 2 and Fig. 3 taken another day when the direct sun light is blocked due to nearby construction scaffolding. The shadow pattern cast on the walk way by the glass-wall and frames is visible. The yellow circle in the Right picture points out the same sewer drain cover as seen in Fig. 2 and Fig. 3. The left and right pictures in this figure can be related by the same glass door bracketed by the overlaid green rectangle. Note that pictures in Fig. 4 are taken with the camera at about the same position and general view direction as when taking pictures in Fig. 2 and Fig. 3. The only difference is that in Fig. 4 the camera zooms out and points more upward in order to put the tall glass-walled building into view.
We performed a controlled laboratory experiment to further confirm the results obtained outdoors. The setup comprises a 150W incandescent light source illuminating a scene from the direction opposite the camera, and a 15W fluorescent light illuminating the same scene from the direction corresponding to right hand side of the picture (Fig. 5).
Fig. 5. Controlled lab experiment of complex overlapping shadow: (a) Overview of the experiment setup. A metal pillar on a optical table illuminated by a strong incandescent light from the side opposite to the camera, while another much weaker fluorescent light illuminating from the right hand side of the picture. The polarization of the observed reflection from the side illuminating fluorescent light is weaker because they are all diffusely scattered reflection, as opposed to the mostly Fresnel reflection [34] coming from the incandescent light shining directly opposing the view of the camera. (b) Intensity-only image. (c) Degree-of-polarization image.
In the intensity-only image only the shadow of the knob cast by the dominant (incandescent) light source is visible. However, in the degree-of-polarization image, additional information is visible and separated clearly in high contrast. Specifically, a “shadow” cast by the much weaker fluorescent light from the right hand side is revealed as a bright area to the left of the metal pillar. The reason that this region appears bright in the degree-of-polarization image is due to the viewing geometry: the strong light reflected from the table is highly polarized, whereas the light reflected to the camera from the side source is only weakly polarized, and so where there is a shadow cast by the weaker source, the degree of polarization is less diluted by the weak unpolarized source and higher degree of polarization is detected. In addition, the area that is not illuminated by either source is very dark in the intensity-only image and is least polarized and seen as the darkest area in the degree-of-polarization image. Similarly, the polarization of the image regions corresponding to areas lit by both strong and weak light sources is lessened by the unpolarized light reflected to the camera from the weak source at the right hand side of the picture. Segmentation algorithms operating on the degree-of-polarization image can readily extract the distinctive “shadow” cast by the weak source. A sample analysis (Fig. 6 left) shows a segmentation obtained by a growing algorithm that starts with 2 × 2 regions. The side shadow area is cleanly separated from the image when 21 or more regions are segmented (Fig. 6, right).
Fig. 6. Left: Segmentation results from region-growing analysis (starting with the entire image divided into 2x2 regions and with adjacent similar regions merging in each iteration) of Fig. 5 (c) into 21 regions. Right: Hidden shadow area extracted from Fig. 5 (c). Note that this pattern is only a portion of a larger shadow of the metal pillar cast by the source at right, and that this larger shadow is partially obscured by both the small knob and the shadow of the small knob cast by the source opposing the camera.
4. Discussion
The processing of shadows in images presents many difficulties for scene segmentation, and all existing methods for analyzing shadows based on intensity-only information have limitations. Many methods are designed for specific applications like aerial photography or traffic monitoring [1–19], so that the lighting condition is simplified or known a priori. Many applications using extant methods require a specific geometry of the illumination and camera, and/or very precise calibrations of the pixel sensitivity of the camera. The use of polarization in shadow analysis and segmentation appears to be robust and certainly provides new and useful information that may facilitate segmentation and reveal new features of the scene and the sources that illuminate it. The polarization based shadow segmentation method suggested here does have its own limitations. While this method is not strictly tied to a specific scene geometry, the method does not work when the scene signals happens to be unpolarized everywhere, a rare but possible scenario. Nonetheless, because signals extracted with Eq. (2) are strongest when there is specular reflection, the use of the degree-of-polarization image for segmentation can be expected to give the best results when the source is opposite and directed toward the imaging system. A valuable feature of the method presented here is that it can readily reveal the presence of multiple sources of widely varying “strength”. As methods have already been developed for estimating the illumination directions of multiple light sources from information in the image [36–39], it can be anticipated that combining polarization analysis with these methods will produce valuable new tools for determining the direction of illumination sources. Investigations along these lines are underway. This use of polarization information in shadow detection, separation and contrast enhancement will also be further enhanced when it is combined with other well known cues like intensity, color, and geometry to achieve more accurate shadow segmentation and give more detailed information on the origin of distinct shadow components. While the present investigation has based its shadow-segmentation solely on degree-of-polarization information, the additional cue provided by the orientation of the local polarization ellipse [θ in Eq. (1)] can also be used for image segmentation (in much the manner in which color is used), and it can also be anticipated that this will further contribute to the method. Moreover, as expected from the independence of polarization from the other physical attributes of light and demonstrated by our experiments, information extracted by polarization about shadows is unique and in general cannot be extracted from other cues alone.
5. Conclusion
We have presented a novel method of shadow segmentation based on the local degree of polarization in images captured by a polarization-sensitive imaging system. This analysis reveals that the polarization of light conveys distinct and valuable information about a scene that can be extracted at modest cost. Polarization have been used in many other vision tasks such as removing glare and target detection, but to the best of our knowledge has not previously been used to aid the segmentation of complex shadows in a scene. Polarization information enables our system to extract information about the complexities of multiple light sources and the overlapping shadows they create. Such shadows are very difficult even to detect in intensity-only images, and their extraction with polarization analysis provides a new means of identifying the direction and nature of light sources illuminating a scene.
Acknowledgments
This work is supported in part by the U.S. Air Force Office of Scientific Research (AFOSR), through grants F49620-02-1-0140, FA9550-05-1-0052, and the DURIP grant F49620-02-1-0241. Correspondence should be addressed to Shih-Schön Lin, TEL: +1-215-898-2102; email: shschon@seas.upenn.edu or Nader Engheta, TEL: +1-215-898-9777; fax; 215-573-2068; e-mail; engheta@ee.upenn.edu.
References and Links
1. M. Nagao, T. Matsutyama, and Y. Ikeda, “Region extraction and shape analysis in aerial photographs,” Comput. Vis. Graph. Image Process. 10, 195–223 (1979). [CrossRef]
2. R. Gershon, A. D. Jepson, and J. K. Tsotsos, “Ambient illumination and the determination of material changes,” J. Opt. Soc. Am. A 3, 1700–1707 (1986). [CrossRef] [PubMed]
3. R. Irvin and D. Mckeown, “Methods for exploiting the relationship between buildings and their shadows in aerial imagery,” IEEE Trans. on Syst. Man..Cybern. 19, 1564–1575 (1989). [CrossRef]
4. J. M. Scanlan, D. M. Chabries, and R. W. Christiansen, “A Shadow Detection and Removal Algorithm for 2D Images,” in Proc. of Int. Conf. on Acoustics, Speech, and Signal Processing (1990) pp. 2057–2060.
5. Y. Liow and T. Pavlidis, “Use of shadows for extracting buildings in aerial images,” Comput. Vis. Graph. Image Process. 49, 242–277 (1991). [CrossRef]
6. C. Wang, L. Huang, and A. Rosenfeld, “Detecting clouds and cloud shadows on aerial photographs,” Pattern. Recogn. Lett. 12, 55–64 (1991). [CrossRef]
7. D. Koller, K. Danilidis, and H.-H. Nagel, “Model-based object tracking in monocular image sequences of road traffic scenes,” Int. J. Comput. Vis. 10, 257–281 (1993). [CrossRef]
8. C. Jiang and M. O. Ward, “Shadow segmentation and classification in a constrained environment,” CVGIP: Image Understanding 59, 213–225 (1994). [CrossRef]
9. G. Funka-Lea and R. Bajcsy, “Combining color and geometry for the active visual recognition of shadows,” in Proc. Int. Conf. on Computer Vision (1995) pp. 203–209.
10. J. Stauder, R. Melch, and J. Ostermann, “Detection of moving cast shadows for object segmentation,” IEEE Trans. Multimedia 1, 65–77 (1999). [CrossRef]
11. J. A. Marchant and C. M. Onyango, “Shadow-invariant classification for scenes illuminated by daylight,” J. Opt. Soc. Am. A 17, 1952–1961 (2000). [CrossRef]
12. G. Finlayson, S. Hordley, and M. S. Drew, “Removing shadows from images,” in ECCV (2002) pp. 823–836.
13. R. Cucchiara, C. Grana, M. Piccardi, and A. Prati, “Detecting moving objects, Ghosts, and shadows in video streams,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 1337–1342 (2003). [CrossRef]
14. T. Gevers and H. Stokman, “Classifying color edges in video into shadow-geometry, highlight, or material transitions,” IEEE Trans. Multimedia 5, 237–243 (2003). [CrossRef]
15. A. Prati, I. Mikic, M. M. Trivedi, and R. Cucchiara, “Detecting Moving Shadows: Algorithms and Evaluation,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 918–923 (2003). [CrossRef]
16. I. Sato, Y. Sato, and K. Ikeuchi, “Illumination from shadows,” IEEE Trans. Pattern Anal. Mach. Intell. 25, 290–300 (2003). [CrossRef]
17. S. Nadimi and B. Bhanu, “Physical models for moving shadow and object detection in video,” IEEE Trans. Pattern Anal. Mach. Intell. 26, 1079–1087 (2004). [CrossRef]
18. E. Salvador, A. Cavallaro, and T. Ebrahimi, “Cast shadow segmentation using invariant color features,” Comput. Vis. Image Understand. 95, 238–259 (2004). [CrossRef]
19. J. M. Wang, Y. C. Chung, C. L. Chang, and S. W. Chen, “Shadow Detection and Removal for Traffic Images,” in Proc. 2004 IEEE Int. Conf. on Networking, Sensing and Control (IEEE Syst. Man Cybern. Society, Taipei, Taiwan, 2004) pp. 649–654. [CrossRef]
20. K. Frisch, “Die polarisation des himmelslichtes als orientierender faktor bei den tanzen der bienen,” Experientia 5, 142–148 (1949). [CrossRef] [PubMed]
21. R. Wehner, “Polarized-light navigation by insects,” Scientific American 235, 106–114 (1976). [CrossRef] [PubMed]
22. R. Schwind, “Zonation of the optical environment and zonation in the rhabdom structure within the eye of the backswimmer, Notenecta glauca,” Cell and Tissue Research 232, 53–63 (1983). [CrossRef] [PubMed]
23. G. Horváth, “Reflection polarization patterns at flat water surfaces and their relevance for insect polarization vision,” J. Theor. Biol. 175, 27–37 (1995). [CrossRef] [PubMed]
24. M. P. Rowe, E. N. Pugh Jr., and N. Engheta, “Polarization-difference imaging: a biologically inspired technique for observation through scattering media,” Opt. Lett. 20, 608–610 (1995). [CrossRef] [PubMed]
25. J. S. Tyo, M. P. Rowe, E. N. Pugh Jr., and N. Engheta, “Target detection in optically scatter media by polarization-difference imaging,” Appl. Opt. 35, 1855–1870 (1996). [CrossRef] [PubMed]
26. S.-S. Lin, K. M. Yemelyanov, E. N. Pugh Jr., and N. Engheta, “Polarization Enhanced Visual Surveillance Techniques,” in Proc. IEEE Int. Conf. on Networking, Sensing and Control (IEEE Syst. Man. .Cybern. Society, Taipei, Taiwan, 2004), 1, pp. 216–221. [CrossRef]
27. S.-S. Lin, K. M. Yemelyanov, E. N. Pugh Jr., and N. Engheta, “Polarization- and specular-reflection-based, non-contact latent fingerprint imaging and lifting,” J. Opt. Soc. Am. A (to be published).
28. K. M. Yemelyanov, S.-S. Lin, E. N. Pugh Jr., and N. Engheta, “Adaptive Algorithms for 2-channel polarization sensing under various polarization statistics with non-uniform distributions,” Appl. Opt. (to be published). [PubMed]
29. J. S. Tyo, E. N. Pugh Jr., and N. Engheta, “Colorimetric representation for use with polarization-difference imaging of objects in scattering media,” J. Opt. Soc. Am. A 15, 367–374 (1998). [CrossRef]
30. K. M. Yemelyanov, M. A. Lo, E. N. Pugh Jr., and N. Engheta, “Display of polarization information by coherently moving dots,” Opt. Express 11, 1577–1584 (2003). [CrossRef] [PubMed]
31. W. G. Egan, “Dark-target retroreflection increase,” in Polarization: Measurement, Analysis, and Remote Sensing II, D. H. Goldstein and D. B. Chenault, eds., Proc. SPIE , 3754, 218–225 (1999). [CrossRef]
32. M. J. Duggin, “Imaging polarimetry in scene element discrimination,” in Polarization: Measurement, Analysis, and Remote Sensing II, D. H. Goldstein and D. B. Chenault, eds., Proc SPIE 3754, 108–117 (1999). [CrossRef]
33. D. H. Goldstein, D. B. Chenault, and J. L. Pezzaniti, “Polarimetric characterization of Spectralon,” in Polarization: Measurement, Analysis, and Remote Sensing II, D. H. Goldstein and D. B. Chenault, eds., Proc. SPIE 3754, 126–136 (1999). [CrossRef]
34. E. Hecht, Optics (Addison Wesley Longman, Inc., Reading, MA, USA1998).
35. K. M. Yemelyanov, S.-S. Lin, W. Q. Luis, E. N. Pugh Jr., and N. Engheta, “Bio-inspired display of polarization information using selected visual cues,” in Polarization Science and Remote Sensing, J. A. Shaw and J. S. Tyo eds. Proc. SPIE , 5158, 71–84 (2003). [CrossRef]
36. A. P. Pentland, “Finding the illuminant direction,” J. Opt. Soc. Am. 72, 448–455 (1982). [CrossRef]
37. Y. Zhang and Y. Yang, “Illuminant direction determination for multiple light sources,” in Proc. CVPR (IEEE, 2000) pp. 269–276.
38. J. Pinel and H. Nicolas, “Estimation 2d illuminant direction and shadow segmentation in natural video sequences,” in Proc. VLBV (2001) pp. 197–202.
39. M. W. Powell, S. Sarkar, and D. Goldgof, “A simple strategy for calibrating the geometry of light sources,” IEEE Trans. Pattern Anal. Mach. Intell. 23, 1022–1027 (2001). [CrossRef]
