Abstract
The simulation of rare edge cases such as adverse weather conditions is the enabler for the deployment of the next generation of autonomous drones and vehicles into conditions where human operation is error-prone. Therefore, such settings must be simulated as accurately as possible and be computationally efficient, so to allow the training of deep learning algorithms for scene understanding, which require large-scale datasets disallowing extensive Monte Carlo simulations. One computationally-expensive step is the simulation of light sources in scattering media, which can be tackled by the radiative transfer equation and approximated by analytical solutions in the following. Traditionally, a single scattering event is assumed for fog rendering, since it is the dominant effect for relatively low scattering media. This assumption allows us to present an improved solution to calculate the so called air-light integral that can be evaluated fast and robustly for an isotropic point source in homogeneous media. Additionally, the solution is extended for a cone-shaped source and implemented in a computer vision rendering pipeline fulfilling computational restrictions for deep learning uses. All solutions can handle arbitrary azimuthally symmetric phase functions and were tested with the Henyey-Greenstein phase function and an advection fog phase function calculated from a particle distribution using Mie’s theory. The used approximations are validated through extensive Monte Carlo simulations and the solutions are used to augment good weather images towards inclement conditions with focus on visible light sources, so to provide additional data in such hard-to-collect settings.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Today’s computer vision methods enable autonomous vehicles and drones to reason about their environment. A key factor behind their success are the underlying datasets used for training, enabling to learn complex neural networks able to distill information from raw camera captures [1–3]. However, rare events like driving in adverse weather are underrepresented in traditional datasets [4]. Nonetheless, enabling the operation in such conditions would lead to beneficial safety gains [4]. Typical approaches to close the data gap are based on the idea of augmenting clear weather scenes to match real world inclement weather recordings [5–7]. Thereby, fog is rendered through empirical models applying Koschmieder’s model and extracting physical parameters from the clear reference. One of such assumptions is the constant ambient airlight, only true for daytime fog [5,6] and which does not hold anymore in the presence of active light sources. Extensions as presented in [7] introduce a handling of light sources but model the spread of the light source illumination through scattering, empirically by matching a Gaussian filter to experimental observations.
One approach to model this physically-accurate broadening is the use of Radiative Transport Equations (RTE). Such equations are also known in related theoretical fields, as neutron transport [8]. Its importance for the simulation of foggy scenarios is elaborated for example by Cerezo et al. [9]. But solving RTE equations numerically is not trivial and requires Monte Carlo simulations [10]. Such methods have an important history in many applications from medicine to astrophysics [11–14], but they require a high amount of computational power and contain statistical noise in contrast to fast analytical solutions. In our example, the Monte Carlo simulation runs multiple days on graphical processing units, whereas the analytical solution takes only seconds for the single scattering solution. Analytical solutions are difficult to find but are important to improve and accelerate rendering applications containing participating media like fog [15]. Due to this challenges, the full RTE can be reduced to only consider single scattered light. It should be noted that the solution employed to compute the single-scattered radiance is typically expressed in terms of the so-called air–light integral. In fact, it has been used for fog rendering already by Nishita, Miyawaki and Nakamae [16]. Pegoraro and Parker [17] derived an air-light integral representation in isotropically and anisotropically scattering media, which they later improved together with Schott [18]. In general, the role of the RTE for light transport in fog is outlined by Bentz et al. [19]. They also developed an analytical model for weak angular dependence based on the diffusion equation, which is limited to the moderate and high scattering regime. More analytical solutions exist taking into account single scattering [20,21] as well as all scattering events [22–24]. In our recent publication [25] we have derived some exact solutions for the single-scattered radiance in semi-infinite media based on radiative transport theory. Further applications of RTE solutions within the computer graphics field can be found in [26–30].
In this publication, two models for the single scattering radiance due to a point source located in an infinitely extended scattering medium are presented as they offer a good approximation for perception systems located within the scattering medium. An example image with traffic and street lights in a foggy atmosphere is shown in Fig. 1. These models differ in the angular radiation characteristics of the applied source. The first model applies an isotropic point source that emits light uniformly in all directions. As stated above, the solution to this problem can be expressed in terms of the air–light integral. The numerical evaluation of this integral representation can become a challenging task. Therefore, we present a modified integral representation as an alternative to the classical air-light integral, which can be efficiently evaluated without numerical problems. The second model assumes a cone-shape source, where light is emitted in a cone with opening angle $\theta _0$. To our knowledge, this is the first solution with a cone-shaped radiation characteristic of the source. It includes the isotropic solution in the limiting cases of $\theta _0={180}^{\circ }$. All models are verified using Monte Carlo simulations and the validity of the single scattering approximation is investigated by comparing it to the solution containing all scattering events. Additionally, an application of the derived solution for fog rendering is presented. It is used to improve the rendering of light sources in adverse weather simulations.
2. Theory and method
In the following section, the theoretical framework for deriving the analytical solution is presented, which is based on the single scattering solution to the radiative transfer equation according to Chandrasekhar [32]. In detail, subsection 2.1 treats the isotropic point source with angular uniform radiation characteristic and subsection 2.2 the cone source with truncated angular reach.
2.1 Isotropic source
The single scattering radiance at position $\mathbf {r}$ in direction $\hat {\mathbf {s}}$ due to an isotropic point source located at the origin can be derived as [21]
The derived representation for the isotropic source was verified using Monte Carlo simulations, see Fig. 5. Additionally, the Monte Carlo simulation considering all scattering events $L_{n\geq 1}^{\textrm {iso}}$ is plotted for comparison. A Henyey-Greenstein phase function [36] with an anisotropy factor of $g={0.8}$ was used. The optical properties are chosen to be typical for fog. The scattering coefficient $\mu _{\mathrm {s}}={0.08}{\textrm {m}^{-1}}$ corresponds to a visibility of approximately $\frac {4}{\mu _{\mathrm {s}}}={50}\textrm {m}$, the distance from the source to the detector is $r={20}\textrm {m}$ and, thus, the optical depth is $\tau =\mu _{\mathrm {s}} r={1.6}$. The absorption coefficient $\mu _{\mathrm {a}}={1{\times }10^{-5}}{\textrm {m}^{-1}}$ is very small, since it can be mostly neglected for fog in the visible wavelength range. These optical properties are chosen as an example and will be used further below. It should be noted that all the solutions were also tested with other values and no significant differences were found. For $\alpha \ \to 0$ the solution diverges to $\infty$, but it is integrable over a solid angle $\alpha \ \in \left [0,\,\alpha _c\right ]$, $\gamma \ \in \left [0,\,2 \pi \right ]$ respecting the Jacobian $\sin \alpha$, it does not depend on the azimuth $\gamma$. The inset shows the relative error between the derived solution and the Monte Carlo simulation of the single scattered radiance. In Fig. 6, the calculation was repeated for smaller angles. In the left image the distance is $r={20}\textrm {m}$ giving an optical depth of $\tau =\mu _{\mathrm {s}} r={1.6}$ and in the right image it is $r={1}\textrm {m}$ giving an optical depth of $\tau =\mu _{\mathrm {s}} r={0.08}$. It can be seen that for small angles and low optical depth the single scattering approximation is a very good approximation to the complete solutions to the RTE considering all scattering interactions. Looking at the relative errors, the statistical noise increases for very small angles due to the Monte Carlo simulation. An additional verification with other phase function can be found in the Appendix A.
2.2 Cone source
This section deals with the single-scattered radiance due to a cone source of the form
Case 1: Opening angles with $\mu _0\geq 0$. In this situation, we have $w\geq 0$. For $w>1$, the inequality (33) cannot be satisfied so that $D=\emptyset$ and hence $L_1^{\textrm {cone}}=0$. If $0\leq w\leq 1$, we define $\beta := \arcsin w\in [0,\pi /2]$ and obtain $\sin (\vartheta +\varphi )>w$ for $\vartheta \in (\beta -\varphi,\pi -\beta -\varphi )$. The intersection with the integration interval results in
Case 2: Opening angles with $\mu _0< 0$. In this case, we have $w<0$. If $w<-1$, the inequality (33) is satisfied for all $\vartheta \in I$, yielding $D=(\vartheta _1,\pi )$. For $-1\leq w<0$, when $\beta \in [-\pi /2,0)$, a slightly longer analysis shows that
3. Validity of the single scattering approximation
In this section we investigate the errors of the single scattering approximation compared to the full solution of the RTE considering all scattering interactions. The relative error between the single scattered radiance and the radiance considering all scattering events versus the optical depth $\tau =\mu _s r$ and the angle $\theta$ for $\mu _a={1{\times }10^{-5}}\;\textrm {m}^{-1}$ is shown in Fig. 10. Two types of phase functions were considered, a Henyey-Greenstein phase function with $g={0.8}$ (left) and a phase function for advection fog (right). It was calculated from a model particle size distribution applying Mie’s theory. Thus, it was assumed that the fog particles are spheres. The used phase function is shown in Fig. 11. We emphasize that this phase function is calculated from a particle size distribution that represents a fictitious fog. The particle size distribution is approximated using a modified gamma distribution like the simulation software MODTRAN, with the parameters for advection fog taken from Gebhart et al. [37]. It serves as an example for a general phase function generated from a particle size distribution with Mie’s theory.
The errors are simulated with a GPU-based Monte Carlo simulation using the lookup table based technique described by Naglič et al. [38]. As expected, for small angles and low scattering distances the errors decrease. Already for moderate optical depths of $\mu _{\mathrm {s}} r={2}$, the relative error is below ${10}\%$, even for the highly forward scattering advection fog.
4. Application of the solution: improved light sources in artificial adverse weather images
Here we show how to apply the solution derived above to improve the generation of artificial adverse weather images from real good weather images. The unscattered light of an isotropic light source in an infinitely extended medium is given by
where $P$ is the source power. For a pixel with solid angle $\sigma$, the directly detected light is5. Summary and conclusion
In summary this work presents an analytical solution for the RTE in case of single scattering for an isotropic and a cone-shaped point source. Here, we assumed the location within a homogeneous infinitely extended scattering medium. To proof the findings, model predictions were validated using Monte Carlo simulations for small radiation angles and equivalent thin optical densities. Further, the solution of the isotropic point source was used to improve the appearance of simulated traffic light sources in dense foggy conditions. For cone-shaped light sources as street lanterns and known opening angles, the introduced solution derived in Eq. (54) could be used. Fog exists in different forms, depending on, e.g., the formation, location and environment, thus many densities and phase functions are possible. Depending on the size of the droplets and the applied wavelength the phase function can represent, both, strongly forward scattering as well as more uniformly scattering behavior. The droplet size distribution also changes during the fog life cycle [41]. We use the selected phase function as an example for a general phase function generated from a particle distribution and using Mie’s theory. The chosen particle distribution does not necessarily coincide with that of real fog, which can be much more complex. The wavelength is also only an example from the visible spectrum [42]. The presented method is not restricted to this particular fog type or wavelength, it can model any phase function. To generate robust datasets, we recommend simulating several fog types. We envision that these results can be used in future work to foster the development of deep learning methods robust to challenging illumination conditions as the one in Fig. 1, through an approach similar to [5,6]. In fact, the extension to challenging illumination conditions includes not only environmental conditions like advertising sign or street lighting, but more importantly also signal lights such as turn signals, traffic lights as shown in Fig. 12, stop lights, warning lights and blue lights of emergency vehicles, which are indispensable in complex urban traffic scenarios. All of those lightning conditions are currently modeled using time consuming Monte Carlo approaches prohibiting the widespread roll out of such of complex simulations due to time constraint issues. The here-proposed analytical solutions offer a reduction of computational time of some orders of magnitude, allowing to process millions of frames as required by state-of-the-art deep learning approaches [43,44]. We further indicate that the single scattering approximation limits the application of the method to certain circumstances. For very dense fog it may not be appropriate. In the future, the solution could be further expanded to include double scattering to reduce this limitation. Extending the solution to consider arbitrary angular radiation characteristics of the source will allow the modeling of vehicle headlamps with known cone illumination profile and differing characteristics for the US or European market.
Appendix
Since the phase function changes with the fog type and wavelength, we additionally show calculations for further phase functions, plotted in Fig. 13. They are again calculated using a modified gamma distribution to model the particle size distribution with parameters from [37]. As one can see, the advection fog is much more forward scattering than the convection fog. The single scattering radiance highly depends on the fog type as shown in Fig. 14. It is calculated using the analytical solution for all four phase functions from Fig. 13 for an isotropic point source at a distance of $r={30}\textrm {m}$. It agrees well to the Monte Carlo simulations that are shown for comparison. Figure 15 shows part of the images from Fig. 12 with an additional image (d), where the single scattering solution with the phase function from convection fog for a wavelength of 550 nm was used. A different fog type would also lead to a different visibility, but to compare only the influence of the phase function, we simulated the image with the same visibility of ${139}\textrm {m}$. Since the convection fog phase function represents a more isotropically scattering behavior, it has a very different effect on the light scattered from the traffic light compared to the advection fog phase function.
Funding
Bundesministerium für Bildung und Forschung (16ME0344).
Acknowledgments
The research leading to these results is part of the AI-SEE project, which is a co-labelled PENTA and EURIPIDES 2 project endorsed by EUREKA. Co-funding is provided by the following national funding authorities: Austrian Research Promotion Agency (FFG), Business Finland, Federal Ministry of Education and Research (BMBF) and National Research Council of Canada Industrial Research Assistance Program (NRC-IRAP).
Disclosures
The authors declare that there are no conflicts of interest related to this article.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2012), pp. 3354–3361.
2. P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, “Scalability in perception for autonomous driving: Waymo open dataset,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2020), pp. 2443–2451.
3. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, “Coco: Commom objects in context,” in Proceedings of the European Conference on Computer Vision (ECCV), D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, eds. (Springer International PublishingCham, 2014), pp. 740–755.
4. M. Bijelic, F. Mannan, T. Gruber, W. Ritter, K. Dietmayer, and F. Heide, “Seeing Through Fog Without Seeing Fog: Deep Sensor Fusion in the Absence of Labeled Training Data,” arXiv:1902.08913 [cs] (2019).
5. S. S. Halder, J.-F. Lalonde, and R. d. Charette, “Physics-based rendering for improving robustness to rain,” in Proceedings of the International Conference on Computer Vision (ICCV), (2019).
6. C. Sakaridis, D. Dai, and L. Van Gool, “Semantic foggy scene understanding with synthetic data,” Int. J. Comput. Vis. 126(9), 973–992 (2018). [CrossRef]
7. Z. Shi, E. Tseng, M. Bijelic, W. Ritter, and F. Heide, “ZeroScatter: Domain Transfer for Long Distance Imaging and Vision through Scattering Media,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2021), pp. 3475–3485.
8. J. C. Stewart, I. Kuščer, and N. J. McCormick, “Equivalence of special models in energy-dependent neutron transport and nongrey radiative transfer,” Ann. Phys. 40(2), 321–333 (1966). [CrossRef]
9. E. Cerezo, F. Pérez, X. Pueyo, F. J. Seron, and F. X. Sillion, “A survey on participating media rendering techniques,” The Vis. Comput. 21(5), 303–328 (2005). [CrossRef]
10. L. L. House and L. W. Avery, “The Monte Carlo technique applied to radiative transfer,” J. Quant. Spectrosc. Radiat. Transf. 9(12), 1579–1591 (1969). [CrossRef]
11. S. A. Prahl, “A Monte Carlo model of light propagation in tissue,” (Berlin, Germany, 1989), p. 1030509.
12. S. Flock, M. Patterson, B. Wilson, and D. Wyman, “Monte Carlo modeling of light propagation in highly scattering tissues. I. Model predictions and comparison with diffusion theory,” IEEE Trans. Biomed. Eng. 36(12), 1162–1168 (1989). [CrossRef]
13. C. Zoller and A. Kienle, “Fast and precise image generation of blood vessels embedded in skin,” J. Biomed. Opt. 24(01), 1 (2019). [CrossRef]
14. U. M. Noebauer and S. A. Sim, “Monte Carlo radiative transfer,” Living Rev. Comput. Astrophys. 5(1), 1 (2019). [CrossRef]
15. T. Ritschel, C. Dachsbacher, T. Grosch, and J. Kautz, “The State of the Art in Interactive Global Illumination,” Comput. Graph. Forum 31(1), 160–188 (2012). [CrossRef]
16. T. Nishita, Y. Miyawaki, and E. Nakamae, “A shading model for atmospheric scattering considering luminous intensity distribution of light sources,” in Proceedings of the 14th annual conference on Computer graphics and interactive techniques, (Association for Computing MachineryNew York, NY, USA, 1987), SIGGRAPH ’87, pp. 303–310.
17. V. Pegoraro and S. G. Parker, “An Analytical Solution to Single Scattering in Homogeneous Participating Media,” Comput. Graph. Forum 28(2), 329–335 (2009). [CrossRef]
18. V. Pegoraro, M. Schott, and S. G. Parker, “A Closed-Form Solution to Single Scattering for General Phase Functions and Light Distributions,” Comput. Graph. Forum 29(4), 1365–1374 (2010). [CrossRef]
19. B. Z. Bentz, B. Z. Bentz, B. Z. Bentz, B. J. Redman, J. D. v. d. Laan, K. Westlake, A. Glen, A. L. Sanchez, and J. B. Wright, “Light transport with weak angular dependence in fog,” Opt. Express 29(9), 13231–13245 (2021). [CrossRef]
20. M. L. Shendeleva, “Single-scattering solutions to radiative transfer in infinite turbid media,” J. Opt. Soc. Am. A 30(11), 2169–2174 (2013). [CrossRef]
21. B. Sun, R. Ramamoorthi, S. G. Narasimhan, and S. K. Nayar, “A practical analytic single scattering model for real time rendering,” ACM Trans. Graph. 24(3), 1040–1049 (2005). [CrossRef]
22. K. M. Case and P. F. Zweifel, Linear transport theory (Addison-Wesley, Reading, Mass., 1967).
23. J. J. Duderstadt and W. R. Martin, Transport theory (Wiley, 1979).
24. M. Machida, “How to Construct Three-Dimensional Transport Theory Using Rotated Reference Frames,” J. Comput. Theor. Transp. 45(7), 594–609 (2016). [CrossRef]
25. A. Liemert, S. Geiger, and A. Kienle, “Solutions for single-scattered radiance in the semi-infinite medium based on radiative transport theory,” JOSA A 38(3), 405–411 (2021). [CrossRef]
26. J. R. Frisvad, T. Hachisuka, and T. K. Kjeldsen, “Directional Dipole Model for Subsurface Scattering,” ACM Trans. Graph. 34(1), 1–12 (2014). [CrossRef]
27. R. Habel, P. H. Christensen, and W. Jarosz, “Photon Beam Diffusion: A Hybrid Monte Carlo Method for Subsurface Scattering,” Comput. Graph. Forum 32(4), 27–37 (2013). [CrossRef]
28. R. Ramamoorthi and P. Hanrahan, “A signal-processing framework for inverse rendering,” in Proceedings of the 28th annual conference on Computer graphics and interactive techniques, (Association for Computing Machinery, New York, NY, USA, 2001), SIGGRAPH ’01, pp. 117–128.
29. W. Jakob, E. d’Eon, O. Jakob, and S. Marschner, “A comprehensive framework for rendering layered materials,” ACM Trans. Graph. 33(4), 1–14 (2014). [CrossRef]
30. H. W. Jensen and P. H. Christensen, “Efficient simulation of light transport in scenes with participating media using photon maps,” in Proceedings of the 25th annual conference on Computer graphics and interactive techniques, (Association for Computing Machinery, New York, NY, USA, 1998), SIGGRAPH ’98, pp. 311–320.
31. fusion-of-horizons, “cavernous,” https://www.flickr.com/photos/fusion_of_horizons/51839030680/ (2022).
32. S. Chandrasekhar, Radiative transfer (Dover Publications, 1960).
33. H. A. DeKleine, “Proof Without Words: Mollweide’s Equation,” Math. Mag. 61(5), 281 (1988). [CrossRef]
34. H. Takahasi and M. Mori, “Double exponential formulas for numerical integration,” Publ. Res. Inst. for Math. Sci. 9(3), 721–741 (1973). [CrossRef]
35. L. N. Trefethen and J. A. C. Weideman, “The Exponentially Convergent Trapezoidal Rule,” SIAM Rev. 56(3), 385–458 (2014). [CrossRef]
36. L. G. Henyey and J. L. Greenstein, “Diffuse radiation in the Galaxy,” The Astrophys. J. 93, 70–83 (1941). [CrossRef]
37. M. Gebhart, E. Leitgeb, S. Sheikh Muhammad, B. Flecker, C. Chlestil, M. Al Naboulsi, F. de Fornel, and H. Sizun, “Measurement of Light attenuation in dense fog conditions for FSO applications,” (San Diego, California, USA, 2005), p. 58910K.
38. P. Naglic, F. Pernuš, B. Likar, and M. Bürmen, “Lookup table-based sampling of the phase function for Monte Carlo simulations of light propagation in turbid media,” Biomed. Opt. Express 8(3), 1895–1910 (2017). [CrossRef]
39. J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2018), pp. 5410–5418.
40. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, (2009), pp. 1956–1963.
41. M. Mazoyer, F. Burnet, and C. Denjean, “Experimental study on the evolution of droplet size distribution during the fog life cycle,” Atmos. Chem. Phys. 22(17), 11305–11321 (2022). [CrossRef]
42. R. Nebuloni, “Empirical relationships between extinction coefficient and visibility in fog,” Appl. Opt. 44(18), 3795–3804 (2005). [CrossRef]
43. T. Sun, M. Segu, J. Postels, Y. Wang, L. Van Gool, B. Schiele, F. Tombari, and F. Yu, “SHIFT: a synthetic driving dataset for continuous multi-task domain adaptation,” in Computer Vision and Pattern Recognition, (2022).
44. E. Wood, T. Baltrušaitis, C. Hewitt, S. Dziadzio, T. J. Cashman, and J. Shotton, “Fake it till you make it: face analysis in the wild using synthetic data alone,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), (2021), pp. 3661–3671.