Willem J. Marais, Robert E. Holz, Yu Hen Hu, Ralph E. Kuehn, Edwin E. Eloranta, and Rebecca M. Willett, "Approach to simultaneously denoise and invert backscatter and extinction from photon-limited atmospheric lidar observations," Appl. Opt. 55, 8316-8334 (2016)

Atmospheric lidar observations provide a unique capability to directly observe the vertical column of cloud and aerosol scattering properties. Detector and solar-background noise, however, hinder the ability of lidar systems to provide reliable backscatter and extinction cross-section estimates. Standard methods for solving this inverse problem are most effective with high signal-to-noise ratio observations that are only available at low resolution in uniform scenes. This paper describes a novel method for solving the inverse problem with high-resolution, lower signal-to-noise ratio observations that are effective in non-uniform scenes. The novelty is twofold. First, the inferences of the backscatter and extinction are applied to images, whereas current lidar algorithms only use the information content of single profiles. Hence, the latent spatial and temporal information in noisy images are utilized to infer the cross-sections. Second, the noise associated with photon-counting lidar observations can be modeled using a Poisson distribution, and state-of-the-art tools for solving Poisson inverse problems are adapted to the atmospheric lidar problem. It is demonstrated through photon-counting high spectral resolution lidar (HSRL) simulations that the proposed algorithm yields inverted backscatter and extinction cross-sections (per unit volume) with smaller mean squared error values at higher spatial and temporal resolutions, compared to the standard approach. Two case studies of real experimental data are also provided where the proposed algorithm is applied on HSRL observations and the inverted backscatter and extinction cross-sections are compared against the standard approach.

S. P. Burton, C. A. Hostetler, A. L. Cook, J. W. Hair, S. T. Seaman, S. Scola, D. B. Harper, J. A. Smith, M. A. Fenn, R. A. Ferrare, P. E. Saide, E. V. Chemyakin, and D. Müller Appl. Opt. 57(21) 6061-6075 (2018)

S. Stamnes, C. Hostetler, R. Ferrare, S. Burton, X. Liu, J. Hair, Y. Hu, A. Wasilewski, W. Martin, B. van Diedenhoven, J. Chowdhary, I. Cetinić, L. K. Berg, K. Stamnes, and B. Cairns Appl. Opt. 57(10) 2394-2413 (2018)

You do not have subscription access to this journal. Citation lists with outbound citation links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

You do not have subscription access to this journal. Cited by links are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

You do not have subscription access to this journal. Figure files are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

You do not have subscription access to this journal. Article tables are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

You do not have subscription access to this journal. Equations are available to subscribers only. You may subscribe either as an Optica member, or as an authorized user of your institution.

A short description of each symbol is given, as well as the type of variable (matrix or scalar) and the location where the symbol was introduced.

Algorithm 1

The standard approach in inferring the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. The tuning parameters of ${\mathsf{F}}^{\mathsf{Low}}$ are set by a lidar expert. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{1}$

Let ${F}^{\mathrm{Low}}(\xb7)$ be a low-pass filter function, and let ${F}^{\mathrm{Diff}}(\xb7)$ be a discrete derivative operator.

1: /*Estimate optical depth*/

2: ${\widehat{\tau}}^{\mathrm{alg}-1}$ ← plug ${Y}_{c}$ and ${Y}_{m}$ into Eq. (11)

The standard approach with averaging in inferring the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. The tuning parameters of ${\mathsf{F}}^{\mathsf{Avg}}$ are set by a lidar expert or an automated system, which are determined by some objective criteria. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{1}\mathsf{a}$

Let ${F}^{\mathrm{Avg}}(\xb7)$ be an averaging function, i.e., block averaging.

4: Execute Algorithm 1 using ${Y}_{c}^{\mathrm{Avg}}$ and ${Y}_{m}^{\mathrm{Avg}}$ to get estimates of the backscatter ${\nu}_{+}$ and extinction $\beta $ cross-sections, the lidar ratio $\mu $, and the optical depth $\tau $. Denote these estimates by ${\widehat{\nu}}_{+}^{\mathrm{alg}-1a}$, ${\widehat{\beta}}^{\mathrm{alg}-1a}$, ${\widehat{\mu}}^{\mathrm{alg}-1a}$, and ${\widehat{\tau}}^{\mathrm{alg}-1a}$.

The new approach in inverting for the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, along with the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{2}$

1: /*Invert the parallel backscatter cross-section*/

2: Set $\ell (\omega ;{Y}_{\iota})={\mathbb{1}}_{N}^{\mathrm{T}}[{f}_{\iota}(\omega )-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}{f}_{\iota}(\omega )]{\mathbb{1}}_{K}$

9: Set $\ell (\tilde{\mu};{Y}_{\iota})={\mathbb{1}}_{N}^{\mathrm{T}}[g(\tilde{\mu})-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}g(\tilde{\mu})]{\mathbb{1}}_{K}$

The TV-PMLE with cross-validation to estimate either ${\mathsf{\omega}}_{\mathsf{\iota}}$ or $\mathsf{\mu}$; see Algorithm 2

Input: A loss function such as $\ell (\psi ;{Y}_{c},{Y}_{m})$, the Poisson noisy matrices ${Y}_{c}$, ${Y}_{m}$ and the constraint set $\mathrm{\Psi}$ for $\psi $.

Another approach to inverting the particulate extinction cross-section $\mathsf{\beta}$. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{3}$

Input: Previously inverted parallel backscatter cross-section $\widehat{\nu}$, such as ${\widehat{\nu}}^{\mathrm{alg}-2}$, with the depolarization coefficient $\rho $ to compute ${\widehat{\nu}}_{+}$.

1: /*Invert the extinction cross-section*/

2: Set $\ell (\beta ;{Y}_{m})={\mathbb{1}}_{N}^{\mathrm{T}}[h(\beta )-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}h(\beta )]{\mathbb{1}}_{K}$

Root Mean Square Error, Bias, and Standard Deviation (Std) Results of the First Experiment^{a}

Backscatter Cross-section ${\nu}_{+}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

Standard approach with block avg.

6

6

6

New approach

190

104

200

Optical Depth$\mathbf{\tau}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

Standard approach with block avg.

2

2

2

New approach

7

2

13

All results are expressed by a factor of reduction relative to the standard approach (Algorithm 1), i.e., the RMSE backscatter cross-section of the new approach is 190 times less than the RMSE backscatter cross-section of the standard approach; the larger the number is, the better. The reduction of, say, the RMSE in regards to the backscatter cross-section ${\nu}_{+}$ was calculated by taking the ratio of RMSE of the standard approach and the RMSE of the new approach (Algorithm 2). From the first five rows, we see that the backscatter cross-section of the new approach attains better performance, compared to the standard approach without (Algorithm 1) and with block averaging (Algorithm 1a). The optical depth $\tau $ estimates are also improved. This is because the new approach uses the actual noise model to find an estimates of the backscatter cross-section and optical depth, and these parameters are inferred as a piecewise smooth images rather than a collection of unrelated values. In contrast, the standard approach infers the backscatter cross-section and optical depth from individual profiles and any spatial and temporal information is not utilized, except for when block averaging is employed, which is clearly suboptimal.

Table 3.

Root Mean Square Error, Bias, and Standard Deviation (Std) Results of the Second Experiment^{a}

Backscatter Cross-section ${\nu}_{+}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

48

12

80

Optical Depth$\mathit{\tau}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

16

5

25

Alternative approach

6

4

7

Extinction Cross-section$\mathit{\beta}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

5

8

3

Alternative approach

0

1

0

Lidar Ratio$\mathit{\mu}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

$5.17\mathrm{e}+10$

$1.06\mathrm{e}+10$

$2.11\mathrm{e}+11$

Alternative approach

$4.10\mathrm{e}+05$

$4.07\mathrm{e}+05$

$4.10\mathrm{e}+05$

All results are expressed by a factor of reduction relative to the standard approach (Algorithm 1), i.e., the RMSE backscatter cross-section of the new approach is 48 times less than the RMSE backscatter cross-section of the standard approach; the larger the number is, the better. The reduction of, say, the RMSE in regards to the extinction cross-section $\beta $ was calculated by taking the ratio of RMSE of the standard approach and the RMSE of the new approach. In all accounts, the new approach (Algorithm 2) is able to infer the optical depth $\tau $, extinction cross-section $\beta $, and lidar ratio $\mu $ at a better accuracy compared to the standard and alternative approaches. The new approach is able to achieve smaller RMSE values for the extinction cross-section because it uses the previously estimated backscatter cross-section to infer the lidar ratio. The lidar ratio is constrained to be spatial piecewise constant using the total variation smoothness constraint. It follows, then, that the inferred optical depth of the new approach also has a higher accuracy, compared to the other approaches. The standard approach lidar ratio estimates are very inaccurate relative to the new and alternative approaches. This is because the lidar ratio of the standard approach is the ratio of the extinction and backscatter cross-sections, and residual noise in the backscatter cross-section increases the standard deviation of the lidar ratio estimate.

Tables (8)

Table 1.

Non-Exhaustive List of Symbols Used in this Paper^{a}

A short description of each symbol is given, as well as the type of variable (matrix or scalar) and the location where the symbol was introduced.

Algorithm 1

The standard approach in inferring the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. The tuning parameters of ${\mathsf{F}}^{\mathsf{Low}}$ are set by a lidar expert. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{1}$

Let ${F}^{\mathrm{Low}}(\xb7)$ be a low-pass filter function, and let ${F}^{\mathrm{Diff}}(\xb7)$ be a discrete derivative operator.

1: /*Estimate optical depth*/

2: ${\widehat{\tau}}^{\mathrm{alg}-1}$ ← plug ${Y}_{c}$ and ${Y}_{m}$ into Eq. (11)

The standard approach with averaging in inferring the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. The tuning parameters of ${\mathsf{F}}^{\mathsf{Avg}}$ are set by a lidar expert or an automated system, which are determined by some objective criteria. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{1}\mathsf{a}$

Let ${F}^{\mathrm{Avg}}(\xb7)$ be an averaging function, i.e., block averaging.

4: Execute Algorithm 1 using ${Y}_{c}^{\mathrm{Avg}}$ and ${Y}_{m}^{\mathrm{Avg}}$ to get estimates of the backscatter ${\nu}_{+}$ and extinction $\beta $ cross-sections, the lidar ratio $\mu $, and the optical depth $\tau $. Denote these estimates by ${\widehat{\nu}}_{+}^{\mathrm{alg}-1a}$, ${\widehat{\beta}}^{\mathrm{alg}-1a}$, ${\widehat{\mu}}^{\mathrm{alg}-1a}$, and ${\widehat{\tau}}^{\mathrm{alg}-1a}$.

The new approach in inverting for the backscatter ${\mathsf{\nu}}_{+}$ and extinction $\mathsf{\beta}$ cross-sections, along with the lidar ratio $\mathsf{\mu}$, and the optical depth $\mathsf{\tau}$. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{2}$

1: /*Invert the parallel backscatter cross-section*/

2: Set $\ell (\omega ;{Y}_{\iota})={\mathbb{1}}_{N}^{\mathrm{T}}[{f}_{\iota}(\omega )-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}{f}_{\iota}(\omega )]{\mathbb{1}}_{K}$

9: Set $\ell (\tilde{\mu};{Y}_{\iota})={\mathbb{1}}_{N}^{\mathrm{T}}[g(\tilde{\mu})-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}g(\tilde{\mu})]{\mathbb{1}}_{K}$

The TV-PMLE with cross-validation to estimate either ${\mathsf{\omega}}_{\mathsf{\iota}}$ or $\mathsf{\mu}$; see Algorithm 2

Input: A loss function such as $\ell (\psi ;{Y}_{c},{Y}_{m})$, the Poisson noisy matrices ${Y}_{c}$, ${Y}_{m}$ and the constraint set $\mathrm{\Psi}$ for $\psi $.

Another approach to inverting the particulate extinction cross-section $\mathsf{\beta}$. Note that whenever the unknown parameters are estimated with this algorithm, it is indicated by a superscript $\mathsf{alg}-\mathsf{3}$

Input: Previously inverted parallel backscatter cross-section $\widehat{\nu}$, such as ${\widehat{\nu}}^{\mathrm{alg}-2}$, with the depolarization coefficient $\rho $ to compute ${\widehat{\nu}}_{+}$.

1: /*Invert the extinction cross-section*/

2: Set $\ell (\beta ;{Y}_{m})={\mathbb{1}}_{N}^{\mathrm{T}}[h(\beta )-{Y}_{\iota}\xb7{\mathrm{log}}_{e}\text{\hspace{0.17em}}h(\beta )]{\mathbb{1}}_{K}$

Root Mean Square Error, Bias, and Standard Deviation (Std) Results of the First Experiment^{a}

Backscatter Cross-section ${\nu}_{+}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

Standard approach with block avg.

6

6

6

New approach

190

104

200

Optical Depth$\mathbf{\tau}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

Standard approach with block avg.

2

2

2

New approach

7

2

13

All results are expressed by a factor of reduction relative to the standard approach (Algorithm 1), i.e., the RMSE backscatter cross-section of the new approach is 190 times less than the RMSE backscatter cross-section of the standard approach; the larger the number is, the better. The reduction of, say, the RMSE in regards to the backscatter cross-section ${\nu}_{+}$ was calculated by taking the ratio of RMSE of the standard approach and the RMSE of the new approach (Algorithm 2). From the first five rows, we see that the backscatter cross-section of the new approach attains better performance, compared to the standard approach without (Algorithm 1) and with block averaging (Algorithm 1a). The optical depth $\tau $ estimates are also improved. This is because the new approach uses the actual noise model to find an estimates of the backscatter cross-section and optical depth, and these parameters are inferred as a piecewise smooth images rather than a collection of unrelated values. In contrast, the standard approach infers the backscatter cross-section and optical depth from individual profiles and any spatial and temporal information is not utilized, except for when block averaging is employed, which is clearly suboptimal.

Table 3.

Root Mean Square Error, Bias, and Standard Deviation (Std) Results of the Second Experiment^{a}

Backscatter Cross-section ${\nu}_{+}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

48

12

80

Optical Depth$\mathit{\tau}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

16

5

25

Alternative approach

6

4

7

Extinction Cross-section$\mathit{\beta}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

5

8

3

Alternative approach

0

1

0

Lidar Ratio$\mathit{\mu}$

RMSE

Bias

Std

Relative to standard approach, reduced by a factor of

Standard approach

1

1

1

New approach

$5.17\mathrm{e}+10$

$1.06\mathrm{e}+10$

$2.11\mathrm{e}+11$

Alternative approach

$4.10\mathrm{e}+05$

$4.07\mathrm{e}+05$

$4.10\mathrm{e}+05$

All results are expressed by a factor of reduction relative to the standard approach (Algorithm 1), i.e., the RMSE backscatter cross-section of the new approach is 48 times less than the RMSE backscatter cross-section of the standard approach; the larger the number is, the better. The reduction of, say, the RMSE in regards to the extinction cross-section $\beta $ was calculated by taking the ratio of RMSE of the standard approach and the RMSE of the new approach. In all accounts, the new approach (Algorithm 2) is able to infer the optical depth $\tau $, extinction cross-section $\beta $, and lidar ratio $\mu $ at a better accuracy compared to the standard and alternative approaches. The new approach is able to achieve smaller RMSE values for the extinction cross-section because it uses the previously estimated backscatter cross-section to infer the lidar ratio. The lidar ratio is constrained to be spatial piecewise constant using the total variation smoothness constraint. It follows, then, that the inferred optical depth of the new approach also has a higher accuracy, compared to the other approaches. The standard approach lidar ratio estimates are very inaccurate relative to the new and alternative approaches. This is because the lidar ratio of the standard approach is the ratio of the extinction and backscatter cross-sections, and residual noise in the backscatter cross-section increases the standard deviation of the lidar ratio estimate.