The average of atmospheric vertical profiles

Simone Ceccherini; Bruno Carli; Piera Raspollini

doi:10.1364/OE.22.024808

1. Introduction

The averaging of atmospheric vertical profiles is an operation that is more and more frequently performed. Profiles acquired in selected time-space regions are averaged to obtain the climatology of the considered species. Profiles measured by different instruments can be averaged in order to obtain a profile that is more precise than the single ones. Profiles of weak species, that are characterized by large random errors, can be averaged in order to obtain a profile with smaller errors even if with worse geographical and time resolutions.

The averaging method depends on the type of information that is available for the characterization of the profile. In the case of direct measurements the profiles are only characterized by the measurement errors. In the more general case of indirect measurements the profiles are retrieved from the observations solving an inversion problem and, being the errors at the different altitudes correlated, a covariance matrix (CM) is needed to describe the uncertainties of the retrieved profile. Furthermore, when the inversion problem is ill-conditioned the retrieval is performed using some a-priori information regarding the profile to be retrieved in the form of a constraint added to the cost function that is minimized in the retrieval [1]. Optimal estimation and Tikhonov regularization are the most frequently used methods. The use of a constraint causes the averaging kernel matrix (AKM), given by the derivatives of the retrieved profile with respect to the true profile [1,2], to be different from the identity matrix. The presence of constraints in the individual retrievals has the undesirable effect of introducing biases in both the value and the shape of the profile and biases which may be acceptable for a single measurement, may not be the best ones for averages that are less ill-conditioned.

We present herewith a new procedure to perform the average that allows changing a-posteriori the strength of the constraint, so that the retrievals of the single measurements can be performed with any constraint and then, on the basis of the number of measurements that we decide to average, this can be subsequently changed without the need of redoing retrievals. The decoupling of the constraint of the average from those of the retrievals makes possible the exploitation of a wider range of retrieval products.

A verification of the effectiveness of the new method for the change of the constraint is provided by its application to the calculation of the average of a weak species retrieved from the measurements of the MIPAS (Michelson Interferometer for Passive Atmospheric Sounding) instrument [3] onboard the ENVISAT satellite.

In Section 2 we present the new method and describe an averaging procedure that uses it. In Section 3 we apply the proposed procedure to real measurements and in Section 4 we draw the conclusions.

2. Average of vertical profiles

We suppose to have N measurements of vertical profiles of an atmospheric species and we want to calculate a profile that represents their average. The profiles are usually obtained with an inversion procedure and the i-th profile is a vector ${\hat{x}}_{i}$ characterized by a CM S_i. The CM is defined as the mean value of the product ε_iε_i^T, where the vector ε_i contains the errors on the vertical profile obtained propagating the errors of the observations through the retrieval process and the superscript T indicates the transposed of the vector [1, 2]. The a-priori assumptions, used as a constraint in the retrieval, are not considered to be a measurement and do not propagate their errors into the CM. This implies that Tikhonov regularization and optimal estimation will be described by the same mathematics, and the calculation of the CM used in the case of the Tikhonov method will be used in both cases.

The most simple way to calculate the average profile is to calculate the arithmetic average of the profiles ${\hat{x}}_{i}$ :

\bar{x} = \frac{1}{N} \sum_{i = 1}^{N} {\hat{x}}_{i} .

An alternative way to calculate the average profile is to weight the profiles with the inverse of the CMs S_i. The use of the weighted mean is optimal when all the measurements refer to the same air mass, but in the case of measurements referring to different geolocations the presence of possible correlations between the retrieval errors and the values of the profiles may introduce a bias in the weighted mean. For this reason here we use the arithmetic average.

Equation (1) is valid when all the profiles are represented on the same vertical grid. If the vertical grids are different it is necessary to resample the N profiles on a common grid before calculating the average. The resampling can be performed with a linear transformation represented by a matrix H_i:

{\hat{x}}_{i} \to H_{i} {\hat{x}}_{i} .

In order to avoid loss of information the common grid used for the resampling has usually more points than the grids of the single measurements.

If the profiles are obtained with constrained retrievals, the averaging process of Eq. (1) adds up the individual constraints and may cause a too strong constraint in the average. The strength of the constraint of the single retrieval should, therefore, be either commensurate with the number of profiles subsequently averaged or as small as possible. However, the reduction of the constraint has some limits because it can cause the instability of the retrieval, with the consequent increase of the number of iterations and of the computing time, as well as the uselessness of single retrieval products made of too oscillating profiles. The best solution is provided by an averaging method in which the strength of the constraint can be changed. Here below we show that when the a-priori profile and the averaging kernels are known indeed we can change the strength of the constraint.

We indicate with x_ai the a-priori profile used in the i-th retrieval and characterize the result of the i-th retrieval ${\hat{x}}_{i}$ by the covariance matrix $S_{i}$ and by the AKM A_i. If we expand at the first order the relationship between the retrieved profile ${\hat{x}}_{i}$ and the true profile x_i, exploiting the definition of the AKM, we obtain the following equation [1, 4]:

{\hat{x}}_{i} = x_{a i} + A_{i} (x_{i} - x_{a i}) + ε_{i} .

Rearranging Eq. (3) we can write:

{\hat{x}}_{i} - (I - A_{i}) x_{a i} = A_{i} x_{i} + ε_{i} .

In Eq. (4) we notice that the vector

α_{i} = {\hat{x}}_{i} - (I - A_{i}) x_{a i},

which can be obtained from known quantities, is equal to the vector A_ix_i, made of the components of x_i along the averaging kernels and, as such, corresponds to a new indirect measurement of the true profile made in the vector space generated by the rows of the AKM. When the CMs are calculated considering the a-priori as a constraint and not as a measurement, the new indirect measurement given by Eq. (5) is characterized by the same CM

S_{i}

of the

{\hat{x}}_{i}

retrieval.

We can use α_i to obtain a new estimate of the profile by minimizing the following cost function:

c_{i} (x) = {(A_{i} x - α_{i})}^{T} S_{i}^{- 1} (A_{i} x - α_{i}) + {(x - x_{a i})}^{T} S_{a i}^{- 1} (x - x_{a i}),

where the a-priori profile x_ai and the a-priori CM S_ai can have different values from those used in the retrievals. In particular we are interested in using larger errors in the a-priori CM in order to obtain a profile that is less constrained than the original one.

The minimum of c_i(x) provides the new estimate of the profile:

{\hat{x}}^{'}_{i} = {(A_{i}^{T} S_{i}^{- 1} A_{i} + S_{a i}^{- 1})}^{- 1} (A_{i}^{T} S_{i}^{- 1} α_{i} + S_{a i}^{- 1} x_{a i})

with a CM, obtained propagating the error of α_i into

{\hat{x}}^{'}_{i}

, equal to:

{S^{'}}_{i} = {(A_{i}^{T} S_{i}^{- 1} A_{i} + S_{a i}^{- 1})}^{- 1} A_{i}^{T} S_{i}^{- 1} A_{i} {(A_{i}^{T} S_{i}^{- 1} A_{i} + S_{a i}^{- 1})}^{- 1} .

The AKM is obtained performing the derivative of

{\hat{x}}^{'}_{i}

with respect to the true profile x_i and is equal to:

{A^{'}}_{i} = {(A_{i}^{T} S_{i}^{- 1} A_{i} + S_{a i}^{- 1})}^{- 1} A_{i}^{T} S_{i}^{- 1} A_{i} .

From these formulas we can see the importance of the retrieval invariant $A_{i}^{T} S_{i}^{- 1} A_{i}$ [5, 6] which provides the calculation of the Fisher information matrix [7] using the retrieval products and describes the information that the observations have about the retrieved parameters.

The new Eq. (7) is a post-processing that under the approximation of linearity makes it possible to change and even remove (when enough information is available in the observations) the a-priori. The same objective was pursued by [8]. However, their method neglects the correlations, providing a correction of the offset that the constraint introduced in the retrieved values, without a modification of the bias introduced in the shape.

We can summarize the new procedure to perform the average in the following way. The first step is to reduce with Eq. (7) the strength of the constraint on the basis of the constraint that we want to have in the average. The second step is (if needed) to resample the profiles on a common vertical grid using Eq. (2). The last step is to calculate the average by means of Eq. (1).

3. Results

The verification of the possibility of changing the constraint in the averaging process is provided by the calculation of averages of a weak species retrieved from MIPAS observations [3].

MIPAS is a limb-viewing Fourier transform spectrometer that sounds the emission of the Earth atmosphere in the spectral range from 685 to 2410 cm⁻¹. It operated successfully onboard the sun-synchronous polar orbiting ENVISAT satellite that was launched on the 1st of March 2002 and ended its operations on the 8th April 2012. The retrieval of the MIPAS measurements used in this paper was performed using the ORM (Optimized Retrieval Model) [9–11], which is the scientific prototype of the ESA operational level 2 processor.

In order to maximize the information derived from the observations the retrieval grid adopted in the analysis of the MIPAS measurements coincides with the tangent altitude grid of the limb scanning sequence. Since the tangent altitude grid is not constant, retrievals of different measurements adopt different retrieval grids and have averaging kernels that peak at different altitudes.

We consider the HCFC-22 species for which a useful measurement of the profile cannot be retrieved from a single MIPAS limb scanning. HCFC-22 is a hydrogenated CFC that, due to its thermodynamic properties similar to those of CFCs, is used as an alternative to CFCs. The primary sink of HCFC-22 is in the troposphere through oxidation with the hydroxyl radical (OH); therefore, its use results in a lower flux of chlorine to the stratosphere and in a smaller impact on the ozone layer than CFCs. However, it was decided that also the production of all the HCFCs needed to be regulated and the Montreal Protocol set limits on HCFC production, with a total phase out planned in developed countries by 2030 and in developing countries by 2040. Therefore, currently HCFC-22 is expected to increase and subsequently, if Montreal Protocol targets are met, a decay is expected to occur first in the troposphere and then in the stratosphere. In order to verify the correctness of our expectations and the effectiveness of the Protocol implementation the monitoring of stratospheric concentrations of HCFC-22 is very important [12].

The ORM code used to perform the operational retrieval uses two different constraints in sequence. First the chi-square function is minimized using the regularizing Levenberg–Marquardt approach [13] and then an a-posteriori Tikhonov regularization with a self-adapting strength is used to eliminate the residual non-physical oscillations [14–16]. The two constraints use a different a-priori (the initial guess in the case of Levenberg-Marquardt and a constant profile in the case of Tikhonov) and their combined use, while providing a good compromise between retrieval speed and minimum constraint, makes it difficult to determine the effective a-priori. Since the method for changing the strength of the constraint requires the knowledge of the a-priori, as described in the previous section, the ORM was modified in order to perform the retrieval with the optimal estimation method [1], for which the a-priori is well defined. The retrievals used in this paper were performed using the optimal estimation method and this procedure is being considered for the future operational analysis of weak species.

The single measurement is the global fit [17] of a few selected spectral intervals (micro-windows [18]) of the spectra observed in a single tangent altitude sequence and the retrieved state vector includes the volume mixing ratio (VMR) profile of HCFC-22, the profiles of the (frequency independent) atmospheric transparency [19] and the (tangent altitude independent) radiometric offset for each of the micro-windows used in the retrieval. A climatological profile [20] is used as the a-priori profile and its CM is built using an error equal to the value of the a-priori profile for the diagonal elements and a correlation length of 10 km for the calculation of the off-diagonal elements. The a-priori value of the transparency profiles is 1 with uncorrelated errors equal to 1%. The a-priori value of the radiometric offsets is 0 with uncorrelated errors equal to 31.6 nW/(cm⁻² sr⁻¹ cm⁻¹).

We performed the average of one hundred HCFC-22 profiles in the latitude band 40-50 North acquired in October 2007. Using Eq. (7) we calculated new state vectors with reduced strength of the constraint, obtained multiplying the original a-priori CM by a factor k. The approach of using a constant scaling factor for the modification of the constraint was also used in [21]. When the CM is multiplied by a factor the errors are multiplied by the square root of that factor. Subsequently we resampled these profiles on a common vertical grid of 1 km step and calculated the average of the resampled profiles. The standard deviation of the mean is calculated using the following expression [22]:

σ_{j} = \sqrt{\frac{1}{N (N - 1)} \sum_{i = 1}^{N} {({({\hat{x}}^{'}_{i})}_{j} - {(\bar{x})}_{j})}^{2}},

where the index j refers to the altitude level.

In order to assess if this new procedure has correctly changed the strength of the constraint we have also repeated the retrievals using ORM with the a-priori CM multiplied by the same factor k used in the post processing. These profiles have been resampled on the common grid and averaged using Eq. (1) for a comparison with the averages obtained using the new procedure.

Figure 1 shows this comparison in the case of k = 10. The average profile of HCFC-22 VMR obtained changing with Eq. (7) the strength of the constraint from k = 1 to k = 10 is shown by the black line and the average profile directly obtained using k = 10 is shown by the blue line. As a reference the average profile obtained with no change of the constraint using k = 1 and the a-priori profile are shown by the red line and by the green line, respectively. The reported error bars are calculated using Eq. (10) and well compare with the errors calculated by the CM of the average obtained propagating the CMs of the single profiles into the average. This agreement is obtained when a filtering, which removes a few outliers identified by their value of the chi-square test larger than a threshold, is applied to the average obtained with the new procedure. Indeed with k = 10 not all ORM retrievals reach convergence, similarly not all ORM retrievals with k = 1 can be used for the change of the a-priori with Eq. (7). Therefore, the filtering applied to the profiles obtained with the new procedure has the purpose to eliminate those profiles corresponding to ORM retrievals that do not converge when performed with k = 10. Similar results are obtained with k = 100 and k = 1000 and are shown in Figs. 2 and 3, respectively.

Fig. 1 Average vertical profile of HCFC-22 VMR obtained changing with Eq. (7) the strength of the constraint from k = 1 to k = 10 (black line). Average of ORM profiles obtained with k = 1 (red line) and with k = 10 (blue line). A-priori profile (green line).

Download Full Size | PDF

Fig. 2 As Fig. 1 when the new strength of the constraint is obtained using k = 100.

Download Full Size | PDF

Fig. 3 As Fig. 1 when the new strength of the constraint is obtained using k = 1000.

Download Full Size | PDF

The agreement between the black and the blue lines shows the performance of Eq. (7) for the change of the strength of constraint. The two lines are well coincident for k = 10, the differences are within the measurement errors in the case of k = 100 and become slightly larger than the measurement errors in the case of k = 1000. Since for large retrieval errors some differences are expected between the linear transformation of Eq. (7) and the non-linear retrieval performed by ORM, the comparison proves the correctness of the method and its capability to change by a large factor the strength of the constraint. It could be argued that when a difference is observed between the two calculations the post processing method provides better results than the non-linear fit because of the distortions introduced in the latter case by the large errors.

In order to evaluate the strength of the constraint, for each of the averages obtained with the four different values of the parameter k we calculated the mean of the number of degrees of freedom (DOF) of the profiles used in the average. The number of DOF of the profile is assumed to be equal to the summation of the diagonal elements of the AKM [1] and the AKM of the profiles with reduced constraint strength is given by Eq. (9). The values of the average DOF as a function of the k values are reported in Table 1. For a reference, in the last row of the table we have also reported the average value of the numbers of points of the retrieval grids, that correspond to the maximum numbers of DOF obtainable when no constraint is used in the retrievals (k = ∞).

Table 1. Average DOF as a function of k.

View Table

We can see that the DOF increase when the k value increases and the strength of the constraint is reduced. In particular when we change the parameter k from 1 to 1000 the DOF change from 6.3 to 14.6. The DOF obtained for k = 1000 are very similar to the maximum DOF obtainable with no constraint.

From the analysis of the three figures and the table we observe that varying the strength of the constraint a different compromise is found between the measurement errors and the DOF. In all cases the retrieved profile has values significantly different from the climatological profile used for the a-priori (green curve). A large constraint (red curve with k = 1) provides a profile with small measurement errors and the shape is made smooth by the few DOF. Reducing the constraint (the black curves with k values greater than 1) both the measurement errors and the DOF progressively increase and a structure is observed in the profile as a result of the vertical resolution improvement. Unfortunately the observed structure does not correspond to the profile that is physically expected in the case of HCFC-22, which should show a VMR that steadily decreases with altitude. The averaging process, while reducing the random measurement errors, does not reduce the systematic errors and very small effects that are negligible when compared with the retrieval errors of a single measurement become important in the case of the average of 100 measurements.

In the case of averages a significant reduction of the measurement errors can be obtained and this improvement can be effectively traded for better vertical resolution. For HCFC-22 the reduction of the strength of the constraint does not provide a better understanding of the shape of the profile but has the effect of unmasking systematic errors.

4. Conclusion

Space borne instruments have collected a very large number of observations from which the atmospheric vertical profile of species with very small concentrations can also be detected, provided that the measurement noise is reduced with sufficient averaging. On the other hand, averaged products are often affected by undesirable biases. Indeed the vertical profile retrieval of weak species requires the use of some constraint for the attainment of a fast and useful product and, since the constraints and the corresponding biases add up in the averaging process, a constraint that is acceptable for a single measurement may not be the best one for the average. In order to overcome this problem we have introduced a new method for the a-posteriori change of the strength of the constraints used in the retrievals. The method can be used whenever the a-priori profile used for the constraint and the AKM of the product are known.

A verification of the effectiveness of the new method is provided by its application to the calculation of the average of a weak species (HCFC-22) retrieved from the measurements of the MIPAS instrument onboard the ENVISAT satellite. The comparison between averaged profiles obtained by applying the same weak constraint either directly in the retrieval or in the post processing (starting from the product with a stronger constraint) shows a very good agreement for a wide range of constraint strength. Only for very large retrieval errors some small differences are observed between the results of the non linear retrieval and the linear approximation of the post-processing method.

In the analyzed test case we observe that reducing the constraint both the measurement errors and the DOF of the profiles progressively increase and a structure is observed in the average profile as a result of the vertical resolution improvement. Unfortunately the observed structure does not correspond to the profile that is physically expected in the case of HCFC-22. The averaging process, while reducing the random measurement errors, does not reduce the systematic errors and very small effects that are negligible when compared with the retrieval errors of a single measurement become important in the average. The subtle problem of systematic errors is in averages often masked by the small number of DOF and can only be unmasked by an analysis in which the vertical resolution of the profile can be freely changed.

References and links

1. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice, Vol. 2 of Series on Atmospheric, Oceanic and Planetary Physics (World Scientific, 2000).

2. S. Ceccherini and M. Ridolfi, “Technical Note: Variance-covariance matrix and averaging kernels for the Levenberg-Marquardt solution of the retrieval of atmospheric vertical profiles,” Atmos. Chem. Phys. 10(6), 3131–3139 (2010). [CrossRef]

3. H. Fischer, M. Birk, C. Blom, B. Carli, M. Carlotti, T. Clarmann, L. Delbouille, A. Dudhia, D. Ehhalt, M. Endemann, J. M. Flaud, R. Gessner, A. Kleinert, R. Koopman, J. Langen, M. Lopez-Puertas, P. Mosner, H. Nett, H. Oelhaf, G. Perron, J. Remedios, M. Ridolfi, G. Stiller, and R. Zander, “MIPAS: an instrument for atmospheric and climate research,” Atmos. Chem. Phys. 8(8), 2151–2188 (2008). [CrossRef]

4. S. Ceccherini, “A generalization of optimal estimation for the retrieval of atmospheric vertical profiles,” J. Quantum Spectrosc. Radiat. 113(12), 1437–1440 (2012). [CrossRef]

5. S. Ceccherini, B. Carli, and P. Raspollini, “Quality quantifier of indirect measurements,” Opt. Express 20(5), 5151–5167 (2012). [CrossRef] [PubMed]

6. S. Ceccherini, B. Carli, and P. Raspollini, “Quality of MIPAS operational products,” J. Quantum Spectrosc. Radiat. 121, 45–55 (2013). [CrossRef]

7. R. A. Fisher, “The logic of inductive inference,” J.R. Stat. Soc. 98(1), 39–54 (1935). [CrossRef]

8. A. B. Burgess, R. G. Grainger, A. Dudhia, and V. H. Payne, “MIPAS measurement of sulphur hexafluoride (SF₆),” Geophys. Res. Lett. 31(5), L05112 (2004). [CrossRef]

9. M. Ridolfi, B. Carli, M. Carlotti, T. Von Clarmann, B. M. Dinelli, A. Dudhia, J. M. Flaud, M. Höpfner, P. E. Morris, P. Raspollini, G. Stiller, and R. J. Wells, “Optimized forward model and retrieval scheme for MIPAS near-real-time data processing,” Appl. Opt. 39(8), 1323–1340 (2000). [CrossRef] [PubMed]

10. P. Raspollini, C. Belotti, A. Burgess, B. Carli, M. Carlotti, S. Ceccherini, B. M. Dinelli, A. Dudhia, J. M. Flaud, B. Funke, M. Hopfner, M. Lopez-Puertas, V. Payne, C. Piccolo, J. J. Remedios, M. Ridolfi, and R. Spang, “MIPAS level 2 operational analysis,” Atmos. Chem. Phys. 6(12), 5605–5630 (2006). [CrossRef]

11. P. Raspollini, B. Carli, M. Carlotti, S. Ceccherini, A. Dehn, B. M. Dinelli, A. Dudhia, J.-M. Flaud, M. López-Puertas, F. Niro, J. J. Remedios, M. Ridolfi, H. Sembhi, L. Sgheri, and T. Von Clarmann, “Ten years of MIPAS measurements with ESA Level 2 processor V6 – Part 1: Retrieval algorithm and diagnostics of the products,” Atmos. Meas. Tech. 6(9), 2419–2439 (2013). [CrossRef]

12. D. P. Moore and J. J. Remedios, “Growth rates of stratospheric HCFC-22,” Atmos. Chem. Phys. 8(1), 73–82 (2008). [CrossRef]

13. A. Doicu, T. Trautmann, and F. Schreier, Numerical regularization for atmospheric inverse problems (Springer, 2010).

14. S. Ceccherini, “Analytical determination of the regularization parameter in the retrieval of atmospheric vertical profiles,” Opt. Lett. 30(19), 2554–2556 (2005). [CrossRef] [PubMed]

15. S. Ceccherini, C. Belotti, B. Carli, P. Raspollini, and M. Ridolfi, “Technical Note: Regularization performances with the error consistency method in the case of retrieved atmospheric profiles,” Atmos. Chem. Phys. 7(5), 1435–1440 (2007). [CrossRef]

16. M. Ridolfi and L. Sgheri, “Iterative approach to self-adapting and altitude-dependent regularization for atmospheric profile retrievals,” Opt. Express 19(27), 26696–26709 (2011). [CrossRef] [PubMed]

17. M. Carlotti, “Global-fit approach to the analysis of limb-scanning atmospheric measurements,” Appl. Opt. 27(15), 3250–3254 (1988). [CrossRef] [PubMed]

18. A. Dudhia, V. L. Jay, and C. D. Rodgers, “Microwindow selection for high-spectral-resolution sounders,” Appl. Opt. 41(18), 3665–3673 (2002). [CrossRef] [PubMed]

19. M. Ridolfi and L. Sgheri, “On the choice of retrieval variables in the inversion of remotely sensed atmospheric measurements,” Opt. Express 21(9), 11465–11474 (2013). [CrossRef] [PubMed]

20. J. J. Remedios, R. J. Leigh, A. M. Waterfall, D. P. Moore, H. Sembhi, I. Parkes, J. Greenhough, M. P. Chipperfield, and D. Hauglustaine, “MIPAS reference atmospheres and comparisons to V4.61/V4.62 MIPAS level 2 geophysical data sets,” Atmos. Chem. Phys. Discuss. 7(4), 9973–10017 (2007). [CrossRef]

21. A. Carissimo, I. De Feis, and C. Serio, “The physical retrieval methodology for IASI: the δ-IASI code,” Environ. Model. Softw. 20(9), 1111–1126 (2005). [CrossRef]

22. D. S. Sivia, Data Analysis (Oxford University, 1996).

The average of atmospheric vertical profiles

Abstract

1. Introduction

2. Average of vertical profiles

3. Results

4. Conclusion

References and links

Cited By

Figures (3)

Tables (1)

Equations (10)

Optics Express

k	Average DOF
1	6.3
10	9.0
100	12.1
1000	14.6
∞	15.7