Lidar measurements of the atmospheric water vapor mixing ratio provide an excellent complement to radiosoundings and passive, ground-based remote sensors. Lidars are now routinely used that can make high spatial-temporal resolution measurements of water vapor from the surface to the stratosphere. Many of these systems can operate during the day and night, with operation only limited by clouds thick enough to significantly attenuate the laser beam. To enhance the value of these measurements for weather and climate studies, this paper presents an optimal estimation method (OEM) to retrieve the water vapor mixing ratio, aerosol optical depth profile, Ångstrom exponent, lidar constants, detector dead times, and measurement backgrounds from multichannel vibrational Raman-scatter lidars. The OEM retrieval provides the systematic uncertainties due to the overlap function, calibration factor, air density and Rayleigh-scatter cross sections, in addition to the random uncertainties of the retrieval due to measurement noise. The OEM also gives the vertical resolution of the retrieval as a function of height, as well as the height to which the contribution of the a priori is small. The OEM is applied to measurements made by the Meteoswiss Raman Lidar for Meteorological Observations (RALMO) in the day and night for clear and cloudy conditions. The retrieved water vapor mixing ratio is in excellent agreement with both the traditional lidar retrieval method and coincident radiosoundings.
© 2016 Optical Society of America
Water vapor is among the most important gases in the atmosphere. Water plays a key role in atmospheric chemistry (primarily as a source of OH) in the radiative budget through its strong absorption and scattering properties in the gaseous, liquid, and solid phases and in atmospheric dynamics through latent heat processes. Water vapor is extremely variable in space and time, and its concentration drops by three orders of magnitude between the surface and the tropopause, which makes it extremely difficult to measure. Mixing ratios greater than 20 g/kg in the lower troposphere are important for weather processes that couple to the stratosphere, where small amounts (a few mg/kg) play a role in the global circulation .
Feedback between the water vapor in the upper troposphere and lower stratosphere (UTLS) and the surface temperature is important, but not understood . An assessment of water vapor measurements in the UTLS by the Stratospheric Processes and their Role in Climate (SPARC) program  concluded that in addition to improved in situ measurements, there was a need for a continuity of measurements to detect trends. Trend detection requires detailed, continuous monitoring of an instrument’s systematic and random measurement uncertainties, including uncertainties due to calibration.
The SPARC report also called for a wider use of lidar water vapor measurements both for validation and for long-term trend detection. Whiteman et al.  discussed the measurement of trends in the upper troposphere using lidar, and concluded that while only about seven measurement periods a month are sufficient, “various real-world problems can plague atmospheric data.” While many major advancements in lidar water vapor hardware have been implemented, in part to avoid jumps in time series due to system changes, the basic analysis of these measurements is still that due to Melfi et al.  in 1969. Though systematic uncertainties are carefully considered and, in some cases, even monitored by using calibration lamps (e.g., Leblanc ), the profile-by-profile analysis of these measurements typically only includes the statistical uncertainties. The measurements are sampled at some native resolution, but the analysis routine, working on the native or any other grid, usually acts as a filter rendering neighboring points of the retrieval dependent and reducing the resolution compared to the grid spacing.
This difference between grid spacing and resolution of the retrieved quantities is often not reported. The maximum height of a useful retrieval is sometimes chosen by an arbitrary cutoff. Furthermore, to capture the large dynamic range of the measurements from near the surface to the stratosphere requires multiple detector channels, as the dynamic range of the signal is beyond the range where a single analog or digital channel is sufficient. In this case, channel can refer to separate physical detectors, or the same detector used simultaneously in analog and digital modes. Currently, multiple detector channels must be glued, with consequences to the interpretation of the retrieval accuracy that are often not considered.
The OEM allows these issues to be addressed on a profile-by-profile basis and can be implemented for any existing lidar. Using the OEM to retrieve the water vapor mixing ratio for a vibrational Raman lidar system in day, night, and cloudy conditions offers consistent results with the traditional techniques and, also has several additional advantages. The OEM is robust and allows researchers to calculate systematic, as well as random, uncertainties. Additionally, it also determines the retrieval’s vertical resolution as a function of altitude, as well as providing a quantitative estimate of the maximum height to which a profile can be retrieved. The retrieval is applied to four channels of measurements: an analog/digital pair for the water vapor returns and an analog/digital pair for the molecular nitrogen returns. Using an OEM allows retrieval of a single water vapor mixing ratio profile consistent with all the input measurements, without the need to correct or glue measurements, or even to have the measurements in the same units or on the same measurement grid. Other system and atmospheric parameters are retrieved as required by the first-principle forward model employed, but this retrieval is explicitly designed for profiles of the water vapor mixing ratio.
2. WATER VAPOR RETRIEVAL USING RAMAN-SCATTER LIDAR MEASUREMENTS
A. Lidar Equation
The measured-backscattered photocounts, , for a lidar system are given by the lidar equation
- • the efficiency of the system, accounting for detector efficiency and nonlinearities as well as losses in the optics;
- • the atmospheric transmission, which for Raman scattering is different in each direction;
- • the number of transmitted photons;
- • the scattering cross section;
- • the area of the receiving telescope; and
- • the geometric overlap.
B. Correction for Detector Nonlinearity
The lidar equation as written above presumes that the count rates depend linearly on the number of received photons per shot and per bin. In other words, it presumes that the photomultiplier counting system does not miss counts because they occur at too high a rate. The count rate is linear for the analog channels, as well as for the digital channels at count rates below about 1 MHz. For the digital channels at higher count rates, the true and observed counts are related by , the counting system dead time
Typically lidar profiles are corrected for dead time effects using empirical corrections or models [7,8]. The OEM retrieval uses data from four channels, one analog and one digital channel for each species. Thus, there is a sufficient amount of information so the dead time for both digital channels can be retrieved. Retrieval of the dead time is of particular importance for systems that operate during the daytime, due to the large amount of solar background present.
C. Lidar Instrument Function
The instrument function for a Raman-scatter lidar can be written as
The overlap can either be implemented into the OEM in a parameterized form (e.g., the model developed by Stelmaszczyk et al. ) or can be specified at each level of the data grid, in which case the values are obtained from measurements or numeric simulations. Povey et al.  have shown for the first case that the overlap parameters can potentially be retrieved. In the second case, overlap is a model parameter. Due to the high complexity of the overlap of the RALMO system, the overlap is specified at each level based on the calculations of Dinoev et al.  and treated as a model parameter. The RALMO system has been engineered to have negligible differential overlap, defined as a difference in the overlap as a function of wavelength between the nitrogen and water vapor wavelengths.
As discussed below, the aerosol optical depth profile is implemented as a retrieval parameter. However, its uncertainty is completely driven by the uncertainty in the overlap and any deviations of the retrieved or specified overlap from the true overlap are compensated by aerosol extinction. In the case of RALMO, below the height of full overlap (approximately 2500 m) the uncertainties in overlap render the retrieved aerosol optical depth not exploitable. However, for RALMO, the extinction coefficient, derived from the height derivative of the aerosol optical depth, is of acceptable quality above the height of full overlap.
3. CURRENT WATER VAPOR RETRIEVAL METHODS
The traditional method is based on the work of Melfi et al. , who showed that the water vapor mixing ratio is proportional to the returns from the water vapor and nitrogen detector channels. Melfi then refined this method to include the height-dependent effect of transmissions . Whiteman et al. and LeBlanc et al. [13,14] further refined the method. In summary, the lidar equations for each species are solved for corrected photocounts; that is, the background term on the right hand side is determined and then subtracted from the observed photocounts. Depending on the lidar’s configuration, the counts may have to be corrected for saturation of the phototube at high count rates, or multiple channels with different gains may be employed to generate a glued profile [15–17]. A particularly difficult issue involves the merging of analog and digital channels. While in the linear region the digital channel variance is appropriate for Poisson statistics, the digitized analog signal often does not follow Poisson statistics, and has uncertainty due to both the intrinsic uncertainty of the measurement and to the uncertainties introduced by the analog-to-digital converter. It should, however, be noted Whiteman et al. found that for their counting system, the analog counting statistics were consistent with a Poisson distribution . Best practice in the field is often to merge the analog counts to the value of the digital counts, and then use the analog signal as if it were a digital measurement. This procedure, though, has not been quantified mathematically as to whether the noise of the analog measurement has been correctly preserved. The OEM allows for multiple sets of measurements at different resolutions and different units, which eliminates the need for gluing of photocount profiles.
In the traditional method, once corrected photocounts have been determined the ratio of the corrected counts is computed. For a system with no differential overlap, this procedure eliminates the effect of the overlap function. The relative measurement is then put into the absolute units of the mixing ratio using a lidar calibration factor, which is typically either estimated from radiosondes or from a combination of radiosondes and microwave radiometers . Venable et al.  recently demonstrated a lidar system that is able to internally calibrate itself to an accuracy of 5%. Atmospheric transmissions can be estimated from models or coincident radiosonde measurements. Whiteman  gives an uncertainty of 10% to 15% for this assumption. Some systems need to account for calibration changes that are dependent on height and not related to transmission, such as the Eureka differential absorption lidar (DIAL) water vapor system . For this method, statistical uncertainties are typically calculated on a profile-by-profile basis.
4. APPLICATION OF THE OEM TO RAMAN-SCATTER LIDAR
While the OEM is well known in the atmospheric satellite and passive optical measurement community, a lidar specialist may be unfamiliar with it. While the OEM has been used regularly to combine measurement sets from lidars and other instruments [22–24], only recently have first-principle forward models been proposed for lidar applications. In these studies, the raw measurement is modeled using a detailed version of the lidar equation. Povey et al.  offers an excellent overview of the OEM and its application to aerosol lidar measurements. We  have recently applied the OEM to the retrieval of Rayleigh-scatter temperature from lidars. Both studies found significant advantages to the use of the OEM over the traditional retrieval method.
In brief, the general solution for the OEM used in this study is found as follows. The measurement and all forward model and retrieval parameters are described using probability density functions (PDFs). Bayes’ theorem then specifies the PDF of the retrieval parameters given a specific measurement and an a priori state of the atmosphere. The optimal solution is then taken as the most likely state of the retrieval parameter’s PDF, hence the state which maximizes the PDF. For a detailed mathematical treatment of the OEM solution used see Rodgers . Further mathematical detail applied to first-principle lidar retrievals can be found in the Povey et al. and our OEM lidar temperature retrieval paper.
The OEM offers several significant advantages over the traditional water vapor lidar retrieval:
- • along with the water vapor mixing ratio, a suite of other parameters can be retrieved simultaneously including geometric overlap (if an analytic function is known), aerosol optical depth profile, Ångstrom exponent, detector dead time, lidar constant, and background for all channels used in the retrieval;
- • a full uncertainty budget can be retrieved including statistical and systematic uncertainties, accounting for forward-model parameter uncertainties;
- • no post or pre-filtering of the retrieval is required and the height resolution of the retrieved profile is the full width at half-maximum of the averaging kernels, which is computed at each height;
- • the method is fast computationally (on the order of the traditional method), and is extremely fast relative to some other inversion methods such as the grid-search technique;
- • OEM can be applied at any required height or time resolution, e.g., nightly averaged profiles that are co-added in height or individual profiles at high temporal-spatial resolution; and
- • OEM is flexible and can be extended with further measurements in the forward model and/or additional retrieval parameters.
These advantages will be explored in detail in the next section.
B. Forward Model for Raman-Scatter Water Vapor Retrieval
For a lidar, a first-principle retrieval begins with the lidar equation. For a water vapor retrieval, the lidar equation must be applied to both the water vapor measurement, , and the nitrogen measurement, . The lidar equation for the true counts due to each species is then
The transmissions are determined from the optical depth, :
The Ångstrom exponent, , allows the aerosol optical depth to be estimated as a function of wavelength via
The forward model as written are true for linear measurements, either digital or digitized analog measurements. For digital measurements, at count rates above about 1 MHz, pulses can begin to pile up and the observed count rate is related to the true count rate using either Eq. (2) or Eq. (3).
One advantage of the OEM is the ability to determine the relative contribution of the a priori information relative to the contribution of the measurements. We used this information in the retrieval of Rayleigh-scatter lidar temperature to determine a height below which the retrieval was primarily due to the measurement and not the a priori. We found that the well-known criteria that the sum of the averaging kernels (e.g., the measurement response) exceeding 0.8 was consistent with the maximum height found from the trace of the averaging kernel matrix, that is the degrees of freedom.
Both criteria are used in this study. We found in clear conditions that the measurement response function is in excellent agreement with the signal-to-noise of the photocount measurements. In cloudy conditions, particularly during daytime, the measurement response often overestimates the height where the measurements have reasonable signal-to-noise levels. For cases such as the cloudy daytime case shown below, the more conservative height based on the degrees of freedom is used to specify the height at which the retrieval is primarily due to the measurements and not the choice of a priori.
In summary, from the set of equations above the OEM retrieves height profiles of the water vapor mixing ratio and the aerosol optical depth. The other retrieved parameters are the lidar constants, the backgrounds, the Ångstrom exponent, and the dead time for each digital channel. Any other information required by the forward model is specified using model parameters. Model parameters will be discussed in the following section.
C. Implementation of the OEM
As in our earlier OEM paper we used the Qpack package for the retrievals. Differences between this retrieval and our OEM temperature retrievals include increasing the number of data channels from two to four and the use of a log-normal retrieval. An interesting parallel between our previous temperature retrieval and the retrieval of aerosol optical depth profile is the fact that aerosol optical depth at a given height depends on all the heights below it. This situation is not unlike that of the hydrostatic equilibrium forward model we considered, where, due to the assumption of hydrostatic equilibrium, the Jacobian of the forward model with respect to temperature is correlated at all heights below a given level.
The OEM requires an estimate of the covariances associated with the measurement and the retrieval and forward model parameters. For the digital channel in the range where the counts are linear (or correctable), the photocounts obey Poisson counting statistics, and the measurement variances are equal to the number of photocounts in a range bin with no correlation between the range bins assumed. The a priori backgrounds and their variances are taken as the mean and variance of the counts above 50 km altitude. Poisson statistics are not necessarily appropriate for the analog channels. The variance of the analog channel was determined for each point by taking three points on either side of a measurement bin and fitting a straight line. This line is then subtracted from the seven values, and the variance of the residuals is determined. We tested this scheme and found it agreed with the Poisson statistics for the digital channels in the linear regime. We then determined the analog a priori background in the same way as for a digital channel.
The a priori water vapor profile used for all retrievals is the U.S. Standard Atmosphere . Due to the high variability of water vapor, the U.S. Standard water vapor profile was scaled to have the same total amount of water vapor as the coincident radiosonde in the altitude range from 500 to 1000 m. Due to water vapor’s large variability the standard deviation of the a priori is taken as 50%.
The calculation of the atmospheric transmissions requires knowledge of the extinction cross sections and the air density. We computed the Rayleigh extinction cross sections using the formula of Nicolet  and assigned a standard deviation of 0.3% of their value. The coincident radiosonde pressure and temperature measurements allowed us to calculate the dry air density, which we assume is known to 1%. We estimated the a priori value of aerosol optical density from the RALMO backscatter ratio standard product . We calculated the aerosol backscatter coefficient, , from the backscatter ratio and then converted it to an extinction assuming a value for the lidar ratio (the ratio of extinction to backscatter). We found the a priori aerosol extinction, , from32] provide the choice of the lidar ratio. For the measurements on clear days, we assumed the lidar ratio to be 80 sr inside the boundary layer (whose altitude was estimated from the RALMO measurement) and 50 sr above. For periods with an extended, low altitude, cloud deck, we took the lidar ratio as 20 sr at all heights. The transmission at the Raman-scatter wavelengths due to aerosols is then computed and related to the elastic wavelength values using Eq. (9). The Ångstrom exponent for aerosols has an a priori value of 1.00 and a standard deviation of 10%. Due to the high variability of the aerosol optical density, the standard deviation of the a priori is taken as 50%, similar to the a priori water vapor standard deviation.
The lidar calibration factor, , is typically found from other instruments such as radiosondes or microwave radiometers. For RALMO, the lidar calibration factor is found using coincident radiosondes as described by Brocard et al. . This technique yields a lidar calibration factor accurate to about 5%. Fitting the forward model to the nitrogen channel measurements (analog and digital) at a nominal height of 3000 m altitude allowed us to find the a priori value of the nitrogen lidar constant. For the digital water vapor channel we determined the lidar constant from the nitrogen lidar constant and lidar calibration factor via1 summarizes the choice of the values and associated uncertainties of the measurements and the retrieval and forward model parameters.
5. RESULTS AND VALIDATION OF OEM WATER VAPOR RETRIEVAL
We retrieved water vapor using measurements obtained by the RALMO system located at the aerological station in Payerne, Switzerland (46.8° N, 6.93° E, 492 m above sea level). RALMO is a fully automated, operational Raman lidar that sends water vapor and aerosol measurements to the MeteoSwiss central database every 30 min for real-time use in numerical weather prediction, nowcasting, and forecasting. The lidar system uses measurements from an external rain detector and a ceilometer to monitor the sky conditions to determine if measurements are possible. If precipitation occurs or low clouds (ceiling below 900 m) are present, RALMO automatically switches to a standby mode, then automatically resumes measurements when weather conditions are favorable again (no rain for 90 min and a cloud base higher than 2000 m). RALMO has operated near continuously since 2008, with the only interruptions to its measurements due to inclement weather or occasional system maintenance. Water vapor profiles are operationally processed and made available in real time. These profiles typically reach approximately 9 km during the night and 4 km during the day for integration periods of 30 min. This good performance, particularly in the daytime, is achieved with the combination of a narrow telescope field-of-view, a narrow receiver bandwidth, and high laser power. The tripled Nd:YAG laser provides up to 9 W at a repetition rate of 30 Hz, however, in standard operation the laser power is reduced to 6 W to extend the laser lifetime. The telescope and the receiver are coupled by optical fibers to ensure good alignment stability. The data acquisition system consists of photomultipliers (Hamamatsu R5600) and analog/digital transient recorders from Licel. The system obtains a measurement every minute (1800 laser shots) with a height resolution of 3.5 m. A unique aspect of RALMO is its light collection system, comprising four telescopes, each 30 cm in diameter, plus a near-range optical fiber, located off the optical axis of one of the four telescopes. The function of the near-range fiber is to provide signal at the lowest heights where the geometric overlap of the four telescopes is very small. All five telescopes are imaged on the holographic grating of the polychromator for spectral separation of the nitrogen and water vapor signals. Dinoev et al. , describes the details of RALMO, while Brocard et al.  gives the instrument’s validation.
A major effort was undertaken to devise a parameterization scheme to allow retrieval of the RALMO overlap function. The geometric overlap model of Stelmaszczyk et al.  was applied to a single telescope, and then the weighted sum of the four telescopes was parameterized, and the telescope tilts (relative to the vertical) and weights at each tilt included in the OEM retrieval. This scheme worked reasonably well above 1000 m altitude. A parameterized method including the off-axis fiber was not possible, however, and since the telescopes and near-range fiber are summed on the photomultiplier, it is impossible to separate the five inputs. We decided to use the overlap as given by Dinoev et al.  as a model parameter, with a standard deviation of 10% below 2500 m. Above this height in the region of complete overlap the standard deviation is taken as 0.01%. Hence, for RALMO it is impossible to separate changes in optical density from uncertainties in the overlap model below the height of complete overlap.
We present three nights in detail to demonstrate the robust nature of the retrieval. The previous RALMO studies of Dinoev et al.  and Brocard et al.  include a detailed discussion of measurements at 1200 UTC on Sept. 5, 2009, and at 0000 UTC on Sept. 6, 2009. We use these same two cases to demonstrate the OEM retrieval of water vapor during clear conditions. Daytime measurements (and retrievals) are more challenging than at nighttime due to the high background level, and the fact that for the RALMO digital water vapor channel, the signal is affected significantly by pulse pileup over the entire height range. For a third detailed example, we choose a period coincident with the 1200 UTC sounding on March 5, 2015, to demonstrate the robustness of the retrieval when clouds are present during the daytime.
We took care to vary the parameters as little as possible between the different cases, anticipating the use of this technique for the operational RALMO retrievals in the near future. The vertical resolution of the data grid is 17.5 m, while quantities were retrieved on a grid with a vertical resolution of 52.5 m for each retrieval. The data grid extends to 17.5 km for the digital channel, and 12 km for the analog channel. Critical to a robust OEM retrieval is the correct specification of the covariance matrices. Often there is correlation between elements of the retrieval state and between the model parameters that is specified by the off diagonal elements of the covariance matrix, but is generally difficult to quantify. For the water vapor mixing ratio and aerosol optical depth profiles, a correlation length of 787.5 m was specified using a tent correlation function as given by Erikkson et al. , which was then applied to generate the full covariance matrices.
A. Clear Sky, Nighttime
The baseline case is for clear sky conditions at night. For this case we chose the measurement period to be the one examined in detail by Brocard et al. , specifically 30 min of RALMO soundings ending at 0030 UTC on Sept. 6, 2009. A coincident radiosonde from Payerne was launched at 0000 UTC. Figure 1 shows the count measurements in the four channels. The backscatter coefficients derived from the traditional method show a layer around 2000 m, with a secondary layer around 4000 m. The a priori value of aerosol optical density was estimated from the RALMO backscatter coefficient using an assumed lidar ratio as previously discussed.
The Jacobians show the sensitivity of the forward model to variations of the retrieval parameters. Figure 2 shows the Jacobians for the logarithm of the water vapor (left hand side) and the aerosol optical depth profile (right hand side) for the analog (top) and digital (bottom) channels. The sensitivity to water vapor of the forward model is limited at the lowest heights due to geometric overlap effects, highlighting the importance of the overlap on the returns below 2000 m. Figure 3 shows the averaging kernels of water vapor mixing ratio and aerosol optical depth. For the logarithm of the water vapor mixing ratio, the averaging kernels are narrow and have an amplitude close to 1.0 below about 5 km, and with significant contributions by the measurements and not the a priori below about 10 km. The aerosol optical density averaging kernels extend much higher, as they only depend on the higher signal-to-noise ratio nitrogen measurements and not the water vapor channels. Figure 1 shows that 10 km altitude is consistent with the height that water vapor signal begins to disappear into the background.
The vertical resolution of the retrieval is not necessarily equal to the spacing of the retrieval grid. The FWHM of the averaging kernels is the retrieval’s vertical resolution as a function of height (Fig. 4). When the averaging kernels are large, below 5 km, the vertical resolution is that of the retrieval grid. Above this height, the resolution begins to increase, reaching a maximum of around 500 m at the top of the usable retrieval around 10 km. Table 2 gives the degrees of freedom of the retrieval, as well as the retrieved Ångstrom exponent and dead time values. The values of the Ångstrom exponent are less than 1, suggesting larger aerosols in the boundary layer. The Licel counting system is specified as being nonparalyzable, hence Eq. (3) is used to retrieve the dead time. The dead times found are about 4 ns, consistent with dead times estimated from the combined analog–digital channels using a fitting approach and close to the dead times specified by the manufacturer.
The residuals are defined as the difference between the forward model and the measurements (Fig. 5). The residuals (blue curves) are unbiased and their magnitude is on the order of the measurement covariance (shown in red) as expected for a good retrieval with cost on the order of unity. The residuals confirmed that the essential physics is contained in the forward model. Figure 6 shows the retrieved water vapor mixing ratio. For comparison, the RALMO measurements processed using the operational retrieval based on corrected photocount ratios are within a few percent of the OEM retrieval at most heights. The differences between the OEM and traditional lidar retrievals and the radiosonde-derived water vapor mixing ratio are at some heights two to five times less than the differences among the lidar retrievals. This difference in the lidar retrievals is due in large part to an approximately 260 m difference in the height of the structure seen at 4500 m by both lidar retrievals.
The OEM-derived mixing ratio decreases rapidly above 9 km following the decrease seen in the radiosonde measurement, while the traditional retrieval does not show this decrease. This difference is explained primarily by the different vertical smoothing between the OEM and traditional retrieval. The traditional retrieval applies a vertical smoothing of 600 m at heights above 9 km while the OEM method conserves a higher vertical resolution of approximately 200 m. The random uncertainty of the OEM retrieval becomes greater than 10% above 9 km (Fig. 7). The OEM provides a complete budget of random and systematic uncertainties. The systematic uncertainties are those due to the Rayleigh scatter cross section, the air density, the lidar calibration factor and the overlap function. The total of the systematic uncertainties is greater than the random uncertainties below about 6 km. Below 1 km the overlap function has the largest effect on the uncertainty. The lidar calibration factor is a constant 5%, while the uncertainty due to air density in the transmission calculation adds a few percent uncertainty, decreasing with increasing height. The MOHAVE-2009 campaign  used lidar, radiosondes and frost point hygrometers measurements to estimate an uncertainty budget. The overlap and calibration uncertainties they deduced were similar to the values we found using the OEM retrieval.
The smoothing error of an OEM retrieval accounts for the limited capability of the retrieval to resolve fine scale structures; that is, the smoothing error quantifies whether or not the temporal–spatial resolution of a retrieval is sufficient to capture the true atmospheric state (the off-diagonal elements of the averaging kernel). Smoothing error is not shown for the retrievals due to limitations related to its application to the RALMO measurements. Both Rodgers and von Clarmann give convincing arguments that the smoothing error in practice cannot be correctly quantified [27,35]. Furthermore, smoothing error does not follow Gaussian error propagation and hence, cannot be propagated between the measurement and retrieval grids. The smoothing error is a useful quantity for comparing retrievals with significantly different vertical resolutions, but for studies such as this one where the differences in vertical resolution between the lidar retrievals and the radiosonde measurements are, over much of the troposphere, relatively small, the smoothing error is not required.
Since optical depth (and transmission) at a given height depends on all the heights below it, the aerosol optical depth retrieved for RALMO at heights where overlap is complete may still be affected by the lower altitudes. However, the extinction due to aerosols does not depend on an integral with respect to height since it is the derivative of the aerosol optical depth. This combination of an incomplete description of the RALMO overlap function and the transmission is seen in the aerosol extinction profile derived from the retrieved aerosol optical depth profile (Fig. 8). Below the height of complete overlap (), the extinction is unrealistically large (and, as discussed in the next paragraph, the systematic uncertainty is much greater than 100%) due to the incompletely specified overlap function. Above the height of complete overlap, the aerosol extinction coefficient is reasonable and is typically about half the value of that due to molecules. There is an increase in the extinction around 3.5 km in the same region where the RALMO backscatter coefficient shows a increase from background values to values greater than 1.1.
Figure 9 shows the uncertainty associated with retrieval of the transmission function due to aerosols. For aerosol transmission, the random uncertainty is relatively small compared to the systematic uncertainty. The sharp spike at 1750 m extends to 795% (that is the uncertainty of the transmission is about eight times its value). Three-quarters of the uncertainty is due to the overlap function, rendering any estimates of transmission unreliable below this height. Uncertainty in the air density is about 20% of the spike, with a small contribution due to the random uncertainty. The spike’s altitude is at the peak uncertainty in the specification of the overlap function.
B. Clear Sky, Daytime
The retrieval setup during the daytime is identical to the one used for nighttime. The major difference between this and the previous retrieval is the large background in the measurements, particularly evident in the digital channels (Fig. 10). This large background reduces the signal-to-noise ratio of the measurements and hence, lowers the maximum height to which water vapor can be retrieved.
The backscatter ratios show aerosol scattering in the boundary layer, with virtually no aerosols in the 1500 to 5500 m region. The Jacobians (not shown) are similar in shape and magnitude to the night case with the exception of a bump in the water vapor Jacobians just below 4000 m. This feature, which is accompanied by a higher signal-to-noise ratio of the water vapor photocount returns, causes a prominent increase in the logarithm of the water vapor averaging kernels (Fig. 11). The vertical resolution is that of the retrieval grid below 2000 m where the averaging kernels are large, decreasing in the region of the bump and then increasing to about 300 m near the top of the useful portion of the retrieval. Table 2 shows the degrees of freedom of the retrieval, the retrieved Ångstrom exponent, and dead time values.
The residuals are unbiased and generally within the limits of the covariance matrix (Fig. 12). The bump at just below 4000 m is not visible in the residuals, meaning the forward model was able to resolve this feature. The OEM retrieval of water vapor mixing ratio is shown in Fig. 13. The bump shows as a sharp increase in both the OEM and traditional retrievals and the radiosonde measurement. However, it is evident that the OEM retrieval reproduced the sharp features more closely to the radiosonde measurement than the traditional processing. This result is again due to the fact that the OEM adapts the vertical resolution in an optimal way as a function of altitude (Fig. 14). In contrast to this adaptation of resolution in the OEM, the operational processing uses a simple scheme to reduce vertical resolution in order to conserve the statistical uncertainty below a certain level. The OEM retrieval above 4000 m shows structure similar to the second decrease in mixing ratio, while the traditional method is not sensitive to this change, again, due to the averaging employed.
The uncertainties for the water vapor mixing ratio show a similar structure to the night case, with the statistical uncertainty greatest at the upper heights and the lidar calibration factor uncertainty constant with a value around 5% (Fig. 15). The overlap uncertainty is largest at the lower heights. The overlap uncertainty, as well as the air density uncertainty, contributes significantly to the uncertainty budget up to 4 km.
C. Cloudy Sky, Daytime
A 20-min period ending at 1230 UTC on March 5, 2015, was chosen to show a worst case scenario, with the presence of a high background plus the constant presence of clouds. The backscatter ratio is on the order of 20 at 1200 m at the cloud base. The cloud base height can be seen in coincident ceilometer measurements taken with a Eliasson CBME80 automatic ceilometer (Fig. 16).
The OEM retrieval used a lidar ratio of 20 sr at all heights, to take into account the fact that the lidar ratio is lower in clouds. The degrees of freedom of the retrieval (Table 2) are used to define the height at which the a priori begins to significantly contribute. The averaging kernels show strong nitrogen returns, but the water vapor averaging kernels become noisy above 2 km (Fig. 17). The vertical resolution is that of the retrieval grid until about 1400 m cut-off altitude (Fig. 18).
The residuals for the water vapor data channels are unbiased (Fig. 19). There is some structure in the nitrogen channels, particularly around 1200 m, the altitude where the extinction rapidly increases. This structure is due to the limited ability of the transmission to vary with height due to sharp concentration gradients in aerosols, although the retrieval is able to adjust itself above the cloud base. The retrieved water vapor agrees with the traditional analysis to within less than 5%, and unlike the traditional retrieval does not have a small () bias with respect to the radiosonde (Fig. 20). At the top of the retrieval, as the radiosonde enters the clouds, it measures a rapid decrease in mixing ratio. The uncertainties are similar to the previous cases (Fig. 21).
The retrievals shown in the previous section, as well as other periods processed using the OEM, show the OEM retrievals are robust and are able to handle daytime and nighttime measurements, both with and without clouds present. Water vapor is a particularly challenging quantity to retrieve, as compared to, for instance, middle atmosphere temperature retrieved from Rayleigh-scatter lidar measurements. For the temperature retrievals presented in our previous OEM lidar retrieval paper , structures in temperature are typically on the order of less than 20%, and temperature cannot “disappear.” However, water vapor can form layered structures with rapid increases and decreases, and regions of the atmosphere can be dry to an extent below what an instrument can detect. While for the temperature retrievals an a priori standard deviation of about 15% can be assumed for the lower thermosphere, the water vapor variability can be on the order of 100%. Despite these challenges the OEM water vapor retrievals are of the same quality or at times better than the traditional retrievals, even while using a featureless a priori that can be different by more than a factor of 10.
The first-principle form of the retrieval used in this study is highly adaptable to different lidar systems. Some water vapor lidar systems, for example, use interference filters sufficiently narrow that changes in atmospheric temperature with altitude can affect intensity of the water vapor returns . It would be straightforward to modify the forward model with an intensity factor that could vary with temperature. The temperature could be from a coincident measurement or from a model. The sensitivity of the mixing ratio retrieval to the assumed temperature could then be included as a systematic uncertainty. The same scheme could be used to account for a system background that is not constant with height.
Use of the OEM retrieval described here will allow better comparisons to be made with different sensors, such as with space-based platforms, where knowledge of the averaging kernels is critical for meaningful comparisons. In addition, the ability to make a profile-by-profile assessment of the uncertainty budget will be important to encourage the use of these measurements by other programs. The World Meteorological Organization’s Global Observing System for Climate, for instance, gives guidelines for satellite measurements of water vapor in the UTLS . For 4-h integrations at 2 km vertical resolution, these guidelines specify an accuracy of 5% and a stability of 0.3% per decade. The OEM retrieval presented will allow the accuracy and stability of ground-based lidar measurements to be assessed in the manner described in the report.
This study, as well as those of Povey et al.  and our paper on OEM Rayleigh lidar temperature retrieval , shows the value of using first-principle OEM retrievals that use the raw (level 0) measurements, as opposed to using corrected measurements. By including the instrument function as part of the forward model, the sensitivity of the retrieved parameters to the instrument function can be determined. OEMs allow the difficult-to-quantify systematic uncertainties such as background (linear or more complicated for some systems), overlap or differential overlap, and merging of multiple digital/digital or analog/digital channels to be estimated on a profile-by-profile basis, with rapid computation times of less than a few seconds on a laptop computer.
Unfortunately, it was impossible to develop an overlap model for the RALMO system. The aerosol extinction coefficient shown, which was a best-case example, still suffered from a poor estimate of the overlap, and was not useful below the region of complete overlap (unlike the water vapor mixing ratio, which could be retrieved down to about 50 m). We look forward to applying this technique to other Raman lidars, where it may be possible to retrieve the overlap as a simple parameterization of telescope and transmitter system parameters.
We have demonstrated that the OEM is an ideal technique to retrieve geophysical and instrument quantities from lidar measurements. The raw measurements can be adequately represented by a first-principle forward model that allows the water vapor mixing ratio, the optical depth, the lidar constants, the system backgrounds, dead times, and Ångstrom exponent to be retrieved. Retrievals have been discussed for three different cases. In summary, the application of the OEM to the retrieval of water vapor mixing ratio showed that
- • The forward model presented contains the essential physics to reproduce the analog and digital measurements leading to unbiased residuals and robust estimates of water vapor mixing ratio.
- • The retrieved mixing ratios agree well with both the traditional analysis and coincident radiosonde measurements.
- • The OEM provides a cutoff height for the retrievals. Below this height the entire profile can be used.
- • Vertical resolution is determined directly at each height, and is adapted in an optimal way accounting for measurement noise.
- • The OEM provides a complete uncertainty budget, including random and systematic uncertainties due to model parameters, including the assumed air density profile and the lidar calibration factor.
- • Simultaneous analog and digital measurements allow the dead time to be retrieved. No conversion of the analog signal into current or virtual photocounts is required.
- • If the lidar’s geometric overlap is sufficiently well determined, aerosol optical depth profile can be retrieved in addition to mixing ratio. If the overlap can be well parameterized, it should be possible to retrieve the overlap function’s parameters.
- • OEMs are computationally fast and practical for routine calculations.
We have demonstrated that the OEM allows retrieval of water vapor mixing ratio from vibrational Raman lidar measurements that are consistent with the traditional method of dividing the photocounts, but with the many additional advantages discussed here. As was the case in our previous OEM lidar paper, the parameter variances chosen for this study are sometimes “best guesses.” The selection of covariances for the model parameters should ultimately be by consensus of the lidar community, perhaps initiated through the Network for Detection of Atmospheric Climate Change (NDACC) Lidar Working Group.
The water vapor OEM retrieval presented here is not yet complete. The next step is to incorporate additional measurements that would allow the lidar calibration factor to be retrieved. If coincident radiosonde flights are available, the retrieval could include the radiosonde measurements combined with an appropriate forward model to describe them, which would allow the retrieval of the lidar calibration factor. If coincident microwave radiometer measurements are available, the brightness temperatures could be included in the forward model for the radiometer and the lidar calibration factor then retrieved. Improvements to the aerosol retrievals will be realized by including the lidar’s elastic channels in the retrievals. These extensions, as well as further inclusions of other measurement sets, point to a future where the combined analysis of multiple sensors for geophysical quantities will be commonplace.
National Science and Engineering Research Council of Canada.
We thank E. Maillard Barras, B. Calpini, M. Hervo, G. Martucci, D. Ruffieux and V. Simeonov for interesting discussions and assistance that enabled us to get these results. We would also like to thank A. Povey and D. Whiteman for critically reading the paper and offering helpful comments and suggestions, as well as the anonymous reviewers. This project has been funded in part by the National Science and Engineering Research Council of Canada.
1. J. Holton, P. Haynes, M. McIntyre, A. Douglass, R. Rood, and L. Pfister, “Stratosphere-troposphere exchange,” Rev. Geophys. 33, 403–439 (1995). [CrossRef]
2. P. M. F. Forster and K. P. Shine, “Assessing the climate impact of trends in stratospheric water vapor,” Geophys. Res. Lett. 29, 10–11 (2002). [CrossRef]
3. D. Kley, J. M. Russell, and C. Phillips, “SPARC assessment of upper tropospheric and stratospheric water vapour,” WMO Report 1043 (2000).
4. D. N. Whiteman, K. C. Vermeesch, L. D. Oman, and E. C. Weatherhead, “The relative importance of random error and observation frequency in detecting trends in upper tropospheric water vapor,” J. Geophys. Res. 116, D21118 (2011). [CrossRef]
5. S. H. Melfi, J. D. Lawrence, and M. P. McCormick, “Observation of Raman scattering by water vapor in the atmosphere,” Appl. Phys. Lett. 15, 295–297 (1969). [CrossRef]
6. T. Leblanc, “Accuracy of Raman lidar water vapor calibration and its applicability to long-term measurements,” Appl. Opt. 47, 5592–5603 (2008). [CrossRef]
7. R. J. Sica, S. Sargoytchev, P. S. Argall, E. F. Borra, L. Girard, C. T. Sparrow, and S. Flatt, “Lidar measurements taken with a large-aperture liquid mirror 1. Rayleigh-scatter system,” Appl. Opt. 34, 6925–6936 (1995). [CrossRef]
8. D. P. Donovan, J. A. Whiteway, and A. I. Carswell, “Correction for nonlinear photon-counting effects in lidar systems,” Appl. Opt. 32, 6742–6753 (1993). [CrossRef]
9. K. Stelmaszczyk, M. Dell’Aglio, S. Chudzynski, T. Stacewicz, and L. Wöste, “Analytical function for lidar geometrical compression form-factor calculations,” Appl. Opt. 44, 1323–1331 (2005). [CrossRef]
10. A. C. Povey, R. G. Grainger, D. M. Peters, J. L. Agnew, and D. Rees, “Estimation of a lidar’s overlap function and its calibration by nonlinear regression,” Appl. Opt. 51, 5130–5143 (2012). [CrossRef]
11. T. Dinoev, V. Simeonov, Y. Arshinov, S. Bobrovnikov, P. Ristori, B. Calpini, M. Parlange, and H. van den Bergh, “Raman lidar for meteorological observations, RALMO–part 1: instrument description,” Atmos. Meas. Tech. 6, 1329–1346 (2013). [CrossRef]
12. S. H. Melfi, “Remote measurements of the atmosphere using Raman scattering,” Appl. Opt. 11, 1605–1610 (1972). [CrossRef]
13. D. N. Whiteman, S. H. Melfi, and R. A. Ferrare, “Raman lidar system for the measurement of water-vapor and aerosols in the earth’s atmosphere,” Appl. Opt. 31, 3068–3082 (1992). [CrossRef]
14. T. Leblanc, T. Trickl, and H. Vogelmann, “Part II: remote sensing sensors: lidar,” in Monitoring Atmospheric Water Vapour: Ground-based Remote Sensing and In-situ Methods, N. Kämpfer, ed. (Springer, 2012), pp. 113–158.
15. D. Petty and D. Turner, “Combined analog-to-digital and photon counting detection utilized for continous Raman lidar measurements,” in Proceedings of the 23rd International Laser Radar Conference, Nara, Japan (2006).
16. D. Whiteman, B. Demoz, P. Di Girolamo, J. Comer, I. Veselovskii, K. Evans, Z. Wang, M. Cadirola, K. Rush, G. Schwemmer, B. Gentry, S. H. Melfi, B. Mielke, D. Venable, and T. Van Hove, “Raman lidar measurements during the international H2O project. Part I: instrumentation and analysis techniques,” J. Atmos. Ocean. Technol. 23, 157–169 (2006). [CrossRef]
17. R. K. Newsom, D. D. Turner, B. Mielke, M. Clayton, R. Ferrare, and C. Sivaraman, “Simultaneous analog and photon counting detection for Raman lidar,” Appl. Opt. 48, 3903–3914 (2009). [CrossRef]
18. D. Whiteman, B. Demoz, P. Di Girolamo, J. Comer, I. Veselovskii, K. Evans, Z. Wang, D. Sabatino, G. Schwemmer, B. Gentry, R. Lin, A. Behrendt, E. Browell, R. Ferrare, S. Ismail, and J. Wang, “Raman lidar measurements during the international H2O project. Part II: case studies,” J. Atmos. Ocean. Technol. 23, 170–183 (2006). [CrossRef]
19. D. D. Venable, D. N. Whiteman, M. N. Calhoun, A. O. Dirisu, R. M. Connell, and E. Landulfo, “Lamp mapping technique for independent determination of the water vapor mixing ratio calibration factor for a Raman lidar system,” Appl. Opt. 50, 4622–4632 (2011). [CrossRef]
20. D. N. Whiteman, “Examination of the traditional Raman lidar technique. II. Evaluating the ratios for water vapor and aerosols,” Appl. Opt. 42, 2593–2608 (2003). [CrossRef]
21. A. Moss, R. J. Sica, E. McCullough, K. Strawbridge, and J. Drummond, “Calibration and validation of water vapor lidar measurements from Eureka, Nunavut, using radiosondes and the atmospheric chemistry experiment Fourier transform spectrometer,” Atmos. Meas. Tech. 6, 741–749 (2013). [CrossRef]
22. U. Löhnert, S. Crewell, and C. Simmer, “An integrated approach toward retrieving physically consistent profiles of temperature, humidity, and cloud liquid water,” J. Appl. Meteorol. 43, 1295–1307 (2004).
23. M. Schneebeli, “Advancements in ground-based microwave remote sensing of the troposphere: calibration, data retrieval and applications,” Ph.D. thesis (University of Bern, 2009).
24. A. Foth, H. Baars, P. Di Girolamo, and B. Pospichal, “Water vapor profiles from Raman lidar automatically calibrated by microwave radiometer data during HOPE,” Atmos. Chem. Phys. 15, 7753–7763 (2015). [CrossRef]
25. A. C. Povey, R. G. Grainger, D. M. Peters, and J. L. Agnew, “Retrieval of aerosol backscatter, extinction, and lidar ratio from Raman lidar with optimal estimation,” Atmos. Meas. Tech. 7, 757–776 (2014). [CrossRef]
26. R. J. Sica and A. Haefele, “Retrieval of temperature from a multiple-channel Rayleigh-scatter lidar using an optimal estimation method,” Appl. Opt. 54, 1872–1889 (2015). [CrossRef]
27. C. D. Rodgers, Inverse Methods for Atmospheric Sounding: Theory and Practice (World Scientific, 2011).
28. P. Eriksson, C. Jiménez, and S. A. Buehler, “Qpack, a general tool for instrument simulation and retrieval work,” J. Quant. Spectrosc. Radiat. Transfer 91, 47-–64 (2005). [CrossRef]
29. “U.S. Standard Atmosphere, 1976,” Tech. Rep. NOAA-S/T 76-1562 (Committee on Extension to the Standard Atmosphere, National Oceanic and Atmospheric Administration, 1976).
30. M. Nicolet, “On the molecular scattering in the terrestrial atmosphere: an empirical formula for its calculation in the homosphere,” Planet. Space Sci. 32, 1467–1468 (1984). [CrossRef]
31. T. S. Dinoev, V. B. Simeonov, B. Calpini, and M. Parlange, “Monitoring of Eyjafjallajökull ash layer evolution over payerne Switzerland with a Raman lidar,” in Proceedings of the TECO, Helsinki, Finland, 2010).
32. A. Ansmann and D. Müller, “Lidar and atmospheric aerosol particles,” in Lidar, C. Weitkamp, ed., Vol. 102 of Springer Series in Optical Sciences (Springer, 2005), pp. 105–141.
33. E. Brocard, R. Philipona, A. Haefele, G. Romanens, A. Mueller, D. Ruffieux, V. Simeonov, and B. Calpini, “Raman lidar for meteorological observations, RALMO–part 2: validation of water vapor measurements,” Atmos. Meas. Tech. 6, 1347–1358 (2013). [CrossRef]
34. D. N. Whiteman, M. Cadirola, D. Venable, M. Calhoun, L. Miloshevich, K. Vermeesch, L. Twigg, A. Dirisu, D. Hurst, E. Hall, A. Jordan, and H. Vömel, “Correction technique for Raman water vapor lidar signal-dependent bias and suitability for water vapor trend monitoring in the upper troposphere,” Atmos. Meas. Tech. 5, 2893–2916 (2012). [CrossRef]
35. T. von Clarmann, “Smoothing error pitfalls,” Atmos. Meas. Tech. 7, 3023–3034 (2014). [CrossRef]
36. D. N. Whiteman, “Examination of the traditional Raman lidar technique. 1. Evaluating the temperature-dependent lidar equations,” Appl. Opt. 42, 2571–2592 (2003). [CrossRef]
37. “Systematic observation requirements for satellite-based data products for climate,” Tech. Rep. GCOS-154 (World Meteorological Organization, 2011).