Dr Ross Bannister

Senior Research Scientist (NCEO)

We need to talk about assimilation …

Modelling the real world is never a perfect process. Errors and uncertainties in all models accumulate in time, consequently placing limits on the value of any forecast, whether it represents a prediction of the weather, or of flooding, etc. It is possible though to achieve good quality forecasts by correcting the model as it evolves with information from fresh observations, where and when they are available. In the case of weather forecasting, such observations constrain the model to “today’s weather”, so that “tomorrow’s weather” can be forecast as accurately as possible.

Merging observations with models is called data assimilation (or DA for short). For weather forecasting, DA helps to determine initial conditions of a numerical weather prediction model, whose output is used by weather forecasters, or used as input into other models that predict flooding depending upon the expected rainfall. The practical application of DA takes a forecast field valid for today and computes a sort of correction field based on the observations. The modified field, is generally a more realistic model state. The correction fields are determined from an algorithm, which uses not only today’s forecast and observations, but also information about how likely these pieces of information are correct (remember the model forecast is erroneous, and even observations are never exact measures of reality). This algorithm is based as closely as possible on a set of equations called the Kalman Filter equations, which were originally developed for engineering applications in the 1950s and 60s. The Kalman Filter equations have a wide range of uses, e.g. the control of trajectories of bodies over large distances (think of space flight and warfare), but have since been adapted for weather forecasting. The weather forecasting problem though has vastly more degrees of freedom than the basic Kalman Filter can cope with so the full Kalman Filter equations are not used; instead approximate equations are used which are solved using either variational procedures, or using a vastly reduced number of variables (e.g. the ensemble Kalman Filter).

Today’s forecast is our very best estimate of today’s weather … probably not

Standard DA methods assume that errors in forecasts and observations obey a normal distribution (otherwise known as a Gaussian distribution). Consider a forecast of today’s relative humidity (RH) above, say London, of about 98% RH (i.e. close to saturation), and let the error (standard deviation) of this forecast be of order 2.5% RH. The Gaussian distribution with this standard deviation is shown in Fig. 1(a). One can interpret this plot by imagining that one has access to a very large number of forecasts, and the distribution is a means of showing the frequency of possible forecast outcomes. Knowing that about 2/3 of the area of a Gaussian distribution is within one standard deviation of the mean, then 2/3 of the forecasts would have values of 98 ± 2.5 % RH. In the absence of multiple forecasts, we assume that the single forecast represents the most likely point of this distribution (the mean, or centre point), and observations can make adjustments to this forecast (examples are the arrows in Fig. 1(a)). The assumed Gaussian specifies how the relative humidity forecast is allowed to be modified by the observations. If an observation suggests that the forecast should really be drier by about 6% (purple arrow), then the distribution would inhibit this change as it has such a small probability. An equally large positive change also has the same degree of unlikeliness (red arrow). Smaller changes though – of either sign – are deemed far more likely (blue and green arrows), and so such modifications would be more likely to happen. Specification of this distribution is how DA can control how observations can update the forecast.

Figure 1: Possible probability distributions of errors in a forecast of relative humidity when the forecast value is 98% RH. Panel (a) has an assumed Gaussian form of this distribution, and panel b has a non-Gaussian form. The arrows in panel a serve to illustrate likely (blue and green), and unlikely (purple and red) changes to the forecast value as a result of assimilating observations. The Gaussian has identical probabilities for positive and negative updates, but the particular non-Gaussian shown has larger probabilities of updating negatively than positively.

Actual distributions are rarely Gaussian shaped. Shown in Fig. 1(b) is a distribution whose shape is far more realistic for a forecast close to saturation (surprisingly this distribution has the same standard deviation as the Gaussian in Fig. 1(a)). This non-Gaussian distribution is asymmetric – it says that there is a higher probability of an observation lowering the relative humidity (negative correction) than raising it. This makes sense given the forecast is at 98% RH – it is more more likely that the true atmosphere has 95% RH (3% RH lower than the forecast), than super-saturated at 101% RH (3% RH higher). There is a similar picture for forecasts of very dry air (e.g. 2% RH), where qualitatively the distribution would be the mirror image of that in Fig. 1(b). These represent examples where the distributions are not only non-Gaussian, but are also strongly flow-dependent, which makes them difficult to specify in operational situations.

In reality the picture is even more complicated than this, as the distribution needs to account for relationships between different variables (e.g. between different positions, and between different kinds of fields like temperature, and winds). There is also the issue of corrections being made that involve moisture phase changes (between ice, water, and vapour), between which there are also potentially strongly flow-dependent relationships. Also generally in weather forecasting problems, most observations made are not of the model’s own variables, but of something related to them, which represents another complication in the DA problem as a whole.

Why is this important?

The aim of doing data assimilation (DA) is to set realistic initial conditions for models that will deliver more accurate forecasts of the future weather than if the model was not updated by the latest observations. Such forecasts include accurate predictions of rainfall for use in models that predict flooding. Some off-line results that we did in the FRANC project suggest that if non-Gaussianity of the forecast distribution is not well accounted for, it is possible for DA to actually give worse initial conditions – worse by the fact that the initial conditions can be unphysical (for instance represent negative, or supersaturated humidities).

A pragmatic solution might be to simply set negative humidities to zero, and supersaturated humidities to 100% RH after the assimilation step, and before the model is run. This `cure’ though could have side-effects, since such adjustments do not necessarily obey the subtle balances that might be at play, e.g., between humidity and temperature. As always prevention is better than cure, and in this case, we believe that prevention may well be to do with improving the assimilation.

Dr Rob Thompson, University of Reading

11October2017

A summer of Floods

We’ve entered autumn (or maybe winter given how it feels!), so its definitely time to look back on the summer we’ve had (or not as it feels). This summer has certainly seemed to have been quite convective, without being warm after June.

This article though will focus on the day of the most high profile UK event, on July 18^th. This day the headlines were floods in Coverack, Cornwall… but they were far from alone, my home of Reading also got hit by a really very intense storm, and one that produced a huge amount of rain and statistics of note. Coverack suffered as a consequence of a very intense thunderstorm, combined with local orography leading to a lot of water pouring through the village. People had to be airlifted to safety and a huge amount of damage was done when flood waters flowing through the village exceeded a metre in depth. 

Coverack’s case is a fine example of the current state of forecasting… the morning’s weather forecast for the area that day was

“Thundery showers are expected to push north across southern parts of the UK through Tuesday evening. Although many places won’t see these showers, there is a chance of localised flooding of homes, businesses and susceptible roads. Frequent lightning may be an additional hazard with possible disruption to power networks. Similarly, but very locally, hail may cause impacts.”

And I can’t argue that was anything but a good forecast, the thunderstorms certainly materialised and were very intense, but small and hence very localised, perfect to demonstrate a problem for the scientists involved in research in to events such as this, observations are sparse. None of the weather stations in the area received a significant amount of rainfall. This is where radar comes in, radar provides areal coverage over a large range (250km from the radar in the UK operational system), with the UK network having coverage over the whole country, almost all with 1km resolution, we can see the rainfall falling even without the presence of rain gauges.

We can see from the radar images that the intense rainfall in the area of Coverack lasted for a few hours, storms triggering to maintain the rainfall rates for several hours.

But we can also see what was happening further East… that last radar image at 1900 shows a large convective system over the New Forest, heading towards Reading. The next few hours would be interesting for certain.

Where we do have numbers and gauge data is our own University of Reading atmospheric observatory. The rain in Reading was phenomenal, and I personally had the misfortune of being out when it occurred. It was like rain I’d previously only experienced in the tropics, driving conditions were horrendous, with incredibly reduced visibility and water simply unable to clear the roads quickly enough. Little appeared in media about the storm, yet there were certainly local impacts from flooding, as evidenced by this photo I took myself in South Reading:

The rain was of most interest to me, but the lightning was also impressive, both sheet and fork lightning. More than 100,000 strikes over the UK.

So I’ll focus on the rainfall rates, very high rainfall rates are not that uncommon, but lasting more than a few minutes is very unusual – and this storm was very, very unusual. The university’s tipping bucket rain gauge got 38.6 mm in the storm (which was later closely matched by the manual gauge), and 1.6 mm in the little shower 20 minutes before the big storm. The gauge experienced 35 mm of that total in 45 minutes, a phenomenal amount, averaging 47 mm/hr for three quarters of an hour. The University of Reading campus gets approximately 640 mm per year on average, so we had 5.5% of the annual rain in 45 minutes – that’s pretty incredible.

Events like these two are exactly what the FFIR programme is all about, about getting better warnings to the right places and more specifically.