By Dr. Sarah Dance (University of Reading) 19th December 20146

 Many would say that Numerical Weather Prediction has been using “big data” for decades. Routine forecasts are produced using computational models with billions of variables, and tens of millions of observations, several times a day. Most of these observations come from scientifically designed observing networks, such as satellite instruments, weather radar and carefully sited weather stations. However, urban areas also present rich sources of data, that to date have not been fully explored or exploited (e.g., citizen science, smartphones, internet of things etc.), and could provide significant benefits when forecasting on small scales, at low cost.

In surface scientific networks, point observations are often sited away from buildings, in locations that are intended to be more broadly representative of larger areas and not designed to reflect local urban conditions. These observations lend themselves more naturally to comparison with discretized models, whose grid-lengths may be much larger than the size of a building. For datasets of opportunity, a key problem is to understand the effects of the urban environment on the observations so that uncertainties can be properly attributed and proper quality control procedures established. Furthermore there are complex issues surrounding use of the data, such as personal privacy and data ownership that must be overcome.

For the rest of this article we focus one dataset of opportunity arising from air traffic control radar reports. Mode Selective Enhanced Surveillance (Mode-S EHS) is used by Air Traffic Management to retrieve routine reports on an aircraft’s state vector at a high temporal frequency (every 4 to 12 seconds). The state vector consists of true airspeed, magnetic-heading, ground-speed, ground-heading, altitude and Mach number. Mode-S EHS reports can be used to derive estimates of the ambient air temperature and horizontal wind at the aircraft’s location. These derived observations have the potential to give weather information on fine spatial and temporal scales, especially in the vicinity of airports, where there are millions of reports per day. For example high-frequency reporting of vertical profiles of temperature and wind may provide extra information for use in numerical weather prediction that would have particular value in the forecasting of hazardous weather.  While some of the problems of understanding and using datasets of opportunity are circumvented (the effects of buildings are less relevant to flying aircraft), all measurements during aircraft turns and other manoeuvres have to be discarded. Furthermore, the reports are transmitted in small data packets, with limited precision, with the result that the uncertainty in the derived meteorological observations is very large, particularly at lower altitudes. For more information see

Mirza, A. K., Ballard, S. P., Dance, S. L., Maisey, P., Rooney, G. G. and Stone, E. K. (2016), Comparison of aircraft-derived observations with in situ research aircraft measurements. Q.J.R. Meteorol. Soc., 142: 2949–2967. doi:10.1002/qj.2864