Big data and the weather forecast
The financial potential of enhanced weather prediction through big data is huge. According to Weather Analytics, 33% of worldwide GDP is affected by the weather. For a whole host of business and public sector organisations, weather predictions can make the difference between profit and loss - or even between life and death. It particularly affects industries such as agriculture, tourism and fishing. So whether the issue is ice cream sales or flood defences, the weather matters.
Big weather data in action
Big weather data analysis helps to transform the data generated by organisations into actionable information.
The Dutch government, which oversees an area of land that is for the most part both flat and under sea level, has become an expert at managing water. Key to this is their understanding of the weather, using metrics such as precipitation measurement, radar data, flood levels from previous events and maintenance data from sluices, dams, pumping stations and locks. This in-depth knowledge allows the country to determine the best course of action when high water levels are expected, thereby reducing the severity of floods and water management costs and preventing environmental degradation.
One motor insurance company is using weather data as a component of its points scoring system when evaluating drivers who opt into its telematics programme. This uses GPS technology to gather journey information and then ranks each trip based on factors such as speed, braking and acceleration.
It also takes into account outside variables often affected by weather (for example road and traffic conditions) to determine driver scoring. By accessing current data through a weather company's API and using a cloud-based analytics service, the insurer is able to construct a more accurate and reliable scoring algorithm based on the precise weather conditions at the place and time of the driver's journey. This then enables the insurer to offer policies that are customised according to the policyholder's driving patterns and performance.
Insurance companies are also using big weather data to triage claims related to property- or crop-damaging conditions, and to verify weather-related crop insurance claims. Meanwhile, agricultural commodity traders use it to predict price changes for particular crops in the commodity market. Additionally, airlines use big weather data to predict wind patterns and then map the most fuel-efficient routes for their aircraft. The UK's Met Office estimates that its global aviation forecasts help save around 20 million tonnes of CO2 emissions annually through increased efficiency.
How is it done?
One weather company explains how this data is generated. Weather Analytics says it ingests and analyses current, historical and forecast data from the US Government's weather service from 1979 to seven days in the future. This global data is placed in a 35km by 35km grid, generating some 650,000 outputs at hourly intervals.
The company says: "We then extract pertinent variables and create/calculate numerous others. We store this massive amount of weather information in databases which allows us to: present the data quickly, calculate additional variables, [and] uniquely package the data."
Clearly, a vast volume of data is involved. Supercomputers, such as those at the UK's Met Office, help meet the demand for speedy processing of vast quantities of data, the volumes of which can only grow as the IoT expands to include sensors on motor vehicles and other outdoor data sources.
For example, the Met Office's 140 tonne supercomputer has 480,000 cores, two million GB of memory and 17,000TB of storage, and performs over 23,000 trillion operations per second. This processing power enhances the resolution of data models, improving the speed of results and enabling weather forecasters to use a smaller grid. The Met Office says that today's four-day forecasts are as accurate as one-day forecasts were 30 years ago.
Conclusion
The volume of data being generated is set to expand massively as the amount of sensors on objects, both moving and static, increases. Big data techniques are essential for analysing that data and for enabling organisations to transform it into meaningful decisions.
The aim of this level of processing is to enhance organisational, agricultural and transport efficiency, improve public safety and better carbon dioxide emissions. Big weather data is at the heart of all our futures, personal and corporate.