High Performance Weather Computing in the Cloud

Consider the volume of data required to create one forecast for one area for one point in time, and it is easy to see why weather is the original “big data.” The ingestion and interpretation of hundreds of thousands of inputs, from surface to sky and across the globe, is a vast amount of information to be processed and modeled, and those inputs change frequently. Engineers at DTN estimate that its global forecast engine processes petabytes of data per day. That complexity has historically limited how often forecast engines can update, which in turn has placed some limitations on how organizations around the world can rely on weather-informed intelligence to make business decisions.

The scalable capacity of cloud computing has changed that. DTN has moved its global forecast engine to the AWS cloud to improve the insights and data that drive its market-leading, weather-based solutions for industries all over the world. “The use of our elastic and scalable cloud infrastructure allows DTN to manage rapidly enough – on demand,” said Chris Wellise, head of sustainability at Amazon Web Services (AWS) in a recent Forbes article.

 

Delivering more timely weather forecasts

DTN began testing the high-performance computing (HPC) capabilities of AWS in the fall of 2020, running data processing and modeling workloads on Amazon Elastic Compute Cloud (Amazon EC2), a service that provides secure, resizable compute capacity in the cloud. While most global weather forecasting organizations run models twice daily, “the intent was to try and increase the frequency of forecast modeling to provide our supply chain customers with intelligence that better reflects how changing weather could impact their operations,” said Doug Chenevert, director of the forecast platform at DTN.
“Because weather changes rapidly, a system that can ingest data quickly and run our models frequently is critical for delivering operational intelligence.”
– Doug Chenevert, DTN

The DTN team constantly works to optimize the weather science workflows, while the weather technology stack on AWS allows for an optimized HPC infrastructure, which has led to improvements across the company’s weather modeling technology stack.

DTN can now reliably, accurately, and consistently generate high-resolution weather forecasts at twice the industry standard. With layers of machine learning and AI now being included, along with industry-specific data in the output from DTN, the “in the moment” intelligence needed for decisioning is more powerful than ever.

 

Delivering operational intelligence for weather-dependent industries

With more frequent model output, DTN can generate more timely and valuable insights for organizations that depend on weather and industry-specific insights for safe and sustainable operations. With quadrillions of processes per second, speed is a critical requirement for blending weather and industry-specific data into the kind of intelligence required for planning and in-the-moment decisions.

 

Ship routing

Weather-optimized routing for maritime shipping can create millions of dollars in efficiency and provide timely options for ship operators to reduce emissions. Ocean-going ships are a disproportionately large part of the global supply chain, carrying nearly 90% of international trade. In 2020, Statista estimated that the volume of global seaborne trade was around 11 billion tons. All that cargo moving all around the world means the data needed to best manage the movement of goods via ship is complex. Ship operators balance the safest, fastest, most time- and fuel-efficient route for each vessel throughout each voyage. DTN weather data, modeled for shipping also includes shipping-specific, ocean, maritime route, and port data. That data underpins the optimization of vessel performance and sailing decisions, including route optimization, start and stop points, and waypoints that factor in oceanic and weather conditions, as well as requests from captains and even piracy warnings. With cloud-based, high-capacity computing and proprietary models, DTN data engineers create outputs that deliver the intelligence used by the global shipping industry to simulate, estimate, and make decisions on the critical ocean-going parts of the global supply chain.

 

Road Management

Changing seasonal highs and lows are creating new challenges for land-based transportation infrastructure as well. According to the U.S. Department of Transportation (DOT), every year, trucking companies lose 6 billion vehicle hours to weather-related incidents, and the U.S. Environmental Protection Agency (EPA) predicts that climate changes will likely further impact roadways, ice roads, vehicles, and railways. The ability to gather and model historical and real-time weather data is critical for DOTs to plan and adapt to climate change.

While road systems have weather stations and road sensors that provide weather data, they only give a portion of the critical data needed and often have data gaps. DTN uses a route-based forecast, which incorporates dynamic weather mapping with environmental data, propriety weather models, and radar and satellite images, in addition to live weather observations to ensure that the forecasts are the most up to date during changing conditions. This huge set of inputs is ingested and modeled in the cloud to enable DTN to provide location-specific alerts — even within a mile along a specific route — so that road managers can make precise decisions on when and where to treat roadways. These complex data sets require large amounts of processing power, constantly updating with the latest intelligence to ensure public safety and efficient movement of freight and the traveling public.

 

Electric utilities

Power companies also require timely intelligence, and DTN weather data feeds the Storm Risk suite, a set of proprietary models and machine learning applications that help electric utilities more accurately predict the power outages and risk to critical infrastructure that a given weather event might create. “You simply cannot have outdated weather intelligence when dealing with severe storms; decisions are being made minute-by-minute, and the best available data has to be there,” commented Chenevert. “The only way to get that kind of speed is with a scalable cloud infrastructure that ramps up as we need to deliver those timely, actionable insights for specific storms.”

By blending historical, current, and geographical data to help utilities protect their grids, emergency response managers at public utilities can better understand how to mitigate risk, better preparing for potential surges in use from extreme temperatures and storms. With timely intelligence, it is easier for a utility to estimate time to restore power and evaluate options to safely stage response crews for efficient restoration response and reporting.

 

Futuristic weather forecasting is firmly in the cloud

Focused on innovation, the DTN team continues to push the envelope of weather-optimized computing capacity in order to best serve its customers. When combining data sets and rendering sophisticated models that are starting to support machine learning and AI, DTN needs quadrillions of processes per second. Additional testing with the Amazon EC2 Hpc6a has shown the potential to further compress rendering time. “Ideally, we want to render high-resolution global forecasts hourly,” said Chenevert. “That kind of output is uncharted territory for weather forecasting, but we’re nearly there.”