The Importance Of Verification: How Accurate Is Your Forecast?

In the weather industry, there is a lot of buzz around data resolution, usually in the context of numerical weather prediction models that provide the most important input to the weather forecast process. The implication is that higher-resolution data is more accurate data. However, resolution is actually a measure of precision, which is not equivalent to accuracy. Accuracy is a measure of how close a prediction is to the target, and precision is more about the granularity of the predictions. It is quite possible to be very precise and not be accurate, and vice versa. If you are going to shoot bullets rather than cannonballs, your aim must be better.

Is it accurate?

Unfortunately, the implication of better accuracy purely due to having a new high-resolution model are not readily or independently substantiated. It’s one thing to claim it, and another to prove it. As we Missourians say, “Show me!” The process of assessing accuracy is what meteorologists call forecast “verification”. It involves saving each forecast when it is issued, and then comparing observed weather conditions at future times to see how well they matched. And while we would like to be able to boil that down to a single number, it just isn’t that simple. The metrics used are dependent upon the parameter being measured and what is important. Furthermore, while individual weather events can provide interesting cases to study, a good verification program is based on large sample sizes – thousands to millions of forecast/observation pairs over seasons or years at many locations. Showcasing a single event as an example of forecast superiority is insufficient. As the saying goes, even a blind chicken finds an occasional kernel of corn now and then.

Two-fold verification

At DTN, we use a combination of internal and independent verification services. Internally, every model coming into our ensemble forecasting system is verified against analyzed conditions to provide a near-term running history of bias and error for key parameters so they can be optimally weighted. We also frequently perform studies when contemplating enhancements to understand the impact of those changes.

Externally, our daily weather forecasts have been routinely collected and verified by Intellovations’ ForecastWatch service since 2016. Using their online tools, our meteorological scientific staff are able to slice and dice by region, parameter, season, forecast lead time and more to see how we stack up against a plethora of other forecast providers in a true apples-to-apples comparison. Their independent validation assures our clients that we are a leader in environmental forecasts that underpin all of our services, while helping our scientists prioritize research and development efforts for continuous improvement. If you are a customer and would like to view results, please contact your account manager and we would be happy to share.

The Bottom Line

The next time you see some buzz about the latest hyperlocalized weather service, you should also look for the proof that it isn’t just prettier or more precise, but that it is actually more accurate and brings substantial value to your decisions.