Computer models utterly fail to predict climate changes in regions.
A few years ago a biologist I know looked at how climate change might affect the spread of a particular invasive insect species. He obtained climate-model projections for North America under standard greenhouse-gas scenarios from two modelling labs, and then tried to characterize how the insect habitat might change. To his surprise, he found very different results depending on which model was used. Even though both models were using the same input data, they made opposite predictions about regional climate patterns in North America.
This reminded me of a presentation I’d seen years earlier about predicted changes in U.S. rainfall patterns under global warming. The two models being used for a government report again made diametrically opposite predictions. In region after region, if one model predicted a tendency toward more flooding, the other tended to predict drying.
Just how good are climate models at predicting regional patterns of climate change? I had occasion to survey this literature as part of a recently completed research project on the subject. The simple summary is that, with few exceptions, climate models not only fail to do better than random numbers, in some cases they are actually worse.
There are two reasons why this is important. First, it tells us something about our lack of understanding of the climate. There are various different theories to explain the rising trend in the global average temperature over the past century. Climate models embed one such theory, based on a relatively high sensitivity to greenhouse gases and strong amplifying effects from a positive water-vapour feedback, and relative insensitivity to other things. In this setup, the only way to get a climate model to mimic the 20th-century average warming is to feed in the observed increase in greenhouse gases. Therefore, the argument goes, greenhouse gases are to blame.
But this kind of argument could be used to support other theories too, if the models are set up just so. To say which theory, if any, is right, we need to look at the spatial patterns. Different theories make different predictions about where the warming should be taking place, a detail that gets missed if we only look at the global average. A valid model should not only get the global trend right, but also the spatial pattern of change.
Second, when policymakers and scientists think about climate change, they are usually not interested in abstract global averages but in potential changes where people actually live, namely at the local level. To say anything meaningful about this requires models that make valid regional predictions.
We already had a clue that something is wrong with spatial details in climate models. Due to the water-vapour feedback, models predict rapid, amplified warming in the troposphere over the tropics. But data collected by weather balloons and satellites fail to show this, and the discrepancy between models and observations is statistically significant.
So how do models do at predicting the spatial pattern of warming over land? Though the 2007 report of the Intergovernmental Panel on Climate Change (IPCC) devoted a whole chapter to model evaluation, it said almost nothing about this question. The IPCC talked mainly about static features, such as whether the model can make the tropics hot and poles cold, and so forth. But it was mostly silent on the spatial changes. A 2008 report of the U.S. Climate Change Science Program went a bit deeper, but only to report on tests of how daily and seasonal variations in models matched the real world (is winter a suitable amount colder than summer, etc.).
The reports weren’t ignoring anything: There just hasn’t been much work on the topic. Over a decade ago one team wrote an editorial in the journal Climatic Change lamenting that, on the few occasions people checked the spatial trend pattern, there was a tendency to use what they called “eyeball assessments”: putting colour plots side by side and declaring that they look similar. More recently, one team actually computed some test statistics, but they set the test up so that a region only failed if models and observations significantly disagreed. That’s a weak test, since lists of random numbers wouldn’t fail it.
Then in 2008 and 2010, a team of hydrologists at the National Technical University of Athens published a pair of studies comparing long-term (100-year) temperature and precipitation trends in a total of 55 locations around the world to model projections. The models performed quite poorly at the annual level, which was not surprising. What was more surprising was that they also did poorly even when averaged up to the 30-year scale, which is typically assumed to be the level they work best at. They also did no better over larger and larger regional scales. The authors concluded that there is no basis for the claim that climate models are well-suited for long-term predictions over large regions.
A 2011 study in the Journal of Forecasting took the same data set and compared model predictions against a “random walk” alternative, consisting simply of using the last period’s value in each location as the forecast for the next period’s value in that location. The test measures the sum of errors relative to the random walk. A perfect model gets a score of zero, meaning it made no errors. A model that does no better than a random walk gets a score of 1. A model receiving a score above 1 did worse than uninformed guesses. Simple statistical forecast models that have no climatology or physics in them typically got scores between 0.8 and 1, indicating slight improvements on the random walk, though in some cases their scores went as high as 1.8.
The climate models, by contrast, got scores ranging from 2.4 to 3.7, indicating a total failure to provide valid forecast information at the regional level, even on long time scales. The authors commented: “This implies that the current [climate] models are ill-suited to localized decadal predictions, even though they are used as inputs for policymaking.”
Indeed. Nor is the problem confined just to a few models. In a 2010 paper, a co-author and I looked at how well an average formed from all 23 climate models used for the 2007 IPCC report did at explaining the spatial pattern of temperature trends on land after 1979, compared with a rival model that all the experts keep telling me should have no explanatory power at all: the regional pattern of socioeconomic growth. Any effects from those factors, I have been told many times, are removed from the climate data before it is published. And yet I keep finding the socioeconomic patterns do a very good job of explaining the patterns of temperature trends over land. In our 2010 paper we showed that the climate models, averaged together, do very poorly, while the socioeconomic data does quite well.
Perhaps the problem is that the models should not be averaged together, but should be examined one by one and then in every possible combination, with and without the socioeconomic data, in case some model somewhere has some explanatory power under just the right testing scenario. That is what another coauthor and I looked at in the recently completed study I mentioned above. It will be published shortly in a high-quality climatology journal, and I will be writing about our findings in more detail. There will be no surprises for those who have followed the discussion to this point.
Ross McKitrick is a professor of economics at the University of Guelph and an expert reviewer for the Intergovernmental Panel on Climate Change. Citations available at rossmckitrick.com.
Next week: Climate models offer millions of ways of getting the wrong answer.