Statistics can be tricky

And for more than one reason.

Sometimes it’s because the math is hard. As the science of statistics has advanced, and computers have made it practical to use ever-more-sophisticated methods, the math has gotten more sophisticated, which often means just plain harder. It’s getting more and more difficult to stay current with the latest developments. But it’s worth the effort; as the methods have gotten more complicated, they’ve also gotten better.

But sometimes it’s for an age-old reason, which few lay people and not enough scientists (even not enough statisticians) fully appreciate, that you have to understand what your data are doing, and why. Often, you really need to feel it in your gut.

Back in 1998 Patrick Michaels et al. (Clim. Res., 10, 27-33) published an analysis of temperature variability. I know enough about Michaels that I don’t trust him as far as I can spit, even after I’m dead. But that’s not related to the problem with this paper; the problem is a failure to understand what the data are doing, and why.

Their approach was simple: take temperature anomaly, which they refer to as “IPCC” data (it’s probably HadCRU data, but that’s not relevant to the topic at hand). Then compute the amount of variation it shows during small time spans (for the case to be illustrated, one year). This generates a time series of variability as a function of year, one which can be studied for trends. Here in fact is their description of their analysis of monthly average temperatures:

“We first looked at the variability of the monthly temperature anomalies within each year using the IPCC monthly gridcell temperature anomaly dataset. For each gridcell in this dataset which contained a complete set of 12 monthly anomalies in any year, we simply calculated the intra-annual variance in the temperature anomalies for that year as:”

They can then take the individual time series for each grid cell to compute a grand average time series. For the monthly anomalies they describe the result thus:

“For the period 1947 to 1996, we produced a single areally-weighted average variance term for each year using the 1041 cells with nearly continuous data. This produced a time-series of intra-annual variance levels from 1947 to 1996 over a large spatial scale (Fig. 3). The trend in this time-series is towards decreasing intra-annual variance of monthly temperature anomalies and is highly significant…”

It’s essentially a correct description of their results, and here’s their graph (their figure 3):

Indeed these data show a decreasing trend. But not for the reason they think; it’s not because of a decrease in intra-annual variability.

If you look closely at the graph, you might notice that their variability measure seems to decrease right around 1960. In fact, to the eye it looks more like a step change (at 1960) than a linear trend. I already know that the two patterns mimic each other very well, so if I only had the data but no idea what it was or how it came about, I doubt I’d proclaim a step change. I’d probably favor the linear hypothesis, based on physics consideration. But I wouldn’t rule out either possibility.

By now some of you might be thinking, “If it’s HadCRU data [which I suspect it is], don’t they use a baseline starting in 1961?” Why, yes they do. It’s the baseline which defines the reference for computing anomaly values.

Anomaly is the difference between a given month’s temperature and the average for that same month during the baseline period. When you subtract that average to convert to anomalies, you’re not necessarily removing “the” annual cycle, you’re removing the average annual cycle during the baseline period.

If the annual cycle itself changes, then subtracting the baseline-period average won’t remove it; it will leave behind a “residue” of the annual cycle, namely, the difference between the then-annual-cycle and the baseline-period annual cycle. That’s why, when I’ve done multiple regression of temperature against such factors as el Nino, volcanic eruptions, and solar variations, I’ve also included an annual cycle — so that if there is a “residue” of the annual cycle due to baseline effects, it can be accounted for in the regression.

I’d bet dollars to donuts that it’s because of the baseline effect that they see different variability before the baseline than they do after.

As I mentioned, I don’t trust Patrick Michaels one little bit. But that’s not the point here. I also see this kind of mistake from scienists I do trust. For example, back in 2012 I criticized Hansen et al. (especially here, but in posts before and after it as well) for making what I believe is a similar (but not identical) mistake. Again, it had to do with the dependence of variability on the baseline period. And make no mistake about it, Jim Hansen is one of those people I do trust.

The real message is that to become a truly good analyst, you have to do more than just learn the math. Anybody can figure out the equations and do the tests, but it takes a good intuition, a good feel for why things might be happening, to notice some of the more subtle effects that can change the result entirely. If I were hiring a statistician, of course I’d want thorough knowledge of standard techniques and how to compute them. But when it came down to selecting the winner out of the finalists, I would disdain the I-know-all-the-latest-techniques guy for the gal who has that innate sense, that “feel” for what’s happening in her gut.

That’s not to say that knowledge of the techniques and methods isn’t necessary — you just can’t get by without them. And it’s not to say that you shouldn’t choose this as your field because you’re worried you might not have that “feel” for things. Unless you’re a super-genius, the intuition doesn’t come from the classroom but from experience; don’t expect to have it fully developed right after graduation, but when you’ve got 10 or 20 years experience under your belt, you’ll know.

And you’ll be glad you’ve got it. After all, statistics can be tricky.