PCA part 4: non-centered hockey sticks

In parts 1, 2, and 3 we’ve looked into the basics of PCA. The essence is that we can view multiple data sets as a vector data set, or as points occupying a (sometimes very) high-dimensional space. The original basis for this space is simply the variables with which we express the raw data; we could call this the canonical basis. In this sense, a “basis” is like a coordinate system for the space, and the canonical basis is the set of variables which define the raw data.

But we don’t have to use just that basis; from a purely geometrical point of view any basis will do. For position in space, e.g., we can define a standard set of x, y, z coordinates, but there’s a great deal of freedom in how to choose even those coordinates. We can pick any point as our origin (where x=y=z=0), as well as any directions for the x-, y-, and z-axes. It simplifies many things if the x-, y-, and z-directions are all perpendicular to each other. It also helps if the distances along each direction are expressed in measures of the same size; if we gave the x-coordinate in meters while y and z were in inches, it could lead to a lot of confusion.

Sometimes there are advantages to a particular basis. For example, we might take as “canonical” coordinates for position, x, y, z coordinates in which the origin is the center of the earth, x points in the direction from the center of the earth to where the prime meridian (which defines longitude 0) meets the equator, y points from the center of the earth toward where longitude 90 deg. East meets the equator, and z points from earth’s center toward the north pole. It’s a perfectly good coordinate system, but if what we’re really interested in is studying the movement of a canoe on a lake, we’ll find it not very convenient. A better choice for the canoe problem, would be to choose a reference point on the lake as our origin (maybe the point on the lake surface where we usually keep the canoe), then choose coordinates so that x points east, y points north, and z points up (this can be referred to as the ENZ system, or “east-north-vertical”). We still have 3 coordinates, which is what we need for a 3-dimensional space, but we’ll find that things are a lot simpler using this system than the other.

We might even reap simplification from the coordinate system itself. Unless we sink the canoe, or lift it high out of the water, we’ll notice that our movements all seem to be restricted to the surface of the water, which conveniently corresponds to z=0. If we took a large number of measurements of position, and performed PCA, we’d likely notice that the last (lowest-variation) PC would be simply the up-down coordinate, and that its variation is tiny compared to the others. This would indicate that we might be able to omit that coordinate (use just the first 2 PCs) without any substantive loss of information, and that would make analysis even simpler and easier.

The real essence of PCA is that it enables us to define a new coordinate system for our data space, one based on the observed variations of the data itself, which gives us an excellent chance to reduce complexity, possibly even reveal some insight. PCA by itself doesn’t alter the data at all, and certainly doesn’t “create” patterns that don’t already exist, it simply offers us new coordinates with which to alter our description of it. A classic example (dealt with in part 1) is the case in which our space is 2-dimensional. The PCs (for normalized data) end up being proportional to the sum, and the difference, between the normalized data values. This does not represent a change to the data — instead of giving x and y we give numbers proportional to x+y and x-y, but the positions don’t change. But as easy as it is to transform from canonical coordinates to sum/difference coordinates, it’s just as easy to transform back. No information is lost, we just look at it from a different viewpoint.

One of the real advantages we may reap is to reduce the number of coordinates we have to use without losing important information. The canoe problem illustrates this perfectly: even if we used some bizarre directions for the coordinate system we would still expect the final PC to be the up-down direction, that its variation would be tiny compared to the others, and that it can probably be discarded without losing anything meaningful. This can be especially desirable when estimating something (like temperature) from a very large number of proxy data sets. Sooner or later we’re likely to do some sort of multiple regression, to calibrate the relationship between our proxies and observed temperature measurements. But if we keep all the proxies, we may have so many degrees of freedom in our “predictor” variables that we end up fitting a lot of the noise in the data, rather than just the signal. This is the classic problem of overfitting, a recurrent problem in analysis, nothing to be sneezed at. So, it’s entirely natural that those who wish to use proxy data sets to estimate past temperature (the most famous being the original “hockey stick” of Mann, Bradley, & Hughes 1998, hereafter called MBH) would use PCA.

If we keep all the PCs, then we don’t reduce the problem of overfitting. But we also have zero information loss, and nobody could accuse anyone of having the massaged the data to achieve a particular result. It’s the same data, just in different coordinates! But when we omit all but the leading PCs, then we might be accused of manipulation by way of our choice of which PCs to include and which to omit. Also, if we keep all the PCs it doesn’t matter whether we normalize the data (so we use the correlation matrix) or not (so we use the covariance matrix). But if we omit PCs, then whether we normalize or not can have a profound impact on the variation present in the various PCs, and could affect our decision which to include and which to omit. In fact, we really don’t even have to zero the data! We can perfectly well do PCA based on variation from the origin rather than the average. But again, it will affect what the individual PCs look like and how much variation they contain, and might therefore affect the decision which to include/omit.

Now to the crux of one of the criticisms levelled against MBH by McIntyre & McKitrick (hereafter MM). The criticism which spurred this series on PCA in the first place is that MBH used non-centered PCA (sometimes called “de-centered” PCA) rather than the usual, centered PCA. In part 1 our very first step was to subtract from each data point the average value of the series — this is the centered case. But MBH, instead, chose to subtract the average value during the calibration interval rather than the entire time span. This makes it non-centered PCA. MM claimed that the hockey stick resulted from the use of non-centered PCA, and that non-centered PCA is just plain faulty anyway.

First let’s dispense with the last claim, that non-centered PCA isn’t right. This point was hammered by Wegman, who was recently quoted in reader comments thus:

“The controversy of Mann’s methods lies in that the proxies are centered on the mean of the period 1902-1995, rather than on the whole time period. This mean is, thus, actually decentered low, which will cause it to exhibit a larger variance, giving it preference for being selected as the first principal component. The net effect of this decentering using the proxy data in MBH and MBH99 is to produce a “hockey stick” shape. Centering the mean is a critical factor in using the principal component methodology properly. It is not clear that Mann and associates realized the error in their methodology at the time of publication.”

Just plain wrong. Centering is the usual custom, but other choices are still valid; we can perfectly well define PCs based on variation from any “origin” rather than from the average. It fact it has distinct advantages IF the origin has particular relevance to the issue at hand. You shouldn’t just take my word for it, but you *should* take the word of Ian Jolliffe, one of the world’s foremost experts on PCA, author of a seminal book on the subject. He takes an interesting look at the centering issue in this presentation.

Wegman’s right about this much: a non-standard centering convention will indeed emphasize the variation of the mean, which turns out to be the main difference between the 20th century and the rest of the time span. What he fails to realize is that that’s the point. He’s also right that the changes associated with it will become more prominent in the PCA analysis, causing those changes to “move up” the PCA list. Again, that’s the point.

PCA (centered or not) doesn’t create patterns at all, they have to be there already even to “exhibit a larger variance.” PCA most certainly cannot “create” a statistically significant relationship between that variation and the observed temperature data. That the relationship is significant emerges from both the regression itself, and the testing done on the verification data (more about that in a moment).

What impact will non-centered PCA really have? Downloading all the data used in MBH and reproducing their analysis is a lot of work, and has already been done (by several researchers), so I decided to take the simpler and easier approach of generating artificial data. I created an artificial temperature time series which follows a distinct “hockey stick” pattern, and also generated 40 separate artificial proxy data time series, all spanning the time interval from 1400 to 1980. I assumed that the temperatures from 1900 to 1980 are known, while those before 1900 are not. The proxy data series were generated by computing a response to the temperature signal and other factors as well. And all the series include random noise. The task is to use PCA to characterize the proxy series, use only a few PCs to characterize it, then determine the relationship between the PCs and observed temperature during the “known” time period (i.e., the calibration interval, from 1900 to 1980), then use that relationship to estimate temperature before 1900. Let’s do this using the non-centered approach of MBH, and the centered approach.

In reality, one wouldn’t use all the known data to calibrate the proxies. Instead, one would use part of the known data, then create the prediction, then check prediction against observation in the remaining part. This is “verificiation,” an independent check on whether or not the method is working. We can’t do this if verification uses the same data we used to calibrate — that would only show that if we fit our proxies to the calibration data, then it fits the calibration data. And we already know that! But this is just an exercise using artificial data, so I’ll skip the verification step.

First let’s use the non-centered method. I subtracted from each proxy series its average during the calibration interval. Then I normalized each series so that they’re all about the same “size.” Then comes PCA. When we do so, the very first PC is by far the biggest one of all, and it looks like this:

It looks strikingly like the (artificial) temperature data:

If we do a regression of temperature against PC1 during the calibration interval, we get this:

That’s an excellent fit! If we use the regression line to take the PC1 value and transform it to temperature, we get an outstanding match with actual temperature, not only during the calibration interval but over the entire time span. It appears that the method works excellently, and that PC1 (which dominates the variation) is sufficient to generate it.

Now let’s use the MM approach and see what happens. They claim to reproduce the MBH approach precisely except for the use of non-centered PCA. So this time, we’ll subtract from each proxy series the average value over the entire time span, normalize, and run PCA. This time, the 1st PC looks like this:

Whoa! That doesn’t look at all like the temperature data. If we now regress temperature against PC1 during the calibration interval we get this:

It’s fair to say that this fit isn’t very good. In fact, it’s fair to say (and the numbers confirm) that this fit sucks. If we use it to estimate past temperature, the estimate doesn’t match observation. At all. It shows spurious warming in the late 16th century and cooling in the mid-19th century, but no real 20th century warming and certainly no hockey stick.

What’s going on here? Didn’t we use exactly the MBH procedure, except to switch to centered PCA (which should be valid)? We ran the PCA, kept only the 1st PC, did the regression, and generated our estimate. But it came out horribly wrong! But something is going on — because the “standard” choice (centered PCA) gave us the wrong result while non-centered PCA gave the right answer! For this artificial data we already know the answer: temperature looks like a hockey stick. This is what MM did; they ran (centered) PCA on the actual data, kept only the first 2 PCs (as MBH had done), ran the regression, and generated the estimate. But it came out dramatically different from the MBH estimate.

The difference in estimates emphasizes the reason for MBH’s choice of non-centered PCA. Its purpose is to emphasize differences between the 20th century (the calibration interval, actually) and the remainder of the time span. That main difference is the 20th-century temperature increase, so that becomes the prime focus of the PCA results; that’s why most of the variation is in the 1st PC (artificial data) or first 2 PCs (real data), including the 20th-century increase which enables us to estimate the proper relationship between proxies and temperature. None of these factors are created by the non-centered PCA, but they are emphasized by it, bringing the relevant variation high up the list of PCs. That enables us to use just a few PCs and get the correct result.

With centered PCA, all the relevant variation is still there. It’s all still statistically significant. But the analysis has less emphasis on isolating the 20th-century changes, so the variation which correlates strongly with observed temperature change (which is the variation we really want) is still among the leading PCs, but no longer in the 1st PC (artificial data) or first 2 PCs (real data). Properly to reproduce the MBH procedure, when we switch from non-centered to centered PCA we also have to change the selection of PCs — because we can expect that the important variation may not be so high up the PC list as it was when the method itself (non-centered PCA) is chosen to emphasize it.

So let’s do it right. Let’s use centered PCA, but this time let’s not restrict ourselves to only the 1st PC (artificial data) or first 2 PCs (real data) just because that’s what emerged from non-centered PCA. With the artificial data, this time it’s the first two PCs that contain the bulk of the variation. We’ve already seen the 1st PC, here’s the 2nd:

Lo and behold, there it is! The variation relevant to temperature is still there, it was always there, it’s still among the leading PCs, it always was, but now there are two big PCs instead of just one, and the temperature-related changes are in the 2nd PC. If we regress temperature during the calibration interval against this PC, we get this:

Now that is a good fit! If we use this regression (more properly, a multiple regression on the first two PCs), it turns out that temperature during the calibration interval is strongly (and significantly) related to the 2nd PC, but not to the 1st PC. When we estimate temperature over the entire time span, we get: a hockey stick. It also matches (quite excellently) the actual temperature (which in this case we know, because it’s artificial data).

All of which emphasizes the fact that PCA doesn’t change the data. It only changes our description of it. The variation in proxies which correlates with temperature was always there, PCA only isolates it in a tractable way. If we use centered PCA, it turns out to be in the 2nd PC, but with non-centered PCA it ends up in the 1st PC. That’s the reason for the choice of non-centered PCA: to make the relevant, i.e. temperature-related, variation end up higher in the PC list. And: it worked.

I’d guess that some of you are thinking, “Can it really be that simple? Did MM really not get this? Did they really discard the relevant PCs just to copy the bare number of PCs used by MBH, without realizing that the different centering convention could move the relevant information up or down the PC list?”

You betcha. When done properly on the actual data, using 5 PCs rather than just 2, the hockey stick pattern is still there even with centered PC — which is no surprise, because it’s not an artifact of the analysis method, it’s a pattern in the data. Here’s a comparison of PC#1 for the North American ITRDB (international tree ring data base) data using the MBH method (red), and PC#4 from using the MM method (blue, which has been adjusted to have the same mean and amplitude as the red curve, but it’s only the relative pattern of variation that matters):

Some of you might be thinking, wouldn’t MM have noticed that their fit (the one lacking a hockey stick) failed statistical significance when they did the verification step to check the correctness of their version? Probably yes. Too bad they simply didn’t do the verification step. But independent researchers have replicated the MM procedure and subjected it to verification. It failed.

The MBH hockey stick result has been the target of more scrutiny, criticism, and downright hostility, than perhaps any other aspect of climate science. That may be because there are things to criticize; not necessarily valid ones, but things nonetheless. It may be because it’s one of the things that has had the greatest impact on public perception of global warming, by making the unusual character of modern warming so vivid. It has certainly made Mike Mann one of the most frequent victims of vicious personal attacks (Mike Mann and Jim Hansen seem to be the people most hated by global warming denialists).

The use of non-centered PCA is not the only criticism of the MBH hockey stick levelled by MM. There are several — but as far as I can tell, none of them are any more valid. I have a strong suspicion that they’ll be repeated in reader comments here. Perhaps some day I’ll address those mistakes by MM as well. In the meantime, I have one request: before you raise other issues about the hockey stick, address this one.

UPDATE UPDATE UPDATE:

Readers referred to figure 2 in Wahl & Amman, so here it is (click for a larger, clearer view):