Sum Fun

A new post on WUWT by S. I. Outcalt concludes that “the modern warming regime ended in 1997.” Outcalt transforms various data sets, and although he refers to his transformation as the “Hurst rescaling” (in order to associate it with the “Hurst exponent”) it’s really just a rescaled version of something statisticians were familiar with long before Hurst, called cumulative sums.


Cumulative sums can show very interesting behavior. They can also be extremely tricky to deal with statistically. That’s not a problem for Outcalt, his post doesn’t seem to have any statistics.

The heart of his claim that “the modern warming regime ended in 1997″ is this graph:

It’s the rescaled version of the cumulative sums of deviations from average, for various global temperature data sets from 1976 through 2008. It begins what seems like a steady rise right around 1997, which Outcalt concludes is the start of a new regime, with modern warming coming to a halt.

One wonders why he stopped at 2008 — it’s 2012 already, data are easy to get, and the “analysis” is pretty simple. Let’s bring things up to date.

I retrieved a different data set (but which one? it’s a mystery) for global average temperature 1976 through 2011 and computed the rescaled cumulative sums of deviations from average:

Wow! This also shows what seems like a steady rise starting in 1997. Could that be a “regime shift”? Could Outcalt be right that modern warming stopped then?

It also indicates a “regime shift” starting in 1984. What could have caused that? Is it the AMO? The PDO? A novel by George Orwell?

Here, by the way, is the same graph but with the result for data from NASA GISS superimposed in red:

The GISS data don’t seem to show the 1984 regime shift. Is it just an artifact of the other data set, or have the bad boys at NASA tampered with their data to eliminate the 1984 regime shift?

Kudos to the first reader who can identify the origin of the mystery data set.

UPDATE UPDATE UPDATE

We have a winner, folks. Reader “L. Hamilton” states

I can get something pretty similar, regime shifts and all, from random data, taking the 1978-2011 GISTEMP trend plus white noise with the residual sd.

Yeah!

It kinda shows how ridiculous is the claim of a “regime shift in 1997″ based on Outcalt’s “analysis.”

About these ads

67 responses to “Sum Fun

  1. UAH!

  2. some reasonable guess would be, a time series of atmospheric co2 content

  3. From WUWT:

    Chiefio Smith examines GHCN and finds it “not fit for purpose”
    Posted on June 21, 2012 by rep49

    ….

    Detecting regime shifts in climate data – the modern warming regime ended in 1997
    Posted on July 3, 2012 by Anthony Watts

    So my question is — If Watts is really claiming that warming ended in 1997, what temperature dataset did he rely on to make that call?

    Couldn’t be GHCN — Chiefio proved that GHCN is worthless.

    Couldn’t be HadCRUT — That has “climategate” written all over it.

    And it sure couldn’t be BEST, given Watts’ refusal to accept the BEST results.

    So, folks — what dataset *does* Watts use to support his claim that warming has ended? Anybody know?

    • Andrew Dodds

      That’d be the ol’ doublethink thing at work: To quote:

      The power of holding two contradictory beliefs in one’s mind simultaneously, and accepting both of them… To tell deliberate lies while genuinely believing in them, to forget any fact that has become inconvenient, and then, when it becomes necessary again, to draw it back from oblivion for just as long as it is needed, to deny the existence of objective reality and all the while to take account of the reality which one denies – all this is indispensably necessary. Even in using the word doublethink it is necessary to exercise doublethink. For by using the word one admits that one is tampering with reality; by a fresh act of doublethink one erases this knowledge; and so on indefinitely, with the lie always one leap ahead of the truth.

  4. I was going to guess some stock market index, but I’m switching to UAH.

  5. Looks like I just repeated Tamino’s question — should have read his entire post before putting up my comment.

    Nonetheless, we can safely assume that Watts would *never* rely on GHCN, HadCRUT, or BEST temperature data to make a “warming ended in XXXX” call.

  6. I can get something pretty similar, regime shifts and all, from random data, taking the 1978-2011 GISTEMP trend plus white noise with the residual sd.

    [Response: Ding ding ding!!! We have a winner.

    I generated random noise, added a linear trend, and voila!

    Which pretty well proves that there's no regime shift.]

  7. The outdated HadCRUT3?

  8. What a wonderful kind of analysis, First rate WUWT science. All that the inflection points at zero indicate is the times, when the data switches between above and below average. That is, the inflection point changes when you change the period of observation. Simple example: a positive linear trend (tamino, could you please demonstrate Hurst rescaling for a linear trend and a parabola. I think this would be instructive to readers). You’ll get an inflection point in the middle of the data window. Move the data window and the inflection point follows. If Outcalt were to start his figure 2 in say 1920 then the inflection points would move drastically. Hurst rescaling may have its applications, but most definetly not the way S.I.Outcalt demonstrates it.

  9. “Kudos to the first reader who can identify the origin of the mystery data set.”

    Practically any time series of an exponentially growing time series with a similar signal:noise ratio as global mean temperature. I didn’t quite follow the series of mean subtractions and trend removals described in the WUWT post, but this sort of asymmetric chevron plot is what you get if you do this to any process that’s remotely well described by exponential growth. A better example might be a less noisy series where there definitely are growth rate shifts that last for some time like the Dow Jones Industrial Average.

    If y = exp(ax) – 1, then C, the cumulative sum of y- over some interval x, is C = (exp(ax) – 1)/a – x*(+1). Which has a minimum at x = ln(+1)/a. For a rate of warming of 0.014C/yr and an average temperature increase of 0.35C since 1975, you get a minimum at 1975+21 = 1996.

  10. Just to fill some of the white space in my earlier graph, the first 6 random trials. Regime changes in every one, although it might have happened in 1992 … or 1999 …
    http://img.photobucket.com/albums/v224/Chiloe/Climate/Outcalt_2.png

  11. Dim sum–only not as tasty nor fulfilling as the Chinese flavor.

  12. “Oh, he’ll try to shoot it down anyway. – Anthony”

    I wonder if Anthony can even understand your debunking?

    -MMM

  13. What if we had less data?
    http://i48.tinypic.com/29n6vxf.png
    Hurst rescaling of hadcrut3 variance adjusted global mean for different timeframes.

    • Bob Loblaw

      If I were generating that graph as part of an analysis I thought was valid, I’d look at it and say “why do all these lines end up converging on exactly the same value at the end?” I’d start to wonder if that result was caused by the analysis method, rather than the data… and isn’t it lovely how the “regime change” always seems to fall somewhere close to the middle of the data? Sure makes it easy to decide what time range to include, if I’m trying to make a particular year stand out.
      Unfortunately, Sam Outcalt had a great reputation in microclimate and modelling during his career. I haven’t heard anything about him in years, though. Has he “gone emeritus”?

      [Response: Because the cumulative sums are of deviations from average, and the sum of the deviations from average are of course zero, all the cumulative sums are zero for the final moment of time.]

      • Because the cumulative sums are of deviations from average, and the sum of the deviations from average are of course zero, all the cumulative sums are zero for the final moment of time.

        This was my very first thought when I saw that Outcalt had graphed cumulative sums of deviations from average. Doing so will inevitably give a trajectory that shows at least one progress from a zero starting point and then a return back. Even in a circumstance where the most recalcitrant of contrarians could not deny a continuing trend!

        I can’t imagine what Watts and Outcalt imagine they’re showing, but they’re certainly not showing that warming stopped in 1997. All that they are showing is that in the dataset graphed, 1997′s residual is the last one to add to the greatest absolute value of the cumulative deviation. It’s probably also instructive to note that the residual with the greatest absolute value does not have to superimpose on this point of the cumulative trajectory – it can occur either before or after this point.

        Still, if they insist on demonstrating to the world how utterly numerically clueless they are, who are we to stop them…

      • Just to be clear, that’s “…continuing trend in the original data“.

      • As tamino has already mentioned, the difference between the last and the first point of the Hurst rescaling need to be the same because of how the series of data points is defined. Actually in my plot the last points are NOT the same, just nearly so. The reason: when you are adding up your data you need to start at some first data point. That one is chosen as zero and this gives you an additional data point. Keeping in the tradition as given by Outcalt I plot the additional point at the beginning and skip the final point, which has value 1.

        The Hurst rescaled curve does not always have to start at 1 or 0, as it does in the case of linear trends. Let’s have a look at a sinusoidal signal and its Hurst rescaled curve.

        http://i47.tinypic.com/34t5noo.png

        Here the latter starts at an “arbitrary” value between 0 and 1. Note that the maxima and minima of the Hurst curve correspond to inflection points of the original data, i.e. to points where the curvature of the orignal data changes. You do not readily see in the Hurst rescaled curves those points where the original data changes from warming to cooling. So for the purpose stated by Outcalt I consider the method to be unsuitable, to put it mildly. Take further notice of another complication: the extrema of the Hurst curve all have different y-values. If you were to attribute any significance to that fact, you’d be in deep trouble.

        P.S.: Whoever is brave enough to wade into WUWT, feel free to use my graphics.

    • bluegrue – Thanks – that makes it very clear!

  14. “I wonder if Anthony can even understand your debunking?”
    No, he definitely can’t, but that does not matter because none of his readers can either. They’ll agree anyway that Tamino is wrong.

  15. I refuse to lend my eyeballs to WUWT in order to find out the details of how that graph was produced. Am I right to think that if Tamino had left out the random noise and just produced a pretend temperature record in which temperature increased at a constant linear rate from year to year, the resulting graph would look more or less like a parabola, with a “regime shift’ in the middle of the period of observations?

    [Response: Yes, the cumulative sums of deviations from average for a straight line will give a parabola. As for "regime shift" -- I think you'd have to mis-apply the term to draw that conclusion.]

    • Sometimes you wonder if when it becomes impossible to fool even his readers Anthony Watts is going to claim his whole global warming denial career was an extended Joaquim Phoenix style piece of comic performance art.

  16. I don’t understand anything. Why would a steady rise in deviation from mean starting in 1997 indicate that “modern warming regime” ended then?

  17. I attempted to post the following on WUWT:

    “Contrary to Gneiss, what Tamino has noticed is that adding white noise to a linear trend and then taking the normalized integral of departures from the mean will produce a similar pattern to that in Fig 3 above. That fatally undermines Outcalt’s claim that the pattern shows a clear departure from a linear trend in the data. In other words, Outcalt’s thesis has been refuted, resoundingly so, by Tamino.

    Now arises the question of honesty. It is the mark of an honest person that they will publicly admit their mistakes lest anyone else should be accidentally deceived by them. As Outcalt’s thesis has been refuted, clearly he has made a mistake. Will he publicly admit it? More importantly, Watts has also made a mistake. He either thought he understood Outcalt’s claim, and thought it a reasonable claim. In that case, his understanding has been shown to be faulty and he ought to publicly say so, and thank Tamino for correcting the error.

    Alternatively he published Outcalt’s article without understanding it. In that case, he shows thereby that his purpose is propaganda. Specifically, it shows that his selection criteria for articles is not that he knows them to be accurate or reasonable, but simply that they come up with conclusions that he likes. In that event, he needs to not only correct the error, but modify his editorial practice to one that is more honest.

    Unfortunately, past practice has shown that Watts does not follow the simple requirement of honesty in correcting his errors, so I will not hold my breath.”

    I was not surprised to recieve the following response:

    “[snip. Labeling our host as being dishonest gets you snipped. My decision alone, Anthony had nothing to do with it. ~dbs, mod.]”

    So, in order to prevent Watts being called dishonest (although I did not, in fact, do so) ~dbs has decided that WUWT must follow the dishonest practise of not acknowledging identified errors. That is in fact no change of policy for WUWT, but the irony that he has confirmed the dishonest practice by censoring my post seems to escape ~dbs.

    • I think you’re splitting semantic hairs if you seriously claim writing “Unfortunately, past practice has shown that Watts does not follow the simple requirement of honesty in correcting his errors…” didn’t amount to calling him dishonest.

      • Rob Honeycutt

        The fact is, Watts is dishonest. He claims not to be a denier of AGW but continually allows posts that specifically deny AGW. And if you question him or anyone whose posts he allows (and he clearly only allows posts from people who deny AGW) he or his mods delete at will.

        These guys are operating under a false pretense that they are creating some new form of peer review in the form of online review. But if you’re going to delete or reject every challenge to what people write it completely defeats the purpose. That makes what Watts engages in propaganda, not science.

        Have you ever seen what real scientists go through in the peer review process? It’s a harsh process that requires serious meddle to get through, even for top researchers.

        I’ll go a step further than Tom has and say that Anthony and his mods are spineless.

        [Response: I'll take a step backward and say that, in my opinion, he actually believes what he posts. And I'll point out that I delete comments which I think are useless, so I certainly edit the content here.

        If there's dishonesty to Watts' blog, it's that he's dishonest with himself. For the stuff that he thinks contradicts global warming, he is NOT the skeptic he likes to believe he is, he's not a skeptic at all but the exact opposite -- the height of gullibility. He will post absolute utter nonsense, with no critical thinking applied to it, which would easily be seen for what it is by a skeptical eye. It's why I prefer the term "fake skeptic." As for Watts, he'll bend over backwards so far to fawn over nonsense, it's embarrassing.

        Yes peer review can be brutal, and when it is it's usually a crucible. But peer review can also be lax. Looking at the paper about Outcalt's method, I suspect the peer review was not nearly severe enough.]

      • Jon, first, it is not splitting semantic hairs at all. Without question, IMO, Watts has engaged in acts which are dishonest. They, however, may have been aberrations from a person who was basically honest. The proof of whether they were aberrations or just representative of his character is whether he modifies his practice to avoid repetition. The distinction between performing a dishonest act, and being habitually dishonest is what is at issue. His moderators seem determined to prove it is the later (which was in doubt) rather than the former (which is what I accused him of).

      • Tamino, the distinction between somebody who tells obvious falsehoods because they cannot be bothered telling truths, and one who does so because they have first deceived themselves is purely academic once that person takes it upon themselves to instruct others. In either even they are a bad instructor and spread falsehoods in the world. Once Watts commenced his website, he has a responsibility to ensure he applied minimum levels of honesty in his posts; the most minimal of which is to acknowledge clear errors. If he eschews that, all that remains is propaganda.

      • Rob Honeycutt

        Tamino… I’ve said the same thing about Anthony in the past, that I believe that he actually believes what he says, but I’m really beginning to have my doubts about that. As time goes on I’m less inclined to believe that he’s merely being dishonest with himself, I’m starting to believe that the dishonesty is deliberate as a way to maintain a fanatical following, and to try to move the political agenda the way he wants.

        As long as Anthony keeps posting things his followers with cheer for he’s a big man. If he were to approach any of the posts honestly he has nothing. He has significant motivation to be dishonest.

      • Bernard J.

        Tom Curtis.

        Exactly.

      • Philippe Chantreau

        I totally agree with Tom. Deniers are given way too nice a range of behaviors while they are not ready to give back anything remotely approaching it. Curry allows blog posts calling for the torture of climate scientists under the pretense that it is all in good fun. When you point out how grotesque and self contradictory deniers arguments are, they scream ad-hominem.

        It has to be called what it is. If it’s stupid, or dishonest, it’s just that. Those who choose to believe in stupid and dishonest arguments are responsible for their choices, no matter how little judgment their emotional reactions leave them. This is why we can think, to bring our best judgment beyond what would normally be dictated by emotional reactions.

    • I have to say that while nothing in your post is _wrong_, I think tactical considerations would indicate that stopping after your first paragraph would have been appropriate.

      eg,
      1) Don’t give them any grounds to claim they are being insulted, which allows them to dismiss (and delete) your comment
      2) Maximize the rather small probability that they will actually read and think about your comment, rather than dismissing it as obnoxious.

      (oddly, this behavior is obvious in both camps: many comments posted by contrarians at RealClimate start or end by going on about how they are sure that the moderators will delete the message because realclimate is such a dishonest place)

      (now, note that I think that RealClimate is honest and scientific, whereas WUWT is a hive of scum and villainy, but in either case, I encourage the same tactics of politely and succinctly stating the relevant science rather than straying into more emotional issues. Not that I always follow what I preach)

      -MMM

      • MMM, denialism thrives on tolerance of falsehood. The reason we have deniers is because Monckton and Watts and the like can assert absurd propositions, and falsehoods easily shown to be such, and be confident that they will not be called on it by their cohorts, and certainly not be asked to retract it. It is about time we started calling them on the more egregious examples. It is also about time we clearly stated the standard we expect in reasonable discourse – that you acknowledge your errors.

        On that basis, the follow on paragraphs are the entire point of my post. The first paragraph added nothing that anybody interested in learning would not already have discovered by reading KR’s earlier comments.

        But if we hammer the point in the later paragraphs enough, eventually it may get through. Particularly with Watts, I might add. He craves intellectual respectability, but can never have it because he will not acknowledge his errors.

    • Horatio Algeranon

      Anthony’s Paradox

      “I deny that I ever deny”

  18. Any time you take a time series (normalized such that mean = 0) that can be well approximated by a linear trend plus some noise you will always get a “bumpy” parabola that looks a lot like the figures Outcalt produced. It’s a forgone conclusion.

    The only questions will be how bumpy will the parabola be (which will depend on how much noise is in the original series relative to the trend) and how deep will the parabola be (which will depend on only the magnitude of the trend). However, Outcalt appears to have normalized the cumulative sums to vary from 0 to 1, so the only think left is how bumpy is the parabola.

    If the noise is small to moderate relative to the trend (as it is with the surface temperature data over this period) then the the parabola will be fairly “clean” and you will almost always be able to eyeball so called “regime shifts” on either side of the bottom of the parabola (i.e. where the curve shifts from steep to flat and back to steep).

    So, all he has demonstrated is that parabolas are shaped like parabolas.

    • You actually do not get a parabola because parabolas are symmetric. For any monotonically increasing, mean-subtracted, y(t) – , you get an asymmetric, chevron shape for the cumulative sum where long times are dominated by the integral of y(t) and short times are dominated by linear decay (the 2nd term’s integral).

      So apply this method to linear growth and you get linear decay transitioning to quadratic growth. Apply it to exponential growth and you get linear decay followed by exponential growth. Some would argue that 1998-2010 data is too flat to be quadratic, exponential, etc but the rate of temperature increase is very low relative to its absolute value and noise.

      [Response: From a linear time series, cumulative sums do follow a parabola. And, sections of parabolas don't have to be symmetric.]

      • “From a linear time series, cumulative sums do follow a parabola.”

        Yes, but this is the cumulative sum of a linear series with an additional, constant term for the mean subtraction. So the cumulative sum of linear growth manipulated in this way shows a shift from being dominated by the mean subtraction term to being dominated by the cumulative sum of the linear growth term, which is parabolic.

        “And, sections of parabolas don’t have to be symmetric.”

        I thought a parabola was a symmetric, conic section by definition. I was responding to “either side of the bottom of the parabola” which seems to be talking about a single parabola. I think it’s worth pointing out that the asymmetry of these plots is an artifact of summing the integrals of a negative, constant term and a positive, increasing term. Otherwise the folks at WUWT will look at the asymmetry Fig. 3 (in the WUWT post) and say there’s some obvious regime change between 1975-1995 and 1995-now.

      • I don’t know if this will show up in the right place, it’s meant to be a response to Zach on 7/5/12 @ 1:56pm:

        the parabola symmetry issue here seems to be semantic–replace Tamino’s “sections” with the more forgiving synonym “pieces” and I think it resolves the problem–parabolas are symmetric, but *pieces* of parabolas can, obviously, be asymmetrical.

  19. I wonder if Outcalt wrote this post in jest; he included this interesting statement in the Abstract:
    “The dramatic “hockey stick” trace, which began in 1976 accompanied by a major transition in the Pacific Decadal Oscillation, ends at the onset of the 21st Century and might be better termed the modern warming regime. This regime was replaced by a pronounced cooling regime. These observations attenuate the demonic interpenetration of the linear trend in the historic global temperature data.”

    Demonic interpenetration of the linear trend? Is he saying that the Devil is behind global warming?

    • I left another comment (currently suspended in moderation) on WUWT:

      Moderator REP: I did ask a question about the science in this post. Yet you snipped it.

      I asked Anthony Watts, or anyone else for that matter, to define and explain the following critical finding in the abstract of this “paper” by Outcalt:

      “This regime was replaced by a pronounced cooling regime. These observations attenuate the demonic interpenetration of the linear trend in the historic global temperature data.”

      I tried to learn what WUWT is telling us. I googled the term demonic interpenetration, and got some strange results; in some cases the term “demonic analogy of interpenetration” is used.

      Lets hear from the WUWT regulars. What does this important conclusion actually mean?

    • Horatio Algeranon

      I wonder if Outcalt wrote this post in jest;

      yes, the “demonic interpenetration” comment is curious (to say the least), especially coming from someone named “Outcalt” which sounds more than a little like “Occult”.

      It would be humorous if Tony’s blog has itself been subject to “demonic interpenetration” by an AGW infidel.

  20. And over at WTFUWT there are half a dozen new posts, and this embarrassing event has been long forgotten, even though it is only a couple of days old.

    • Collective behavior remindful of Sacks’ brain damaged patients; highly selective and specific cognitive dysfunction.

  21. Horatio Algeranon

    “Sum Fun”
    – by Horatio Algeranon

    Zero equals zero
    And one it equals one
    A parabola’s a parabola
    Isn’t math sum fun?

    • Bernard J.

      Horatio,

      You could have gone the zen route, and titled it “Outcalt’s Pendulum”…

      • Horatio Algeranon

        Great title

        “Outcalt’s Pendulum”

        Down and Up
        And To and Fro
        The pendulum swings
        With Watts in tow

        The happy cycle
        Never ends
        Mathturbation
        Betwixt friends.

  22. arch stanton

    Clearly, “demonic interpretation” is a legal term, even if “denialist” isn’t. (but don’t use it in regards to Smokey)…

    • arch, please be more careful with your terminology.
      A “demonic interpretation” would mean that the demons have caused a misinterpretation of the data. OTOH, a “demonic interpenetration” indicates that demons have actually changed the system creating the data, or changed the data measurement system. And according to sources, the process they use involves the demonic interpenetration of human spirits, causing the humans to act under the influence of the demons.

      In short, in the Outcalt theory publicized by Anthony Watts, demons invading human spirits cause the system changes that result in the linear trend of rising temperatures.

      And you know what? For once, they just might be right!

      • arch stanton

        Ah, thank you for pointing out the subtle but significant difference Zin, and I get your point about them being right (in a Pogoesqe sort of way).

  23. It always amazes me how desperate deniers are and try every trick to tease out the result they want.
    But the facts are simple, if we have had something like 11 of the warmest 13 years since 2000, how can warming have stopped?

  24. Horatio Algeranon

    Horatio thought “demonic interpenetration” was what happened in Rosemary’s Baby

  25. Yeah still over head:)

  26. Interesting comment by Gneiss (well, the mod reply is the interesting part) on WUWT: http://wattsupwiththat.com/2012/07/03/detecting-regime-shifts-in-climate-data-the-modern-warming-regime-ended-in-1997/#comment-1024942

    “Basically, Outcalt’s post is transparently wrong, as anyone who thought about his arithmetic has figured out by now. Why the post was written and published without noticing that is a fair question. How Outcalt and Anthony respond now will shed light on the answers.

    [REPLY: Anthony publishes lots of stuff, not all of which he agrees with. It is simply "interesting". WUWT commenters are fully capable of critiqueing the work and have done so. Anthony is not a co-author on this article and is not required to justify or explain anything, especially to anonymous individuals using anonymous proxy servers. Check site policy regarding that. -REP] ”

    I’m pretty sure I’ve come across this sort of thing before on WUWT. It’s a typical get-out clause for Watts that if he posts something he thinks he agrees with that is later shown to be a load of rubbish, he can say actually he didn’t agree with it, it was just “interesting”. And what can we say about WUWT commenters being capable of critiqueing [sic] the work? Several echoes went around the chamber before MattN homed in on the problem – and I don’t know whether he and the others who actually pick this work apart are WUWT regulars…

  27. Horatio Algeranon

    The 3 kinds of denies: “Denies, damned denies and no statistics.”

    “Cumulative deviations from the mean are often used in the analysis of homogeneity.

    “Though graphs are useful for the detection of shifts in the mean it is usually not obvious how real changes can be distinguished from purely random fluctuations. Therefore it is always necessary to test the significance of departures from homogeneity by statistical methods.” — T.A. Buishland

    From Buishand, T.A., 1982. Some methods for testing the homogeneity of rainfall records. J. Hydrol., 58: 11–27.

  28. I wonder what would happen if you generated synthetic data using the linear trend from UAH, plus white noise with residual sd and fit a 4th order polynomial to it. I’m sure someone here sees where I am going with that…

  29. I’m belatedly appreciating the delicious irony here: McI and McK famously (and unsuccessfully) purported to show that the hockey stick was an artifact of processing. Now Outcalt has made the inverse case a reality, by creating his own iconic “artifact of processing.”