Cherry-Picking is Child’s Play

Anybody can do it.

Fake “skeptics” of global warming do it all the time. One of the latest and most extreme — this one is a real doozy — comes from John Coleman. Of course it’s regurgitated by Anthony Watts.


Coleman tells us this:


However, skeptical scientists have produced studies that show that the last 15 years have seen a cooling in the United States. This is the NOAA NCDC Climate at a Glance US annual mean temperature trend the last 15 years.

1819777_G

Problem: That trend line suggesting “the last 15 years have seen a cooling in the United States” isn’t statistically significant. Not even close. Other problem: That’s not the last 15 years.

Here’s all the data from the National Climate Data Center (the source for Coleman’s graph), together with Coleman’s trend line in blue, and dashed lines showing the time span chosen (click the graph if you want a larger, clearer view):

usa48_1

Why, do you suppose, did Coleman pick that particular set of data for his “trend”? Answer: Because it gives him what he wants.

When you choose your data because it gives you what you want, you’re cherry-picking.

Here, in red, is an analysis using all the data:

usa48_2

Coleman’s choice is so very, very cherry. It’s championship cherry. Olympic cherry-medal cherry. He should be nominated for “cherry-picker of the year.”

But let’s not think him clever for finding this juicy little over-ripe cherry. Remember that cherry-picking is oh so easy. Anybody can do it. Fake “skeptics” of global warming do it all the time.

How easy is it? Suppose you had some temperature data, say 40 years of annual averages, which was just plain random noise. No trend. Not even any autocorrelation. Plain vanilla bland random white noise.

If you analyze all of it, looking for a trend going either way (warming or cooling) by applying linear regression, insisting on 95% statistical confidence (i.e., insisting on a p-value no more than 5%), then there’s a 2.5% chance you’ll find a downward slope — a “false positive” cooling if you will. There’s also a 2.5% chance you’ll find an upward slope — a “false positive” warming if you will. So there’s a 5% total chance you’ll find a “statistically significant” slope even though there really isn’t one, it’s just a fluke due to random noise. Heck — that’s what “95% confidence” (a 5% p-value) means.

But you want to show that there’s a cooling trend. Not warming. Just cooling. You insist on it. By the usual method, you’ve only got a 2.5% chance of getting what you want. It turns out, just by unfortunate accident, that you didn’t get that lucky 2.5% result. Your raffle ticket just wasn’t the winner this time. Sad Panda.

Solution: don’t use all of the data! Instead, try starting with the 2nd year instead of the 1st. If that doesn’t work out, start with the 3rd year instead of the 2nd. If that doesn’t work out … well, you get the idea. Try all kinds of different start years until you get what you want. Persistent Panda! Wouldn’t that give you a better chance of getting what you want?

Yes. Yes, it would. Although doing it the honest way only gives you a 2.5% chance of getting what you want by sheer dumb luck, when you’re allowed to pick the start year because it gives you what you want, you’ve got a 21.3% chance to get lucky. It’s like taking 8.5 raffle tickets when you only payed for one.

Some bunnies might protest that taking 8.5 raffle tickets when you only payed for one is not the honest way.

But wait! There’s more! Seein’ as how you didn’t require yourself to start at the start, why force yourself into a corner by having to end at the end? Why not let yourself leave out the final year if that happens to make it harder to get what you want? Hey, it’s only one year. Who’s going to quibble about one single year?

Now you’re allowed to pick the start year you want and to leave out the final year if that makes it easier to get what you want. In consequence, even though the data are nothin’ but noise, you have a 28.6% chance to declare “statistically significant cooling”! This is terrific!!! It’s like taking 11.4 raffle tickets, even though you only paid for one.

Some bunnies might protest that taking 11.4 raffle tickets when you only payed for one makes you less-than completely honest Panda.

But wait! There’s more! Maybe even after you try all the start years, and leave out the last year if it doesn’t go your way, you still don’t get what you want. Guess what? That’s not the only raffle in town! If this temperature data set doesn’t agree with your story, try another. Or try sea ice data. Or sea level data. Or glaciers. Or — keep looking through data for all the world’s glaciers until you find one that you think looks good. Or, just wait for an especially heavy snowfall, somewhere, some time (winter is your best bet) and declare that you’ve found “cooling.”

It’s kinda like paying for one ticket in one raffle, then taking 11.4 tickets from every raffle. Hundreds of tickets — thousands — you’re sure to have a winner somewhere! If, a few years later, that ticket no longer pays off, switch to a different raffle.

John Coleman isn’t the only one playing this game. Most if not all of the fake “skeptics” do it. Regularly. The latest incarnation is the “16 years of no warming” meme. They’re playing it to the hilt. But it’s just cherry-picking — they picked the data because it gives them what they want. In a year or two, when that game no longer pays off, they’ll switch to another raffle.

Or you could do like John Coleman did. Even after picking the start year you want and leaving out the final year, when you still don’t get a supposedly “significant” result — call it “cooling” anyway.

Pathetic Panda.

About these ads

41 responses to “Cherry-Picking is Child’s Play

  1. Excellent post, exposing once again the routine dishonesty of fake ‘skeptics’.

    To me, one of the biggest causes for concern about AGW is that the ‘skeptics’ rely entirely on dishonesty to cast doubt on climate science. If there was any valid argument or evidence at all to reassure us that global warming isn’t going to be as bad as predicted, you can be 100% sure that the so-called skeptics would be shouting it from the rooftops. We’d never hear the end of it. Instead, all we get are false accusations of fraud against decent and honest climate scientists… myths and lies… cherry-picked data… distorted data… unfounded claims… bogus arguments… the whole gamut of dishonesty and dirty tricks, and they can’t seem to find a single shred of valid science to show that the situation is better than we think, rather than worse.

    That’s really, deeply worrying.

    • The bleeding obvious: if climate science really was flawed, why would the fossil fuel industry with such a massive interest in showing it to be flawed rely on propaganda, lies, character assassination and cranks to make their case?

  2. We are too kind in labeling it “cherry picking” (as if it was impish and insignificant). This is deceitful, misdirection and a media fraud. If the risk was nuclear secrets or an active conflagration, such misdirection would be criminally prosecuted. Coleman and Watts do real harm and should be held accountable.

    • Susan Anderson

      Yes, every day that lies are treated as misstatements, they spread a little further and poison the wind a bit more.

  3. “He should be nominated for “cherry-picker of the year.”

    And an early winner he might turn out to be. It’s hard to beat this.

    I understand that people at first are skeptical when it comes to AGW, but doesn’t stuff like this quickly tell them to take John Coleman off of their list of trustworthy sources? And by extension, Anthony Watts and his lucrative anti-science blog?

    That’s how it all started for me. Checking stories on WUWT every day, impressed each and every time, but then finding out it was only part of the truth or just plainly lied. That’s when I started to get convinced that AGW might be a problem.

    • Second that–I had much the same experience, albeit on a chat group. And it is very, very persuasive for those who know how to recognize these kinds of debater’s tricks.

  4. Now, given the awareness factor between Open Mind and WUWT, everyone knows that Johnnie Boy is going to see this post. Tamino is questioning Coleman’s integrity. Plain and simple. The proper response–the only response–would be for John to come here and defend his methodology. Otherwise I, and any other rational person, would have to conclude that John has no integrity toward the science, that his integrity is placed elsewhere. I’m all for reasoned, evidence-based dialogue with “skeptical scientists.” Can you provide such, John? Certainly robust methodology can withstand the scrutiny given here — sort of like passing through the gauntlet.

    Puh puh puh please?

  5. Not knowing the precise file he used, I used my best eyeball interpolation to get each temp value. For raw stats, I get 54.08±.173 for the mean/std err of the 15 years plotted. Therefore, as best I can see Coleman has shown that the mean of those 15 years is about 7.5 std errs greater than the baseline period he cites (which does admittedly overlap a bit). R–t.test(Data,mu=52.79)–works this out to t=7.46, d.f., 14, p≤3.07e-06.

    Looks to me as if he’s pretty shown this 15 year period is significantly higher than his chosen baseline.

    Of course I doubt if he or anyone else over there will notice that one thing.

  6. tamino, a very small nit to pick. You need to start and end your cherry pick just a few moths later to align exactly with what Coleman has done. His starts at ~53F and ends at ~54F, whereas yours starts at ~52.2F and ends at ~53.1F.

    I know that it doesn’t change the substance of what you’re saying one iota, but these idiots will seize on the tiniest little thing to claim that your analysis is invalid. It’s how they roll, as you well know.

  7. When you have a claim, and you try to find a way to get science to prove your claim, you are plainly and simply not using science. Full stop.

    I get this. You get this. Your readers get this. How is it that so many people do NOT get this?

  8. Horatio Algeranon

    “Noise Pickers”
    – by Horatio Algeranon

    Pick the beginning
    Pick the end
    If you want
    A cooling trend

    Pick the data
    Pick the source
    If you want
    A fooling force

    Pick your methods
    Pick your ploys
    If you want
    To pick your noise.

  9. Lars Karlsson

    I have to commend Coleman for bringing new blood into the art of cherry-picking by omitting the last, record-breaking year. Only a truly creative mind could have come up with something like that.

    And I also have to commend Watts for the high intellectual standard at his award-winning blog.

  10. Susan Anderson

    Accepting information from indirect sources is like responding to a phishing attack.

    As fast a new information comes out, it is carefully parceled and people are prevented from checking originals. Hard to counter, though.

    If the metaphor is helpful, you are welcome to it.

  11. Susan Anderson

    A nice compendium of stories going the right way, for once, here (thanks to Andy Skuce)

    http://www.skepticalscience.com/CCCMpersonal2.html

  12. I also have to question the dishonesty and/or incompetence of centering the green “trend” line (apparently to my eyes at least) on 52.97–the 1901-2000 average reported average value–rather than on the much higher mean value actually observed in even this cherry-picked data?

    What possible valid reason could be at work?

  13. And now for the Procrustean of the Anthropocene.

  14. What really gets me is that many of my fellow realists think we should be kind and patient to these assholes.

  15. I have always thought this comment from Judith Curry is priceless:

    “Is it “cherry picking” to start a trend analysis at 1998? No, not if you are looking for a long period of time where there is little or no warming”

    (http://judithcurry.com/2012/02/07/trends-change-points-hypotheses/)

    [Response: Yeah. That's the ticket. Yeah.]

  16. Alex the Seal

    …. and that chart is just the tip of a distorted logic ice-berg. I just read that entire article and now need to clean my brain.

    It’s like these guys rediscovering the reasons why data is homogenised in the first place. Tomorrow there will be an article about how those pesky weather bureaus massage their data. Sigh.

  17. Another terrific post!
    [Minor nit: "payed" for "paid," a couple of times.]

    The ‘artistic’ aspect of Mr. Coleman’s cherry pick is that he started with ’97, not ’98–so much more superficially ‘impartial’ looking, that way…

  18. W. Edwards Deming had a wonderful example of a newspaper headline as an example of public misunderstanding of statistics:
    50% of American babies are below average weight!
    I suppose the editor who chose that headline was named Coleman.
    Or even better:
    “Welcome to Lake Wobegon, where all the women are strong, all the men are good-looking, and all the children are above average. ” Garrison Keillor
    “Welcome to Watts up with That, where all are trends are cooling, all the ice is recovering, and the temperature is always below average”

    • A: “how many legs do you have?”
      B: (confused) “two…”
      A: “Ah! Just as I thought… more than average”

    • Toby.

      I occasionally like to point out that half the population is of below average intelligence…

  19. Thank you Tamino. The whole article was just so bad even Wattsonians found fault with it. (Watts looked like he wanted to distance himself from it by saying he ‘was asked to’ post it. He could have said “No”, but didn’t. He’s running out of denialisms and will take anything no matter how contradictory or ridiculous eg Tisdale.)

    I attempted to tear the Coleman post apart the other day (minus statistical critique) – WUWT Recipe: Baked Denier on Toast

    • he ‘was asked to’ post it

      By whom?

      And there was me thinking it was his blog.

      So who’s the puppet-master?

      • Horatio Algeranon

        “Neither heat nor droughts
        Nor melting ice
        Stays these Curryers
        From denies”

        – Post(it) Master Watts

  20. Horatio Algeranon

    “The Spin Cycle”
    – by Horatio Algeranon

    The Wattsing machine
    Has always been
    On dippy-cycle
    Endless spin

  21. John Coleman is a rank amateur.
    Werner Brozek manages 7 with one blow.
    Here is an excerpt from his comment at Roy Spenser’s blog

    On all data sets, the different times for a slope that is at least very slightly negative ranges from 8 years and 3 months to 16 years and 1 month.

    1. UAH: since October 2004 or 8 years, 3 months (goes to December)
    2. GISS: since May 2001 or 11 years, 7 months (goes to November)
    3. Combination of 4 global temperatures: since November 2000 or 12 years, 2 months (goes to December) (Needs to be confirmed)
    4. HadCrut3: since April 1997 or 15 years, 9 months (goes to December) (Needs to be confirmed)
    5. Sea surface temperatures: since March 1997 or 15 years, 10 months (goes to December)
    6. RSS: since December 1996 or 16 years, 1 month (goes to December)
    RSS is 193/204 or 94.6% of the way to Santer’s 17 years.
    7. Hadcrut4: since November 2000 or 12 years, 2 months (goes to December.)

    See the graph below to show it all.
    Graph

    This analysis indicates for how long there has not been significant warming at the 95% level on various data sets.
    For RSS the warming is NOT significant for over 23 years.
    For RSS: +0.126 +/-0.136 C/decade at the two sigma level from 1990
    For UAH, the warming is NOT significant for over 19 years.
    For UAH: 0.143 +/- 0.173 C/decade at the two sigma level from 1994
    For Hacrut3, the warming is NOT significant for over 19 years.
    For Hadcrut3: 0.098 +/- 0.113 C/decade at the two sigma level from 1994
    For Hacrut4, the warming is NOT significant for over 18 years.
    For Hadcrut4: 0.098 +/- 0.111 C/decade at the two sigma level from 1995
    For GISS, the warming is NOT significant for over 17 years.
    For GISS: 0.116 +/- 0.122 C/decade at the two sigma level from 1996
    (If you want to know the times to the nearest month that the warming is not significant for each set, they are as follows: RSS since September 1989; UAH since April 1993; Hadcrut3 since September 1993; Hadcrut4 since August 1994; GISS since October 1995 and NOAA since June 1994.)

  22. I would argue that monthly is a very wrong level of aggregation to look at with climate data. I am unsure why anyone uses it (even tamino). To my mind, annually is a minimum aggregation to consider and even that is going to be overnoisy w.r.t. the phenomenon being measured and heavily autocorrelated. 5 to 10 year aggregations are much more defensible.

    No one would measure pressure in a vessel by aggregating individual instantaneous measurements over the picosecond. No one would measure sound level by aggregating instantaneous measurements over the microsecond. Or at least I don’t think anyone would! Why does anyone measure climate by aggregating over months (or days for that matter which is also possible to do)?

  23. I don’t think they have cherry picked well enough. I’d have gone for the period 2006-2008.

  24. hinschelwood

    “Some bunnies”?

    Am I on the right blog here?

    [Response: You're just a hare's breadth away.]