BS and DK

[Note: see the UPDATE at the end of the post]

Brandon Shollenberger has written a blog post about my last post. I consider it an example of the Dunning-Kruger effect, that unskilled individuals overrate their ability.

It starts with a comment on this graph:

I was sure there was no way I saw a smoothed line go up while the unsmoothed values went down. Only, I had.

It’s followed by a comment on this graph:

Again, I was shocked. This figure showed an even greater uptick than the last! My mind couldn’t understand how the data was going down, but Tamino’s derived lines were going up.

He commented here, asking why, but also making sure to emphasize at length how much it defied belief. I tried to explain it to him. Really. Repeatedly. The problem was that his intuition — that if the data go down the smoothed line couldn’t possibly still go up — was wrong. He was misinterpreting the noise in the data as the signal. A smoothed fit which took that into account could easily continue up.

He wasn’t buying it. He got more and more obstinate, and frankly, his comments got more and more insinuating. It culminated in his making a statement that I can only interpret as an outright accusation. How nice of him.

In his blog post, Shollenberger does his own Lowess smooth and gets this (note: his graph says “USHCN” whereas I used data from NCDC, but that’s not really important):

Shollenberger says:

You’ll note in my figure, the smoothed line goes down at the end, exactly as I would expect, and exactly as my intuition told me.

But in my smoothed curve, the line does not go down at the end (it curves slightly that way, but doesn’t actually decline). He concludes that the reason is my modification to the Lowess algorithm:

There’s only one answer. For some reason, Tamino’s modified version of LOWESS gives extremely low weight to the data near the ends of the series.

Once again, Shollenberger is wrong.

Let’s take the data I used (anomalies of monthly data from NCDC) and compute a Lowess smooth — not using my modified version, just the built-in vanilla “lowess” function in R. Then let’s compare the unadorned Lowess smooth (in blue) to my modified version (in red):

Whoa! That’s different! It certainly does not go down at the end. But then, it doesn’t really follow the data that well either, as we can see by overlaying the 5-year averages (black dots):

What’s up with that?

A Lowess smooth includes a tuneable parameter which controls how much of the data is used in the “window” for each estimated value. A wider window means more smoothing — maybe too much, so we might fail to pick up some variations which are genuine signal. A shorter window means less smoothing — maybe too little, so we pick up some variations which are really due to the noise, not the signal.

The default (in R) is to include 2/3 of the data at each moment of time. That’s too wide, so although the Lowess smooth correctly identifies the overall increase, it fails to pick up some variations which are genuine signal. The default in my program is to use a window a bit less than 25 time units wide (note: the tricube function has finite support so it has a well-defined window width, my weighting function doesn’t). For these data, that corresponds to a parameter value of about f=0.2 in the R implementation. Let’s compare the R “lowess” result using f=0.2 (in blue) to my modified version (in red):

Now the two results are quite similar. There are differences because my algorithm is different. And no, I don’t care to go into it in depth simply to satisfy Brandon Shollenberger’s lust for fault-finding. Notice that the vanilla Lowess does not go down at the end. It curves slightly that way (as does mine) but doesn’t slope downward. Imagine that.

How did Shollenberger get such a different result? Let’s compare the vanilla R verion of “lowess” but with parameter value f=.15 (blue line) to my original smooth (red line):

Well whaddaya know? Now it goes down at the end.

He thinks this is the “right” result — exactly as his intuition told him. I say it’s the wrong result, that the smoothing time scale is too short so it’s picking up variations which are really due to noise, not signal. Let’s test that idea, shall we?

Since about 1975, USA temperature has risen dramatically. Let’s take the data since then and fit a straight line — we all agree there’s an increase. Then lets look at the residuals for some sign of statistically significant departure from the linear increase. Here are those residuals:

One way to look for other signal is by fitting higher-order polynomials, and we can even test such fits for statistical significance. I used polynomials of degree 2 through 6 and computed the p-value for each. In all cases, the p-value was nowhere near significant. I didn’t bother to correct for autocorrelation, but it’s not very large and would only make the fits less significant.

Maybe … maybe since 1975 is too much data to detect a brief downturn near the end. Let’s repeat the experiment using only the data since 2000. Result: the same.

Although not statistically significant, the smooths you get from polynomial fits show an interesting behavior. Here’s the 4th-degree polynomial fit to data since 2000:

Notice that if we allow statistically insignificant fits we can get a rather dramatic upturn at the end. It’s the kind of false impression that noise can create.

If you set the window width to too small a value, then you can get the recent downturn in the smoothed curve with my version of Lowess too — not because of the different weighting function. Brandon Shollenberger was wrong. Again.

Why did I choose the degree of smoothing I used? Because it’s the default for my smoothing program. I chose it as the default because considerable experience in its use has shown it to be a good general-purpose value for climate time series. If I notice what looks like real signal that isn’t detected, I’ll try a shorter window, if I notice what looks like response to noise I’ll try a longer time window. If I’m working on a peer-reviewed paper I’ll probably run some statistical tests (like above) to confirm or deny what is signal and what is not. In this case, my intuition told me the default was good. I didn’t try a bunch of values until I got the result I wanted.

Now let’s talk about the other graph. Shollenberger says:

Naturally, I suspected that graph was questionable too, so I decided to test something. What if when calculating the five-year averages, we don’t include 2012, which only had four months of data?

Then he gives us this graph:

And he says:

I was speechless. If I took the four months from 2012 as a whole year and averaged them with the data for 2008-2011, I got Tamino’s results. If I took the average from 2007-2011, I got a result nearly half a degree lower. Not only does this have a huge visual impact, it means Tamino used one data set in three different ways:

1) Annual averages, excluding data from 2012.
2) Monthly smoothed, using all data.
3) Five-year averages, using four months from 2012 as a whole year.

No, Brandon, you’re wrong. Again.

I did compute the smooth using all monthly data. I also computed the 5-year averages using all monthly data. Damn me for not calling them “60-month averages.” I certainly didn’t use four months from 2012 as a whole year. I did use 28 months from January 2010 through April 2012, so the final average is based on less data than its predecessors. Damn me again for not mentioning that, especially since it has no effect on the subject at hand — that Tom Harris’s claims about the USA temperature record are wrong. But at least I had the courtesy to show the (2-sigma) error bars for the averages.

Let’s recap. For 5-year averages I used all the monthly data. You added a 5-year average for 2007-2011, which means the 2007-2009 (the below-the-smooth stuff) gets used twice. And you omitted everything from 2012 (the way-over-the-smooth stuff).

The only departure from using all the monthly data was for the graph of 1-year averages. Using only January through April to represent 2012 gives this:

If I had shown that, I suspect Brandon Shollenberger would have had a conniption. If, on the other hand, I had used May-through-April for the 1-year averages it would have looked like this:

If I had shown that, I suspect Brandon Shollenberger would have accused me of deliberately choosing a period which makes the final data point the warmest.

So instead, in this and only this instance, I omitted the first four months of 2012 — the hottest third-of-a-year on record — entirely. How alarmist of me.

Shollenberger closes with this:

Does any of this have any impact on the point of his blog post? No. To be honest, I still know nothing about the Tom Harris his post criticizes. The point of this is people promoting global warming concerns should make it possible to understand their work without having to figure out undisclosed details based on arbitrary and unexplained decisions. In fact, everyone should.

I suppose I could turn every blog post into an exact elucidation of every technical detail. Lord knows I never get very technical.

How about this instead: people criticizing global warming concerns should have sufficient competence to know what they’re doing.


It seems Brandon Shollenberger is having difficulty understanding the difference between his result and mine, since he says he used a parameter f=.2, not f=.15.

Brandon Shollenberger (Comment #95500)
May 10th, 2012 at 10:11 pm


Can you verify that using that parameter value generates Tamino’s graph? If it’s true that the difference between “standard LOWESS” and Tamino’s “fancy smoothing method” is a 0.05 difference in one parameter, the dark mutterings about “very weird and undisclosed arithmetical choices” seem a bit overdramatic.

It’ll take me a little bit to see if I can figure out what is going on in Tamino’s post, but I can say this much right off the back. I did not use f=0.15 as Tamino portrays. I tested a number of different values for f to see which matched his line best, and I found 0.2 gave a near perfect match. 0.15 gives a terrible match, and it looks dramatically different than the graph I posted. There is no way anyone trying out LOWESS filters would actually think I used f=0.15, and I have no idea why Tamino implies I used it.

Heck, if you just look at the line he comes up with using f=0.15, it’s obviously different than my figure by a large margin. As an example, the last rising slope in my figure pretty much rises steadily, but his supposed replication has a huge flat portion. There’s also the obvious fact the period from 1950-1960 is basically flat in my figure, but in his, it’s rising notably. Given how obvious these differences are, I’m at a loss as to how you can say:

A parameter value of f=0.15 seems to give the graph that Brandon shows.

All it takes to see that wrong is to look at the two graphs. That’s it.

He’s confused.

It took me about a minute to figure out his problem. He computed his Lowess smooth using 1-year averages, not the monthly data. This in spite of the fact that he was undoubtedly aware I had used monthly data. It’s number “2” in his list of ways I had used the data. Damn me for assuming that he would attempt to reproduce my result by doing the same thing!

Gosh Brandon, you could have told us. Why didn’t you follow your own advice to “make it possible to understand their work without having to figure out undisclosed details”?

Perhaps he’s surprised that using annual averages instead of monthly data could have that much effect on a Lowess smooth. What does that suggest to you, Brandon? Is it possible that the sensitivity of the result to annual averaging indicates that the wiggle you are so fond of is just an artifact of the noise?

Here’s something else for you to try, Brandon. Compute 1-year averages for May-through-April since you don’t want to bias the result by leaving out the most recent data. Then study the time span from 1975 to the present. I’ll get you started:

59 responses to “BS and DK

  1. “If I notice what looks like real signal that isn’t detected, I’ll try a shorter window, if I notice what looks like response to noise I’ll try a longer time window.”

    What do you mean by ‘signal’ and ‘noise’ in temperature series. Is it:
    signal = externally forced response
    noise = Internal variability

    Because from the time series’ themselves we have no way of knowing which is which. The division is arbitrary.

    In fact there is some evidence that the externally forced response is closer to the blue line in your 2nd figure. In other words, a lot of the multidecadal variability is explained by natural ocean circulations (e.g.,

    [Response: Loosely speaking, I use the term “noise” for fluctuations which are essentially random. For example, volcanic eruptions can be treated as noise in spite of representing externally forced response — unless you can show that they exhibit a detectable trend.]

  2. Game, set, and match. Brilliant, Tamino!

    “What’s up with that?” Cackle!

    Brandon, you were pushing it but for me the absolute stunner is this last paragraph of yours Tamino has quoted.

    Does his reply here have any impact on the point of his original blog post? Yes, it helps confirm that his analysis was right and that yours and Tom Harris’ was wrong.

    “I still know nothing about the Tom Harris his post criticizes.”

    Hmm. Either way, you lose here, Brandon. That doesn’t put you in a favorable light. Knowing such stuff would help establish some credibility on your part. Knowing the back story on Harris would help you understand the context of Tamino’s original post.

    “… people promoting global warming concerns should make it possible to understand their work without having to figure out undisclosed details based on arbitrary and unexplained decisions.”

    Well, he did make it clear and he patiently tried to explain it to you. You admitted, *after you had done the math,* that he was right.

  3. Or how about this….written specifically for Dan H, but it certainly seems widely applicable. So what’s your point? If you’re convinced that climate scientist’s work is so seriously flawed, stop with the links (and analyses..) that don’t make any sense and don’t stand up to even the most fundamental of reviews . Put together a coherent argument and submit it to a reputable scientific journal. Stop wasting electrons and our time.

  4. Tamino: Let’s follow another piece of advice given here, and study the data ourselves.

    Tamino, you shouldn’t encourage us to study the data on the one hand and slap us down with the other as we work our way up the learning curve. Brandon was right that your previous post wasn’t as transparent as it could have been. Although I am no fan of his cloying use of !!! and insulting feigned shock-surprise.

    Thank you for the clarifications. While the slap-downs may be entertaining for some, the gold in this blog is in the exposure to (and attempted replication of) your methods.

    [Response: I didn’t respond as I did because he inquired about methods in order to work his way up the learning curve. I answered his question, and tried, repeatedly, to clarify the origin of his mistaken interpretation.

    But he wasn’t having any of it. And now he’s gone out of his way to refute my result when he really doesn’t know what he’s doing. If he were interested in learning that would be a welcome relief. Don’t hold your breath.

    And seriously — if I included all the technical details all the time, posts would be weighed down in excruciating detail. The point — that Tom Harris’s claims were false — was quite clear.

    Speaking of details, has anyone asked Shollenberger what value he used for the window width parameter? Why he chose that value? How many different ones he tried before he got what he wanted?]

  5. My own totally unscientific and almost certainly utterly unfair judgment on Brandon is that he’s a small-town kid with a sharp mind and a ginormous chip on his shoulder. I think that’s a little different than Dunning-Kruger, but fine for the nonce.

    Lucia’s rush to close the book on this is in my opinion much more classic DK.

    In fact, only by making very weird and undisclosed arithmetical choices is it possible to concoct a graph with the 5 year average ending going up.

    Nobody can do head-shaking Dunning-Kruger like Lucia Liljegren.

  6. Tamino, you shouldn’t encourage us to study the data on the one hand and slap us down with the other as we work our way up the learning curve.

    Oh, c’mon, Brandon didn’t come here asking for help “working his way up the learning curve”.

    He came here insisting Tamino must work the way up the learning curve in order to meet his own expertise.

    Which, as it turns it, is more or less non-existent.

    And his post at Lucia’s demostrates further D&K, i.e. arrogance that he knows Tamino’s job better than Tamino does.

    Brandon’s not interested in working his way up the learning curve. He’s only interested in “proving” mainstream science “wrong”, because, you know, he knows that he knows blah blah.

  7. has anyone asked Shollenberger what value he used for the window width parameter?

    Yes. And I didn’t get an answer. I did get what values he got running parameters of .15 and .2, however:

    [Response: See the UPDATE at the end of the post.]

  8. OK, I know this is the wrong thread, but I could not find any recent open thread, and since DK was mentioned…I present to you yet another poster child of the DK effect:
    Tim Curtin!

  9. See the UPDATE at the end of the post.

  10. Ha Ha, Marco. TC riaises his head above the parapet again. And gives new meaning to the term “gone emeritus”.It’s not even a position bestowed by the univeristy – it’s just a club he’s joined!! Fabulous!

  11. RE: Dunning-Kruger

    I recommend that anyone with an interest in biases and errors have a read of Daniel Kahneman’s “Thinking Fast and Slow”, now published a few months. It provides several case studies of how the human race as a whole are not good intuitive statisticians. Undue overconfidence in one’s own intuitions is another facet of human psychology.

    The Dunning-Kruger effect (though he does not refer to it explicitly) is just one of many biases described by Kahneman.

    You will recognise climate science deniers as poster childs for many of the biases described by Kahneman, though even more useful is the ability to self criticise, if we could attain that humble state!

    • Bernard J.

      It’s a shame that Daniel Kahneman didn’t explicitly mention the Dunning-Kruger effect in “Thinking, Fast and Slow”. He could probably have bolstered much of his work by doing so.

      Still, it’s not on the level of omission as is, say, Gadsby, by Ernest Vincent Wright. It simply messes with my head to think how Wright achieved what he did!

      • Bernard J.

        Hi Tamino.

        The link in my previous post doesn’t work.

        Apparently it should be:

        If you could change it that would be great, but if the whole post is just off-thread and you want to delete it, that’s fine. And it goes without saying that this one can go straight to the bin!

  12. Brandon Shollenberger has a comment on his own post on this contretemps blaming sleep depravation, saying…”In other words, I’m a huge fool, and I’m unbelievably embarrassed.
    “I obviously cannot do calculations on annual data and compare it to calculations done on monthly data. Once I fixed that data handling error, I got the exact same graphs Tamino displays in his current post. This means he and I are now in complete agreement about what was done. To be clear, his description of the calculations are completely accurate.

    He goes on to say he will modify his post after a good night’s kip & then adds some comments showing that he is not actually “now in complete agreement.” But he does end by saying “Anyway, I apologize to everyone for making such a stupid mistake. I hope you can forgive me for it and not let my idiocy distract you too much from the real issue being discussed.

    • When someone waves the flag, respect it. Closure.

      Next issue.

      • Well, someone waving a white flag while taking a parting shot is somewhat different.

        Tamino has admitted to using a 28-month period in a graph of five-year (60 month) averages, and that caused the graph to curve up rather than down.

        And Lucia has doubled down:

        The facts that Tamino’s choice of f is very near the transition from where higher values show upticks and lower values, that the value results in a smoothing window that is now where near the size he uses in the graph that is supposed to make us feel confident in his undisclosed choice? Those make me think the failure to disclose the choice and it’s effect on the behavior of the end point more dubious not less.

        One has questions as to whether “closure” is possible here.

      • Yep. It definitely puts him one up on someone like, say, “Poptech.”

      • The facts that Tamino’s choice of f is very near the transition from where higher values show upticks and lower values…

        That’s sort of the point. Choose too small a value for f, and your smooth is unduly influenced by noise. Too large, and you lose features of the signal. An f which is just large enough to adequately deal with the noise, and no larger, is a good thing.

      • One has questions as to whether “closure” is possible here.

        …aaaaaaaand one was right to question.

      • Just to extract the, er, pertinent message from Brandon Shollenberger’s latest position within this contretemps. He says “So yes, I messed up in a big way, but that wouldn’t have happened if not for me trusting Tamino’s inaccuracies/deceptions.” This is after B.S. having had a sleep to allow him to reflect on this intractable conundrum that faces him. It’s always possible he may wake up again & find that his innate infallibility entirely in tact as it was all just a very bad dream.

        And this is still presumably qualified by his statement in The Blackboard post “Does any of this have any impact on the point of his blog post? No.
        Indeed, the post still has an amending preamble stating “I have a confession to make. I was wrong. Tamino was right.” So his post at “The Blackboard is getting rather like that old philosophical paradox – “This statement is a lie!”

      • I’ve read the followups. Did I get taken in or what?

  13. This is an example of a growing phenomenon in America–the extension of “democracy” into nonpolitical realms where it makes no sense. More and more Americans believe that any opinion on any subject is equally valid–whether or not the holder knows anything about the subject. The opinion of a right-wing political blogger on a science issue is just as reliable as that of a scientist who actually studies the issue.

    Picture a guy who never put two pieces of stone together in his life. He goes up to a thirty-year-veteran union stonemason. The latter is building a wall. Our hero say to the mason, “You’re doing that all wrong, stupid!”

    • BPL, that actually happens now and again. Someone wants a job done within a certain amount of time, for a certain cost, or in a particular way, so they push push push against the experts until the experts bend or break. How many high school principles know very little about the actual conditions of the classroom and instead force a particular model on teachers (to their extreme chagrin)? How much fudging takes place on the build site when the money says “do it this way”? And how often do powerful economic forces encourage people to think of themselves as experts, knowing full well that those people will fail at a certain rate, for a certain time, and to a certain extent–all to the profit of the force (direct sales, eTrade, etc.)? When moral authority, epistemological authority, economic authority, and cultural authority begin to converge in the same group (is Monckton the uber-DK of this condition?), one must lash oneself to the twin masts of history and science with the rope of integrity, and get ready to kick the “masters of reality” in their reproductive nodes, literally and figuratively. There: summer’s coming early, and I needed enough pith for a new summer helmet.

      • dsl350:

        BPL, that actually happens now and again. Someone wants a job done within a certain amount of time, for a certain cost, or in a particular way, so they push push push against the experts until the experts bend or break.

        Challenger, cold weather, o-rings, boom.

        Management push against the engineers …

  14. L. Hamilton

    Approaching with a different toolkit (Stata) and defaults, here’s another consistent-although-not-strictly-comparable view of smoothing. I simply took NCDC annual averages and drew four graphs, the first using lowess tricube with bandwidth=80% of the data (Stata’s default). In the next three panels I narrowed the bandwidth (still tricube weighting) to find out what it takes to get the graph Shollenberger wanted to see. Only at bandwidth=40% or below does it start turning down at the end, tracking those last four years.

  15. Edgar Walsh

    “Naturally, I suspected that graph was questionable too, ”

    Of course he did.

  16. TrueSceptic

    I must say that Shollenberger had me fooled with his first comment in the Harris thread: it looked like a genuine request for explanation. His subsequent comments quickly gave him away, however, as someone who simply refuses to learn (another definition of D-K).

    On another point, is it plausible that someone who frequents climate blogs and writes “articles” on climate science had never heard of Tom Harris?

  17. Horatio Algeranon

    “Tiger Tailer”

    — by Horatio Algeranon

    I’m a tiger tailer
    By trade, a wagger-trailer
    I track the trend
    In the tiger end
    A vector,
    Not a scalar.

  18. I agree with what Ron Broberg posted above. Ron also posted something in the comment thread at Lucia’s that people ought to keep in mind.

    On the original issue — the content of Tom Harris’s course — it’s obvious that Tom Harris is badly misinformed on climate issues, and thanks to his ill-considered teaching appointment, his students are now misinformed as well.

    But a little more clarity in the posts, and a little more patience in the comments, would have been preferable IMVHO. Endpoints in a time series present some thorny issues, and while there are some obviously wrong ways to handle them, there’s no universally agreed upon right way. It would be good to recognize right up front that different analytical choices will lead to different appearances at the end of those graphs.

    There are undoubtedly some people you’ll never be able to reach. But people who really are open-minded are more likely to be convinced when the methodological choices are clear and people who raise questions about them aren’t slapped down quite so aggressively.

    For example, in the graph of five-year averages in the original post, all the data points are based on 60-month intervals except for the last one, which is based on 28 months. Making that clear up-front would have taken only a few words, and might have reduced the opportunity for confusion. I do notice that the final data point has wider confidence intervals, and is spaced differently than the previous points, so there are subtle indications in the graph itself that this point is different. (I don’t quite understand the spacing — 28 months is less than half of the 60-month interval, but while the final point is spaced more closely than the preceding points, it’s not 50% closer…?) Alternatively, one could do 60 month intervals that run up to the present (e.g., 4/2007-3/2012, 4/2002-3/2007, 4/1997-3/2002, etc.)

    Not trying to second-guess all your choices, or say that you should have done more work, or come across as a “concern troll”. I just think that the previous post and the discussion that ensued represented a bit of a missed opportunity.

    [Response: You don’t come across as a concern (or any kind of) troll, neither did Ron Broberg. But I think you both utterly miss the point.

    It’s just not possible to make a post satisfy all readers in terms of detail — what’s too little for some is too much for others. In my opinion, it’s easy for long-winded explanation to become tedious, and that makes for poorer communication in the long run. You’re free to disagree, or agree with my general statement but think I miss the mark. Different strokes.

    The point is this: Shollenberger asked and I answered. But he wouldn’t accept the answer. It became increasingly obvious he wasn’t going to, no matter how much explanation or detail I provided. He turned from inquisitive to inquisitor, which makes the suggestion of a “little more patience” very bad advice.]

  19. [Response:…]

    Thanks for the reply.

  20. If Shollenberger want’s to understand smoothing, he should try a simple smooth, i.e. an exponentially weighted moving average. The smooth is just a state variable, i.e. it has the differential equation
    dsmooth = (data-smooth)/tau
    It happens to be the optimal forecast for a noisy time series under some conditions, and with a decent choice of tau is a reasonable general heuristic.

    Consider the time series

    A smooth with time constant tau=2 updates halfway toward the data at each point:

    Notice that the last point of the smooth goes up, though the data went down. This is just an instance of bathtub dynamics ( ).

    A fancier smooth, like Lowess, uses a more general weighting scheme, but it’s not fundamentally different, and therefore is quite likely to go up when the data goes down, violating naive (but wrong) intuition.

    • Bernard J.

      Or one could point out the statistics underpinning the circumstance “best-of-three; OK, best-of-five; no, best-of-seven…”

      I’m guessing that right now Shollenberger knows exactly how that feels. If he can extrapolate from that, to the tail of that damned curve, he might actually learn something.

  21. Huh. So I downloaded that USHCN lower-48 data set.

    The most recent 12 months (May 2011-Apr 2012) are the warmest 12-month period in the whole data set (not just the warmest May-to-April period, the warmest any 12-month period).

    But May 2011 was quite cool in the lower 48 … so it’s highly likely that the next 12-month period (June 2011-May 2012) will be even warmer.

  22. The D-K effect is truly a trademark of climate fake skpetics. This does not happen in other fields of science.

    If you explain a Smith chart to a layman, and he does not understand, he says “sorry, it’s too complicated for me to grasp it.”

    If these self-proclaimed ‘skeptics’ don’t understand climate science, they write blog posts, articles or even books saying “it’s a hoax”.

    • TrueSceptic

      I’m not sure about that. There are many who refuse to believe that Relativity can possibly be true, and even more so when it comes to Quantum Mechanics.

      I think that the crucial difference is that the findings of climate science have a huge bearing on our long-term way of life: we simply cannot carry on as if we have no effect on the environment.

      • You may be right. Maybe I should add “mostly” or “in my experience” to my first paragraph above.

        Your alternative explanation is ofted cited by those who try to understand denialism, but I think the skeptics’ reaction is less elaborate and more immediate than that. To me, I seems more like the “us versus them” or “left versus right” attitude. If you start saying that Al Gore is a moron, and the solution is something along the lines of negotiable rights of pollution, then they’re much more prone to agree, even if it implies in potential change in lifestyle. Just my personal perception.

  23. Musing naively about all of this, I’m struck how often the ‘noise’ is specifically the high-frequency stuff, when in principle it could perhaps be the other way ’round–that is, you might want to remove low-frequency information to look at shorter timespans. (Of course, that’s in essence what’s done in de-trending and calculating residuals.)

    Are there general issues involved here–perhaps in terms of what humans tend to define as ‘interesting,’ or perhaps in purely statistical terms?

    • Bernard J.


      For a moment there I thought that you were talking about the noise of denialism…

      That’s something that definitely requires filtering.

  24. Tom Passin

    Since I wrote my first LOWESS routine some 20 years ago, I’ve found that you have to analyze a *lot* of data, a *lot* of different ways, before you can really get a good sense of what’s real and what’s noise/artifact/whatever. It helps if you get to wrestle with data sets that all your attempts fail on. It helps a lot if you generate your own data sets with known signal and noise.

    This process is your apprenticeship. If you skip it, you can get overconfident about your notions of what data really shows, and how significant variations are. Tamino does a great job of easing the rest of us through some of these stages. Thanks, Tamino!

    BTW, my implementation of LOWESS uses a gaussian weighting function, such that at the edge of the specified window, the weight is down by 2 sigma. There’s no really good justification for this weighting function, except that it was easy to write, and it worked well in my first implementation so I kept porting it to newer ones.

  25. What it really is is the Wikipedia effect, where when an expert writes in she gets trashed by the locals.

  26. Horatio Algeranon

    “Stupid’s Arrow”
    — by Horatio Algeranon

    “You’ve got an arrow in your butt”
    Said Doc, ” I will remove it.”
    “Nay, tell me Doc, which way it points.
    So Heartland will approve it.”

  27. Horatio Algeranon

    Not sure about LOESS filter, but with the Savitzsky-Golay filter (which is also based on a local least squares polynomial fit to a function) , the “best” filter width (to filter out noise and preserve signal) is between 1 – 2 x the width (FWHM) of the “features” of interest (eg, spectral peaks). Any “peaks” with width less than that are effectively filtered out (ie, treated as noise).

    Given that the “peaks” for “weather noise” in global temp (eg, from El nino, la nina and volcanic eruptions) have a duration roughly of the order 1-2 years, a width of “a bit less than 25 time units” (25 months) would certainly seem to be in the right ballpark (at least for Savitzky-Golay) if one wishes to filter out the “noise” and largely preserve signal.

    So, one actually has to understand the characteristics of the noise and signal to do optimal filtering (imagine that!)

    What may seem like arbitrary choices of filter width to someone who has no experience with this stuff are actually well justified.

    And one really has to wonder about those who know (or certainly should know) about this stuff and still persist in purposely creating doubt.

    Finally, some may be better than others but all filters have problems near the endpoint (because you only have data points on one side).

    And focusing on what the graph is doing near the endpoint has more than a little in common with focusing on short term trends obtained with linear least squares. So it should probably come as no surprise that it is some of the same folks focusing on both.


    • Tom Passin

      “Finally, some may be better than others but all filters have problems near the endpoint (because you only have data points on one side).”

      Right. Well, you can write a filter (even a LOWESS-sytle filter) that only uses data from the past of the current point, that is to say a causal filter, of which a typical moving average filter is an example. The more usual LOWESS implementation uses a window centered about the current point (a non-causal filter, since the value at a point is affected by future data values), but when you approach the endpoint at the most recent end, the filter properties start to switch over to a moving-average style as you get fewer and fewer data points from the future of the current point. This changes the autocorrelation properties and other time behavior of the smoothed points near the end of the the data. So you can’t pay *too* much attention to the details of the smoothed curve near the ends of a data set.

    • Is it not also the case that any smoothing applied to a series which has a long-term upward trend is likely to give (provisional) values, (for annual temperatures for example), towards the end-point of the graph that are lower than what their final values will be in the future when they are no longer at the end of the graph?
      For example, HADCRUT3, here: is smoothed with a 21 point binomial filter in which the smoothed value of a particular year is calculated from the ten years either side of it, weighted according to Pascal’s triangle such that more distant years have lower weightings. For the last ten years of the black smoothed curve, where there are not ten years of data beyond the year in question, the smoothed value of a particular year is calculated using (increasingly) the unsmoothed value of the final complete year of the series, since, obviously, future values do not yet exist. Since the long-term trend is upwards that final year value is likely to be less on average than the subsequent years that will eventually replace it in the calculation of the smoothed value, thus giving provisional smoothed values lower than their final value.

      Are more sophisticated smoothings able to overcome this ‘end droop’ of series with long-term positive trends, or is it a problem with all of them?

      [Response: It is indeed a common problem with smoothing methods. However, Lowess doesn’t exhibit that behavior.]

    • Exactly!

      All finite time series data suffer from the end point problem, so larger error bars must be assigned as we approach either end point, as we only have half the information available as compared to the interior data given any particular filter bandwidth.

      Instead of discussing f = 0.15 or f = 0.2, I rather we kept to the halfpoint analogy used in FIR/IIR filter design, that being 1/N data points (or conversely an N-point moving average (as the simplest example of an FIR filter)). That way no matter the run length of the time series, the same halfpoint chariteristics are always used throughout (the degree of the filter also defines the filter attenuation characteristics, so the problem is never as simple as some would like).

    • TrueSceptic

      It’s worse than that. He’s accusing Tamino of making things up.

      He just makes things up, trying to smear me by implying I haven’t read the post I discussed and quoted from. Why?

      The idiot also claims it’s got nothing to do with Tom Harris.

    • sod
      Brandon may proclaim “I was right!“. He may even argue (as in his most recent comment #95636 replying to you, one of so many comments he makes over there) to the position where he feels he can say “…I was right. I just wasn’t always explicit about describing my intuition because… it’s intuition.” but, cut through the nonsense, primacy still goes to the amending preamble of that post over there that establishes “I have a confession to make. I was wrong. Tamino was right.
      While the veracity of Tamino’s position is thus established, that of Brandon’s remains unclear. Branden is probably both right and wrong, a situation which can occur when you are continually shifting you position.

  28. Horatio would say that you might have a new candidate for Mathturbation King (the Queen is well established and quite safe)

    Diddling the window adjustment factor for LOESS, LOWESS (or other filters) with no regard for the characteristics of the relevant signal and noise (as Shollenberger and others are doing) is the very epitome of mathturbation.

    The tip of the graph even moves up and down with the diddling.

    Sure can’t beat that!

    The latest “debate” about what end of the time series is ‘doing” is simply absurd given the uncertainties which are quite obvious on tamino’s graph of the five year averages

    Talk about cycles.

    This is “short term trend” deja doo-doo all over again.

    • Rattus Norvegicus

      I don’t think that anyone will be able to take the crown and scepter from Nicola Scafetta. He is truly the reigning and undefeated champion.

      • Horatio Algeranon

        Well perhaps there is room for more than one king.

        The more the merrier, right?

        We three kings of Disorient are
        Bearing graphs we traverse afar.
        Foolishness fountain, molehill into mountain,
        Following McIntyre

      • Rattus Norvegicus

        Thanks, you made me laugh out loud. And I don’t mean LOL — really, I was laughing out loud.

      • “The jester stole his thorny crown??”

  29. Michael Hauber

    Even Tamino’s chart has a downturn around 1960. Obviously global warming is a hoax…..