Statistics: not for Cliff Mass

It all started with Cliff Mass saying “… with huge transient peaks and troughs (see below). With such variability …” when talking about CalFire data of area burned by wildfire in California from 1987 through 2016. His comment led me to make a very serious mistake: I thought he might actually know what he was talking about.


I figured he must be talking about the fact that the “huge transient peaks and troughs” indicated that the noise in the data was severely non-Gaussian. That would weaken a trend test like ordinary least squares (OLS), which might suggest a significant trend when there is none or fail to detect a significant trend when the data actually demonstrate it.

So I did OLS, then subjected the residuals to the Shapiro-Wilk test for normality. Result: they definitely don’t follow the normal distribution, and when you can show that with just 30 data points you shouldn’t ignore it. Fortunately, there are many robust trend tests. My favorite is the Theil-Sen method, so I used that. Result: the trend is statistically significant.

There are other ways. For example Keeley & Syphard, when analyzing annual data, decided to log-transform the data and anlyze those values. Just for fun, I did the same with the CalFire data. Result: the trend is statistically significant. The p-value is 0.0462, that’s 95.4% confidence. Also, the Shapiro-Wilk test applied those residuals does not demonstrate departure from the normal distribution — so ordinary least squares is not a weak test.

But if you use ordinary least squares on the raw data the p-value is 0.0722. That’s significant at 92.8% confidence, but not at 95% confidence.

Here’s something for Cliff Mass to think about: why is it that when you use a test which is demonstrably weak it doesn’t give 95% confidence, but when you use a test which is not demonstrably weak is does give 95% confidence?

Cliff Mass finally got around to doing a statistical test, after he had drawn a conclusion with no test at all. Here’s what he has to say about it:


Cliff Mass said…

Several of you are asking about the significance of the trend in acreage (e.g., Sofistek). I did the trend significance analysis today, using the same approach I and other used in our peer-reviewed paper on snowpack trend (found here, https://journals.ametsoc.org/doi/abs/10.1175/2009JCLI2911.1, based on the approach of Casola et al., 2009). This type of trend analysis takes in consideration the variability of the time series (which is very important). I found that that trend over the entire period was NOT statistically significant (generally we used the 95% level…a trend is considered significant if there is less than a 5% chance of it happening by chance).

A global warming activists (someone named Tamino) claims otherwise, but his method is not appropriate (I won’t get into the technical issues here). He also likes to mock folks he disagrees with, which is not the way scientists interact with each other….cliff

August 12, 2018 at 5:17 PM

If you read the paper he refers to, then the one by Casola et al., you discover that the approach used is: ordinary least squares. The only thing Casola et al. do to it is combine some algebaric manipulation with a simplifying approximation (which weakens the test but not by much).

So the method Cliff Mass used was: ordinarly least squares. Seriously. That’s it.

Now for the truly bizarre part: Cliff Mass keeps repeating that “This type of trend analysis takes in consideration the variability of the time series (which is very important).” Cliff, they all do. Least squares, Theil-Sen, L1 regression, Mann-Kendall, you name it — they all do.

When Cliff Mass talked about “the variability of the time series” I thought he was talking about something other than the fact that there’s noise in time series and every statistical test takes that into account. Giving Cliff Mass too much credit — that was my mistake.

Here’s my prediction: Cliff Mass will probably not respond to this criticism of his faulty analysis, but if he does, he’ll be sure to end his comment with an ad hominem comment about me personally.


This blog is made possible by readers like you; join others by donating at My Wee Dragon.


Advertisements

21 responses to “Statistics: not for Cliff Mass

  1. For someone who actually knows stats, it would be embarrassing to accuse someone of using a method not tailored to the error structure of the data, just to be caught doing precisely that. I can’t say that Mass would find that embarrassing though.

    FWIW the OLS estimate isn’t necessarily bad. Depending on what we want out of it, it may only be sub-optimal. If the expected value of burn acreage is a linear function of the input variables, then the OLS estimator is unbiased; if there is no autocorrelation (there probably is), then OLS is also consistent. The data y|X are not Gaussian so the estimate is not very efficient, which may be a pointless observation if there is no consistency.

    But given what type of data this is, I would say that we should try to find a better estimator than OLS. The assumption of linear relationship seems tenuous, I’d bet money there’s real autocorrelation, and the errors are certainly not Gaussian. The mean and variance are probably correlated. A log-normal GLM (essentially OLS on the log-data) is conceptually better. A Gamma regression would also be better.

    [Response: I too suspect there is autocorrelation, but the Box-Pierce and Box-Ljung tests fail to establish it, so at the least it’s not very strong.]

  2. I appreciate this blog’s careful description of the various adequate and inadequate approaches to trend analysis. I’m a regular reader and have learned a number of interesting things here. This time around, I have a standard comment to add that I felt was not mentioned or only in passing: the debate is missing the big-picture view, which in my opinion is “attribution”. A significant trend does not necessarily mean attribution. It’s often a strong indicator in climate change analysis, but it need not mean (maybe even can’t mean) attribution. Since we learned about the large internal variability in the climate system that can occur over a time period of even just three decades, I’ve started to not trust a significant trend itself. As often as possible I’m instead turning to ensemble simulations to verify how often I might get a significant trend by chance. For example, we recently showed that the declining water year precipitation trend in the US Southwest over the recent decades is significant at the 95% confidence level, yet, it cannot be attributed to anthropogenic climate change at all (https://flaviolehner.files.wordpress.com/2018/07/lehner18grl.pdf). How about that?

    Of course, it is almost impossible to create a trustworthy ensemble of simulations of area burned, since it depends on so many factors, some with a strong random component (precipitation trends, weather sequencing, humans), others with a more systematic component (temperature trends, fire suppression, population growth), and many poorly constrained. So that’s a challenge. However, broadening the discussion from the details of trend analysis (very valuable from a technical perspective!) to the big picture could make this blog more attractive to other readers. I think the blog deserves it. Meanwhile, I completely agree that Cliff Mass is once again struggling to see differing views as valuable, and might once again be plain wrong.

  3. Cliff: “He also likes to mock folks he disagrees with, which is not the way scientists interact with each other.”

    No. It’s the way scientists interact with fricking poseurs.

  4. I read Cliff’s comment late last night.
    From experience in the climate shuffle you know that, as Cliff failed to give a value after crunching the numbers, it would be very close to the arbitrary 95% level.
    Honesty is never a factor in their evasion of reality.

  5. http://www.rifters.com/crawl/?p=886

    ====
    what I want to address here is the attitude of the scientists, and how that relates to the way science actually works.

    I keep running into recurring commentary on the snarkiness of the scientists behind these e-mails. They’re really entrenched, people seem surprised to note. Got a real siege mentality going on, speak unkindly of the skeptics, take all kinds of cheap shots unbecoming of the lab coat. These people can be downright assholes.

    No shit, Sherlock. I was a scientist myself for the longest time, and the people I’d gladly drop into a vat of nitric acid start with the Pope and go all the way down to anyone who voted for Stephen Harper’s conservatives.

    The apologists have stepped up, pointed out that these were private conversations and we shouldn’t expect them to carry the same veneer of civility that one would expect in a public presentation. “Science doesn’t work because we’re all nice,” remarked one widely-quoted NASA climatologist. “Newton may have been an ass, but the theory of gravity still works.”

    No. I don’t think he’s got it right. I don’t think most of these people do.

    Science doesn’t work despite scientists being asses. Science works, to at least some extent, because scientists are asses. Bickering and backstabbing are essential elements of the process. Haven’t any of these guys ever heard of “peer review”?

    There’s this myth in wide circulation: rational, emotionless Vulcans in white coats, plumbing the secrets of the universe, their Scientific Methods unsullied by bias or emotionalism. Most people know it’s a myth, of course; they subscribe to a more nuanced view in which scientists are as petty and vain and human as anyone (and as egotistical as any therapist or financier), people who use scientific methodology to tamp down their human imperfections and manage some approximation of objectivity.

    But that’s a myth too. The fact is, we are all humans; and humans come with dogma as standard equipment. We can no more shake off our biases than Liz Cheney could pay a compliment to Barack Obama. The best we can do— the best science can do— is make sure that at least, we get to choose among competing biases.

    That’s how science works. It’s not a hippie love-in; it’s rugby. Every time you put out a paper, the guy you pissed off at last year’s Houston conference is gonna be laying in wait. Every time you think you’ve made a breakthrough, that asshole supervisor who told you you needed more data will be standing ready to shoot it down. You want to know how the Human Genome Project finished so far ahead of schedule? Because it was the Human Genome projects, two competing teams locked in bitter rivalry, one led by J. Craig Venter, one by Francis Collins — and from what I hear, those guys did not like each other at all.

    This is how it works: you put your model out there in the coliseum, and a bunch of guys in white coats kick the shit out of it. If it’s still alive when the dust clears, your brainchild receives conditional acceptance. It does not get rejected. This time….
    =====

    Weather people must be much nicer to each other, eh?

  6. Thanks Hank, I’ve made frequent reference to Peter Watts’s piece since you linked to it on Deltoid nine years ago, subsequent to the CRU email hack.

    “Science works, to at least some extent, because scientists are asses. Bickering and backstabbing are essential elements of the process. Haven’t these guys heard of ‘peer review’?”

    Great stuff 8^D!

    • Yes.

      One of the most annoying habits of deniers on all sites is their propensity to lecture professionals on how science “really” works and how “real” scientists should think and act using all their deep knowledge of how science is done. And not just on technical details. Oh no. For example we get lectures that scientists need to “prove” things without resorting to a supposedly dreaded “appeal to authority”. Or, we get lectures on just how to implement the hypethetico-deductive method appropriately. And even that the hypothetico-deductive method stages must be followed as described in high school texts. We hear a lot of turn-of-the-last century naive Logical Positivism. We hear lots about Popper but nary a word about Pepper.

      Over on RC just yesterday a denier talked about how the tobacco-cancer connection was established in double blind clinical studies which makes that finding ever so much stronger than the claims of climate scientists who cannot perform such experiments!

      Argghhh.

      • Ah, that would be “Victor”. I responded to his double-blind randomized nonsense, hope it gets through.

        They didn’t (at least I think so) let an earlier comment through where I showed using his own words that Victor’s dismissals are clearly due to his own desperation: he cannot see a solution to the climate change problem as presented by climate scientists that does not involve complete destruction of the economy. And hence he has two option: despair or deny. We know what he has chosen.

      • …as I was saying, before my palm inadvertently brushed the trackpad, when investigated, the ‘climate scientist’ had impeccable credentials, all right–as an electrical engineer.

        (A discipline whose intellectual demands I certainly don’t disparage–but on the other hand, I wouldn’t call Mike Mann if I needed someone to design a super-duper power inverter.)

  7. I feel like I’m watching a statistics professor get into an argument with a particularly obtuse undergraduate with a Dunning-Kruger issue.

  8. Looking at the residuals after OLS, my non-statistician’s eye makes me wonder if the normal distribution does sort-of fit those residuals in that because there is massive variability and because a negative level of burning is inadmissible, then it is transformed into a Poisson’s distribution, a form I always understood to be what a normal distribution turns into when it is constrained by the axis.

    [Response: No, a Poisson distribution is something very different. You might be thinking of the log-normal distribution? In any case, the Shapiro-Wilk test demonstrates that it’s definitely not normal, not even approximately. Maybe the “take-away” from your impression is how hard it is to tell “by eye” when there’s so little data.]

  9. Thought i’d throw in my two cents.

    I’ve been aware of Cliff Mass’s obtuseness and denial of what’s in front of his face for years. As one meteorologist to another, I greatly resent his obfuscation of reality in his efforts to push the idea that what we’re seeing globally is no big deal.

    Keep pouring it on, Tamino! At least you’re doing humanity a great service by exposing the idiocy and duplicity of all these ideological disciples of Ayn Rand who shamelessly subvert real science in their quest to further their ridiculous faith in the current status quo.

    • Lamont Granquist

      I’m not sure Cliff Mass is motivated by right wing ideology.

      He’s more of the old man shouting at the clouds who has a terminal case of needing to be contrarian. And the problem is that where he’s successful is arguing against reporting that comes out on the nightly news, which is legitimately always sensationalized and overblown about literally everything, so climate change is included. He can often be right that they’re spouting nonsense.

      But it spills over into actual scientific debate where he doesn’t believe california wildfires are increasing or have anything to do with climate change or where for several years now he’s been denying arctic amplification because one of his grad students wrote a paper. He also makes the argument that the 2C global warming only means you add 2C to the lows, highs and median temperature uniformly, which is a hopelessly naive viewpoint of climate change. He must be late 60s based on when he graduated from college and I think he has that disease of old scientists where they think they know everything and have forgotten how to learn.

      • Lamont Granquist,
        You talk of “the argument that the 2C global warming only means you add 2C to the lows, highs and median temperature uniformly” which is quite a powerful one for a US weather presenter. It is perhaps a revised version of the classical Dickie Lindzen “thin red line” argument which compares the level of AGW-to-date (back in the day, that was well less than 1ºC) with the weather experienced by Lindzen in Massachusetts on a spring day – the diurnal range and the daily max/min record temperatures with a bit of spring warming thrown in for good measure. AGW appears on the graph as a very thin red line. (Of course, as a climatologist, Lindzen was taking the piss with this disingenuous argument. Indeed he is on record saying it would have to be “twenty-times greater” to be “remarkable”. That would be a global warming of +12ºC.)
        Yet in revising the classical argument, Mass has broken it. Firstly, why is it only 2ºC he speaks of? It will easily be 4ºC unless we take it seriously and switch off our global emissions very quickly. Is Mass advocating such actions? Or is he arguing against it?
        And secondly, even if it is only 2ºC, the effect on the billions living in the tropics (who are mainly not responsible for AGW) is massive. A 6ºC rise will make the tropics literally unlivable outside air-con and that 2ºC which would be no-big-thing on a US weather chart, something the population wouldn’t even notice (unless they work outdoors), is thus advancing a third the way to making someone else’s environment literally deadly. In the tropics the change is going to be a very big thing and very noticeable for all who live there outside a perpetual air-conned environment.

      • Right. Just to dot the ‘i’ for casual readers, the problem with the ‘just add 2 C’ argument is that it ignores the frequency with which temperatures recur.

        It might be true that your maximum high only increases by 2 C. But because of the shape of typical temperature distributions, the frequency with which present maximum highs occur can increase by many multiples.

        Here’s a (qualitative) illustration:

        So if those current maximum temperatures are/were sufficient to cause distress now/previously, there will be/is a heck of a lot more distress under climate change.

  10. Ralph Peterson

    Tamino. Perhaps you have a better chance of engaging with Cliff Mass and other scientists if you refrained from unnecessary (and quite frankly, childish) name calling— like accusing Cliff Mass of putting a burning item on folks doorstep. I also wish you would evaluate the source of his information and get beyond an arcane discussion of significance of a linear fit to the data–which seems silly in any case (who says a linear variation means anything?)

    [Response: When he puts shit on your doorstep I’m not going to call it a rose to spare his feelings. This isn’t a case of a scientific dispute, it’s *another* case of Cliff Mass exhibiting astounding statistical naiveté and/or outright deceptive cherry-picking to support his persistent agenda to deny any harmful effect of climate change, then responding to correct and purely scientific criticism with total bullshit. As for other scientists, I have no problem engaging; we sometimes disagree, but they are at least honest about it and we often learn from each other.

    Who cares about a linear fit? Cliff Mass did, for one. And it’s far more important than you may think; it doesn’t prove a linear trend, but it does demonstrate that it’s going up, not down and not staying the same.

    You stand a much better chance of not being thought a “concern troll” if you stop acting like one.]

  11. twofeathersuk

    It’s climate change not just global warming; therefore we would expect wetter years as well as drier periods? Can a stronger statement be supported statistically about the increased variance of the figures?

  12. Philippe Chantreau

    It seems to me there is a larger picture to this. I wonder how the situation is if we look at the Pacific coast including British Columbia and Alaska. BC has 600 active fires right now and we are seeing terrible air quality in Southwest WA because of that. Their worst fire season was last year, with 65,000 people evacuated and some very large fires. I don’t have the statistic chops but it would be interesting to take a look at that bigger picture.

  13. I’m not able to evaluate technical/statistical arguments, but any discussion of long term trends in US wildfires, IMO should, obviously, include mention of trends in resources devoted to, and efficiency in, firefighting (as Zeke mentions in his article that was linked downstairs). For anyone serous about these discussions to not do so (excepting discussions such as this one, which are focused on evaluating specific statistical analyses) is inexcusable, IMO.

    Can anyone link to where Cliff discussed those mitigating factors?

  14. O/T, but begs for assessment here–a new paper outlining a novel method of probabilistic interannual forecasting of GMST and SST (representing what the authors term a “severe truncation” of the phase space, i.e., only that one parameter is considered.) Also–and give chutzpah points here–they state that this method depends upon 4 assumptions, none of which are strictly true (though they think all 4 are ‘close enough’, as evidenced by assessed hindcast skill).

    Perhaps gratingly, they use the “p” word–that’s “pause” for causal readers–in connection with the “post-1998 decade”, and claim skillful hindcasting of it. To be completely clear, they are attempting to model variability via machine learning, so the implication would be that they attribute the so-called pause to internal variability, too.

    Intriguing, but a lot of it is over my head, so I–and probably a lot of folks–would be very interested in more expert reactions.

    https://www.nature.com/articles/s41467-018-05442-8

    (I’ll cross-post at RC as well.)

    [Response: I saw the paper and read it, and thought of posting about it. I guess I’ll move a bit faster on that.]

    • I’m interested in a post on this one too.

      Post-1998 did see evidence of a cooling contribution from natural factors – plenty of evidence links this to changes in cloudiness and heat uptake. Of course the obs still fell within our expected range for forced warming + natural variability.

      If their statistical model actually does help explain some of the natural variability component, that would be cool.