In the last post I discussed what I thought was a mistaken application of Bayesian analysis. I didn’t claim that Bayesian analysis isn’t appropriate for the problem, in fact I showed the kind of Bayesian analysis which I think is appropriate. But some readers objected to my claims; I think we have some Bayesian zealots out there.
Let’s try again. Two treatments, A and B, are applied to attempt to forestall a disease (“microsoftus”). B is just a placebo, and was given to 10 subjects, 3 of whom were subsequently infected. A was given to 30 subjects, only 1 of whom came down with the disease. The Bayesian analysis I objected to concluded that the probability treatment A was better than B is 99% — definitely significant (one might even say strongly so). A frequentist analysis using an exact test rejected the null (that treatment A has no effect different from B) at 95% confidence, but just barely so.
Let’s change the numbers a bit. Only one subject was given treatment B (placebo), and the unfortunate soul subsequently contracted microsoftus. Only four subjects got treatment A, none of whom got the disease. Do we have statistically significant evidence that treatment A works better than placebo?
The scientist conducting the trial starts with:
We scrap the hypothesis that the two treatments have exactly equal effectivenesses, since we do not believe it.
Then he does the exact same analysis I objected to earlier, which led to 99% confidence in the former case. In this case, the probability that turns out to be 0.9524. That's more than 95%, so he says we have significant — publishable — evidence that the treatment is working.
One of the referees for the paper is a frequentist, who says that this is mistaken. One of five subjects got the disease. Under the null hypothesis, the chances that the diseased individual was the one of five who got treatment B is: one of five. That might be good at 80% confidence, but it’s nowhere near 95%. No significant result yet.