Bayes vs Frequentist

I hate to link to XKCD for the second comic in a row, but the most recent one really struck a chord with me.  While it is a little unfair, it hits what is to me the essential weakness of frequentist statistics: that the standard null-hypothesis rejection only considers how unlikely something is to happen by chance.  In contrast, the Bayesian analysis weighs the different possibilities using prior information.

Also, the willingness of the Bayesian character in the strip to place a bet relates to the characterisation of probability as degrees of belief; intuitive, meaningful in a Bayesian approach, but impossible under a strict frequentist interpretation.

Statistics

I'm planning another post on a recent research paper, but in preparation I want to talk about statistics.  Specifically, I want to talk about Frequentist versus Bayesian perspectives for the interpretation of experiments.  I won't get technical, or go into the actual details about how calculations are done, but rather talk about philosophy.

My casual impression, based on reading papers to stay abreast of the field, is that most experiments use Frequentist methods in analysing their data.  In this approach, discovery of new phenomena is based on disproving the null hypothesis, the assumption that there is nothing to discover.  In this sense, Frequentist methods are very Popperian.  Frequentists will argue that this ensures their methods are objective, which is more or less true.¹

The problem with Frequentism is that it has a tendency to be misinterpreted.  For example, let's say in a particular experiment we can exclude the null hypothesis at 95% confidence level.  What does that mean?  It is tempting to interpret it as saying that there is a 95% probability that the null hypothesis is false.  However, this is wrong.  The strictly correct statement is: if the null hypothesis is true, the probability of getting this experimental result is 5% or less.