3 Foundational disagreements: interpretations of probability theory
Notions of chance...are the preoccupation of men engaged in rash undertakings.
Blood Meridian, by Cormac McCarthy
It is unanimously agreed that statistics depends somehow on probability. But as to what probability is and how it is connected with statistics, there has seldom been such a complete disagreement and breakdown of communication since the Tower of Babel.
— Leonard J. Savage, The Foundations of Statistics
In the last chapter, we analyzed two attempts at solving the problem of induction. The first Popper’s falsificationist solution, systematized by statisticians as hypothesis tests. Such hypothesis tests—along with confidence intervals, maximum likelihood estimation, and many other statistical modeling methods—are often labeled as “frequentist methods”, for reasons that will become clear later in this book. The second attempt at solving the problem of induction was Bayesian probabilism, named for its use of Bayes’ theorem. Frequentist and Bayesian methods are not just proposed solutions to the esoteric problem of induction; almost all inference and modeling methods used in contemporary statistics and data science fall into one of these paradigms.[^1]
Frequentist and Bayesian methods have some similarities. Common to both paradigms is the use of probability theory, perhaps in conjunction with some “substantive” scientific theory, to model the observed data (Howson & Urbach, 2005; Spanos, 2019). Suppose that we observe data \(\mathbf{x} = (x_1,...,x_n)\). The typical assumption is that the data are realizations of a stochastic (probabilistic) process \(\mathbf{X} = (X_1,...,X_n)\). For example, let \[\mathbf{x} = (1.0, -2.6, -4.8,1.2,-1.5,-1.3,-2.3, 1.1, 3.5, 6.2),\] be the change in body weight over six weeks for \(n = 10\) individuals randomly sampled from the Mandelbaum Recreation Center.[^2] The random sampling process could have produced other data values, e.g., \[\mathbf{x} = (2.0, -4.5, 4.4,-1.5,-1.1,-2.8, 0.6, 0.1, -7.2, -1.2),\] if different individuals were selected for the random sample. The random sampling process is modeled by a set of random variables, \(\mathbf{X}\), that give way to \(\mathbf{x}\). Probability theory provides the rules by which \(\mathbf{X}\) varies.
From there, frequentist and Bayesian methods diverge in their use of probability theory. Frequentist methods only use probability as described above: to model the way that data vary. Any probabilistic statements used in a frequentist method are downstream of that data model. The Bayesian paradigm takes a similar approach to modeling data as realizations of a probabilistic process; but Bayesians also use probability in another, very different way: to quantify uncertainty in hypotheses about statistical parameters. This quantification of uncertainty in hypotheses happens through the prior and posterior distributions. Bayesian methods assign probabilities directly to the hypotheses in question. In our example of Bayesian probabilism in Chapter 2, the prior distribution quantified our degree of belief in \(H\): a high carbohydrate diet causes an increase in body weight before observing data. Similarly, the posterior distribution quantified our degree of belief in \(H\) after observing data.
What justifies this Bayesian use of probability? Can probability really model an individual’s subjective degree of belief? Isn’t probability theory a mathematical theory, and thus, in some sense, objective? Many popular textbooks define probabilities, not in terms of subjective degrees of belief, but in terms of relative frequencies (for example, see Ugarte et al. (2016)). Is that the best way to interpret probabilities?
The interpretation of probability sits at the core of debates about frequentist and Bayesian inference. But how we interpret probabilities matters in other ways, too. Statisticians and non-statisticians alike use probability statements all the time. Consider the following claims:
The probability that a fair coin will land on heads is \(0.5\).
The likelihood of being dealt a flush in 5-card poker is approximately \(0.2\%\).
The odds are \(1.1:1\) that the Colorado Avalanche will beat the New York Rangers.
For a particular woman about to undergo IVF treatment, the probability that she will give birth to a healthy child is \(1/3\).
There’s a \(50\%\) chance of rain in Boulder, CO tomorrow. How are we to interpret the words ‘probability’, ‘likelihood’, ‘odds’, and ‘chance’ in the claims above? What does it even mean to interpret probability? In this chapter, we will consider some answers to these questions; our goal will be to gain a deeper understanding of probability theory, both in the use of statistical inference, and our use of probability in our everyday lives.
3.1 What is an interpretation of probability?
The mathematical theory of probability emerged out of attempts to understand probability in the context of gambling on “games of chance” (Clayton, 2021; Hacking, 2009). Games of chance are interesting, in the sense that they are concrete physical processes that—in theory and often in practice—can be repeated. Over such repetitions, patterns emerge. We can flip a coin and note that the proportion of heads stabilizes after many flips (if the coin is fair, this proportion will be close to \(0.5\)). We can deal poker hands, and note that the proportion of hands with a flush gets close to \(0.2\%\) after many hands. Sixteenth and seventeenth century mathematicians—including Gerolamo Cardano, Pierre de Fermat, Blaise Pascal, and Jacob Bernoulli—exploited this notion of repeatability to mathematize some of these law-like patterns. We might label this conception of probability—the conception that probability describes patterns that emerge from repeatable processes like games—objective probability. Objective probability describes aleatoric uncertainty, or, uncertainty inherent in the empirical world.
Other mathematicians—motivated by a different set of problems—used probability in a different way. In An Essay towards solving a Problem in the Doctrine of Chances, the Reverend Thomas Bayes developed theorems related to conditional probability and what is now called Bayes’ theorem (Bayes & Price, 1763).[^3] Rather than describing law-like patterns that emerge from repeatable processes, probability was used to describe human uncertainty about the world. Pierre-Simon Laplace’s (1749 – 1827) definition of probability suggests a similar use of probability (Laplace, 1902/1951 [1902]):
The theory of chances consists in reducing all events of the same kind to a certain number of equally possible cases, that is to say, to cases whose existence we are equally uncertain of, and in determining the number of cases favorable to the event whose probability is sought. The ratio of this number to that of all possible cases is the measure of this probability, which is thus only a fraction whose numerator is the number of favorable cases, and whose denominator is the number of all possible cases.
Though it might not be immediately obvious, Hájek (2019) suggests that this definition is epistemic in the sense that it presupposes a “we” that are “equally uncertain” of a set of events. That is, probability was not directly linked to a repeatable physical process, but was linked to our subjective uncertainty about the world.[^4]
In modern parlance, we might say that Bayes and Laplace were attempting to assign probabilities to hypotheses or parameters. What is the probability that this coin is fair? For a particular woman about to undergo IVF treatment, what is the probability that IVF will be successful? The answers to these questions—e.g., the probability that this coin is fair is \(0.75\); the probability that this particular woman’s IVF experience will be successful is \(1/3\)—involve assigning probabilities to fixed features of the world that are, at least in practice, not easily subject to repeatability conditions. In assigning probabilities to fixed features of the world, 18th century thinkers like Bayes and Laplace were making use of the concept of probability in a way different from their 17th century predecessors. We might label this conception of probability—the conception that allows probability assignments to apply to fixed, non-repeatable features of the world like hypotheses or parameters—epistemic (or subjective) probability. Epistemic probability describes uncertainty that arises from a lack of human knowledge about the empirical world.
Does probability describe something objective about the empirical world? Or, does it describe something about our human uncertainty? Mathematicians and others interested in probability never really settled these conceptual or philosophical tensions between objective and subjective notions of probability. However, they did (mostly) settle important mathematical questions about how to systematize and calculate probabilities. The culmination of the systematization of mathematical probability came in the 20th century, when Andrey Kolmogorov provided a mathematically rigorous formulation for probability using the language of measure theory.[^5] Kolmogorov’s axioms of probability are taught in most introductory probability courses; they provide the rules for specifying a probability model, calculating probabilities of more complex events, and ultimately, performing statistical inference.
Get the full book: Buy Patterns from Static