5 The Bayesian statistical paradigm
Where you’ve nothing else construct ceremonies out of the air and breathe upon them.
The Man, The Road, by Cormac McCarthy
*Should probability enter [the chapter on frequentist inference](https://www.brianzaharatos.com/book), we saw that frequentist statistical methods are grounded on a relative frequency interpretation of probability. Frequentists reject subjective or evidential interpretations of probability and the philosophical probabilism that arises from those interpretations. Instead, frequentists embrace inference procedures that are calibrated to probe the world in relevant ways and are unlikely to be wrong in the long run. For example, in a frequentist hypothesis test, statisticians specify a statistical model—which includes a joint probability distribution over random variables that model data—and then use the probability assignments over the data to reject or fail to reject a null hypothesis. In newer versions of hypothesis testing, statisticians are able to say more about the extent to which various hypotheses are severely tested or corroborated.
As eluded to throughout this book, Bayesian statisticians take a very different approach. Bayesian statistical methods embrace either the subjective or logical interpretation of probability, allowing for statisticians to account for observed relative frequencies and epistemic uncertainty. As we saw in the chapter on probability, the the subjective and logical interpretations of probability are at least plausible interpretations. The former is often justified by the use of Dutch book arguments; the latter is justified by deriving probability from propositional logic. But are methods derived from epistemic interpretations of probability robust enough to support an entire statistical inference framework? That is what we’ll explore in this chapter!
5.1 The Bayesian statistical model
As with a frequentist statistical model, a Bayesian statistical model starts with a random process, \(\mathbf{X} = (X_1,...,X_n)\), that models the DGP in question. As random variables, \(\mathbf{X}\) has a probability distribution; that distribution is parameterized by a vector of parameters, say, \(\boldsymbol\theta = (\theta_1,...,\theta_p)\). Consider again the BK example from the chapter on frequentist inference:
DGP\(_H\): Randomly sample \(n\) Americans and record whether they have (\(1\)) or have not (\(0\)) read BK.
In DGP\(_H\), \(\mathbf{X}\) is a Bernoulli random process with parameter \(\theta = p\). In contrast to frequentist inference, a Bayesian model takes the parameters \(\boldsymbol\theta\) themselves to be random variables. The parameters are random in the sense that they are unobserved (or unobservable), and thus, their values are uncertain to observers. Because both \(\mathbf{X}\) and \(\boldsymbol{\theta}\) are random entities, we can (at least in theory) write down the joint distribution of \((\mathbf{X}, \boldsymbol\theta)\), namely, \(f(\mathbf{x}, \boldsymbol\theta)\). From this joint distribution, we can make inferences about the values of \(\boldsymbol\theta\) by finding the posterior distribution of \(\boldsymbol\theta\) given data \(\mathbf{x}\) (i.e., realizations of \(\mathbf{X}\)). We obtain the posterior distribution, or other conditional or marginal distributions, through basic probability theory. Specifically, the posterior distribution—the holy grail of Bayesian inference—often denoted \(\pi(\boldsymbol{\theta} \, | \, \mathbf{x})\), is given by
\[ \pi(\boldsymbol{\theta} \, | \, \mathbf{x}) \,\, \overset{\text{def}}{=} \,\, \frac{f(\mathbf{x}, \boldsymbol\theta)}{f(\mathbf{x})} \,\, \overset{\text{Bayes}}{=} \,\,\frac{f(\mathbf{x}\, | \, \boldsymbol\theta)\pi(\boldsymbol\theta)}{f(\mathbf{x})}. \tag{5.1}\] The first equality holds by definition of conditional probability. However, Bayesian statisticians do not use the middle expression in their inference methods, mainly because, in practice \(f(\mathbf{x}, \boldsymbol\theta)\) is not known. Thus, the second equality, which holds by Bayes’ theorem for random variables, is more useful. Its components are known or assumed as part of the standard Bayesian modeling assumptions. The likelihood function, \(f(\mathbf{x} \, | \, \boldsymbol\theta)\), represents the probability (density) of the data, given a value of the parameter. This function incorporates assumptions related to the uncertainty in the DGP. For example, it may represent the probability of observing exactly five individuals who have read BK in a random sample of \(n = 15\) individuals, given a value of \(p\). This likelihood function is (almost) the same likelihood function that is used in frequentist inference.[^1]
The prior distribution, or, prior, \(\pi(\boldsymbol\theta)\), is defined as an individual’s epistemic probability distribution over the parameter(s) in question, before observing the actual data. This prior probability is one of the most philosophically controversial aspects of Bayesian inference. As such, we will return to a longer discussion of it in Section ?sec-PhilBayes.
The denominator of Bayes’ theorem, \(f(\mathbf{x})\), is the probability distribution of the observed data, unconditional on any parameters. For example, what is the probability of observing five individuals who have read BK in a random sample of \(n = 15\), without conditioning on a known probability of success, \(p\)? That’s a hard question to answer directly. The law of total probability (LoTP) converts this computation into one we can answer directly. The LoTP says that we can find the marginal distribution of the data by summing (or integrating) over the likelihood times the prior for all values of the parameters. In the discrete parameter case, if there are \(m\) possible values of \(\theta\), say, \(\theta_1,...,\theta_m\), each having prior distribution \(\pi(\theta_i)\), then the LoTP states that \[ f(\mathbf{x}) = \sum^m_{i=1}f(\mathbf{x} \mid \theta_i)\pi(\theta_i). \tag{5.2}\] In the more realistic continuous case (e.g., \(\theta\) is on the interval \((0,1)\)), the LoTP is written as \[ \displaystyle f(\mathbf{x}) = \int f(\mathbf{x}\, | \, \boldsymbol\theta)\pi(\boldsymbol\theta) d\boldsymbol\theta, \tag{5.3}\] where the integral is over the entire parameter space. The only difference between equations Equation 5.2 and Equation 5.3 is that, summing the likelihood times the prior is not possible because the parameter is continuous. The integral replaces the sum. For the intuition and derivation of the law of total probability, see [app:math].
With all of this in mind, assuming a continuous parameter \(\boldsymbol\theta\), we can write Bayes’ theorem as: \[ { \pi(\boldsymbol{\theta} \, | \, \mathbf{x}) = \frac{f(\mathbf{x}\, | \, \boldsymbol\theta)\pi(\boldsymbol\theta)}{\int f(\mathbf{x}\, | \, \boldsymbol\theta)\pi(\boldsymbol\theta) d\boldsymbol\theta} }. \tag{5.4}\] Since the denominator of Equation 5.4 is an integral over \(\boldsymbol\theta\) (or, in the case of a discrete parameter space, a sum over \(\boldsymbol\theta\)), the result will not contain \(\boldsymbol\theta\), i.e., it is a constant with respect to \(\boldsymbol\theta\). That means that the denominator of Equation 5.4 does not impact the shape of \(\pi(\boldsymbol{\theta} \, | \, \mathbf{x})\), but instead, only impacts the magnitude of the area under \(\pi(\boldsymbol{\theta} \, | \, \mathbf{x})\). Therefore, we can write Bayes’ theorem as \[ { \pi(\boldsymbol{\theta} \, | \, \mathbf{x}) \propto f(\mathbf{x}\, | \, \boldsymbol\theta)\pi(\boldsymbol\theta) }, \tag{5.5}\] where \(\propto\) is read as “proportional to”. From this formulation, we can see that our posterior beliefs about \(\boldsymbol\theta\) given data \(\mathbf{x}\) are proportional to how likely the data are given \(\boldsymbol\theta\), times our prior beliefs about \(\boldsymbol\theta\). This formulation should make some intuitive sense. If a value of \(\boldsymbol\theta = \boldsymbol\theta_0\) was thought to be relatively unlikely before observing data—i.e., \(\pi(\boldsymbol\theta)\) placed a low probability (density) on \(\boldsymbol\theta = \boldsymbol\theta_0\))—then it would take highly likely data given \(\boldsymbol\theta = \boldsymbol\theta_0\) to produce a high posterior probability (density) on \(\boldsymbol\theta = \boldsymbol\theta_0\).
Get the full book: Buy Patterns from Static