4 The frequentist statistical paradigm
Let me ask you something. If the rule you followed brought you to this, of what use was the rule?
—Anton Chigurh, No Country for Old Men, by Cormac McCarthy
Should probability enter [statistics] to capture degrees of belief about claims? To measure variability? Or to ensure we won’t reach mistaken interpretations of data too often in the long run of experience? Modern statistical methods grew out of attempts to systematize doing all of these.
— Deborah Mayo, Statistical Inference as Severe Testing
In the previous chapter, we saw that there are many different ways to interpret probability. Some take probability to be describing some objective feature of the world, for example, relative frequencies that arise in repeated measurements. Others take probability to be describing something epistemic, for example, partial belief or incomplete information. These different interpretations of probability have given rise to different ways of producing statistical inferences. Frequentist statisticians take the so-called objective approach, assigning probabilities only to relative frequencies that arise over repeated measurements and repeated sampling. They use those probabilities to produce inferences to hypotheses about how the data were generated. In this chapter, we’ll study the philosophical foundations and practical methods of the frequentist statistical inference paradigm. First, we’ll consider the formal construct—a statistical model—that provides the basis from which probability statements are made. Then, we’ll study the foundational tools of frequentist inference—including hypothesis testing, point estimation, and interval estimation—with special attention to philosophical justifications (or lack thereof).
4.1 Statistical models
Consider the following hypothesis about reading trends in America:
\(H\): At least ten precent of Americans have read The Brothers Karamazov (BK).
As with all statistical hypotheses, \(H\) refers to a population level parameter. In this case, the parameter is the proportion of Americans who have read BK: \[p = \frac{\# \text{ individuals in the population who have read {\it BK}}}{\# \text{ number of individuals in the population}}.\]
\(H\) is difficult to confirm or falsify. Part of the difficulty lies in the fact that the population in question—all Americans—is very large.[^1] It is not practical to observe all of the relevant information at the population level. Instead, we can gather data relevant to \(H\) in a sample—a subset of the population. There are many different ways that a sample can be gathered from a given population. A data generating process (DGP) is a description of the sampling or measurement process that gives rise to the data. Consider the following DGP related to hypothesis \(H\):
DGP\(_H\): Randomly sample \(n\) Americans and record whether they have (\(1\)) or have not (\(0\)) read The Brothers Karamazov (BK).
A key feature of this DGP is the notion of a random sample. Randomness is notoriously difficult to define with precision (Eagle, 2010). For our purposes, we define a random sample as a subset of the population such that each individual—in this case, each American—has the same probability of being included.[^2] DGPs give way to actual or observed data, \(\mathbf{x} = (x_1,...,x_n)\). For example, imagine a random sample of size \(n = 15\) arising from DGP\(_H\): \[\mathbf{x} = (0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0).\] Intuitively, there is information in \(\mathbf{x}\) relevant to \(p\), and thus, \(H\). Generally, sample information relevant to a parameter or hypothesis can be summarized through a function of the data, sometimes called a statistic. In this case, the relevant statistic is the proportion of individuals in the sample that have read BK: \[\widehat p = \frac{\# \text{ of individuals in the sample who have read {\it BK}} }{n}.\] By convention, in statistics, when we have a sample quantity that corresponds to a population level quantity, we use the same notation, but add a “hat” on top.[^3]
For the data \(\mathbf x\) above, \(\widehat p = 2/15\). \(\widehat p\) does not precisely answer questions related to \(p\) and \(H\); it contains information from the sample only. It’s easy to see how \(p\) and \(\widehat p\) might be different. Imagine that researchers had gathered a different random sample of size \(n\), say, \(\mathbf{x'} = (x'_1,...,x'_n)\). Using \(\mathbf{x'}\), the sample proportion, \(\widehat p'\) would likely be different, based on the fact that \(\mathbf{x'}\) would (very likely) include different individuals with different reading habits. It is likely that both \(\widehat p\) and \(\widehat{p'}\) would be different from \(p\). How different? A statistical model can help us answer this and related questions.
A statistical model is a set of formal assumptions about the DGP that gives rise to observed data. Specifically, a statistical model makes the assumption that actual data \(\mathbf{x} = (x_1,...,x_n)\) are realizations of a stochastic (probabilistic) process \(\mathbf{X} = (X_1,...,X_n)\). \(\mathbf{X}\) is chosen to encode information from the DGP (e.g., DGP\(_H\)), with a connection to the theory or hypothesis of interest (e.g., \(p\), \(H\)). The assumption that actual data, \(\mathbf{x}\), come from a stochastic process, \(\mathbf{X}\), is often justified in terms of repeated sampling or measurement error. If we had observed the same phenomena (e.g., sampling process, experiment, physical process) again under sufficiently similar conditions, we would have observed different values. The statistical model quantifies how likely all of the possible datasets are under various hypotheses. For example, possible data \[\mathbf{x'} = (1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1),\] with \(\widehat{p'} = 9/15\), would be relatively likely under the hypothesis that \(p = 0.7\); on the other hand, the actual data \(\mathbf x\) (with \(\widehat{p} = 2/15\)) would be relatively unlikely under that hypothesis. For frequentist statisticians, these probabilities—probabilities assigned to various actual and hypothetical datasets—are relevant to what we ought to infer about parameters and hypotheses. Crucially, these probabilities are the only relevant ones for inferences about parameters and hypotheses. Probability theory enters the frequentist statistical model to quantify and subsequently test expectations we have about which data sets would and would not arise under various hypotheses.
In the simplest cases, the DGP produces independent and identically distributed (iid) data. Informally, for \(i=1,...,n\), and \(j = 1,...,n\):
\(X_i\) is independent from \(X_j\) if the occurrence of \(X_i\) does not influence the probability of occurrence of \(X_j\), for all \(i,j = 1,...,n\), \(i \ne j\).
\(X_1,...,X_n\) are assumed to have the same probability distribution, i.e., the same “shape” (e.g., normal, binomial), center, scale, etc. With respect to DGP\(_H\), if we assume that (i) no individual influences any other with respect to the reading BK, and that (id) the probability, \(p\), that any randomly selected person in the population has read BK is the same for each person, then DGP\(_H\) is iid.[^4] Further, it can be modeled by \(\mathbf{X} = (X_1,...,X_n)\), where each \(X_i\) has a Bernoulli probability distribution. That is, each \(X_i\), takes on the value \(x = 1\) with probability \(p\); and the value \(x = 0\) with probability \(1-p\). The probability distribution function (pdf)—the function that describes how to compute the probability that \(X_i\) takes on one or zero—is given by: \[\begin{aligned} f(x_i; p) = p^{x_i}(1-p)^{1-x_i}. \end{aligned}\] Some basic algebra reveals that, if \(x_i = 1\), \(f(x_i; p) = p\), and if \(x_i = 0\), \(f(x_i; p) = 1-p\). This function describes the probability that a single individual in the sample has read BK. The function that describes the probability of any given dataset (e.g., that two out of 15 individuals in a sample have read BK) is called the joint probability distribution function. Under the iid assumptions, the joint pdf associated with the process is the product (represented by the symbol “\(\prod_{i=1}^n\)”) of the pdfs for each \(X_i\): \[ \begin{aligned} f(\mathbf{x}; p) &= \prod_{i=1}^n f(x_i; p) \\ &\overset{\text{def}}{=}\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} \\ &\overset{\text{algebra}}{=} p^{\sum_{i=1}^n x_i}(1-p)^{n - \sum_{i=1}^n x_i}. \end{aligned} \tag{4.1}\] The joint distribution \(f(\mathbf{x}; p)\) answers questions like: what is the probability of observing \(\mathbf{x}\)? The statistical model, \(\mathcal{M}_{p}(\mathbf{x})\), for DGP\(_H\) is comprised of:
A stochastic process, \(\mathbf X\). In the BK example, \(\mathbf X\) is a binomial process.
A specification of the family of joint pdfs, \(f(\mathbf x; p)\). In the BK example, \(f(\mathbf x; p)\) is given by Equation 4.1.
A specification of what values all of the variables can take on. In the BK example, \(p\) is any value between zero and one, and each \(X_i\) can be either zero or one.
By a family of probability distribution functions, we mean the collection of \(f(\mathbf{x}; p)\) for all possible values of \(p\) in \((0,1)\). Since we do not know which value of \(p\) generated the observed data \(\mathbf x\), the statistical model encodes the family of distributions as the set of all possible explanations of the data. Intuitively, some distributions in this family are much more plausible than others as explanations for the actual data; for DGP\(_H\), the “sub-family” of distributions singled out by \(p \in (0.9,1)\) are implausible as explanations for the actual data \(\mathbf{x}\). More generally—that is, not just in the context of DGP\(_H\)—we can denote the statistical model associated with data \(\mathbf{x} = (x_1,...,x_n)\) and parameterized by \(\boldsymbol\theta\) as \(\mathcal{M}_{\boldsymbol\theta}(\mathbf{x})\).
Get the full book: Buy Patterns from Static