(This is the second of two posts on the textbook All of Statistics. Click here for post I.)

4. Fundamentals of Statistical Inference

Probability theory is about using distributions to derive information about outcomes. Given the assumption that $X \sim Poisson (3)$ , we can compute probabilities of outcomes the form $Pr (X = k)$ . Statistics is about the opposite: using outcomes to derive information about distributions.

The book exclusively considers the setting where we observe independently sampled data points $x_{1}, . . ., x_{n} \sim X$ where the distribution of $X$ is unknown. It is often convenient to talk about the RV that generated the $k$ -th sample point, the logical notation for which is $X_{k}$ . The book uses upper-case letters for both (e.g., 'we observe data $X_{1}, . . ., X_{n}$ "), not differentiating between the RV that generates the $k$ -th point (i.e., $X_{k}$ ) and the k-th point itself ( $x_{k}$ ).

On the highest level, one can divide all inference problems into two categories: parametric inference and non-parametric inference. In parametric Inference, we assume that we know the family of distributions that $X$ belongs to (Binomial, Poission, Normal, etc.). In this case, the problem reduces to inferring the parameters that characterize distributions in that family. For example, if we observe data on traffic accidents, we may assume that $X \sim Poisson (λ)$ , and all that's left to estimate is $λ$ . Conversely, in the context of non-parametric inference, one does not assume that $X$ belongs to any one family and thus has to estimate the distribution directly.

5. Bayesian Parametric Inference

5.1. The concept

Let $Θ$ be the parameter we wish to learn about. Since we don't know its value, we treat it as a RV that ranges over several possible values. Furthermore, let $x = (x_{1}, . . ., x_{n})$ be our observed data, and let $X = (X_{1}, . . ., X_{n})$ be the RVs that generate this data.

Bayes' theorem is an equation relating ${Pr}_{B} (A)$ to ${Pr}_{A} (A)$ . (With $P_{Y} (X) := P (X | Y) .)$ In our case, we are interested in terms of the form $P_{[X = x]} (Θ = θ)$ , and we know how to compute terms of the form $P_{[Θ = θ]} (X = x)$ because once we fix a value of $Θ$ , the distribution is known and we can compute the required probabilities. Thus, Bayes' theorem gives us a way to reduce a problem of statistical inference to a problem of probability theory.

Alas, we have

$P_{[X = x]} (Θ = θ) = P_{[Θ = θ]} (X = x) \frac{P (Θ = θ)}{P (X = x)}$

Our prior on $θ$ will virtually always be continuous. Let $h$ be the pdf of this prior. The $X_{i}$ may be continuou...

Litereature Summaries