Summary

I'll begin with the notion of sufficient statistics: why they matter for bounded agents, the 90 year old obstruction one encounters, and why this isn't so bad. Then I'll present an alternative approach with proofs and exercises, following Probability Theory: The Logic of Science (Jaynes 2003) extensively.

In pt.1 I use a symmetry argument to show that KPD assumes too much. All the benefits of sufficient statistics can be realized without assuming any kind of independence.

In pt.2 I generalize Maxent to elicit a class of non-iid models exhibiting sufficient statistics that are compatible with pt.1 by construction. Taken together, this can be viewed in some sense as a representation theorem for finite exchangeability.

Pt. 0 -- preliminaries

Let's start with a couple historical remarks on the notion of sufficient statistics, as introduced by RA Fisher (ca. 1920) in On the Mathematical Foundations of Theoretical Statistics.

That the statistic chosen should summarise the whole of the relevant information supplied by the sample. This may be called the Criterion of Sufficiency.

Not long after, three independent (and presumably identically distributed) authors, e.g. Koopman 1936, discovered an obstruction to sufficiency. Today this result is known as the Koopman-Pitman-Darmois (KPD) theorem.

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases. Intuitively, this states that nonexponential families of distributions on the real line require nonparametric statistics to fully capture the information in the data.
Less tersely, suppose $X_{n}, n = 1, 2, 3...$ are independent identically distributed real random variables whose distribution is known to be in some family of probability distributions, parametrized by $θ$ , satisfying certain technical regularity conditions, then that family is an exponential family if and only if there is a $R^{m}$ -valued sufficient statistic $T (X_{1}, . . ., X_{n})$ whose number of scalar components $m$ does not increase as the sample size $n$ increases.

At first blush, this seems to place a considerable restriction on an agent's ability to compress data for the purpose of inference. Our brains summarize ...

Posts

Wikitag Contributions

Comments