Named Distributions as Artifacts

[-][anonymous]6y100

I just learned about exponential families which seem to encompass most of the distributions we have named. It seems like the distributions we name, then, do sort of form this natural group with a lot of nice properties.

[-]johnswentworth6y100

This post has a lot of misconceptions in it.

Let's start with the application of the central limit theorem to champagne drinkers. First, there's the distinction between "liver weights are normally distributed" and "mean of a sample of liver weights is normally distributed". The latter is much better-justified, since we compute the mean by adding a bunch of (presumably independent) random variables together. And the latter is usually what we actually use in basic analysis of experimental data - e.g. to decide whether there's a significant different between the champagne-drinking group and the non-champagne-drinking group. That does not require that liver weights themselves be normally distributed.

That said, the CLT does provide reason to believe that something like liver weight would be normally distributed, but the OP omits a key piece of that argument: linear approximation. You do mention this briefly:

Most processes don't "accumulate" but rather contribute in weird ways to yield their result. Growth rate of strawberries is f(lumens, water) but if you assume you can approximate f as lumens*a + water*b you'll get some really weird situation where your strawberries die in a very damp cellar or wither away in a desert.

... but that's not quite the whole argument, so let's go through it properly. The argument for normality is that f is approximately linear over the range of typical variation of its inputs. So, if (in some strange units) lumens vary between 2 and 4, and water varies between 0.3 and 0.5, then we're interested in whether f is approximately linear within that range. Extend this argument to more variables, apply CLT (subject to conditions), and we get a normal distribution. What happens in damp cellar or desert is not relevant unless those situations are within the normal range of variation of our inputs (e.g. within some particular dataset).

(The OP also complains that "We can't determine the interval for which most processes will yield values". This is not necessarily a problem; there's like a gazillion versions of the CLT, and not all of them depend on bounding possible values. CLT for e.g. the Cauchy distribution even works for infinite variance.)

Now, a better argument against the CLT is this one:

Most processes in the real world, especially processes that contribute to the same outcome, are interconnected.

Even here, we can apply a linearity -> normality argument as long as the errors are small relative to curvature. We model something like a metabolic network as a steady-state $x$ with some noise $ϵ$ : $x_{i} = f_{i} (x, ϵ_{i})$ . For small $ϵ$ , we linearize and find that $Δ x \approx (I - \frac{\partial f}{\partial x})^{- 1} \frac{\partial f}{\partial ϵ} ϵ$ , where I is an identity matrix and the partials are matrices of partial derivatives. Note that this whole thing is linear in $ϵ$ , so just like before, we can apply the CLT (subject to conditions), and find that the distributions of each $x_{i}$ are roughly normal.

Takeaway: in practice, the normal approximation via CLT is really about noise being small relative to function curvature. It's mainly a linear approximation over the typical range of the noise.

Next up, triangles.

"The left-hand side is Pyathagor's formula, the right-hand side is this artifact which is kind of useful, but there's no property of our mathematics that exactly define what a slightly-off right-angled triangle is or tells us it should fit this rule."

There absolutely is a property of mathematics that tells us what a slightly-off right-angled triangle is: it's a triangle which satisfies Pythagoras' formula, to within some uncertainty. This is not tautological; it makes falsifiable predictions about the real world when two triangles share the same right-ish corner. For instance, I could grab a piece of printer paper and draw two different diagonal lines between the left edge and the bottom edge, defining two almost-right triangles which share their corner (the corner of the paper). Now I measure the sides of one of those two triangles very precisely, and find that they satisfy Pythagoras' rule to within high precision - therefore the corner is very close to a right angle. Based on that, I predict that lower-precision measurements of the sides of the other triangle will also be within uncertainty of satisfying Pythagoras' rule.

On to the next section...

I think [Named Distributions] can also be dangerous because of their simplicity, that is, the lack of parameters that can be tuned when fitting them.

Some people seem to think it's inherently bad to use complex models instead of simple ones when avoidable, I can't help but think that the people saying this are the same as those that say you shouldn't quote wikipedia.

I fully support quoting Wikipedia, and it is inherently bad to use complex models instead of simple ones when avoidable. The relevant ideas are in chapter 20 of Jaynes' Probability Theory: The Logic of Science, or you can read about Bayesian model comparison.

Intuitively, it's the same idea as conservation of expected evidence: if one model predicts "it will definitely be sunny tomorrow" and another model predicts "it might be sunny or it might rain", and it turns out to be sunny, then we must update in favor of the first model. In general, when a complex model is consistent with more possible datasets than a simple model, if we see a dataset which is consistent with the simple model, then we must update in favor of the simple model. It's that simple. Bayesian model comparison quantifies that idea, and gives a more precise tradeoff between quality-of-fit and model complexity.

[-]George3d66y30

And the latter is usually what we actually use in basic analysis of experimental data - e.g. to decide whether there's a significant different between the champagne-drinking group and the non-champagne-drinking group

I never bought up null-hypothesis testing in the liver weight example and it was not meant to illustrate that... hence why I never bought up the idea of signfiance.

Mind you, I disagree that signficance testing is done correctly, but this is not the argument against it nor is it related to it.

(The OP also complains that "We can't determine the interval for which most processes will yield values". This is not necessarily a problem; there's like a gazillion versions of the CLT, and not all of them depend on bounding possible values. CLT for e.g. the Cauchy distribution even works for infinite variance.)

My argument is not that you can't come up with a distribution for every little edge case imaginable, my argument is exactly that you CAN and you SHOULD but this process should be done automatically, because every single problem is different and we have the means to dynamically see the model that best suits every problem rather than stick to choosing between e.g. 60 names distributions.

Even here, we can apply a linearity -> normality argument as long as the errors are small relative to curvature.

I fail to see your argument here, as in, I fail to see how it deals with the interconnected bit of my argument and I fail to see how noise being small is something that ever happens in a real system, in the sense you use it here, as in, noise being everything that's not inference we are looking for.

There absolutely is a property of mathematics that tells us what a slightly-off right-angled triangle is: it's a triangle which satisfies Pythagoras' formula, to within some uncertainty.

But, by this definition that you use here, any arbitrary thing I want to define mathematically, even if it contains within it some amount of hand wavyness or uncertainty, can be a property of mathematics ?

I fully support quoting Wikipedia, and it is inherently bad to use complex models instead of simple ones when avoidable. The relevant ideas are in chapter 20 of Jaynes' Probability Theory: The Logic of Science, or you can read about Bayesian model comparison.

Your article seems to have some assumption that increase complexity == proneness to overfitting.

Which in itself is true if you aren't validating the model, but if you aren't validating the model it seems to me that you're not even in the correct game.

If you are validating the model, I don't see how the argument holds (will look into the book tomorrow if I have time)

Intuitively, it's the same idea as conservation of expected evidence: if one model predicts "it will definitely be sunny tomorrow" and another model predicts "it might be sunny or it might rain", and it turns out to be sunny, then we must update in favor of the first model. In general, when a complex model is consistent with more possible datasets than a simple model, if we see a dataset which is consistent with the simple model, then we must update in favor of the simple model. It's that simple. Bayesian model comparison quantifies that idea, and gives a more precise tradeoff between quality-of-fit and model complexity.

I fail to understand this argument and I did previously read the article mentioned here, but maybe it's just a function of it being 1AM here, I will try again tomorrow.

[-]philip_b6y10

Let’s start with the application of the central limit theorem to champagne drinkers. First, there’s the distinction between “liver weights are normally distributed” and “mean of a sample of liver weights is normally distributed”. The latter is much better-justified, since we compute the mean by adding a bunch of (presumably independent) random variables together. And the latter is usually what we actually use in basic analysis of experimental data—e.g. to decide whether there’s a significant different between the champagne-drinking group and the non-champagne-drinking group. That does not require that liver weights themselves be normally distributed.

I think your statement in bold font is wrong. I think in cases such as champagne drinkers vs non-champagne-drinkers people are likely to use Student's two-sample t-test or Welch's two-sample unequal variances t-test. It assumes that in both groups, each sample is distributed normally, not that the means are distributed normally.

[-]johnswentworth6y30

No, the student's two-sample t-test does not require that individual samples are distributed uniformly. You certainly could derive it that way, but it's not a necessary assumption. All it actually needs is normality of the group means via CLT - see e.g. here.

[-]Pattern6y40

When we teach people the formula we should draw attention to the difference: "The left-hand side is Pyathagor's formula, the right-hand side is this artifact which is kind of useful, but [1] there's no property of our mathematics that exactly define what a slightly-off right-angled triangle is or [2] tells us it should fit this rule".

[1] There probably is. (Though the idea that real triangles are exactly like mathematical triangles and that can be proved via logic might be wrong.)

[2] And it says tells you exactly how wrong the rule is based on what the triangle is actually like.

Well, that assumes the phenomenon fits a 2,000,000 parameter equation using some combination of the finite set of operations provided by the network (e.g. +, -, >, ==, max, min,avg).

Or that a 2,000,000 parameter equation will make a good approximation. (I'm not sure if that's what you meant by "fit".) If you have some assumptions, and use math correctly to find that the height of something is 4 ft, but it's actually 5 ft, then the assumptions aren't a perfect fit.

So I'm going to disregard it for now; if anyone knows of a good article/paper/book arguing for why simple models are inherently better, please tell me and I'll link to it here.

Here's an argument made in response to your article:

Suppose I have 100 datapoints and I come with a polynomial that fits all of them, with "degree 99". How close do you think that the polynomial is to the real function? Even if I the datapoints are all 100% accurate, and the real function is a polynomial, there is no redundancy at all. Whereas if the polynomial was of degree 3, then 4 points is enough to come up with the rule, and the other 96 points just verify it (within it's paradigm). When there's no redundancy the "everything is a polynomial" paradigm doesn't seem justified. When there's 96 redundant points, out of 100 points, it seems like polynomials are a really good fit.

(In other words, it's not clear how a complicated model compresses rather than obfuscates the data - though what is "complicated" is a function of the data available.)

We have models that can account for more complexity, without any clear disadvantages, so it's unclear to me why we wouldn't use those.

This article focused heavily on Named Distributions, and not a lot on these alternatives. (NNs were mentioned in passing.)

You see, the invention of the CLT, whilst bringing with it some evil, is still the best possible turn of events one could hope for, as we are living in the best of all possible worlds.

That sounds like an interesting bit of history.

[-]Donald Hobson6y30

Here is why you use simple models.

The blue crosses are the data. The red line is the line of best fit. The black line is a polynomial of degree 50 of best fit. High dimensional models have a tendency to fit the data by wiggling wildly.

[-]johnswentworth6y30

That problem would be handled by cross-validation; the OP is saying that a simple model doesn't have an obvious advantage assuming that both validate.

Given that both models validate, the main reason to prefer a simpler model is the sort of thing in Gears vs Behavior: the simpler model is more likely to contain physically-realistic internal structure, to generalize beyond the testing/training sets, to handle distribution shifts, etc.

[-]Donald Hobson6y10

It depends on what cross validation you are using. I would expect complex models to rarely cross validate.

LESSWRONG
LW

LESSWRONG
LW

22

Named Distributions as Artifacts

22

22

On confusing priors with models

Artifacts that became idols

Are Named Distributions harmful?

In conclusion

Epilogue