Bathing Machines and the Lindy Effect

[-]shirisaya5y100

Without doing the math to check, nothing that you said seems wrong. However, I take a very different lesson from the idea of the Lindy Effect than you do. Specifically, the Lindy Effect tells us that when making predictions about future lifetimes of non-perishable things, we should assume a power law distribution. If you've never dealt with predictions under thick-tail assumptions, you might be surprised how little intuition you will have for it. (The 80-20 rule is another example of assuming thick-tails.)

For example, a Pareto distribution (the easiest of the power law distributions) will typically have a mean larger than the median, and the mode (most common single outcome) is essentially instant failure. If the constant of proportionality in the Lindy Effect is greater than one, this implies an infinite variance even with a finite mean. Also, the force of mortality (instantaneous rate of death, aka hazard function) is a decreasing function of time.

The reason that this is typically a good rule of thumb for making future estimates is a consequence of making predictive distributions (i.e. integrating out uncertainty in a fit parameter). As the most familiar example, remember that even if the maker of the Matrix came down and told that a random variable was normally distributed, you will still need to estimate the parameters from observation. This will lead to your mean estimates being distributed as a Student-t, which is a power law.

[-]DirectedEvolution5y10

If I'm reading you right (low confidence), then I think our lessons are compatible. The longer something's been around, the longer we should expect it to continue, in absolute terms. At the same time, our best outside view guess is always that the thing is getting toward the end of its life, in relative terms.

I notice the Lindy Effect getting tossed out often as a counterargument to an inside view. So for example, if John says "the Catholic church is on its last legs," Alice might say "it's been around for almost two millennia, so the Lindy Effect suggests it'll probably be around for a long time to come."

I think the way to synthesize their approaches is to start with Alice's point of view, then modify it with John's. And this makes perfect sense. If you told me that X has been around for 2,000 years, then . without knowing what X is, I'd feel pretty confident that it's not going to disappear tomorrow. But I'd also want to know what X is, so I can modify my expectations accordingly.

The Lindy Effect makes a little less intuitive sense when X is only a few seconds old. But that's because I can't stop my imagination from filling in what X might be by imagining the social circumstances. Anything you can tell me is 2 seconds old is probably something you made, and it's probably an object. Most objects don't self-destruct seconds after they were manufactured.

More generally, anything that requires work to make requires an input of energy. That means evolution's fighting entropy for it, and probably wouldn't invest in it if it was likely to be fragile. Anything the living make has a life expectancy in proportion to the energy it took to build it.

But likewise, the more energy it takes to make a thing, the smaller a fraction of the total output of things per unit time. If the Lindy Effect doesn't seem intuitive, that's because we're so used to paying attention to big, old things that we don't think to use the countless small and temporary things as examples.

[-]Bucky5y30

You can generalise this for other required accuracies. If instead of 25% we use "a" then the optimal guess is $100 % + a$ of the current life which is correct $\frac{2 a}{100 % + a}$ of the time.

If we use an alternative optimisation criterion where we compare any two prediction methods and see, over the life of the bathing machine, which is closer to the correct answer most often then 200% (i.e. the halfway rule) is best.

So which rule of thumb you use depends on what you're looking to achieve - a guess which will be fairly good for as much of the lifetime as possible or a guess which is better for most of the lifetime, even if sometimes it's way off.

[-]DirectedEvolution5y10

Another ramification of the 125% longer/40% confidence Lindy Effect is that it'll always be reasonable to expect the thing you're considering to have already ended if you're using it to establish your priors for forward-looking life expectancy. You could deal with that paradox by shrinking the early bound of what you'll consider a correct prediction and being more generous on the late bound of when the thing ends.

The Lindy Effect is thus an anti-conservative heuristic. With a generous confidence interval, It's reasonable to expect, in the absence of other information, that anything could end this year. What Lindy does is provide a maximum value on our super-low-information prior for longevity.

At least a quarter of the world's population live on farms. That's been true for the last 12,000 years. The Lindy Effect suggests that state of affairs might end this year, but that we shouldn't expect it to last more than another 4,000 years or so.

The Lindy Effect is also scope insensitive. It's clearly more likely that at least 10% of the world's population live on farms as of next year. But the Lindy heuristic generates the same estimate no matter what the cutoff is, as long as it's at least the present-day level. I suppose the scope is inside-view information, so perhaps that makes sense if we're looking for a strictly outside-view prior.

[-]DirectedEvolution5y10

If the Lindy Effect is useful for forward-looking predictions, it would seem to be a way of calibrating our priors before modifying them with evidence from the inside view.

Example: We've had Christian states since Christianity became the official s̶o̶f̶t̶ ̶d̶r̶i̶n̶k̶ religion of the Roman Empire 313 AD. To estimate how much longer they'll last, we start by recognizing that absent any inside view information, our best bet would be to estimate it'll last to around the year 2500, plus or minus 500 years. Then we move from this starting point by considering inside view evidence and heuristics.

What's the trend on proportion of world population living in a Christian state over time? What's its status in the nine nations where it's still the official state religion? What if the AI foom permanently disrupts all our institutions?

Do we think that pre-industrial institutions have demonstrated durability by transcending technological and social shifts? Are they especially fragile, since they aren't built to respond to the demands of the modern era? Or are these heuristics too ambiguous to be useful?

[-]DirectedEvolution5y10

start = 1800
end = 2000
lifespan = end - start
ci = lifespan * .25
end_i = end - ci
end_f = end + ci

p_max = 0
max_correct = 0
for i in range(10000):
p = i * .001
correct = 0
for year in range(lifespan):
end_prediction = year * p + start
if end_prediction >= end_i and end_prediction <= end_f:
correct += 1

if correct > max_correct:
max_correct = correct
p_max = p
print("correct %:", max_correct / lifespan)
print("p: ", p_max)
works_i = (end_i - start) / p_max
works_f = (end_f - start) / p_max
print("Works from", start + works_i, "to", start + works_f)

LESSWRONG
LW

LESSWRONG
LW

15

Bathing Machines and the Lindy Effect

15

15