[This is the fourth piece in my series on the Parametric Process Assumption (PPA), a new model of anthropics. You should start here if you are just seeing this for the first time.
Before diving into the PPA and its implications, let’s take a moment to motivate it philosophically: what do we mean when we say that we are uncertain about something?
One idea I really like, but which hasn’t really made its way into the LessWrong water supply as much as I might expect, is that there are really at least two different types of uncertainty. The first kind, known as aleatoric uncertainty, is what we conventionally think of as randomness. The roll of a die, the flip of a coin, the configuration of a deck of cards dropped to the floor - that kind of thing.
The second kind is epistemic uncertainty. These are the things that have definitive, immutable truth values, but which are nevertheless unknown. We saw this in the digit-of-pi Sleeping Beauty problem from the introduction - a specific digit of pi is not, in any aleatoric sense, random, but we nonetheless may not know it and, for that reason, may choose to assign a probability to it. One could reasonably say, for instance, that the googolth digit of pi has a 1 in 10 chance of being 7.
A few notable properties separate these concepts. Epistemic uncertainty, for instance, is reducible. If I were to compute the googolth digit of pi, then I would know it forever, and in general the more evidence I have about a given epistemic fact, the more confident I would be in my estimates, at least in the limit. Aleatoric uncertainty, by way of contrast, never reduces. No matter how many times I flip my fair coin, there will always be a 50% chance that the next flip will come up heads.
In that sense, aleatoric uncertainty is the uncertainty of repetition, of what will happen “next time” or “in the limit.” When we assign a probability to an aleatoric uncertainty, we are doing so in a philosophically objective way: if you flipped a fair coin over and over and over again, the percentage of flips that went heads would get closer and closer to 50%. Even if you never actually make those flips, even if in practice it’s impossible to make that many flips, the abstraction is still mathematically well-defined.
Dice converging to their average value over many rolls. Courtesy of Wikipedia.
This definition of probability doesn’t apply to epistemic uncertainty. The googolth digit of pi cannot be resampled; it has always been whatever it currently is, and it will always go on being whatever it currently is. The one property that allowed us to define aleatoric uncertainty in an objective way - the hypothetical ability to resample - is of no use here. The digits of pi, along with countless other mathematical truths, and likely much of physics, are not and have not ever been random. The googolth digit of pi does not have a 10% chance of being 7; it has a 100% chance of being whatever it is, and we do not know what that might be.[1]
And yet, it seems clear that there must be some sense in which we are able to say that each digit has probability 10%. This is the driving force behind Bayesian statistics: that when confronted with epistemic uncertainty, we can quantify it by assigning our beliefs to a probability distribution and applying the same laws of probability that we used in the aleatoric setting.
That we can do this is by no means a given. There is an entire other school of statistics - the Frequentist school - that rejects the concept of epistemic probabilities. Mind you, Frequentists don’t reject the actual concept of epistemic uncertainty, but they don’t admit methods which assign probabilities to epistemic statements; in the world of Frequentism, one literally could not say that the googolth digit of pi had a 10% chance of being 7, or indeed being anything else. Confidence intervals, p-values and their ilk are all elaborate attempts to learn about the world without ever explicitly assigning credences to epistemic unknowns, and that is what makes them so famously counter-intuitive.
This aversion to epistemic probability is not completely unfounded. Unlike aleatoric uncertainty - which has an objective repetition-based interpretation - Bayesian/epistemic probabilities are inherently subjective. Bayesian priors represent a quantification of one’s beliefs, but there are no hard mathematics that can tell us where those beliefs come from. Priors are just guesses, and everyone’s guess might be different. You can’t even really say for sure if any one person’s prior is better than another’s. One prior may indeed prove closer to the truth than another, and any reasonable prior will converge to the truth with enough data, but it’s mathematically impossible to build a prior that’s always better in all situations when data is limited; this is known as the No Free Lunch Theorem.
I don’t want to oversell these flaws. If you’re hanging out on LessWrong, you probably already know the incredible effectiveness of Bayesian reasoning. But even with something as trivial as assigning a probability to pi’s googolth digit, No-Free-Lunch still applies: while only a sociopath would decide that there was a 99% chance that the googolth digit was 7, it’s still possible that the digit actually is 7, and in that case the sociopath’s prior is in some sense better than that of the normie who only gave 7 a 10% chance.[2] Which is all to say that while assigning probabilities to beliefs about epistemic uncertainty may be a highly effective and intuitive way to reason about the world, it is not an objective way to reason about the world.
Bayesian Parametric Modeling 101
[If you are already familiar with Parametric Bayesian models, you may wish to skip this section. Please do read the Closing Thoughts section though.]
Generally, in any real world situation, both types of uncertainty will be present. If you’re Bayesian then the most common way to model this is with what’s known as a parametric model.
The textbook example of a parametric model would be repeated flips of a biased coin. In this setup, you have a coin with some probability of coming up heads, but you have no idea what this value is - the coin could have high and often come up heads, low , or anything in between.
If we could flip the coin an infinite number of times, then we could know exactly. But in the real world we can’t actually do that, and simply calling an epistemic unknown isn’t enough to tell us how we would estimate it with only a limited number of flips. What, for instance, is the exact probability that the coin is fair ( =50%) if we flip it once and get heads? What if we flip it 10 times and get the sequence HTHTHHHHHH?
One way to answer that, the standard Bayesian way, is to start with a probability distribution over possible values of . This distribution, of course, is what we call our prior.
For the sake of illustration, we’ll use an example prior where we believe there are only three possible values for : 25%, 50%, and 75%. We are 100% certain the coin comes up heads one of those three percentages of the time, we just don’t know which. Before any coins are flipped, we can start with the assumption that all three of these possibilities are equally likely:
The true value of may be an epistemic unknown, but each individual coin flip is random in an aleatoric sense. If we think of the coin flips as a random process that evolves by repeatedly producing heads and tails, then is sort of like an input to that process. It does not tell us how the randomness will resolve on any individual flip, but it does give us the overall shape of how that randomness will proceed. A coin-flip process with input parameter =25%, for instance, will average out to have 25% heads. For that reason, is referred to as a “parameter.” If you're familiar with programming, the analogy might be that our process is a function and is the input that controls its progression.
Each possible value of produces a different random process
Trying to apply this model is where one encounters the famous Bayes Rule. In LW and adjacent spaces, Bayes Rule is often presented as a fundamental idea everyone should be familiar with, but if you aren’t the type of person who likes looking at equations, I don’t think it’s often clear exactly what the rule truly means, beyond perhaps some generic statement about “updating on evidence.”
A more refined explanation of Bayes Rule, however, is that it inverts the parametric model we just introduced. The is dead is that parameters are the things we really want to know, but we can’t actually observe them directly, so instead we must observe other aleatoric randomness and try to guess the parameters underneath. Bayes Rule is just the formula we use to do that; it’s how we extract knowledge about parameters without actually seeing them.
For instance, suppose we kept our uniform prior over =25%, =50%, or =75%. After seeing a single coin flip go heads, we would apply Bayes Rule, do a bit of math, and perform an update, thus giving us a new prior (technically, a “posterior”) which looks like this:
In this particular case, seeing a single heads has really only caused us to move some of the probability mass away from =25% to =75%. This was just a particular quirk of the prior we started with however. If we go on to observe, say, another nine flips giving us the sequence HTHTHHHHHH, then our prior would update even more sharply:
Now we have a very sharp peak at =75%, and the possibility of a coin that comes up tails 75% of the time has been basically eliminated.
Parametric techniques are far and away the most common way Bayesian statistics is applied in the real world. Countless areas of fields like cosmology, medicine, and engineering rely on parametric models. Even modern LLMs are parametric in a sense - the size of neural networks is often measured in “parameters.” In that case, the sampling of the next output token is the equivalent of the individual coin flips in this section’s example; the model’s parameters define the process which is the LLM’s output.
Most formulations of anthropics that I’ve seen, however, don’t wrestle with this framing in a deep way. Just look at standard problems like Sleeping Beauty: a single coin is flipped, and then either one or two observers are created. There is no epistemic “prior” in a situation like that, there is simply a coin with completely known odds of heads. That's not a sufficient analogy for all the myriad complexities of real-world physics and epistomology. Something is clearly missing.
Closing Thoughts
A good way to think about parametric models is that they are mostly about nesting. In the previous section’s example (the coin of unknown bias), we saw that each possible value of created a new self-contained process where the coin behaved differently. In some sense, it’s as if the aleatoric randomness - the flips of the coin - has actually been nested within the epistemic uncertainty.
If you aren’t sticking around for the more technical portions of this series, this nested structure should be your biggest takeaway from this piece. Hopefully, the connection to the “digit-of-pi” Sleeping Beauty problem is clear: parameters behave SSAishly under repetition, but when we nest a process within them, that process behaves SIAishly. The PPA is (mostly) just a simple extension of parametric modeling to the case of indexical reasoning.
That said, there’s still an elephant in the room, which is that the line between epistemic and aleatoric is actually quite fuzzy. For instance, in the last section we imagined a coin where the value of was epistemic. But the real world is filled with coins. If I reach into my pocket and pull out a coin, wouldn’t that constitute a random coin? That would make the choice of aleatoric, not epistemic.
What if we push the reasoning up a layer? The coin in question must surely have come from a particular factory, and this factory must have some probability distribution over the coins it produces - perhaps it produces a fair coin only X% of the time, and we’d like to know X. Does this make X an epistemic unknown?
Maybe. Or maybe this factory is just one of a million factories built by the same construction company, and each one has a different chance of producing a fair coin, and the quality of our factory is just another random variable?[3]
You can play this game for a very long time, but you can’t play it forever. At some point, you’re going to hit a ceiling. Perhaps the Laws of Physics, governed by mathematical truths that are as fixed as the digits of pi, say that across all factories that would ever get built over the course of an infinite number of universe samples, the average coin would have a bias of 55%, or the average factory would produce fair coins 36% of the time, or the average Earth would produce coin factory construction companies that yada yada yada. Sooner or later, you have to arrive at an unknown that could not possibly be construed as aleatoric.
And I’m not saying that that has to be where you draw the line! We just spent a whole essay discussing the inherent subjectivity of building a prior. What you choose to model as epistemic or aleatoric (or rather, what you make your top-level parameter) is inevitably going to be subjective simply because you don't have complete knowledge of how the world works; the best you can do is find the point where “things could be different” stops feeling coherent to you, and draw the line there.
But the distinction is not completely imagined, either. And as we saw in the “digit-of-pi” Sleeping Beauty problem, it can have real consequences for your anthropic predictions.
Maybe we could try to be clever and say the probability of each digit is 1/10 because if we select a digit at random the distribution would approach uniform? Even that wouldn’t necessarily work, because it has not been proven that pi is a normal number - it’s entirely conceivable that, say, the digit 2 appears more often than others. You would still need to appeal to some other notion of probability.
In the real world, we tend to measure predictors by the aggregate of multiple predictions. Presumably if our sociopath guessed “99% 7” for every digit, we would learn not to trust him.
[This is the fourth piece in my series on the Parametric Process Assumption (PPA), a new model of anthropics. You should start here if you are just seeing this for the first time.
The previous piece is "A Brief Defense of Indexical Reasoning."
TLDR:
The next piece is "Improper Priors and the Infinite SIA."]
Ways of not knowing
Before diving into the PPA and its implications, let’s take a moment to motivate it philosophically: what do we mean when we say that we are uncertain about something?
One idea I really like, but which hasn’t really made its way into the LessWrong water supply as much as I might expect, is that there are really at least two different types of uncertainty. The first kind, known as aleatoric uncertainty, is what we conventionally think of as randomness. The roll of a die, the flip of a coin, the configuration of a deck of cards dropped to the floor - that kind of thing.
The second kind is epistemic uncertainty. These are the things that have definitive, immutable truth values, but which are nevertheless unknown. We saw this in the digit-of-pi Sleeping Beauty problem from the introduction - a specific digit of pi is not, in any aleatoric sense, random, but we nonetheless may not know it and, for that reason, may choose to assign a probability to it. One could reasonably say, for instance, that the googolth digit of pi has a 1 in 10 chance of being 7.
A few notable properties separate these concepts. Epistemic uncertainty, for instance, is reducible. If I were to compute the googolth digit of pi, then I would know it forever, and in general the more evidence I have about a given epistemic fact, the more confident I would be in my estimates, at least in the limit. Aleatoric uncertainty, by way of contrast, never reduces. No matter how many times I flip my fair coin, there will always be a 50% chance that the next flip will come up heads.
In that sense, aleatoric uncertainty is the uncertainty of repetition, of what will happen “next time” or “in the limit.” When we assign a probability to an aleatoric uncertainty, we are doing so in a philosophically objective way: if you flipped a fair coin over and over and over again, the percentage of flips that went heads would get closer and closer to 50%. Even if you never actually make those flips, even if in practice it’s impossible to make that many flips, the abstraction is still mathematically well-defined.
Dice converging to their average value over many rolls. Courtesy of Wikipedia.
This definition of probability doesn’t apply to epistemic uncertainty. The googolth digit of pi cannot be resampled; it has always been whatever it currently is, and it will always go on being whatever it currently is. The one property that allowed us to define aleatoric uncertainty in an objective way - the hypothetical ability to resample - is of no use here. The digits of pi, along with countless other mathematical truths, and likely much of physics, are not and have not ever been random. The googolth digit of pi does not have a 10% chance of being 7; it has a 100% chance of being whatever it is, and we do not know what that might be.[1]
And yet, it seems clear that there must be some sense in which we are able to say that each digit has probability 10%. This is the driving force behind Bayesian statistics: that when confronted with epistemic uncertainty, we can quantify it by assigning our beliefs to a probability distribution and applying the same laws of probability that we used in the aleatoric setting.
That we can do this is by no means a given. There is an entire other school of statistics - the Frequentist school - that rejects the concept of epistemic probabilities. Mind you, Frequentists don’t reject the actual concept of epistemic uncertainty, but they don’t admit methods which assign probabilities to epistemic statements; in the world of Frequentism, one literally could not say that the googolth digit of pi had a 10% chance of being 7, or indeed being anything else. Confidence intervals, p-values and their ilk are all elaborate attempts to learn about the world without ever explicitly assigning credences to epistemic unknowns, and that is what makes them so famously counter-intuitive.
This aversion to epistemic probability is not completely unfounded. Unlike aleatoric uncertainty - which has an objective repetition-based interpretation - Bayesian/epistemic probabilities are inherently subjective. Bayesian priors represent a quantification of one’s beliefs, but there are no hard mathematics that can tell us where those beliefs come from. Priors are just guesses, and everyone’s guess might be different. You can’t even really say for sure if any one person’s prior is better than another’s. One prior may indeed prove closer to the truth than another, and any reasonable prior will converge to the truth with enough data, but it’s mathematically impossible to build a prior that’s always better in all situations when data is limited; this is known as the No Free Lunch Theorem.
I don’t want to oversell these flaws. If you’re hanging out on LessWrong, you probably already know the incredible effectiveness of Bayesian reasoning. But even with something as trivial as assigning a probability to pi’s googolth digit, No-Free-Lunch still applies: while only a sociopath would decide that there was a 99% chance that the googolth digit was 7, it’s still possible that the digit actually is 7, and in that case the sociopath’s prior is in some sense better than that of the normie who only gave 7 a 10% chance.[2] Which is all to say that while assigning probabilities to beliefs about epistemic uncertainty may be a highly effective and intuitive way to reason about the world, it is not an objective way to reason about the world.
Bayesian Parametric Modeling 101
[If you are already familiar with Parametric Bayesian models, you may wish to skip this section. Please do read the Closing Thoughts section though.]
Generally, in any real world situation, both types of uncertainty will be present. If you’re Bayesian then the most common way to model this is with what’s known as a parametric model.
The textbook example of a parametric model would be repeated flips of a biased coin. In this setup, you have a coin with some probability
If we could flip the coin an infinite number of times, then we could know
HTHTHHHHHH?One way to answer that, the standard Bayesian way, is to start with a probability distribution over possible values of
For the sake of illustration, we’ll use an example prior where we believe there are only three possible values for
The true value of
Each possible value of
Trying to apply this model is where one encounters the famous Bayes Rule. In LW and adjacent spaces, Bayes Rule is often presented as a fundamental idea everyone should be familiar with, but if you aren’t the type of person who likes looking at equations, I don’t think it’s often clear exactly what the rule truly means, beyond perhaps some generic statement about “updating on evidence.”
A more refined explanation of Bayes Rule, however, is that it inverts the parametric model we just introduced. The is dead is that parameters are the things we really want to know, but we can’t actually observe them directly, so instead we must observe other aleatoric randomness and try to guess the parameters underneath. Bayes Rule is just the formula we use to do that; it’s how we extract knowledge about parameters without actually seeing them.
For instance, suppose we kept our uniform prior over
In this particular case, seeing a single heads has really only caused us to move some of the probability mass away from
HTHTHHHHHH, then our prior would update even more sharply:Now we have a very sharp peak at
Parametric techniques are far and away the most common way Bayesian statistics is applied in the real world. Countless areas of fields like cosmology, medicine, and engineering rely on parametric models. Even modern LLMs are parametric in a sense - the size of neural networks is often measured in “parameters.” In that case, the sampling of the next output token is the equivalent of the individual coin flips in this section’s example; the model’s parameters define the process which is the LLM’s output.
Most formulations of anthropics that I’ve seen, however, don’t wrestle with this framing in a deep way. Just look at standard problems like Sleeping Beauty: a single coin is flipped, and then either one or two observers are created. There is no epistemic “prior” in a situation like that, there is simply a coin with completely known odds of heads. That's not a sufficient analogy for all the myriad complexities of real-world physics and epistomology. Something is clearly missing.
Closing Thoughts
A good way to think about parametric models is that they are mostly about nesting. In the previous section’s example (the coin of unknown bias), we saw that each possible value of
If you aren’t sticking around for the more technical portions of this series, this nested structure should be your biggest takeaway from this piece. Hopefully, the connection to the “digit-of-pi” Sleeping Beauty problem is clear: parameters behave SSAishly under repetition, but when we nest a process within them, that process behaves SIAishly. The PPA is (mostly) just a simple extension of parametric modeling to the case of indexical reasoning.
That said, there’s still an elephant in the room, which is that the line between epistemic and aleatoric is actually quite fuzzy. For instance, in the last section we imagined a coin where the value of
What if we push the reasoning up a layer? The coin in question must surely have come from a particular factory, and this factory must have some probability distribution over the coins it produces - perhaps it produces a fair coin only X% of the time, and we’d like to know X. Does this make X an epistemic unknown?
Maybe. Or maybe this factory is just one of a million factories built by the same construction company, and each one has a different chance of producing a fair coin, and the quality of our factory is just another random variable?[3]
You can play this game for a very long time, but you can’t play it forever. At some point, you’re going to hit a ceiling. Perhaps the Laws of Physics, governed by mathematical truths that are as fixed as the digits of pi, say that across all factories that would ever get built over the course of an infinite number of universe samples, the average coin would have a bias of 55%, or the average factory would produce fair coins 36% of the time, or the average Earth would produce coin factory construction companies that yada yada yada. Sooner or later, you have to arrive at an unknown that could not possibly be construed as aleatoric.
And I’m not saying that that has to be where you draw the line! We just spent a whole essay discussing the inherent subjectivity of building a prior. What you choose to model as epistemic or aleatoric (or rather, what you make your top-level parameter) is inevitably going to be subjective simply because you don't have complete knowledge of how the world works; the best you can do is find the point where “things could be different” stops feeling coherent to you, and draw the line there.
But the distinction is not completely imagined, either. And as we saw in the “digit-of-pi” Sleeping Beauty problem, it can have real consequences for your anthropic predictions.
The next piece is "Improper Priors and the Infinite SIA"
Maybe we could try to be clever and say the probability of each digit is 1/10 because if we select a digit at random the distribution would approach uniform? Even that wouldn’t necessarily work, because it has not been proven that pi is a normal number - it’s entirely conceivable that, say, the digit 2 appears more often than others. You would still need to appeal to some other notion of probability.
In the real world, we tend to measure predictors by the aggregate of multiple predictions. Presumably if our sociopath guessed “99% 7” for every digit, we would learn not to trust him.
Obligatory reference to Bayesian hierarchical modeling, which explicitly nests parameters inside other parameters