Introduction

This sequence is an attempt to sketch a unified framework for several interconnected questions: Where do Bayesian priors come from? What even are probabilities? How should we deal with infinite ethics? What's going on with anthropics? I hope to lay out both some of the existing answers and my own preferred synthesis.^[1]

I understand that many people have already thought about these questions, and I have only read portions of the existing literature. I think most of what I will write here, even in the section about my preferred synthesis, is not novel. People whose writing I'm building on include Wei Dai, Paul Christiano, Joe Carlsmith, Scott Garrabrant and Richard Ngo. I've also listened to some people like Lukas Finnveden, Vivek Hebbar and Ryan Greenblatt talk about related topics, which was also influential on me.^[2]

This first post will look at some possible definitions of probabilities and why I think they don't really work. Later posts will examine what we can best replace probabilities with.

What even are probabilities?

What do I mean when I say that I give a 10% probability that it's going to rain in my town tomorrow? This 10% probability doesn't refer to any tangible fact about the real world. Sure, there is some amount of objective randomness in whether it will rain or not tomorrow, due to quantum randomness. But I have no idea how big the quantum effects are on the weather tomorrow, and when I say I give a 10% chance for rain, I'm clearly not referring to the true quantum probabilities.

I'm also not satisfied with the frequentist view where you need to look at a series of sufficiently similar events in the past, and count the frequency with which the event happens. This view may be tenable for rain (though I still don't know how you define "sufficiently similar" days), but I don't know how you would apply it to any less generic question, like the probability that the Russia-Ukraine war ends in 2026.

The classical Bayesian view holds that probabilities are just my subjective credences; they only live in my head. I find this view appealing. Still, if someone tells me he thinks there is a 50% chance that Bigfoot is standing in the next room, I wouldn't just shrug and say "Yep, it's all subjective, like liking chocolate and vanilla ice cream. He says 50%, that's as good as any other probability estimate."

I intuitively think that giving a 50% probability for Bigfoot standing next door must be wrong in some important sense, so we will need to investigate more deeply what probabilities mean instead of just saying they are all subjective.

I will explore two common answers - one based on defining an objective prior for Bayesianism, and another based on defining probabilities through betting odds. I think both answers offer valuable insights that I will build on in later posts, but neither of them give a satisfactory definition of probabilities.

Probabilities from priors

When I try to predict what will happen next, I rely on past evidence. The reason I believe there is less than a 50% chance of Bigfoot standing in the next room is that I have looked into many rooms in my life and Bigfoot was in none of them, plus I have read about other people not encountering Bigfoot, plus I have some broader evidence on what kind of animals are found where.

However, relying on past evidence runs into the problem of induction.

The sun has risen every day, so I expect it will rise again tomorrow. But it is an equally valid hypothesis, equally fitting the evidence, that the laws of nature dictate the sun will rise every day until June 1, 2026, and never again. Why, on May 31st, do I still think the sun will probably rise?

Galilei observes that all objects fall at the same rate, and then encounters a tropical fruit he has never seen before. Should he assume this fruit also falls the same way? Russell playfully conjectures that there might be an intact teapot floating between Earth and Mars. Why do I expect our probes won't find it?

The traditional answer is something like a simplicity prior, also referred to as Occam's Razor. The laws of nature are supposed to be simple: they shouldn't differ for every particular object, they shouldn't contain arbitrary date-specific caveats, and complex objects like teapots shouldn't appear without a cause. But it's unclear what "simplest explanation" actually means, so we will need to explore that further.

Solomonoff induction

In Bayesian terms, everything I've observed in my life is evidence for and against various hypotheses. I started with some set of hypotheses that had some initial prior probabilities, and all my observations updated them. The question is: what were these starting hypotheses and prior probabilities, before I had any evidence at all?

One common answer is the Solomonoff induction. All hypotheses are assumed to be computable: everything I've observed was produced by a computer program, and the next observations will be produced by the same program. My prior distribution is based on program length on a Universal Turing Machine. A program of length n gets prior probability proportional to, let's say, .^[3] This way, the sum of all priors is finite and can be normalized to 1.

Then, I look at all the observations I have made so far, I do the Bayesian updating starting from this above-described prior, and that's how you make predictions about unknown events.

This matches our intuition nicely. If we have no evidence about whether the sun will cease to exist on June 1st, we should assign this low probability, because the program encoding a special caveat for June 1st is longer than one without it.

Problems with Solomonoff induction

It's tempting to say that one should define probabilities as the result of Solomonoff induction. Probabilities would be still subjective in the sense that no one can actually run the full Solomonoff induction, so we are all just giving our best guesses. But I can at least still say that the guy who gives 50% probability to Bigfoot standing next door is wrong in the sense that I'm confident that's not close to what the Solomonoff induction says.

There are several problems, however. I will not engage with the problem of Solomonoff induction being uncomputable^[4] - I think it would still provide a valuable philosophical grounding of probabilities even without it being computable. I will also not engage with the problems of the agent reasoning about itself, explained in the Embedded agency post.^[5] But there are some other problems I plan to engage with:

1. Why assume computability? I can't find it anymore, but Wei Dai has a very old post asking what we would do if an advanced alien civilization, who otherwise showed themselves to be trustworthy and benevolent, told us they had a halting oracle. Should we give 0% probability that they are telling the truth, given that our prior only contains computable universes and those can't have halting oracles in them? Why should we be so certain that all our observations are produced by a computer program? Isn't this a kind of arbitrary assumption?

2. Which Universal Turing Machine? Solomonoff induction weighs hypotheses by how long they are to write as programs on a Universal Turing Machine. But there are many different Universal Turing Machines - which one should we rely on? After all, there exists some convoluted Universal Turing Machine on which "the laws of physics plus Bigfoot standing next door in this particular moment" is actually a very short program, because Bigfoot-next-door is baked into the programming language.

Proponents of the Solomonoff induction like to point out that different choices of the UTM only lead to a finite constant factor difference in how big a probability Solomonoff induction assigns to various predictions, and with unlimited evidence, the results converge. But in practice, I don't have unlimited evidence. I want to decide whether to go next door, and I don't want to be eaten by Bigfoot. If my friend says Bigfoot is 50% likely to be there, I want to have some counter-argument, instead of just shrugging that there exists a UTM under which this is a reasonable estimate.

3. Description length of my observations, not the universe. Our intuition is that the laws of nature should be simple. But if I naively apply Solomonoff induction to my observations, the shortest program producing what I, David Matolcsi, am observing is not just a description of the laws of the universe. It's the laws of nature plus a pointer to my specific location in the universe. These two pointers together are hopefully still shorter than a raw dump of my observations.^[6] But now the simplicity prior operates not just over the laws of the universe, but also over my place in it. According to the Solomonoff-prior that gives probability to all n-long descriptions, the probability that I am in a moment whose shortest description is at least N-long should only be 1/N.^[7] This would imply that I'm probably in a simple-to-describe place in the universe, but it doesn't really look like it, especially if I take into account the quantum multiverse.

4. Simulations and malignity. As I explained in my previous post, and as discovered by Paul Christiano and others, the Solomonoff induction is malign. You can read my full post, but here is a brief summary.

It really looks like we are in a very special small region of space-time.

We live in the millennium when it's likely that our species either goes massively multi-planetary or dies. Every species goes through this crucial millennium at most once. Planets absorb only a small fraction of stellar energy, most planets don't naturally spawn life, a millennium is vanishingly short compared to a planet's history, and only a tiny fraction of energy during that millennium sustains biological minds reflecting on things.

This means an extremely small fraction of all negentropy^[8] in the history of the universe is used to power biological minds living in their species' crucial millennium. On the other hand, it seems plausible that a technologically mature, galaxy-spanning civilization can capture and put to their own use a large fraction of the negentropy of the universe.

I have no reason to think that the universe that looks like this one has an especially high prior in the Solomonoff-prior compared to many other, similarly large universes that sustain intelligent life. If there is even a one-in-a-billion chance that a powerful space-faring civilization dedicates even a one-in-a-billion fraction of its harvested resources to simulating minds that believe they are biological beings living through their crucial millennium, this vastly outweighs the real instances.^[9]

So if it looks like you are living in the crucial millennium of your species' history, you are probably in a simulation. But there are many different possible simulations, some quite short, some quite weird, many basically solipsistic (only simulating one decision of one or a few people). Given that short, solipsistic simulations are much cheaper to run, there are plausibly more of them.

So if you find yourself making a decision that might be important for the future of humanity (and this decision might be as mundane as publishing a blog post), then you should have a significant probability of being in a short solipsistic simulation. But then every probability estimate you make about your future ("will it rain when I step outside?") is heavily influenced by your expectations on what kind of simulation you might be in, and this can lead to very unintuitive results, which are contrary to how we normally think about probabilities.

In particular, if you try to make any important decision based on your all-things-considered probability estimate, then plausibly your probability estimates will be dominated by aliens trying to simulation-capture you to influence the predictions of your copies in base reality. Being influenceable by these simulation-captures is what's called the malignity of Solomonoff induction.

—-

While I think Solomonoff induction is a good starting point, and I will get back to it later in this sequence, I think these problems are serious enough that it's not reasonable to define probabilities as the result of Solomonoff induction. I think Problem 3 may be solvable with a different formalism (I will write more about this in my next post), but Problems 1, 2 and 4 afflict all formalized priors I can think of.

This makes me think that defining probabilities based on a formal prior is not a very useful concept, and doesn't really match how we normally think about probabilities.

Probabilities as betting odds

For most confusing philosophical questions, I think the best way to get out of the definitional quagmire is to try to form the questions in a way that is action-relevant. If I need to make an actual decision in a (possibly hypothetical) situation, that often clarifies my thinking, and dissolves the semantic squabbles that were irrelevant to the main question.

In the case of probabilities, I think it's often best to think of them as the betting odds at which I'd be indifferent between betting in either direction.

If the weather forecast says 37% chance of rain, and I trust it, then I'd accept a bet at 30% odds on rain but not at 40%. The point of indifference is 37%, so that's my probability. There must always be one set of betting odds at which I'm indifferent to betting, so this can be a coherent definition of probabilities.

Some people don't like these betting-based definitions, and insist that there must be something more real in probabilities than just how one would bet.^[10] I will write more about this in a future post, but for now I will just say that I'm myself very sympathetic to thinking in terms of bets. I believe basically everything can be formulated as a "bet", and I don't quite see what could be there about probabilities that can't be phrased this way.

"What do you anticipate happening?" From my perspective, anticipation is nothing else than thinking about the consequences of an event. That's useful if the event happens, and a waste of time if it doesn't. Therefore, whether I anticipate an event translates to whether I want to bet my time on thinking about it.

"Aren't you surprised by this event?" To me, surprisal is just getting into a situation that I didn't make plans for. It's equivalent to losing a bet: I wagered my time on thinking about the consequences of the other possibility, but the outcome that I didn't bet on had come to pass.

This leads me to believe that thinking in terms of what bets I would make is all there is to say about probabilities. However, the terms of the bets often get confusing, and I will eventually need to conclude that in some cases, thinking about probabilities is just not the right thing to do at all.

Sleeping Beauty

Before I go further in exploring this betting-based definition, I will introduce a famous puzzle in anthropics which will help illustrate some difficulties.

Sleeping Beauty is put to sleep by researchers. During the two days that her sleep will last, the researchers will briefly wake her up either once or twice, depending on the toss of a fair coin (heads: once; tails: twice). After each waking, they will put her back to sleep with a drug that makes her forget that waking. When Sleeping Beauty is woken up, what probability should she give that the coin toss is heads?

Some argue the answer should be ½: after all, she is predicting the result of a fair coin flip. Some argue it should be ⅓: if the experiment happened many times, then only about ⅓ of Sleeping Beauty's wake-ups would happen in situations where the coin landed on heads.

Sleeping Beauty taking bets

Let's try to solve this puzzle in terms of the betting-based definition.

Whenever Sleeping Beauty wakes up, she is offered a choice to bet $1 on the coin coming out heads. What are the betting odds where Sleeping Beauty should be indifferent to entering the bet?

With this operationalization, the answer is clearly 1/3: that translates to Sleeping Beauty making a bet at each awakening that she will pay $1 if the coin came up tails, and will gain $2 if it came up heads. Looking at this from before the experiment started: with 50% probability, the coin will land on heads, Beauty will be awakened once and will gain $2 on the bet. With 50% probability, the coin will land on tails, she will be awakened twice, and will lose $1 twice. This strategy generates 0 money in expectation, so a bet with an implied probability of 1/3 is what makes Sleeping Beauty indifferent.

The trouble with money-based definitions

However, operationalizing probabilities through monetary bets gets funky pretty quickly. What's the probability of hyperinflation in the next 10 years? If I operationalize "is it above 10%?" as "would I prefer one dollar conditional on no hyperinflation, or ten dollars conditional on hyperinflation?"—well, ten dollars during hyperinflation isn't worth much.

And it's not just inflation. Money's value correlates with all sorts of things. A marginal dollar has different value depending on how rich you will become. For a utilitarian, the value of a dollar is also dependent on how much leverage you have over the future; a dollar is more valuable if you have more leverage. For example, the number of alien civilizations affects your estimate of humanity's expected share of cosmic resources, and therefore affects how much you can expect to influence the cosmos from spending a dollar on AI safety work today. So it becomes confusing to operationalize your probabilities on whether aliens exist in the lightcone via hypotheticals on which odds you would bet on it.

All of this means that defining probabilities in terms of monetary bets is often not the right choice.

Betting on experiences

It might be more useful to imagine betting on experiences. The probability of an event is 10% if I'm indifferent between savoring a piece of chocolate if the event occurs versus savoring a piece of chocolate if a random number generator rolls below 0.10.^[11] I think Paul Christiano uses a definition like this in this comment to operationalize the probability of being in a simulation.

However, this seemingly reasonable definition also leads to some pretty strange places. For example, let's see how this changes the Sleeping Beauty analysis.

Suppose that whenever Beauty wakes up, she can receive a piece of chocolate if the coin landed on heads, or receive a piece of chocolate if an independent random number generator produces a number below p. We can define the p for which she is indifferent between the two choices as her probability of the coin landing heads.

This boils down to a value judgement: is waking up twice, eating the same type of chocolate both times, then forgetting both, twice as valuable as eating it once then forgetting it? If you think yes, it's exactly twice as good, then you should bet with ⅓ implied probability.

But you could also think that eating a chocolate once, or going through the exact same experience twice in a memory-wiped state are equally good. Then if you bet on heads, you get the experience with ½ probability, and if you bet on the random number generator, you get the experience at least once with probability. So the point of indifference is when , so according to this definition, Sleeping Beauty should give a probability to the coin landing on heads.

If you believe that eating two identical chocolates and forgetting them is somewhat better but not exactly twice better than eating the chocolate once,^[12] then under this definition, your probability of heads should be somewhere between 0.333 and 0.382, depending on your exact philosophical views.

I think the Sleeping Beauty problem is not just an edge-case. This dependency on your philosophical views on copied experiences is something that pops up whenever you try to reason about simulations and infinite universes if you define probabilities using the bets on experiences.

This is a pretty unnatural way for probabilities to work, so if you insist on defining probabilities, we should look for something else.

Betting on terminal values

Perhaps the cleanest definition uses an even more hypothetical terminal value: a new happy planet appearing somewhere far away, unaffected by anything on Earth. "Would I prefer a happy planet to appear if there's hyperinflation, or a happy planet to appear if the RNG rolls below 0.10?" If I'm indifferent, hyperinflation has a 10% probability, because the planet is far away and unaffected by indirect correlations.

In the Sleeping Beauty question, I think I'm back at ⅓ implied probability with this definition.

Unfortunately, even this breaks down for sufficiently abstract questions. "What's the probability of being in a simulation?"—where does the planet appear, inside or outside the simulation? "How many alien civilizations exist?"—depending on some philosophical considerations, at some point adding an extra planet to the already teeming alien life might have diminishing returns in value.

Altogether, I don't think there is a clean definition of probabilities based on betting that makes probabilities a useful concept in full generality.

Probabilities for the exotic and the mundane

Ultimately, what matters is not how I define probabilities, but how I make decisions. I will argue in my next two posts why I am mostly acting in a way as if I was assuming a materialistic world-view and that we are outside the simulation.

Under these assumptions, probabilities are a useful abstraction.

Probabilities in the mundane world

For mundane questions—rain, hyperinflation, AGI timelines—I mentally translate "probability" to what implied probabilities I would bet with if I was betting on far-away planets appearing, assuming that we don't live in a simulation and assuming a materialistic world-view.

Imagining probabilities in terms of these bets on terminal value is a useful definition for me. When I'm deciding whether to bring an umbrella with myself, I have some intuitive estimate of how much productivity it would cost me to get drenched in the rain and how much productivity it would cost to spend time on carrying and storing my umbrella. I try to work on things that matter for my terminal values, so productivity translates to value. So once I know how I would bet in terms of terminal values (e.g. far-away happy planets appearing), I can use that information in an expected value calculation for various decisions related to rain: whether to bring an umbrella, whether to bring a rain jacket, whether to invest in farm-land, etc. This makes probabilities a useful abstraction for mundane questions.^[13]

Letting go of probabilities

For philosophically confusing questions involving anthropics and the simulation hypothesis, I refuse to answer with probabilities and instead ask what exact bet we are hypothetically making, or what action we need to decide on. This makes me reluctant to pick a side in the SIA vs SSA debate in anthropics; I just don't believe it's the right level of abstraction to ask these questions. (Though SIA is generally closer to the mark in my opinion.)

Similarly, I can't in good-faith respond with probabilities to questions that don't make sense under materialistic assumptions, like "what is the probability that Jesus rose from the dead?" Amending "…assuming a materialistic universe" defeats the purpose of the question. It's a somewhat awkward position that I can't give straightforward probabilities if someone asks about Jesus, and instead I need to say that "for complicated philosophical reasons, I'm mostly acting as if he was an ordinary human".^[14] But I maintain that there is no good way to put probabilities on this question - Jesus rising from the dead is deep into the territory where probabilities stop being a useful abstraction.

Once I give a probability to Jesus rising from the dead, how do I deal with Pascal's Wager, with infinite reward standing on one side? In my next posts, I will discuss infinite ethics and dealing with the supernatural, but this will require going beyond natural notions of probabilities.

Also, if you insist on using probabilities, what is the probability that you are in a short solipsistic simulation now? And given that you are reading about Jesus right now, what's the probability that Jesus is indeed a centrally important character in a larger simulation and now the simulators are just testing how you are thinking about this character? As I said above, I ignore simulations when asked for probabilities of mundane events, and I will present arguments for this choice in a later post. But given how similar gods and simulators are, it feels unfair to silently add "assuming we are not in any kind of simulation" when someone asks a question about the Son of God.

Finally, if you want to define probabilities outside mundane questions, you need to have some resolution to the SIA vs SSA question in anthropics. I'm sympathetic to Joe Carlsmith's arguments that SIA is generally more reasonable, and this would imply that we should accept the Presumptuous Philosopher's logic that we are more likely to be in worlds with more observers similar to us. But how does this interact with the supernatural? Did you know that a prominent strain within Mormon theology claims that we are in an infinite causal chain where people ascend to godhood and create new worlds - a chain of creation without start or end?^[15]

I will try to deal with all these considerations about the supernatural in a later post, but that will not be based on the concept of probabilities anymore.

Conclusion

Altogether, I think probabilities are a useful abstraction under some circumstances, but for the more complex questions I need to fall back to a basic question:^[16] I want to choose between action A and B, and taking into account all considerations, I want to know which action leads to a better world according to my values.

Of course, this is easier said than done. When I'm deciding whether to bring an umbrella with myself, I'm helping the versions of myself that live in worlds where it's going to rain, and I'm inconveniencing the versions of myself that live in worlds where it's not going to rain. So I will need some method to weigh against each other the consequences of my actions in infinite possible worlds. I will write more about my proposed solutions in my next posts, but I believe that probabilities are not the right abstraction to handle these questions in general.

1. ^{^}

   For the avoidance of doubt: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

2. ^{^}

    I’m only familiar with the LessWrong line of thought on these topics. I’m woefully unaware of the academic philosophy tradition, and I’m possibly rediscovering ideas that appeared there too.

   It's also the case that most of the prior work I read is scattered across many, often very confusingly written blog posts, and I can't easily tell where I first came across various ideas I'm exploring here. Therefore, I will not try to do a full exegesis of where each idea came from, and will instead present the arguments as a unified flow, with only occasional direct references to the work of prior authors. It's also very possible that there are important insights that I missed that people have already written on these topics - in that case, feel encouraged to link to them in the comments.

3. ^{^}

    If the prior probabilities were only proportional to then the overall probabilities of n-length programs would add up to 1 for every n, and the full sum would be infinite. So we need a somewhat stronger decay in probabilities - now the overall probability of n-length programs is , and the sum of these is finite. We could have also chosen a different decay factor that ensures a finite sum.

4. ^{^}

    That means there is no algorithm that can compute the Solomonoff-prior of strings up to arbitrary precision.

5. ^{^}

    I think the problems of embedded agency might be important; I just haven’t really engaged with them yet.

6. ^{^}

    Otherwise, if I believed there were no universe laws plus location pointer that were simpler than my raw observations, then I’d basically think of myself as a Boltzmann-brain and I couldn’t predict any next observations.

7. ^{^}

    The overall prior of all n-long descriptions is , and summing from N to infinity is approximately 1/N.

8. ^{^}

    I’m not a physicist and I’m not actually sure that negentropy is the right term here, but something like this seems right.

9. ^{^}

    There is some complication that maybe the real crucial millennium has unusually short description-length, so it gets relatively large weight within the universes. But I believe that the rest of space-time likely still holds much larger weight, so turning a fraction of that into simulations still outweighs the real crucial millennium.

10. ^{^}

    For example, Joe Carlsmith expresses skepticism of defining everything through betting in this post.

11. ^{^}

     I love chocolate.

12. ^{^}

     This is the view that matches my intuition.

13. ^{^}

     Of course, in practice, when I’m deciding whether to bring an umbrella with myself, I’m not thinking exclusively in terms of work productivity. I’m often thinking in terms of how things would make me feel. Ideally I would only take my well-being into account to the extent it matters for productivity and wisdom to make the world better. In the rest of this series, I will implicitly rely on the assumption that my only goal is trying to pursue the scope-sensitive Good (otherwise, the entire theory I’m building here kind of goes haywire). I actually aspire to live like that, though of course I can’t promise I’m always living up to this ideal - the spirit is willing but the flesh is weak. 

14. ^{^}

     I will write a bit more about how I relate to existing religions in a later post.

15. ^{^}

     I would love to read someone sincerely making this SIA argument for Mormonism. Unfortunately, I couldn’t find any examples of this on the internet.

16. ^{^}

     Arguably the only important type of question that exists