I have similar thoughts on this and I appreciate that you articulated these things clearly. And I look forward to reading about your proposed solutions!
Especially agree with this: "For philosophically confusing questions involving anthropics and the simulation hypothesis, I refuse to answer with probabilities and instead ask what exact bet we are hypothetically making, or what action we need to decide on. " I have found myself saying something like "I don't want to give an answer to P(doom), because I think answering this question ends getting into things like the simulation hypothesis and anthropics and the existence of god and such."
Perhaps there's ultimately a "better" (less wrong) conception of things that would replace the concept of probabilities with something else. I think the same is true for the concepts of truth and morality, although I have no idea what the better conceptions would be. I hope to write a post about this. I do think that truth has a lot of parallels with probability in terms of what the issues for it look like (and they are also deeply related concepts); maybe your proposed solution will give me sone ideas.
(Side note: "The Solomonoff induction is malign." Wtf is this usage of 'malign'? I know that people have used it to describe the Solomonoff induction in the past, but why did they choose that word?)
I like my strategy of silently amending "assuming a materialistic world-view and not being in a sim" to all mundwne questions. I think that makes it possible to non-misleadingly communicate about the probability of AI takeover.
(This is assuming that you agree that you should mostly act as if you were in a materialistic non-sim world.)
"The Solomonoff induction is malign." Wtf is this usage of 'malign'? I know that people have used it to describe the Solomonoff induction in the past, but why did they choose that word?
This is referring to the claim made by paulfchristiano that an AI using SI concludes that it is in a simulation due to its extremely high-leverage position in the universe(as explained in the post) and also that the simulators are doing the simulation in order to manipulate the prior probabilities in a way which favors the AI performing actions the simulators happen to desire. Leading the AI to be malign in the sense that it acts in line with the interests of whatever kind of simulators it considers likely, instead of the interests someone tried to program into it, because its prior probabilities are warped to believe that actions taken towards the former interests actually help the latter.
I may be misunderstanding the intended scope of the post, but currently the argument reads to me more like a critique of some probabilistic frameworks than a critique of probabilistic reasoning in general.
Epistemic status: similar to author, most prior work I read is scattered across many, often very confusingly written blog posts, and I can't easily tell where I first came across various ideas. I tried to focus on "general" deductive logic based on my reading of the post (which may be wrong) instead of applying stuff that is too framework-specific.
I will also provide feedback on some wording (seems like author tried too hard to make post streamlined and/or conform to style norms.)
This first post will look at some possible definitions of probabilities and why I think they don't really work
Don't really work for what, more exactly? Descriptive account of how agents (humans) interpret probability? Account of how probability should be conceptually seen? How we should use probability for epistemic rationality? Instrumental rationality? It seems that you want something for epistemic and instrumental rationality based on the rest of the post, but I think it would be better if you clarified from the start.
What do I mean when I say that I give a 10% probability that it's going to rain in my town tomorrow? This 10% probability doesn't refer to any tangible fact about the real world. Sure, there is some amount of objective randomness in whether it will rain or not tomorrow, due to quantum randomness. But I have no idea how big the quantum effects are on the weather tomorrow, and when I say I give a 10% chance for rain, I'm clearly not referring to the true quantum probabilities.
I'm not sure if this is a general problem? If A told me their probability, the following interpretation seems reasonable: "According to A's computations, which probably weren't at a level of precision/detail involving quantum uncertainty, in 10% of their predicted worlds rain would happen tomorrow in my town.)".
Formal form:
I'm also not sure if the frequentist critique is that relevant. I'm unsure whether frequentists wanted to generalize work based on based instances to everything, including the Russian invasion of Ukraine. Some space could be dedicated to other frameworks (the one above is based on my intuition).
The classical Bayesian view holds that probabilities are just my subjective credences; they only live in my head. I find this view appealing. Still, if someone tells me he thinks there is a 50% chance that Bigfoot is standing in the next room, I wouldn't just shrug and say "Yep, it's all subjective, like liking chocolate and vanilla ice cream. He says 50%, that's as good as any other probability estimate."
I assume you wanted to move quickly to the Solomonoff induction part, but that is not sufficient evidence against general bayesianism. Dismisssing it via an example of aburd priors, which most Bayesians would disagree with and (presumably) try to fix from inside the framework, is also suspicious.
On Solomonoff priors producing unintuive results:
I don't see how being unintuitive compared to a naive conception of probability is evidence against probability-in-general/Solomonoff induction, instead of being evidence either the naive conception of probability being bad or some assumption being false. In fact, lacking other adequate explanations for the (apparent) plausibility other worlds being solipsistic simulations, I would be rationally be forced to take them into consideration and, implicitly, heighten my credence in the Solomonoff prior, if your link to them is assumed.[1]
This makes me think that defining probabilities based on a formal prior is not a very useful concept, and doesn't really match how we normally think about probabilities.
I agree that formal priors like Solomonoff induction are bad (or at least incomplete). However, you are forced to base your theory on some priors (more exactly, priors derived from biology). I don't think priors being a formal component would make a hypothetical theory "worse", or that lacking formal priors would make a theory intrinsically better.
Also, basing your logic solely on the failures of Solomonoff priors is not valid by itself[2]. Why wouldn't the conclusion be something like "I don't know"/"We don't know"? In general, it seems to me that your attempts to make it streamlined makes it feel like the post is overly focused on defeating a selection of theories with well-known flaws and implying "ergo, only this framework can save us"[3] instead of accepting uncertainty.
For most confusing philosophical questions, I think the best way to get out of the definitional quagmire is to try to form the questions in a way that is action-relevant. If I need to make an actual decision in a (possibly hypothetical) situation, that often clarifies my thinking, and dissolves the semantic squabbles that were irrelevant to the main question
In the case of probabilities, I think it's often best to think of them as the betting odds at which I'd be indifferent between betting in either direction.
In regards to your proposed solution for operationalization: Why is winning hypothetical bets action-relevant? Why would an agent want to calibrate their probabilities of X via optimizing bets specifically on money, chocolate, or hypothetical terminal values? It seems like redundant mental gymnastics to get an equivalent result at best. At most, you can imagine hypothetical bets where you get fixed utility instead of chocolates etc., but you can equivalently frame that as optimizing a number instead of gambling.
I personally haven't been blocked by such definitional quagmires from directly calculating probabilities and expected utilities. I think that the framing it as "this philosophical problem highlights a possible bias/irrational aspect in my calculation algorithm" is better than the "These paradoxes prove that the concept of probability is incoherent/not useful".
Some people don't like these betting-based definitions, and insist that there must be something more real in probabilities than just how one would bet.[10] I will write more about this in a future post, but for now I will just say that I'm myself very sympathetic to thinking in terms of bets. I believe basically everything can be formulated as a "bet", and I don't quite see what could be there about probabilities that can't be phrased this way.
Stuff that can't be phrased this way: the definition; conceptual clarity. "Winning hypothetical bets" and "measure of how likely something is" seem conceptually distinct, even if you can apply bets for equivalent results.
For philosophically confusing questions involving anthropics and the simulation hypothesis, I refuse to answer with probabilities and instead ask what exact bet we are hypothetically making, or what action we need to decide on. This makes me reluctant to pick a side in the SIA vs SSA debate in anthropics; I just don't believe it's the right level of abstraction to ask these questions. (Though SIA is generally closer to the mark in my opinion.)
Isn't the "bet" in such problems implied to be simply the "truth"/"best representation of (some part of) reality under the conditions of the problem"? Probabilities abstract utilities away, yes, but are also work for every utility function.
Forcing probability discussions into stuff like "if the Sleeping Beauty woke up and got the utility equivalent of a chocolate under [betting conditions]" is logically suspicious (why are conceptions of probability other than bets not explored, more precisely? The post seems to assume by default bets are better.).
On probabilities for infinitesimal/supernatural scenarios, including pascalian wagers: using hyperreals or surreal probabilities works mathematically, and arguably still count as "natural notions" of probability. I don't know how common this position is, or how to calibrate those infinitesimals specifically[4], but I think it merits consideration.
So I will need some method to weigh against each other the consequences of my actions in infinite possible worlds. I will write more about my proposed solutions in my next posts, but I believe that probabilities are not the right abstraction to handle these questions in general.
I kind of agree with the conclusion, but as a kind of lemma based on properties of expectationalism. You can ignore the probabilities of nihilistic worlds or other "decisionally irrelevant" stuff, simply because the utilities would be forced to be 0. You are also right that ultimately we want to make decisions, but this post hasn't convinced me why one should abolish probability-weighed expectationalism to determine the right action instead of using computational tricks or refining the framework.[5]
I personally don't think that "Current technological abilities imply we likely live in nihilistic simulations" is that positively correlated specifically with Solomonoff priors, but I may be wrong.
I think you agree with your "This makes me think" hedging, but I wanted to point it out explicitly.
Sorry for exaggerated phrasing.
As you implied, it's enough to calibrate enough in an action-relevant way (e.g. "whether to follow the wager"), though I consider that to be more of a computational trick.
By the way, the way you phrased it in conclusions made it too similar to the fallacious "Abstractions are too weak for real phenomena" for my liking.
In the general case, I ultimately think probabilities are caring measures, or to put it another way this is just another thing UDT got totally right (and the mainstream decision theories got this point very wrong).
The main reason for this is that when we attempt to try to focus on arbitrary worlds/thought experiments, we forget that any prior is just as good as any other based on only objective measures, and priors/probabilities become as arbitrary as values.
I'd say one of the main insights of UDT (and possibly FDT/EDT) is that probabilities are caring measures, not about the states of the worlds in and of themselves.
Yes, I will make this point in my next post. (I'm not sure though if probabilities being caring measures is a necessary consequence of UDT. I thought this was a different axis.)
Cool post! I have random thoughts you may or may not find interesting:
In your discussion of Sleeping Beauty eating chocolate, you are assuming (like many others, including the originator of the problem, Elga) that Beauty has exactly the same experiences each time she is woken, if she is woken twice. (Or at least, you are assuming that one can stipulate this without changing the answer.) If not, two experiences of delightful chocolate fairly clearly should count double only one such experience, just as we would count them double if two different people ate chocolate.
But this is not consistent with Sleeping Beauty being human. Humans cannot have identical experiences at different times, even in principle, assuming present physical theory is correct. It would contradict the "quantum no-cloning" theorem. Also, it would turn the Sleeping Beauty problem from one that is almost doable - just needing a good memory erasure drug, which is quite conceivable seeing as we know of things (like a blow to the head) that can cause memories to be lost - into a completely fantastic problem. Highly fantastic thought experiments are dubious guides to anything.
Similar problems arise when considering Boltzman brains, infinite universes, and the possibility that we are in a simulation. These all raise numerous philosophical issues. Trying to use them to figure out how to reason with (or without) probabilities seems dubious, unless you resolve all the other philosophical issues they present at the same time. Otherwise, you run the risk of assuming a strange, wild, highly unintuive universe and then reasoning about what it says concerning probability using arguments that would be seen to contradict this assumtion if one truly understood what it implied.
These comments relate somewhat to my paper at https://arxiv.org/abs/math/0608592
Re the questions of "Why assume computability" and "Which universal Turing machine": I have a strong suspicion that if you compare your favorite UTM with no halting oracle and any other "natural non-obnoxious" UTM with a halting oracle to the whole arithmetic hierarchy (or beyond, if you wish), you get basically the same posterior probabilities of events given your observation history.
Re "Description length of my observations, not the universe": my physics is spotty so this phrasing might not be exactly right but keep in mind that you don't need the exact "starting seed" of the universal wave function + your exact "spot" in it; you just need enough to describe the simplest-to-describe seed/spot that aligns with your observations. My hunch is that this is going to be much shorter than the raw dump of your observations
Probability is a measure function, representing how often events are realized among iterations of probability experiment. The latter is a certain approximation of some real world scenario to the best of your knowledge. A map to a territory, if you will. Probabilities are "subjective" in a sense that they are properties of the map. But they are "objective" in a sense that this map represents the territory. I'm leaving you a link to my sequence on probability theory. It's unfinished, but I believe that even in its current state it can be quite helpful for some of the questings you are raising here.
I'd recommend to be careful with invoking betting odds. Yes, it is a great validator fo correctness of probabilistic estimates, but it requires to invoke an additional measure function - the whole mathematical apparatus for utilities which is an extra complication and therefore an extra opportunity to get confused. Probability is one thing, utility is the other. If you already feel confused with the former, adding the latter probably isn't going to make you less confused. It's better to go back to the basics.
Even more so with "anthropic" scenarious. Appealing to Sleeping Beauty problem while trying to resolve a general confusion about probabilities is like trying to use a metaphor from quantuum mechanics while discussing philosophy. Unless you and the audience are experts in the field, most likely you are going to make yourself and everyone else only more confused.
Cudos for not wanting to pick side in the SSA vs SIA debate. It is a false dylema between two terrible options. You can do so much better than either of them.
I have no reason to think that the universe that looks like this one has an especially high prior in the Solomonoff-prior compared to many other, similarly large universes that sustain intelligent life. If there is even a one-in-a-billion chance that a powerful space-faring civilization dedicates even a one-in-a-billion fraction of its harvested resources to simulating minds that believe they are biological beings living through their crucial millennium, this vastly outweighs the real instances.
Specifically in this situation, I don't think our actions should be any different. (see also) Assuming that simulations and reality are indistinguishable (to us), then I think we should regard our actions as affecting both cases at once.
"What were these starting hypotheses and prior probabilities, before I had any evidence at all?" This maps on nicely to the classic Zen koan, "What was your original face before your parents were born?".
Introduction
This sequence is an attempt to sketch a unified framework for several interconnected questions: Where do Bayesian priors come from? What even are probabilities? How should we deal with infinite ethics? What's going on with anthropics? I hope to lay out both some of the existing answers and my own preferred synthesis.[1]
I understand that many people have already thought about these questions, and I have only read portions of the existing literature. I think most of what I will write here, even in the section about my preferred synthesis, is not novel. People whose writing I'm building on include Wei Dai, Paul Christiano, Joe Carlsmith, Scott Garrabrant and Richard Ngo. I've also listened to some people like Lukas Finnveden, Vivek Hebbar and Ryan Greenblatt talk about related topics, which was also influential on me.[2]
This first post will look at some possible definitions of probabilities and why I think they don't really work. Later posts will examine what we can best replace probabilities with.
What even are probabilities?
What do I mean when I say that I give a 10% probability that it's going to rain in my town tomorrow? This 10% probability doesn't refer to any tangible fact about the real world. Sure, there is some amount of objective randomness in whether it will rain or not tomorrow, due to quantum randomness. But I have no idea how big the quantum effects are on the weather tomorrow, and when I say I give a 10% chance for rain, I'm clearly not referring to the true quantum probabilities.
I'm also not satisfied with the frequentist view where you need to look at a series of sufficiently similar events in the past, and count the frequency with which the event happens. This view may be tenable for rain (though I still don't know how you define "sufficiently similar" days), but I don't know how you would apply it to any less generic question, like the probability that the Russia-Ukraine war ends in 2026.
The classical Bayesian view holds that probabilities are just my subjective credences; they only live in my head. I find this view appealing. Still, if someone tells me he thinks there is a 50% chance that Bigfoot is standing in the next room, I wouldn't just shrug and say "Yep, it's all subjective, like liking chocolate and vanilla ice cream. He says 50%, that's as good as any other probability estimate."
I intuitively think that giving a 50% probability for Bigfoot standing next door must be wrong in some important sense, so we will need to investigate more deeply what probabilities mean instead of just saying they are all subjective.
I will explore two common answers - one based on defining an objective prior for Bayesianism, and another based on defining probabilities through betting odds. I think both answers offer valuable insights that I will build on in later posts, but neither of them give a satisfactory definition of probabilities.
Probabilities from priors
When I try to predict what will happen next, I rely on past evidence. The reason I believe there is less than a 50% chance of Bigfoot standing in the next room is that I have looked into many rooms in my life and Bigfoot was in none of them, plus I have read about other people not encountering Bigfoot, plus I have some broader evidence on what kind of animals are found where.
However, relying on past evidence runs into the problem of induction.
The sun has risen every day, so I expect it will rise again tomorrow. But it is an equally valid hypothesis, equally fitting the evidence, that the laws of nature dictate the sun will rise every day until June 1, 2026, and never again. Why, on May 31st, do I still think the sun will probably rise?
Galilei observes that all objects fall at the same rate, and then encounters a tropical fruit he has never seen before. Should he assume this fruit also falls the same way? Russell playfully conjectures that there might be an intact teapot floating between Earth and Mars. Why do I expect our probes won't find it?
The traditional answer is something like a simplicity prior, also referred to as Occam's Razor. The laws of nature are supposed to be simple: they shouldn't differ for every particular object, they shouldn't contain arbitrary date-specific caveats, and complex objects like teapots shouldn't appear without a cause. But it's unclear what "simplest explanation" actually means, so we will need to explore that further.
Solomonoff induction
In Bayesian terms, everything I've observed in my life is evidence for and against various hypotheses. I started with some set of hypotheses that had some initial prior probabilities, and all my observations updated them. The question is: what were these starting hypotheses and prior probabilities, before I had any evidence at all?
One common answer is the Solomonoff induction. All hypotheses are assumed to be computable: everything I've observed was produced by a computer program, and the next observations will be produced by the same program. My prior distribution is based on program length on a Universal Turing Machine. A program of length n gets prior probability proportional to, let's say, .[3] This way, the sum of all priors is finite and can be normalized to 1.
Then, I look at all the observations I have made so far, I do the Bayesian updating starting from this above-described prior, and that's how you make predictions about unknown events.
This matches our intuition nicely. If we have no evidence about whether the sun will cease to exist on June 1st, we should assign this low probability, because the program encoding a special caveat for June 1st is longer than one without it.
Problems with Solomonoff induction
It's tempting to say that one should define probabilities as the result of Solomonoff induction. Probabilities would be still subjective in the sense that no one can actually run the full Solomonoff induction, so we are all just giving our best guesses. But I can at least still say that the guy who gives 50% probability to Bigfoot standing next door is wrong in the sense that I'm confident that's not close to what the Solomonoff induction says.
There are several problems, however. I will not engage with the problem of Solomonoff induction being uncomputable[4] - I think it would still provide a valuable philosophical grounding of probabilities even without it being computable. I will also not engage with the problems of the agent reasoning about itself, explained in the Embedded agency post.[5] But there are some other problems I plan to engage with:
1. Why assume computability? I can't find it anymore, but Wei Dai has a very old post asking what we would do if an advanced alien civilization, who otherwise showed themselves to be trustworthy and benevolent, told us they had a halting oracle. Should we give 0% probability that they are telling the truth, given that our prior only contains computable universes and those can't have halting oracles in them? Why should we be so certain that all our observations are produced by a computer program? Isn't this a kind of arbitrary assumption?
2. Which Universal Turing Machine? Solomonoff induction weighs hypotheses by how long they are to write as programs on a Universal Turing Machine. But there are many different Universal Turing Machines - which one should we rely on? After all, there exists some convoluted Universal Turing Machine on which "the laws of physics plus Bigfoot standing next door in this particular moment" is actually a very short program, because Bigfoot-next-door is baked into the programming language.
Proponents of the Solomonoff induction like to point out that different choices of the UTM only lead to a finite constant factor difference in how big a probability Solomonoff induction assigns to various predictions, and with unlimited evidence, the results converge. But in practice, I don't have unlimited evidence. I want to decide whether to go next door, and I don't want to be eaten by Bigfoot. If my friend says Bigfoot is 50% likely to be there, I want to have some counter-argument, instead of just shrugging that there exists a UTM under which this is a reasonable estimate.
probability to all n-long descriptions, the probability that I am in a moment whose shortest description is at least N-long should only be 1/N.[7] This would imply that I'm probably in a simple-to-describe place in the universe, but it doesn't really look like it, especially if I take into account the quantum multiverse.
3. Description length of my observations, not the universe. Our intuition is that the laws of nature should be simple. But if I naively apply Solomonoff induction to my observations, the shortest program producing what I, David Matolcsi, am observing is not just a description of the laws of the universe. It's the laws of nature plus a pointer to my specific location in the universe. These two pointers together are hopefully still shorter than a raw dump of my observations.[6] But now the simplicity prior operates not just over the laws of the universe, but also over my place in it. According to the Solomonoff-prior that gives
4. Simulations and malignity. As I explained in my previous post, and as discovered by Paul Christiano and others, the Solomonoff induction is malign. You can read my full post, but here is a brief summary.
It really looks like we are in a very special small region of space-time.
We live in the millennium when it's likely that our species either goes massively multi-planetary or dies. Every species goes through this crucial millennium at most once. Planets absorb only a small fraction of stellar energy, most planets don't naturally spawn life, a millennium is vanishingly short compared to a planet's history, and only a tiny fraction of energy during that millennium sustains biological minds reflecting on things.
This means an extremely small fraction of all negentropy[8] in the history of the universe is used to power biological minds living in their species' crucial millennium. On the other hand, it seems plausible that a technologically mature, galaxy-spanning civilization can capture and put to their own use a large fraction of the negentropy of the universe.
I have no reason to think that the universe that looks like this one has an especially high prior in the Solomonoff-prior compared to many other, similarly large universes that sustain intelligent life. If there is even a one-in-a-billion chance that a powerful space-faring civilization dedicates even a one-in-a-billion fraction of its harvested resources to simulating minds that believe they are biological beings living through their crucial millennium, this vastly outweighs the real instances.[9]
So if it looks like you are living in the crucial millennium of your species' history, you are probably in a simulation. But there are many different possible simulations, some quite short, some quite weird, many basically solipsistic (only simulating one decision of one or a few people). Given that short, solipsistic simulations are much cheaper to run, there are plausibly more of them.
So if you find yourself making a decision that might be important for the future of humanity (and this decision might be as mundane as publishing a blog post), then you should have a significant probability of being in a short solipsistic simulation. But then every probability estimate you make about your future ("will it rain when I step outside?") is heavily influenced by your expectations on what kind of simulation you might be in, and this can lead to very unintuitive results, which are contrary to how we normally think about probabilities.
In particular, if you try to make any important decision based on your all-things-considered probability estimate, then plausibly your probability estimates will be dominated by aliens trying to simulation-capture you to influence the predictions of your copies in base reality. Being influenceable by these simulation-captures is what's called the malignity of Solomonoff induction.
—-
While I think Solomonoff induction is a good starting point, and I will get back to it later in this sequence, I think these problems are serious enough that it's not reasonable to define probabilities as the result of Solomonoff induction. I think Problem 3 may be solvable with a different formalism (I will write more about this in my next post), but Problems 1, 2 and 4 afflict all formalized priors I can think of.
This makes me think that defining probabilities based on a formal prior is not a very useful concept, and doesn't really match how we normally think about probabilities.
Probabilities as betting odds
For most confusing philosophical questions, I think the best way to get out of the definitional quagmire is to try to form the questions in a way that is action-relevant. If I need to make an actual decision in a (possibly hypothetical) situation, that often clarifies my thinking, and dissolves the semantic squabbles that were irrelevant to the main question.
In the case of probabilities, I think it's often best to think of them as the betting odds at which I'd be indifferent between betting in either direction.
If the weather forecast says 37% chance of rain, and I trust it, then I'd accept a bet at 30% odds on rain but not at 40%. The point of indifference is 37%, so that's my probability. There must always be one set of betting odds at which I'm indifferent to betting, so this can be a coherent definition of probabilities.
Some people don't like these betting-based definitions, and insist that there must be something more real in probabilities than just how one would bet.[10] I will write more about this in a future post, but for now I will just say that I'm myself very sympathetic to thinking in terms of bets. I believe basically everything can be formulated as a "bet", and I don't quite see what could be there about probabilities that can't be phrased this way.
"What do you anticipate happening?" From my perspective, anticipation is nothing else than thinking about the consequences of an event. That's useful if the event happens, and a waste of time if it doesn't. Therefore, whether I anticipate an event translates to whether I want to bet my time on thinking about it.
"Aren't you surprised by this event?" To me, surprisal is just getting into a situation that I didn't make plans for. It's equivalent to losing a bet: I wagered my time on thinking about the consequences of the other possibility, but the outcome that I didn't bet on had come to pass.
This leads me to believe that thinking in terms of what bets I would make is all there is to say about probabilities. However, the terms of the bets often get confusing, and I will eventually need to conclude that in some cases, thinking about probabilities is just not the right thing to do at all.
Sleeping Beauty
Before I go further in exploring this betting-based definition, I will introduce a famous puzzle in anthropics which will help illustrate some difficulties.
Sleeping Beauty is put to sleep by researchers. During the two days that her sleep will last, the researchers will briefly wake her up either once or twice, depending on the toss of a fair coin (heads: once; tails: twice). After each waking, they will put her back to sleep with a drug that makes her forget that waking. When Sleeping Beauty is woken up, what probability should she give that the coin toss is heads?
Some argue the answer should be ½: after all, she is predicting the result of a fair coin flip. Some argue it should be ⅓: if the experiment happened many times, then only about ⅓ of Sleeping Beauty's wake-ups would happen in situations where the coin landed on heads.
Sleeping Beauty taking bets
Let's try to solve this puzzle in terms of the betting-based definition.
Whenever Sleeping Beauty wakes up, she is offered a choice to bet $1 on the coin coming out heads. What are the betting odds where Sleeping Beauty should be indifferent to entering the bet?
With this operationalization, the answer is clearly 1/3: that translates to Sleeping Beauty making a bet at each awakening that she will pay $1 if the coin came up tails, and will gain $2 if it came up heads. Looking at this from before the experiment started: with 50% probability, the coin will land on heads, Beauty will be awakened once and will gain $2 on the bet. With 50% probability, the coin will land on tails, she will be awakened twice, and will lose $1 twice. This strategy generates 0 money in expectation, so a bet with an implied probability of 1/3 is what makes Sleeping Beauty indifferent.
The trouble with money-based definitions
However, operationalizing probabilities through monetary bets gets funky pretty quickly. What's the probability of hyperinflation in the next 10 years? If I operationalize "is it above 10%?" as "would I prefer one dollar conditional on no hyperinflation, or ten dollars conditional on hyperinflation?"—well, ten dollars during hyperinflation isn't worth much.
And it's not just inflation. Money's value correlates with all sorts of things. A marginal dollar has different value depending on how rich you will become. For a utilitarian, the value of a dollar is also dependent on how much leverage you have over the future; a dollar is more valuable if you have more leverage. For example, the number of alien civilizations affects your estimate of humanity's expected share of cosmic resources, and therefore affects how much you can expect to influence the cosmos from spending a dollar on AI safety work today. So it becomes confusing to operationalize your probabilities on whether aliens exist in the lightcone via hypotheticals on which odds you would bet on it.
All of this means that defining probabilities in terms of monetary bets is often not the right choice.
Betting on experiences
It might be more useful to imagine betting on experiences. The probability of an event is 10% if I'm indifferent between savoring a piece of chocolate if the event occurs versus savoring a piece of chocolate if a random number generator rolls below 0.10.[11] I think Paul Christiano uses a definition like this in this comment to operationalize the probability of being in a simulation.
However, this seemingly reasonable definition also leads to some pretty strange places. For example, let's see how this changes the Sleeping Beauty analysis.
Suppose that whenever Beauty wakes up, she can receive a piece of chocolate if the coin landed on heads, or receive a piece of chocolate if an independent random number generator produces a number below p. We can define the p for which she is indifferent between the two choices as her probability of the coin landing heads.
This boils down to a value judgement: is waking up twice, eating the same type of chocolate both times, then forgetting both, twice as valuable as eating it once then forgetting it? If you think yes, it's exactly twice as good, then you should bet with ⅓ implied probability.
But you could also think that eating a chocolate once, or going through the exact same experience twice in a memory-wiped state are equally good. Then if you bet on heads, you get the experience with ½ probability, and if you bet on the random number generator, you get the experience at least once with probability. So the point of indifference is when , so according to this definition, Sleeping Beauty should give a probability to the coin landing on heads.
If you believe that eating two identical chocolates and forgetting them is somewhat better but not exactly twice better than eating the chocolate once,[12] then under this definition, your probability of heads should be somewhere between 0.333 and 0.382, depending on your exact philosophical views.
I think the Sleeping Beauty problem is not just an edge-case. This dependency on your philosophical views on copied experiences is something that pops up whenever you try to reason about simulations and infinite universes if you define probabilities using the bets on experiences.
This is a pretty unnatural way for probabilities to work, so if you insist on defining probabilities, we should look for something else.
Betting on terminal values
Perhaps the cleanest definition uses an even more hypothetical terminal value: a new happy planet appearing somewhere far away, unaffected by anything on Earth. "Would I prefer a happy planet to appear if there's hyperinflation, or a happy planet to appear if the RNG rolls below 0.10?" If I'm indifferent, hyperinflation has a 10% probability, because the planet is far away and unaffected by indirect correlations.
In the Sleeping Beauty question, I think I'm back at ⅓ implied probability with this definition.
Unfortunately, even this breaks down for sufficiently abstract questions. "What's the probability of being in a simulation?"—where does the planet appear, inside or outside the simulation? "How many alien civilizations exist?"—depending on some philosophical considerations, at some point adding an extra planet to the already teeming alien life might have diminishing returns in value.
Altogether, I don't think there is a clean definition of probabilities based on betting that makes probabilities a useful concept in full generality.
Probabilities for the exotic and the mundane
Ultimately, what matters is not how I define probabilities, but how I make decisions. I will argue in my next two posts why I am mostly acting in a way as if I was assuming a materialistic world-view and that we are outside the simulation.
Under these assumptions, probabilities are a useful abstraction.
Probabilities in the mundane world
For mundane questions—rain, hyperinflation, AGI timelines—I mentally translate "probability" to what implied probabilities I would bet with if I was betting on far-away planets appearing, assuming that we don't live in a simulation and assuming a materialistic world-view.
Imagining probabilities in terms of these bets on terminal value is a useful definition for me. When I'm deciding whether to bring an umbrella with myself, I have some intuitive estimate of how much productivity it would cost me to get drenched in the rain and how much productivity it would cost to spend time on carrying and storing my umbrella. I try to work on things that matter for my terminal values, so productivity translates to value. So once I know how I would bet in terms of terminal values (e.g. far-away happy planets appearing), I can use that information in an expected value calculation for various decisions related to rain: whether to bring an umbrella, whether to bring a rain jacket, whether to invest in farm-land, etc. This makes probabilities a useful abstraction for mundane questions.[13]
Letting go of probabilities
For philosophically confusing questions involving anthropics and the simulation hypothesis, I refuse to answer with probabilities and instead ask what exact bet we are hypothetically making, or what action we need to decide on. This makes me reluctant to pick a side in the SIA vs SSA debate in anthropics; I just don't believe it's the right level of abstraction to ask these questions. (Though SIA is generally closer to the mark in my opinion.)
Similarly, I can't in good-faith respond with probabilities to questions that don't make sense under materialistic assumptions, like "what is the probability that Jesus rose from the dead?" Amending "…assuming a materialistic universe" defeats the purpose of the question. It's a somewhat awkward position that I can't give straightforward probabilities if someone asks about Jesus, and instead I need to say that "for complicated philosophical reasons, I'm mostly acting as if he was an ordinary human".[14] But I maintain that there is no good way to put probabilities on this question - Jesus rising from the dead is deep into the territory where probabilities stop being a useful abstraction.
Once I give a probability to Jesus rising from the dead, how do I deal with Pascal's Wager, with infinite reward standing on one side? In my next posts, I will discuss infinite ethics and dealing with the supernatural, but this will require going beyond natural notions of probabilities.
Also, if you insist on using probabilities, what is the probability that you are in a short solipsistic simulation now? And given that you are reading about Jesus right now, what's the probability that Jesus is indeed a centrally important character in a larger simulation and now the simulators are just testing how you are thinking about this character? As I said above, I ignore simulations when asked for probabilities of mundane events, and I will present arguments for this choice in a later post. But given how similar gods and simulators are, it feels unfair to silently add "assuming we are not in any kind of simulation" when someone asks a question about the Son of God.
Finally, if you want to define probabilities outside mundane questions, you need to have some resolution to the SIA vs SSA question in anthropics. I'm sympathetic to Joe Carlsmith's arguments that SIA is generally more reasonable, and this would imply that we should accept the Presumptuous Philosopher's logic that we are more likely to be in worlds with more observers similar to us. But how does this interact with the supernatural? Did you know that a prominent strain within Mormon theology claims that we are in an infinite causal chain where people ascend to godhood and create new worlds - a chain of creation without start or end?[15]
I will try to deal with all these considerations about the supernatural in a later post, but that will not be based on the concept of probabilities anymore.
Conclusion
Altogether, I think probabilities are a useful abstraction under some circumstances, but for the more complex questions I need to fall back to a basic question:[16] I want to choose between action A and B, and taking into account all considerations, I want to know which action leads to a better world according to my values.
Of course, this is easier said than done. When I'm deciding whether to bring an umbrella with myself, I'm helping the versions of myself that live in worlds where it's going to rain, and I'm inconveniencing the versions of myself that live in worlds where it's not going to rain. So I will need some method to weigh against each other the consequences of my actions in infinite possible worlds. I will write more about my proposed solutions in my next posts, but I believe that probabilities are not the right abstraction to handle these questions in general.
For the avoidance of doubt: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.
I’m only familiar with the LessWrong line of thought on these topics. I’m woefully unaware of the academic philosophy tradition, and I’m possibly rediscovering ideas that appeared there too.
It's also the case that most of the prior work I read is scattered across many, often very confusingly written blog posts, and I can't easily tell where I first came across various ideas I'm exploring here. Therefore, I will not try to do a full exegesis of where each idea came from, and will instead present the arguments as a unified flow, with only occasional direct references to the work of prior authors. It's also very possible that there are important insights that I missed that people have already written on these topics - in that case, feel encouraged to link to them in the comments.
If the prior probabilities were only proportional to then the overall probabilities of n-length programs would add up to 1 for every n, and the full sum would be infinite. So we need a somewhat stronger decay in probabilities - now the overall probability of n-length programs is , and the sum of these is finite. We could have also chosen a different decay factor that ensures a finite sum.
That means there is no algorithm that can compute the Solomonoff-prior of strings up to arbitrary precision.
I think the problems of embedded agency might be important; I just haven’t really engaged with them yet.
Otherwise, if I believed there were no universe laws plus location pointer that were simpler than my raw observations, then I’d basically think of myself as a Boltzmann-brain and I couldn’t predict any next observations.
The overall prior of all n-long descriptions is , and summing from N to infinity is approximately 1/N.
I’m not a physicist and I’m not actually sure that negentropy is the right term here, but something like this seems right.
There is some complication that maybe the real crucial millennium has unusually short description-length, so it gets relatively large weight within the universes. But I believe that the rest of space-time likely still holds much larger weight, so turning a fraction of that into simulations still outweighs the real crucial millennium.
For example, Joe Carlsmith expresses skepticism of defining everything through betting in this post.
I love chocolate.
This is the view that matches my intuition.
Of course, in practice, when I’m deciding whether to bring an umbrella with myself, I’m not thinking exclusively in terms of work productivity. I’m often thinking in terms of how things would make me feel. Ideally I would only take my well-being into account to the extent it matters for productivity and wisdom to make the world better. In the rest of this series, I will implicitly rely on the assumption that my only goal is trying to pursue the scope-sensitive Good (otherwise, the entire theory I’m building here kind of goes haywire). I actually aspire to live like that, though of course I can’t promise I’m always living up to this ideal - the spirit is willing but the flesh is weak.
I will write a bit more about how I relate to existing religions in a later post.
I would love to read someone sincerely making this SIA argument for Mormonism. Unfortunately, I couldn’t find any examples of this on the internet.
Arguably the only important type of question that exists