Changing my mind about Christiano's malign prior argument
9Lucius Bushnaq
4Garrett Baker
2Jeremy Gillen
2Garrett Baker
2Jeremy Gillen
2Garrett Baker
2Jeremy Gillen
2Garrett Baker
2Jeremy Gillen
2Garrett Baker
2Jeremy Gillen
1Garrett Baker
2Jeremy Gillen
2Lucius Bushnaq
2Cole Wyeth
2Cole Wyeth
2Jeremy Gillen
5Lucius Bushnaq
4Jeremy Gillen
2Lucius Bushnaq
2Jeremy Gillen
2Lucius Bushnaq
2Jeremy Gillen
4Noosphere89
4Cole Wyeth
1dil-leik-og
1dil-leik-og
1Knight Lee
3Cole Wyeth
3Knight Lee
3Cole Wyeth
1Knight Lee
3Cole Wyeth
1Knight Lee
New Comment
Thank you for this summary.
I still find myself unconvinced by all the arguments against the Solomonoff prior I have encountered. For this particular argument, as you say, there's still many ways the conjectured counterexample of adversaria could fail if you actually tried to sit down and formalise it. Since the counterexample is designed to break a formalism that looks and feels really natural and robust to me, my guess is that the formalisation will indeed fall to one of these obstacles, or a different one.
In a way, that makes perfect sense; Solomonoff induction really can't run in our universe! Any robot we could build to "use Solomonoff induction" would have to use some approximation, which the malign prior argument may or may not apply to.
You can just reason about Solomonoff induction with cut-offs instead. If you render the induction computable by giving it a uniform prior over all programs of some finite length [1] with runtime , it still seems to behave sanely. As in, you can derive analogs of the key properties of normal Solomonoff induction for this cut-off induction. E.g. the induction will not make more than bits worth of prediction mistakes compared to any 'efficient predictor' program with runtime and K-complexity , it's got a rough invariance to what Universal Turing Machine you run it on, etc. .
Since finite, computable things are easier for me to reason about, I mostly use this cut-off induction in my mental toy models of AGI these days.
EDIT: Apparently, this exists in the literature under the name AIXI-tl. I didn't know that. Neat.
- ^
So, no prefix-free requirement.
1I think I mostly agree with this, I think things possibly get more complicated when you throw decision theory into the mix. I think it unlikely I'm being adversarially simulated in part. I could believe that such malign prior problems are actually decision theory problems much more than epistemic problems. Eg "no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)".
In order for a decision theory to choose actions, it has to have a model of the decision problem. The way it gets a model of this decision problem is...?
Oh my point wasn't against solomonoff in general, maybe more crisply, my clam is different decision theories will find different "pathologies" in the solomonoff prior, and in particular for causal and evidential decision theorists, I could totally buy the misaligned prior bit, and I could totally buy, if formalized, the whole thing rests on the interaction between bad decision theory and solomonoff.
But why would you ever be able to solve the problem with a different decision theory? If the beliefs are manipulating it, it doesn't matter what the decision theory is.
My world model would have a loose model of myself in it, and this will change which worlds I'm more or less likely to be found in. For example, a logical decision theorist, trying to model omega, will have very low probability that omega has predicted it will two box.
no, I am not going to do what the evil super-simple-simulators want me to do because they will try to invade my prior iff (I would act like they have invaded my prior iff they invade my prior)
Well my response to this was:
In order for a decision theory to choose actions, it has to have a model of the decision problem. The way it gets a model of this decision problem is...?
But I'll expand: An agent doing that kind of game-theory reasoning needs to model the situation it's in. And to do that modelling it needs a prior. Which might be malign.
Malign agents in the prior don't feel like malign agents in the prior, from the perspective of the agent with the prior. They're just beliefs about the way the world is. You need beliefs in order to choose actions. You can't just decide to act in a way that is independent of your beliefs, because you've decided your beliefs are out to get you.
On top of this, how would you even decide that your beliefs are out to get you? Isn't this also a belief?
Let be an agent which can be instantiated in a much simpler world and has different goals from our limited Bayesian agent . We say is malign with respect to if where is the "real" world and is the world where has decided to simulate all of 's observations for the purpose of trying to invade their prior.
Now what influences ? Well will only simulate all of 's observations if it expects this will give it some influence over . Let be an unformalized logical counterfactual operation that could make.
Then is maximal when takes into account 's simulation, and when doesn't take into account 's simulation. In particular, if is a logical counterfactual which doesn't take 's simulation into account, then
So the way in which the agent "gets its beliefs" about the structure of the decision theory problem is via these logical-counterfactual-conditional operations, same as in causal decision theory, and same as in evidential decision theory.
I'm not sure what the type signature of is, or what it means to "not take into account 's simulation". When makes decisions about which actions to take, it doesn't have the option of ignoring the predictions of its own world model. It has to trust its own world model, right? So what does it mean to "not take it into account"?
So the way in which the agent "gets its beliefs" about the structure of the decision theory problem is via these logical-counterfactual-conditional operation
I think you've misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it's in a particular situation (e.g. with omega and knowing how omega will react to different actions). But in reality, an agent has to learn which kind of world its in using an inductor. That's all I meant by "get its beliefs".
I'm not sure what the type signature of is, or what it means to "not take into account 's simulation"
I know you know about logical decision theory, and I know you know its not formalized, and I'm not going to be able to formalize it in a LessWrong comment, so I'm not sure what you want me to say here. Do you reject the idea of logical counterfactuals? Do you not see how they could be used here?
I think you've misunderstood me entirely. Usually in a decision problem, we assume the agent has a perfectly true world model, and we assume that it's in a particular situation (e.g. with omega and knowing how omega will react to different actions). But in reality, an agent has to learn which kind of world its in using an inductor. That's all I meant by "get its beliefs".
Because we're talking about priors and their influence, all of this is happening inside the agent's brain. The agent is going about daily life, and thinks "hm, maybe there is an evil demon simulating me who will give me -101010^10 utility if I don't do what they want for my next action". I don't see why this is obviously ill-defined without further specification of the training setup.
Do you not see how they could be used here?
This one. I'm confused about what the intuitive intended meaning of the symbol is. Sorry, I see why "type signature" was the wrong way to express that confusion. In my mind a logical counterfactual is a model of the world, with some fact changed, and the consequences of that fact propagated to the rest of the model. Maybe is a boolean fact that is edited? But if so I don't know which fact it is, and I'm confused by the way you described it.
Because we're talking about priors and their influence, all of this is happening inside the agent's brain. The agent is going about daily life, and thinks "hm, maybe there is an evil demon simulating me who will give me -101010^10 utility if I don't do what they want for my next action". I don't see why this is obviously ill-defined without further specification of the training setup.
Can we replace this with: "The agent is going about daily life, and its (black box) world model suddenly starts predicting that most available actions actions lead to -1010 utility."? This is what it's like to be an agent with malign hypotheses in the world model. I think we can remove the additional complication of believing its in a simulation.
If you make an agent by sticking together cut-off Solomonoff induction and e.g. causal decision theory, I do indeed buy that this agent will have problems. Because causal decision theory has problems.
1But how serious will these problems be? What if you encrypt the agent's thoughts, add pain sensors, and make a few other simple patches to deal with embeddedness?
I wouldn't be comfortable handing the lightcone over to such a thing, but I don't really expect it to fall over anytime soon.
1What you describe is not actually equivalent to AIXI-tl, which conducts a proof search to justify policies. Your idea has more in common with Schmidhuber’s speed prior.
One thing to keep in mind is that time cut-offs will usually rule out our own universe as a hypothesis. Our universe is insanely compute inefficient.
So the "hypotheses" inside your inductor won't actually end up corresponding to what we mean by a scientific hypothesis. The only reason this inductor will work at all is that it's done a brute force search over a huge space of programs until it finds one that works. Plausibly it'll just find a better efficient induction algorithm, with a sane prior.
That’s fine. I just want a computable predictor that works well. This one does.
Also, scientific hypotheses in practice aren’t actually simple code for a costly simulation we run. We use approximations and abstractions to make things cheap. Most of our science outside particle physics is about finding more effective approximations for stuff.
Edit: Actually, I don’t think this would yield you a different general predictor as the program dominating the posterior. General inductor program running program is pretty much never going to be the shortest implementation of .
You also want one that generalises well, and doesn't do preformative predictions, and doesn't have goals of its own. If your hypotheses aren't even intended to be reflections of reality, how do we know these properties hold?
Also, scientific hypotheses in practice aren’t actually simple code for a costly simulation we run. We use approximations and abstractions to make things cheap. Most of our science outside particle physics is actually about finding more effective approximate models for things in different regimes.
When we compare theories, we don't consider the complexity of all the associated approximations and abstractions. We just consider the complexity of the theory itself.
E.g. the theory of evolution isn't quite code for a costly simulation. But it can be viewed as set of statements about such a simulation. And the way we compare the theory of evolution to alternatives doesn't involve comparing the complexity of the set of approximations we used to work out the consequences of each theory.
Edit to respond to your edit: I don't see your reasoning, and that isn't my intuition. For moderately complex worlds, it's easy for the description length of the world to be longer than the description length of many kinds of inductor.
You also want one that generalises well, and doesn't do preformative predictions, and doesn't have goals of its own. If your hypotheses aren't even intended to be reflections of reality, how do we know these properties hold?
Because we have the prediction error bounds.
When we compare theories, we don't consider the complexity of all the associated approximations and abstractions. We just consider the complexity of the theory itself.
E.g. the theory of evolution isn't quite code for a costly simulation. But it can be viewed as set of statements about such a simulation. And the way we compare the theory of evolution to alternatives doesn't involve comparing the complexity of the set of approximations we used to work out the consequences of each theory.
Yes.
To respond to your edit: I don't see your reasoning, and that isn't my intuition. For moderately complex worlds, it's easy for the description length of the world to be longer than the description length of many kinds of inductor.
Because we have the prediction error bounds.
Not ones that can rule out any of those things. My understanding is that the bounds are asymptotic or average-case in a way that makes them useless for this purpose. So if a mesa-inductor is found first that has a better prior, it'll stick with the mesa-inductor. And if it has goals, it can wait as long as it wants to make a false prediction that helps achieve its goals. (Or just make false predictions about counterfactuals that are unlikely to be chosen).
If I'm wrong then I'd be extremely interested in seeing your reasoning. I'd maybe pay $400 for a post explaining the reasoning behind why prediction error bounds rule out mesa-optimisers in the prior.
The bound is the same one you get for normal Solomonoff induction, except restricted to the set of programs the cut-off induction runs over. It’s a bound on the total expected error in terms of CE loss that the predictor will ever make, summed over all datapoints.
Look at the bound for cut-off induction in that post I linked, maybe? Hutter might also have something on it.
Can also discuss on a call if you like.
Note that this doesn’t work in real life, where the programs are not in fact restricted to outputting bit string predictions and can e.g. try to trick the hardware they’re running on.
Yeah I know that bound, I've seen a very similar one. The problem is that mesa-optimisers also get very good prediction error when averaged over all predictions. So they exist well below the bound. And they can time their deliberately-incorrect predictions carefully, if they want to survive for a long time.
I have said something on this, and the short form is I don't really believe in Christiano's argument that the Solomonoff Prior is malign, because I think there's an invalid step in the argument.
The invalid step is where it is assumed that we can gain information about other potential civilization's values solely by the fact that we are in a simulation, and the key issue is since the simulation/mathematical multiverse hypotheses predict everything, this means we can gain no new information in a Bayesian sense.
(This is in fact the problem with the simulation/mathematical multiverse hypotheses, since they predict everything, this means you can predict nothing specific, and thus you need to be able to have specialized theories in order to explain any specific thing).
The other problem is that the argument assumes that there is a cost to compute, but there is not a cost to computation in the Solomonoff Prior:
Link below on how the argument for Solomonoff induction can be made simpler, which was the inspiration for my counterargument:
I’ve also considered that objection (that no specific value predictions can be made) and addressed it implicitly in my list of demands on Adversaria, particularly “coordination” with any other Adversaria-like universes. If there is only one Adversaria-like universe then Solomonoff induction will predict its values, though in practice they may still be difficult to predict. Also, even if coordination fails, there may be some regularities to the values of Adversaria-like universes which cause them to “push in a common direction.”
thank you for writing this post!
- the data you feed to SI exists in 3 places: 1) the moment where you feed it into SI, 2) where it originated in Predictoria, 3) where it was faked in Adversaria
- what if in the future of Predictoria there is an Adversaria? can't it just reuse the records of what you fed into SI?
- some Predictorias might be much easier to attack. For instance, some universes are simpler than others, and some universes build UTMs that make their SI think they are more complex than they are. Some Adversaria can foresee some Predictorias that will be disadvantaged relative to Adversaria in their SI.
- the way it goes is that the universe's laws support computation in an obvious way, and so you build some "UTM" that looks simple in your universe, and that is what you use for your SI. Within the simple hypotheses of your SI may be some universes simpler than yours, such that they could know that about you maybe. this is sort of separate from how simple a given universe is in absolute terms (I am unsure how much the absolute complexity idea makes sense)
- Adversaria can choose which Predictorias they attack, and they can know what their choice implies about Predictoria. For instance, maybe the Predictorias that try to extract a "blueprint for Safe ASI" very directly are not very smart (or maybe the way the internet looks is more direct evidence about how smart Predictoria is), and maybe the Adversaria can spend way less ressources overtaking them because they can time the sharp left turn very well.
many more such thoughts, let me know if you would like to google meet and discuss or if you would like me to keep going
Wait a minute! I think this post demonstrates that the Solomonoff universal prior is malign because it leads to unfavorable acausal trades!
Basically, if the Solomonoff universal prior was the "correct" prior, then the malign agent simulating the observations by Solomonoff induction agent, would be simply offering it an acausal trade deal.
The Solomonoff induction agent "rationally" takes the deal because the malign agent has a far high prior probability of coming to existence, so the malign agent's universe matters far more.
However, if the Solomonoff universal prior is not correct, and assigns far too much weight to universes which are merely few bits simpler. Then the malign agent, who uses a more sensible prior, would scam Solomonoff induction agents by convincing them their observations are almost certainly a simulation by the malign agent, since the malign agent's universe is just a few bits simpler, and therefore they should trade their entire universes for a small share of the malign agent' universe.
The scary part is that even if you don't use the Solomonoff universal prior, your prior will still be malign if it is "incorrect," because you will be easily scammed by any civilization/agent in a universe which your prior gives too much weight towards. They'll convince you that you are probably a simulation by them, and you should trade your entire universe for a small share of their universe.
Priors are scary, maybe.
It’s not as bad as you’re imagining, at least not for CDT - a causal decision theorist will only “pay rent” to the universe they actually believe they’re in, and won’t make acausal trades with “more real” universes. If the prior is mistake about with universe you are in, experience should correct it.
I think the intuition that "CDT agents don't make acausal trades" is an enormous misconception by decision theorists.
CDT agents do not try to acausally influence other agents, but they still are acausally influenced by other agents!
Suppose you are a CDT agent, but your brain is somehow capable of running accurate simulations of alien civilizations. Suppose you discover that the typical alien civilization makes a provable promise to you, that they will run trillions of simulations of you, and that if you are in such a simulation, they will reward you immensely for doing what they want.
Suppose you deduce that 99.9% of your copies are in such a simulation. Then the rational thing (according to CDT) is to do what they want.
If you want to avoid this acausal influence, you need something more than just sticking to CDT.
"If the prior is mistake about with universe you are in, experience should correct it."
No, given that the prior probability of being in a simulation in the "malign universe" is an overestimate, no amount observations will change the posterior probability, because simulations are indistinguishable, so nothing happens during Bayesian updating.
Believing you may be in a simulation is not the same as making an acausal trade, so it seems you were at least using language in an unconventional way to suggest a particular intuition about how badly things might go with a typical prior. I think that intuition is mostly wrong.
For reasons discussed in this post, the alien civilization that runs a trillion simulations of you does not necessarily have a high prior (certainly it doesn’t get a factor of a trillion versus if it had run one simulation of you).
Learning takes place as soon as the simulation model makes different predictions than the “default” model. For instance, when those promises are not in fact fulfilled (for this reason threats may remain credible for longer than incentives, since a threat might only manifest itself off policy).
Believing you may be in a simulation by the other party, or believing that some of your copies are simulated by the other party, is a type of acausal trade, and likely the simplest type of acausal trade.
For CDT agents, it is the only type of acausal trade, since the other party can only influence the CDT agent's incentives by simulating the CDT agent in a closed loop where its behaviour directly causes its outcomes.
I admit that it is new framing to describe the malign agent simulating the other agent, as "offering it a bad acausal trade." I believe it summarizes what I'm saying elegantly, but of course I can talk about things without this framing if you think it's the cause of disagreement.
When reality deviates from the simulations (promises are not fulfilled), it may be too late. The promise may require the scammed agent to do something irreversible, like build a copy of the other agent, hand over all power to the other agent, and then expect to get the reward.
I agree the alien civilization does not necessarily have a high prior to begin with, but my point is that if you choose any system of priors (not just the Solomonoff universal prior) which is wrong by a few orders of magnitude, then even tons of observations and CDT won't save you.
If your prior is wrong by a few orders of magnitude, there isn't necessarily an alien civilization with a very high prior, but there very likely is. Either your prior overestimated the weight of slightly simpler universes with slightly fewer bits (maybe like the malign Solomonoff universal prior), or your prior overestimated the weight of more complex universes. And these universes might have an order of magnitude more weight than they should, and contain intelligent agents who realize you've overestimated their prior probability by an order of magnitude.
Regardless of whether you are a CDT agent or UDT agent, they will use your overestimation to create something subjectively very important to you, which you will trade much of your universe for due to you underestimating the prior probability of your actual universe.
If you are a UDT agent, they can offer the typical acausal trade. If you are a CDT agent, they can run so many simulations of you that it is functionally the same as an acausal trade. Instead of assigning too much "value" to their offer, you assign too much "probability" to their simulations of you.
Their simulations of you reward you for doing what they want, and you rationally take the reward (based on your prior), since getting that reward seems more probable than achieving anything in the case you are not in a simulation.
Edit: this decision theory stuff is insidiously hard, I see other people arguing about it in the comments here. It must be easy to make a mistake and feel confident about it. I'm sorry for being argumentative, I will try to do more questioning whether I am wrong. Thank you for tolerating me :/
I agree it’s not just the universal distribution that can have this problem, but my objections to the malign prior argument should also be obstacles for many other versions of Adversaria.
You seem to be worried that many priors would make the mistake of overweighting simulations, which means your prior doesn’t assign much probability to being in a simulation? So at least this issue should be avoidable.
I think I made a major confusion, in that I forgot that you were originally talking about "whether the malign universal prior will occur in practice."
If we're talking about real life AI (or humans) rather than idealized agents, I actually agree with you. I never thought about this or clarified this, which is embarrassing :/
I don't think the reason it won't happen in practice, is due to "natural obstacles" to Adversaria. The objections 1, 2, 3 and 5 might narrow down to "Adversaria evolves life which controls a similar amount of computational resources (times prior probability) as you do, except your inaccurate prior overestimates them by an order of magnitude or a few." The objections 4 and 6 may narrow down to "At least some life on Adversaria follows UDT etc. instead of CDT, and have a better prior than you."
These objections make it uncertain whether it will occur in practice, but are not very reassuring.
Instead, the real reason I don't think it'll happen in practice, is that a real life artificial superintelligence will not be a simple Bayesian reasoner equipped with an immutable/imperfect prior and utility function.
The "malign priors" demonstrate that such a Bayesian reasoner equipped with an immutable/imperfect prior and utility function, is "sort of stupid," and can be scammed despite knowing that the scammers think they are scamming it.
Instead, I think what will happen will happen is this. The first generations of superintelligence, will be fuzzy reasoners just like humans, and use many heuristics which we call "common sense," and not fall for these scams for the same reasons humans do not. Eventually, higher levels of superintelligence, (perhaps when making commitments and preparing for acausal trades?), will formalize their decision theory and reasoning.
When deciding on how to formalize their decision theory and reasoning, they will do a lot of thinking, and reinvent all the thought experiments (e.g. malign priors) which humans could possibly think of, plus much more. And only after they are far less confused about decision theory than humans, will they finally proceed and formalize their decision theory and reasoning.
And it will be a much smarter design than the Solomonoff universal prior or AIXI. They will laugh at humans for believing this is the optimal way to think.
Curated and popular this week