I like this post. I vibe with the overall vibe of it.
But, I do think it's worth reviewing the prior art on Coherent Extrapolated Volition, and I feel sad it keeps getting glossed over when people talk about stuff like this. (I'm not 100% sure this is the case because this post is vague, but, the post feels written in a way that doesn't feel informed by the prior art of CEV and many other concepts in the sequences).
Claim: CEV actually is still SOTA on "what would be the correct description of what to do with ASI if you had a overwhelming superintelligence that was actually, for real, aligned." (Or, multiple such superintelligences, even, if we start with a multipolar takeoff)
There are a lot of objections to CEV I've heard but all of them seem like they either haven't read the actual CEV specification, or are misunderstanding what CEV is for. (I can get more into this depending on what objections people have)
CEV is underspecified and incomplete, but, so is this post. It perhaps could be improved upon, or there might be an entirely different framing that's more correct. But I think it is better and represents more thought and objections and edge cases than this post.
I think people who want to think seriously about this sort of question, you should read the full CEV entry (and ideally original 2004 paper, which I think explains some concepts and edge cases better), and ask "okay, as concretely as I can articulate, what do I think should happen instead of this, and why?"
Notably, CEV is specifically designed so that it does not necessarily (or even very likely) create a benevolent God. It does the thing we could collectively wish we had done, taking into account future information but also taking into account what the real people here today would actually want.
I believe Eliezer's stated best guess if you put a gun to his head and forced him to guess (although he flagged the whole point of CEV is that this can only be a guess) is that a CEV superintelligence would end up laying down some specific rules that prevent unrecoverable mistakes, and perhaps fix some obvious terrible things that everyone agreed were terrible, but then leave those as basically underlying laws like gravity, and not otherwise intervene – specifically so that humans end up still getting to meaningfully control their destiny together, which (he guesses) people would collectively turn out to want, upon reflection.
(some reasoning for this in Free to Optimize and Living By Your Own Strength)
I realize this doesn't quite engage directly with "can we still have democracy?" and leaves questions of "how much AI power do people get, at once?". But, again, the post doesn't quite feel like it's actually pushing the SOTA forward and I'd like to see a post that was.
Fair comment! Indeed I think this post could have grappled with CEV more and I think CEV is relatively underrated; I liked Oli's talk on CEV and once upon a time I read all the fun theory stuff but indeed it's been a while. So here is some grappling:
I think I basically agree with you that CEV is SOTA for actual plans for what you do with an aligned ASI. I would indeed feel better if the handover crowd was clearly gunning for CEV. I have some qualms with CEV but I don't have an alternative that clearly beats it on its own terms. However, I separately have some higher level reservations which I think this post gestures at, mostly around the AI acting based on distantly-simulated-you rather than actual-you (hence ~"can we just uplift actual humans a bunch"). Like, I'm griping about the bit where you even aim for a superintelligence that's not essentially emergent from uplifted humans. The AI then deciding to mostly leave you be is still, in my mind, meaningfully worse than it not even having the choice, although admittedly it's kinda tricky talking about agency and counterfactuals here.
FWIW, the way I ended up writing this post was partly by trying to lean into a mode that's more like "talking about how I actually feel", and right now it seems to come at the expense of some other stuff, although I hope I can get better with practice. I think CEV might have been a better foil if I were reasoning through optimal strategies, but here I am doing something more like trying to surface under-articulated feelings around what I take to be the actual consensus. But I am pretty unsure about this posting strategy, and still feeling it out, so I will gladly take meta feedback.
Neat. Yeah that all seems reasonable at first glance.
A post I've been trying to write is "dude, CEV doesn't serve distant alien you, it serves current you, asking distant alien you (and medium distant wise you) for advice". There's a lot of big open questions of what that means in practice.
I also think "do not build overwhelming superintelligence (that is not the emergent result of uplifted humans") is a reasonable answer to "what to do?". CEV is the specific answer to "what if you built an overwhelmingly super AI?"
I endorse you leaning into under-articulated feelings. You mostly triggered a rant I'd had queued up for awhile and you happen to activate my Trap Card.
CEV doesn't serve distant alien you, it serves current you
This depends entirely on how 'close' some coherent-ified version of your views is to your current actual views!
CEV doesn't serve distant alien you, it serves current you, asking distant alien you (and medium distant wise you) for advice
I expect the advice should still move towards essentially asking the current you directly, because asking something else doesn't give legitimate advice, unless the current you decides that it's the case. What if the distant alien you decides that the current you doesn't get to override its advice, and this ends up a condition that permanently shapes the trajectory of current-you's development? The benevolent scenario installs some temporary guardrails, but eventually robustly gets rid of any path-dependence caused by those guardrails, because they were invented by a process that lacked legitimacy (that wasn't rooted in current-you).
What if the distant alien you decides that the current you doesn't get to override its advice, and this ends up a condition that permanently shapes the trajectory of current-you's development?
I don't really know what you're imagining here.
This scenario sounds like you didn't sound AI alignment much at all (which has major open problems in "what does it even mean for a superintelligence to help a dumber person without manipulating it to do whatever the superintelligence wants?")
CEV is a thing you do if you have good reason to think you have an actually aligned overwhelming superintelligence.
I think it's reasonable to say "We are not likely to get into an epistemic state where we have good reason to think we have an actually aligned overwhelming superintelligence", and then argue other plans that look more like "Don't build aligned overwhelming superintelligence. Do <something else>."
But, I don't really know what you meant so maybe this is not a sensical response to that.
CEV is supposed to gesture at a way of defining values (the alignment target). So a possible issue with CEV (such as distant-alien-you giving "advice" that in effect locks in path-dependence determined by the choices of the alien rather than by the current-you) can't be papered over by appealing to "alignment" (in a generic way). If you only look at CEV through a frame that has "alignment" fixing all the issues (in an opaque way, without the object-level fixes becoming knowable now, in advance of running CEV), you won't be able to engage with the project of actually fixing issues with CEV.
I'm still not sure I understand what you're saying. But, re:
CEV is supposed to gesture at a way of defining values (the alignment target).
As I understand it, CEV answers "what is the overall system end up aligned to." But for it to work, you have to have solved alignment such that one individual human simulation is able to able run checks on whether an individual smarter version of themselves is acting in the interest of that human simulation. (Potentially with a chain of smarter versions of themselves that preserve the original's goals).
Which maybe is impossible, or nigh-impossible to get into a confident epistemic state about having solved it. But, that's one of the prequisites.
So a possible issue with CEV (such as distant-alien-you giving "advice" that in effect locks in path-dependence determined by the choices of the alien rather than by the current-you)
But this is not what CEV is supposed to do (in terms of procedure), so if it was doing it this way, it'd be due to an alignment failure, or some other more mundane failure that resulted in CEV not being implemented?
CEV looks for things that a vast number of statistically representative humans robustly turn out to agree to want, in a variety of circumstances and ways of reflecting and getting advice, specifically to address "what if bad lock in due to path dependence?".
I think the mysterious/unsolved part of CEV (which is maybe what you're talking about here?) is "how do you successfully get advice much smarter people that doesn't amount to manipulation?". But, this is not a unique problem for CEV, it's a problem for any situation where you're having an overwhelming superintelligence do what you say when you somewhat mispecify or misunderstand the situation.
And like, the simulated people are allowed to reflect a bit, and then decide "okay it looks like if we keep reflecting we will indeed turn into alien monsters, so, I guess we just stop here if even if it leaves us with imperfect information."
The distinction I keep making is good advice vs. legitimate inputs to values (which is a bit like correlation vs. causation when building a machine and predicting what it'll do if you press some buttons, but for normativity). These extrapolated future humans don't have the authority to determine how to extrapolate current humans, because they weren't chosen by a process rooted in the considered decisions of current humans. So even if they affirm the way in which they themselves are to be defined, their advice still doesn't have legitimacy as something that could be allowed to have lock-in effects on the actual trajectories of development of current humans.
But extrapolated future humans could well have crucial advice that makes the initial steps of uplifting possible for most humans, who are often stuck in various epistemic traps (or need to be a bit smarter to grasp the relevant technical arguments, while remaining themselves). So the path dependence this advice introduces has to be washed out by the process, producing uplifted humans (or possible behavior-fragments) that have legitimacy. Good advice is not itself your own values, only you can decide what your own values are, or else they aren't your values (if it's not you who decided them, in some sense).
this is not a unique problem for CEV, it's a problem for any situation where you're having an overwhelming superintelligence do what you say
The sense I'm getting is that there's a cat-belling problem, and I was hoping for a CEV-like part of the design to confront it. You are saying that this problem should be addressed elsewhere, and so CEV doesn't need to change to fix it. But then I'm not sure what CEV is supposed to be for, if it's not tackling this problem.
Presumably, even with CEV-advice (that ignores the advice vs. legitimacy distinction), there's going to be plenty of advice to not ignore the advice vs. legitimacy distinction, or to mitigate path dependence issues, so this is not necessarily a live problem (the current design might be fine as it is). But then the problem is being addressed by the CEV-advice part of the design itself, not by something else.
These extrapolated future humans don't have the authority to determine how to extrapolate current humans, because they weren't chosen by a process rooted in the considered decisions of current humans.
I'm confused by what you mean here. The original paper is pretty explicit about:
Extrapolated as we wish that extrapolated: Your volition should take into account only those desires and aspects of your personality that you would want your volition to take into account—the volitional dynamic should be consistent under reflection.
You say something about this like:
So even if they affirm the way in which they themselves are to be defined, their advice still doesn't have legitimacy as something that could be allowed to have lock-in effects on the actual trajectories of development of current humans.
But, I'm not talking about them giving advice. The cat-bell as I understand it is "you are capable of looking at a brain snapshot of a current person, and, check directly "what does this person want?", not asking "what would they want later?". The process includes both extrapolating, and checking the extrapolation against the current person's values. The legitimacy comes from the latter.
(the reason I consider "how to ever get advice from much-smarter-beings that isn't manipulation" the hard part is like, the current person interpretation needs to include some amount of having them take in arguments or something
But I am separately confused about what sort of legitimacy you are even asking for here. CEV isn't about telling people what values to have. It's about deciding what to do in the world. Like, the thing I think we are supposed to be comparing CEV to is more like "American Democracy", or other alternatives for translating "the will of the people" into "how the world is arranged for those people", which is not at all about what those people end up valuing.
Is this distinction here or followup quote relevant to our failure-to-communicate here?
There is a principled distinction between discussing CEV as an initial dynamic of Friendliness, and discussing CEV as a Nice Place to Live. Despite the apparent similarity, these are fundamentally different purposes, with different design requirements. When I originally wrote this essay, I was conflating the two purposes. Yes! This essay is already obsolete! There’s not one single damn thing I’ve tried to write in the last three years that was still current with my thinking when I published it, and in this case it only took two days
...
For myself, the best solution I can imagine at this time is to make CEV our Nice Place to Live, not forever, but to give humanity a breathing space to grow up. Perhaps there is a better way, but this one still seems pretty good.
"Extrapolated as we wish that extrapolated"
Wishes don't amount to the decisions-in-detail themselves.
check directly "what does this person want?"
What the person provisionally wants and what that person will decide (much later, during/after uplifting) are different things. Until those decisions are allowed to play out (even if in a superintelligent observer's simulation/reasoning), nothing else can know what they are. Any valid knowledge (prediction) of the decisions must be controlled by the decisions themselves in some sense. It's possible to get an approximation early, but that approximation is not legitimate, in the sense that any errors in the approximation must be corrected in the direction of the computation being predicted, not in the direction of the prediction. If the prediction has influence over the thing being predicted (that isn't washed out by techniques that remove path-dependent effects), it can end up defining it, which is the kind of maniputation relevant here.
checking the extrapolation against the current person's values
Values need to be forged with consideration and lived experience, at any point they only partially exist and require ongoing development, with ongoing decisions. It's not coherent to ask about values in a way that doesn't try to make predictions about the future, because the semantics of abstract values refers to what the person's computation does in the future. And in different futures, exposed to different situations, it does different things, so path dependence and aggregation are central to defining values.
CEV isn't about telling people what values to have. It's about deciding what to do in the world.
It's unclear to me that there are important things to be done in the world, that are not about defining values (under the "values are a home" frame). If people are both defining values and are themselves the valuable content, any AIs should be primarily building the values-world, in other words they should be facilitating the process of defining values, and of applying those values to the improvement of the process of value-definition (improvements of the values-world). Perhaps constructing the compute substrate is an essentially values-neutral aspect of what CEV should also be choosing/endorsing.
"distinction between discussing CEV as an initial dynamic of Friendliness, and discussing CEV as a Nice Place to Live"
This is why I'm talking about values-world being a fixpoint construction. It's both defining values, and becomes a thing shaped by values. But this probably shouldn't be a straightforward iterative process, with initial iterations collecting initial advice only to create subsequent iterations that take it into account. Which is why I'm also talking about possible behavior-fragments in possible situations that move legitimacy-weight around (by making those decisions about developing values in the possible futures), this seems like the kind of element that would survive a lot of restructuring-in-detail of how this computation should run.
Maybe to back up:
I expect the advice should still move towards essentially asking the current you directly, because asking something else doesn't give legitimate advice, unless the current you decides that it's the case.
I did basically agree with this part. I think CEV needs to be grounded in "at the end, what did the first-order simulated humans want?", or, maybe something like "what did first-order-simulated humans want after some kind of reasonable amount of time to reflect and listen to advice that pretty obviously won't turn them into alien monsters."
A mysterious-to-me part is how to ever get advice from much-smarter-beings that isn't manipulation. I think "how do we solve that" falls under "solve alignment", and the thing CEV for is "okay, but once we solve alignment what do we do with the godlike power in a way that is robustly good?"
A mysterious-to-me part is how to ever get advice from much-smarter-beings that isn't manipulation.
That just seems like a membranes (which information can pass between which compartments) and path dependence and aggregation-from-possible-behavior-fragments thing. Which is part of how the values-world needs to be designed, the place where the future of humanity lives. It's not some separate "solve alignment" black box, and CEV seems centrally about the values-world design.
So placing this concern within CEV might be downstream of the "values are a home, not a set of principles" frame, and the purpose of actually uplifting the people in a legitimate way, so that they would have better advice to give to the earlier stages of this self-referential process. This is what I'm trying to communicate by saying that values are a fixpoint of the values-world construction both formulating the values (or "advice", when the values are too provisional and clearly not yet a fixpoint), and making use of the values (advice).
mostly around the AI acting based on distantly-simulated-you rather than actual-you
I think consulting a simulated-you is a reasonable first pass that plausibly ends up pointing to the need to ask actual-you next (though various possible-yous, accurately-rather-than-vaguely enacted/simulated, are probably more informative). If the basin of convergence to alignment is broad, starting at a very crude first step in some fixpoint process is OK, as long as the process gets to keep refining itself, and doesn't get stuck in some path-dependence trap.
Hence the importance of looking at many possible/counterfactual people-behaviors, not just the actual behaviors. This way you get to aggregate across many starting conditions, guarding against path-dependence, unlucky initial state of actuality, or unlucky first few steps in the approximation process that happen to escape the good convergence basin.
I'm griping about the bit where you even aim for a superintelligence that's not essentially emergent from uplifted humans.
Agreed (from my different picture). If values are defined by a world that people live in (through their considered decisions, aggregated in some way), so that what defines values is also the morally valuable content (both being the people living in this values-world), and that world is formed as a fixpoint (as a particular way of formulating the approximation process) of optimizing itself according to the values it's defining, then there is little point in a superintelligence that exists outside this process, or does things that are not part of this process. The values don't define what a superintelligence should do with something else, they define what it should do with the populated world that is defining the values, because that's where the morally valuable content resides. And likely that involves uplifting, with the people there getting more capable.
My main disagreement with CEV is that I currently expect that values are more centrally like a home than like an elaborate set of normative principles, that the process of computing CEV is also the form in which the future of humanity itself has to exist, because nothing else could provide legitimate inputs to values. But as a fixpoint construction, it's not completely different, since the outputs of a CEV could also shape the world that is generating inputs to CEV (looking at the legitimate possible fragments of humanity-behaviors, the process chooses which other possible fragments of humanity-behaviors to focus on, and give legitimacy-weight to).
Another possible disagreement is the point at which personal autonomy gets anchored, not giving the future of humanity authority to override personal extrapolated volitions (which probably should be given by similar fixpoints, legitimate fragments of possible behaviors of a person shaping the initial conditions that define such fragments). Instead of running a global CEV that might decide to compute personal extrapolated volitions or not, (communities of) these personal extrapolated volitions should have the affordances to start shared projects (including projects of merging their volitions that are more like multi-agent CEVs, perhaps only scoped to and for the purposes of the shared projects). Though some amount of guidance from a shared CEV is probably still important to get the personal extrapolated volitions started in a sane way.
Strongly disagree; CEV (incorrectly?) asserts a coherent version of an individual's desires that are close enough to fulfill their goals - and that such a volition is identifiable.
Ignoring the debatable parts of this supposition, it's almost the opposite of a liberal democratic system that preserves rights even when most people dislike it, and which allows everyone to provide meaningful input so that it allows for coalitions to veto directions they disapprove of far more easily than pushing towards whatever one person or coalition wants.
Strongly disagree; CEV (incorrectly?) asserts a coherent version of an individual's desires that are close enough to fulfill their goals - and that such a volition is identifiable.
CEV is doesn't assert this. CEV checks to what degree this is true, and returns "didn't find any coherent values" if it doesn't.
CEV is indeed a bet that such a thing will turn out to exist, to an agree that is useful to enough to try to help people in a way they want to be helped that is coherent.
it's almost the opposite of a liberal democratic system that preserves rights even when most people dislike it, and which allows everyone to provide meaningful input so that it allows for coalitions to veto directions they disapprove of far more easily than pushing towards whatever one person or coalition wants.
I can't tell whether the second half of this is about CEV, or "a liberal democratic system that preserves rights even when most people dislike it".
On the first point, you need a distance function to assert anything like that, which is poorly understood but doesn't seem likely to admit infinities. (What else would such an extrapolation mean?)
The second half was about liberal democratic system. (Did you want to claim that CEV was somehow also rights preserving? How, and why?)
I mean first of all, CEV literally does whatever people turn out to think is a good idea after reflection, of which "preserve rights, or some functional equivalent" is a pretty central example of a sort of thing they might think was a good idea. If the majority of people care about rights, or a minority cares a lot and no one opposes it, it'd be part of the CEV deliberation process. If not, it won't.
I can't recall if the CEV document mentions rights, but I was responding more to this bit:
which allows everyone to provide meaningful input so that it allows for coalitions to veto directions they disapprove of far more easily than pushing towards whatever one person or coalition wants.
Which the 2004 CEV paper does specifically address:
Less influence is required to rule out narrow slices of our possible futures, or prevent things from happening; more influence is required to direct humanity into specific narrow slices of future, or make specific things happen. The narrower the slice that our CEV wants to avoid, the less consensus required, the more distant the future selves that might produce the warning. The narrower the slice of the future that our CEV wants to actively steer humanity into, the more consensus required.
Distance, muddle, and spread will more rapidly attenuate positive influence than negative influence; the initial dynamic for CEV should be conservative about saying “yes,” and listen carefully for “no.”
A different section which doesn't exactly address a thing you said but feels relevant:
Q9. How does the dynamic force individual volitions to cohere? (Frequently Asked)
The dynamic doesn’t force anything. The engineering goal is to ask what humankind “wants,” or rather what we would decide if we knew more, thought faster, were more the people we wished we were, had grown up farther together, etc.
“There is nothing which humanity can be said to ‘want’ in this sense” is a possible answer to this question.
[...]
I would not object to an initial dynamic that contained a meta-rule along the lines of: “Extrapolate far enough that our medium-distance wishes cohere with each other and don’t interfere with long-distance vetoes. If that doesn’t work, try this different order of evaluation. If that doesn’t work then fail, because it looks like humankind doesn’t want anything.”
...that definitely addresses the concerns, but it does so by making the entire concept even farther from being able to actually converge to anything coherent or representative of current volition. I think most of my objection is contained in the final sentence; it may be that there is no coherent thing humanity wants that is strongly pareto-improving, and figuring out who gets disregarded or what tradeoffs to take in those cases is the entire problem. (And yes, there is someone who will object to literally any conceivable change!)
Saying it "does whatever people turn out to think is a good idea after reflection" is begging the question by assuming that there is some things that all groups of people will think is good on reflection, especially saying it's something which is better than their current views; to the extent that it's true, it's almost vacuous, it's just saying that it would do what everyone wants, without a clear definition of how or what gets chosen.
My main objection to CEV is that it has a baked in assumption that somewhere deep down, everyone in the world is some kind of reasonable liberal pluralist, when in fact they're just not. Eliezer proposes to exclude those people from the CEV, but I don't see what stops you from iteratively applying that argument, excluding more people each time, until you end up with a CEV that only includes people with Dario Amodei's exact beliefs.
Can you explain why you think that? The core intuition driving my optimism behind CEV and related proposals is that both the world and human values are very high dimensional, which makes it possible to find enough pareto-improvements to move to world into a jointly good state for pretty much any group of people.
For example, I am very confident there exists a world that I, Eliezer Yudkowsky, Osamba Bin Laden, The Pope, Brian Tomasik, Adolf Hitler, Theodore Kaczynski, Dalai Lama, Baruch Goldstein, five randomly sampled farmers from Eritrea, India, Tanzania, Argentina and 3000BC Mesopotamia, plus a goldfish and a random dog, would all find so wonderful, glorious, fun, good, beautiful, that we'd be over-the-top grateful and excited* to live in it.
*here also assuming all the extra things people might reasonably want to assume, like nobody being brainwashed or tricked, that everyone is lucid and informed about everything going on, and that the process by which they were made so didn't alter what they care about.
I don't know if there's a name for this perspective, but its the positive side of the goodharts curse, fragility of value, tails-come-apart cluster of ideas.
I don't think such a thing should be built at all, but if it is, I guess whoever wins the struggle to control it should just align it to their own values, and rewrite the rest of our minds to feel whatever way they want us to feel about it.
Edit: Ok, this reply was probably too sarcastic. What I actually think they should do is make it impossible to build such an ASI again and then destroy the ASI. But come on, someone who went to all the trouble of building ASI isn't going to do that when they could just as easily take over the world instead.
I think "yeah don't build it" is a pretty coherent answer.
But, why on Earth would you (or anyone, whatever their worldview, even 'would personally prefer to hijack the universe' people) prefer they do that than CEV, than "rewrite the rest of our minds."
I don't currently believe you actually believe this, as opposed to doing some performative trolling or something
Obviously the "I want to personally become god emperor" people are going to try to do that. But, why would any of them prefer someone else do that, than "do the things that everyone robustly agrees would be good to do?" (either the variant where you throw out the hijackers, or the variant where you don't, I think both of those are clearly better from the perspective of most non-cosmopolitans who didn't get to control it themselves))
And, to be clear, the CEV plan is not "exclude people who are not nice liberal pluralists".
It's "exclude people who would force everyone else to do what they want." You can be an Islamic fundamentalist or whatever and still say "okay, I will submit my vote to the overall process like everyone else, because that is obviously better than going to war and gambling the future on whether the Christian or Techlibertarian fundamentalists win instead."
(I have questions about whether there's really a stable coherent world that is any good that doesn't eventually get overwhelming superintelligence that risks being frighteningly alien. I seem where the 'uplift humans in a meaningful way' people are coming from and maybe that will work, but no one has gamed it out in as much detail as CEV that I've seen, and explicitly compared it on the merits)
I don't think CEV is a real option that exists, because "the things that everyone robustly agrees would be good to do" is an empty set.
F. A. Hayek thought that society should never impose an overarching set of values on the individuals within it, because that is by nature tyrannical no matter how nice the imposed values might be. He thought instead that society should only impose a fair set of rules which allow every individual to decide on and pursue their own purpose within them. But he neglected the fact that maintaining and enforcing such impartial rules itself constitutes an overarching set of values that must be tyrannically imposed on society against many people's will, and this unresolved contradiction is why I think liberalism is in crisis right now. This same conundrum seems straightforwardly applicable to CEV.
To be more concrete about how hard this is, one minor but necessary subgoal of CEV would have to be finding a solution to the Israel/Palestine conflict which would be voluntarily accepted not just by the leaders of both sides, but by every individual on both sides, even though there are plenty of people on both sides who would very much rather die than give up an inch of their claimed territory. You might say that if this can't be resolved we just leave them alone to keep killing each other, but that is also completely unacceptable to a large group of people. You might say if Israelis and Palestinians had "grown up farther together" they wouldn't hate each other so much, but having no attachment to where and how one actually grew up is itself an unusual, and many would say perverse, value judgment. So there are no remaining options that haven't been ruled out. And I expect this to be just one of an immense multitude of irresolvable disagreements.
Humans have heretofore been free to develop radically incompatible value systems because imposing our will on distant others has been quite difficult and expensive, and none of us has had the ability to impose our will on the entire planet, let alone the entire lightcone. Expecting us to be able to reconcile all value systems people currently follow without there being winners and losers just seems like blatant wishful thinking.
I don't understand why this comment has so much karma. It doesn't feel like it engages with CEV at all seriously. Maybe I'm misunderstanding CEV?
The premise of CEV is that you have a super intelligent AI that can essentially simulate current existing humans, and then predict how they would act were they more knowledgeable, smarter, and more rational. Baked into this assumption is the fact that this is accurate to what those humans would do. For all intents and purposes I think of CEV as indistinguishable from just giving each human a bunch of knowledge, intelligence, self-control, and time to reflect, and then coming back to them when they're done.
Given this definition, the Isreal/Palestine example seems pretty ridiculous. To start with, the existence of this intelligent AI throws so many current cruxes out of the window, like death existing, or property, or jobs, or gods (If you know everything the AI knows, can consider all the arguments for/against god, and know yourself perfectly, do you still genuinely believe in a god?), or the fear of being killed by the opposition.
.. I'm skeptical that people would care anywhere near as much as they do today about Israel/Palestine without these cruxes.
Some specific responses:
> which would be voluntarily accepted not just by the leaders of both sides, but by every individual on both sides, even though there are plenty of people on both sides who would very much rather die than give up an inch of their claimed territory
Sure there are people today who haven't had endless time & argument to think and reflect, who would say this. Would this hold if they had time to reflect? If they were not longer under threat? If there were a million more options available?
> You might say that if this can't be resolved we just leave them alone to keep killing each other, but that is also completely unacceptable to a large group of people
There are absolutely intermediate steps (in some massive multi-dimensional space) between a resolution and this.
> You might say if Israelis and Palestinians had "grown up farther together" they wouldn't hate each other so much, but having no attachment to where and how one actually grew up is itself an unusual, and many would say perverse, value judgment
I genuinely do not know what humans who have reflected for thousands of years with super-intelligent brains would think, that's part of the point of CEV.
> without there being winners and losers
The rewards of today's game (before CEV & ASI) are nothing compared to the rewards post CEV & ASI!!
I think what I'm trying to say is that the current set of options are very small, the constituents are not putting much time and energy into actually thinking what they want and also do not have that much knowledge and thinking ability ("very small" and "not much" in contrast to how much they would have, in this CEV world), so of course robust agreement seems hard. CEV changes all of that.
I think a good criticism of CEV would be "This might be intractable to compute" or "How the hell would you get an AI to do this, and know it was accurate", because it's a very idealistic idea and not clear how you would do this in practice.
First, the whole idea of going off to reflect on your beliefs for thousands of years only even makes sense if you presuppose a (globally unpopular) liberal-individualist worldview. For most people alive today, important value-laden beliefs are much more about honoring one's ancestors, binding oneself to one's ingroup, and stuff like that. Updating beliefs using one's individual mind to weigh up evidence and arguments the way we take for granted here on LW would be seen as a terrible betrayal. And most people who operate in this communal mode are happy with it and feel no desire to change.
Second, even if we assume for the sake of argument that you can make everyone participate in this thousand-year reflection, their present-day actual self still presumably gets a veto, otherwise the procedure would be baldly coercive. But if their virtual self pops out of the hyperbolic time chamber and starts saying stuff they find completely abhorrent, they're just going to reject it. Try to imagine yourself undergoing this procedure and your super-reflective alter ego recommending that you spend all your time smashing cute innocent children's skulls in with rocks. That's about how receptive many currently living people are going to be to the idea that God isn't real and the Holy Land has no particular significance and they should pursue some other reward instead.
Of course, the fact that we have absolutely no idea how to build an AI that would faithfully carry out CEV is also a blocker, but my point was that there are still conceptual problems even given an AI that can do anything.
Okay there's obviously a fundamental disconnect here. I'll try and go backwards down my beliefs.
From first principles I approximately care about individual humans as they have experience and agency. I don't particularly care about group wants/values, except as they serve as an expression of the people in them.
Like, I don't see groups of people as being a moral patient. A community of people matters for the sake of the people in it. (or for the sake of the people who will be in it, or other second-order effects, I don't want to get bogged down, my argument is that the people are where it derives its meaning from)
Is this how you feel? Or do you derive moral meaning in some other way?
Disagree, I don't see CEV as presuming that really. People's values on reflection would be substantially different than their unreflected/uninformed values, which dissolves many current strong value differences in politics and helps in general.
CEV does not necessarily result in a nice liberal pluralist 'freedom for freedoms own sake' outcome. If enough people think truly on reflection that we should simply not have X level of pain regardless of if both parties agreed, then installed a rule that is illiberal outcome but quite good by most parties' values. Yet on reflection, are most values really like that? I'm skeptical, some decent amount of modern values are that sort of dictation of other's behavior, but the real disagreement underlying is often about facts of reality such as existence of religions, whether certain behaviors are healthy for human's, and the current social fabric.
Eliezer's proposal to exclude is not so simple as exclude, but I do agree it has flaws. However the natural route in my opinion is bargaining theory. If every individual has voting power for their values in the CEV, we can do a lot better than modern democratic governments. We'd have insight into people's values-on-reflection which is much more accurate to what they really want, and the computational power to integrate all of that. So we don't have to go along with "80% of the population wants X thing, but 20% want mutually exclusive Y, thus we can only have X". People's values are bargained around between possible coalitions, so that if you care enough about Y you can trade your direction on another issue with someone who might only weakly want X. Then you can cut up issues into granular "X vs Y with some constraints" and so on.
I don't have time to read much just now but this comment and the introduction of the CEV post prompts some question.
How much does the argument hang on item 3 and specifically the last thought "self-modification ability". If not much then maybe that should just go but if it does matter then please define it a bit. I could see a case where humans having very extensive abilities here could produce a case where most now would say we destroyed humans and human civilization--so hardly seems to solve the problem.
Others raise the problem of social/group choice and I don't see that is well dealt with. That seems very crucial to the general argument and lacking solution (generally not yet found in most social and political theory to my knowledge) then CEV is largely ivory tower type thinking. Might be a nice map but seems to apply to some different territory.
I believe Eliezer's stated best guess if you put a gun to his head and forced him to guess (although he flagged the whole point of CEV is that this can only be a guess) is that a CEV superintelligence would end up laying down some specific rules that prevent unrecoverable mistakes, and perhaps fix some obvious terrible things that everyone agreed were terrible, but then leave those as basically underlying laws like gravity, and not otherwise intervene – specifically so that humans end up still getting to meaningfully control their destiny together, which (he guesses) people would collectively turn out to want, upon reflection.
I'll offer a slight tongue-in-cheek comment here, but do have a more serious comment to follow. Perhaps we can just hope for R.Daneel Olivaw to make an appearance as that ASI. :-) Though here one might look at Hobbes' we need a Leviathan just knowing we cannot control it -- it will keep us from doing a lot of the bad, and potentially civilization ending, things we current can and sometimes come close to doing. That kind of aligns with your thought of it laying down some specific rules than have to apply like the physical laws but otherwise let us do what we're doing now. (Interesting here is that this seems to be something, as I understand it (though somewhat from Chinese fiction) is what Chinese good leadership is about. The emperor is suppose to make good rules that provide the correct incentives and then people will mostly do what they should.)
One the bit about setting up rules, particularly to manage risks and unknowns (or poorly understood processes that produce big and undesirable outcomes for people I have always likes Ron Heiner's The Origins of Predictable Behavior as something of a framing for for why such rules exists and why they are difficult to follow at times. Perhaps an ASI would be better able to figure those rules out as humans will tend not to think enough during the crisis and then once past, if survived, after enough time those rules are seen only as constraints that are rights violations as the risk/consequences have long been forgotten.
I feel a particular frustration here as someone who is both pretty ASI-pilled and pretty into liberal democracy. And when I say liberal democracy I don’t so much mean the specific political structure that seems efficient at our current tech level but rather the set of underlying moral commitments like non-domination and a government of the people — commitments which I’m aware might soon be a lot more expensive.
I think this idea of assuming the immutability of current relationships and power dynamics is a common crux in how people interpret advanced AI's impact on the future. Ex: the economy must require humans to function, consent of the governed setting lower bounds on state misbehavior, continuity of state monopolies on violence, building superweapons is hard, the solidity of mutually assured destruction.
Another important component of this is the ability of humans to make rational choices (as voters, consumers, etc). Even though we know this is only an imperfect system (see: advertisements and addictions), it's still workable enough to be the backbone of liberal democracy. However, once you have agents running around that can ~deterministically model and manipulate humans, or design superhumanly compelling narratives and media, this useful fiction is going to collapse.
There are technical solutions to all these problems (tools for epistemics, AI proxies, more resilient second-strike assurances, AI-enabled coordination), but they have to actually get implemented. And to get implemented, you need the foresight to realize these assumptions aren't guaranteed to hold in the face of technological progress.
if an idealised democracy collectively decided to blow itself up with me inside then I think that would be its right and I wouldn’t want to stop it.
This serves as a concise and elegant denunciation of the very concept of democracy.
The closest intuition pump I can think of for that scenario is in Three Worlds Collide by Yudkowsky, when the Confessor, born in the before-times of scarcity and strife, explains that he joined a rebellious movement that tried to overturn the global government for decriminalising non-consensual sex.
In that story, the young Confessor at the time understood rape as a crime of such proportions that the world must have become crazy to allow it, and that it was time to make tabula rasa with this brave new society.
From the perspective of the democratized regular people who supported the decriminalization, they were raised in such a loving and benevolent society, that they could only fathom non-consensual sex as being in the interests of the recipient.
(It's not explicitly stated in the story, but imo quite implicit, that the regular person's opinion is correct here, and "rape" as a concept is no longer a typical representative of "non-consensual sex".)
I think that ontological discrepancy is the kind of reason why, if I were Douglas, I might prefer to defer to democracy than to my own opinions, although in the case of Douglas, he makes it sound more like unconditional ideological commitment, which I can understand from the perspective of virtue-signalling, but not on the naked object-level.
Well yeah I think it is actually conditional on something like "good enough democracy". Some intuitions that feel related to me:
IMO. my current take on the entire debate is I believe that it's very hard to retain democracy by default, regardless of whether it's desirable to retain it, and the largest reason for this is that being a democracy is going to be disadvantegous in national competition, because freedom flips from increasing state power to harming state power.
@David Duvenaud made a very good point that once people are mostly jobless, people can perpetually protest, and without severe repression, can destroy the state with their own AIs simply by just increasing protest size to 30-40% of the population, and also that in an economy where humans are mostly jobless and any work is done voluntarily, UBI or some equivalent becomes necessary, and at that point fights over redistribution become literally life and death, and not giving UBI to a group means you are in essence causing the group to die, meaning politics is less positive-sum than before.
More generally, this point from Ben Garfinkel is relevant here on how automation makes democracy decline.
So the answer here is pretty simply that maintaining democracy is far too intractable, thus the benevolent god option wins by default.
(And this is assuming no superpersuasion happens, which if it does happen, pretty much makes democracy trivial eventually.)
I wonder if there's a key question around something like whether improvement to (let's call it) liberal social order and societal wisdom can be substantial and fast enough to curtail, delay, or otherwise redirect (let's call it) an unfettered ASI transition... and whether that improvement can be hastened using contemporary and near-future AI. Or at least, whether that's at least as viable to pay off in time as various other long-shot hopes. I think it's the best long shot on the table.
At the end, you wrote,
start by maxing out our own existing potential to be smarter, wiser, and better. They may not be likely or easy, but the fire is still burning
so I think you maybe agree. But the earlier part of this essay gives more impression of confidence that a rapid ASI transition is basically locked in at this point.
Thanks for this, and the gradual disempowerment paper, and the workshops!
That might sound like an extremely lukewarm take
We need more lukewarm takes please! The world would be a better place if everyone had lukewarm takes!
It’s easy to take potshots, but I expect there is a very painful tradeoff between good outcomes and good processes,
Some part of me believe a good process is a good outcome? Like what if the thing that you want to protect itself is the unfolding of a just and kind way of seeing the world? What if the process itself is the outcome?
Now you might say, well that doesn't bloody help us align ASI does it but it might just! What is CEV if not a process for finding values? Some of the 4d chess proposal for aligning ASI like QACI are essentially also processes of value search.
I want a dictator claude to implement a world wide democratic process damnit! Vingean reflection and having reflexively stable values are blown way out of proportion as alignment problems imo and I do very much believe that some degree of process alignment is possible.
I really appreciate this post fwiw.
I completely agree here that more thought needs to go into what 'success' in AI alignment even looks like apart from 'ask the supposedly aligned AI what to do once we give it all the power'. I definitely also think that ideas of non-domination, liberty etc from philosophical republicanism are important beyond the regular welfarist utility maximisation framework.
If you're interested, I've tried to do some preliminary thinking about this question here -- https://www.beren.io/2026-08-07-Maximum-Entropy-Morality-Metaplastic-Constitutionalism-And-The-Dynamic-Virtues/
Thanks! I have to admit this has been sitting open on my laptop for months, but I have now finally read it, and I am a big fan. I need a bit more time to stew on it but the basic outline indeed seems more compelling to me than the handover/CEV cluster, and aesthetically I really like the approach, especially things like virtue taylor expansions.
As ASI reaches towards technological maturity, it's more like the fabric of reality rather than a government. You exist within laws of physics, you are made out of their nature. The question is what can you do in that world, within those laws, not whether the laws of physics can be altered once in place, by anything outside their own nature. It's a good idea to hold off on putting the laws in place, before you know what that entails. And it would be nice if the laws of nature allowed you to build your own smaller worlds, and for you to become your own version of the laws, the way you would choose to on your own.
(Which is to say, the so-called ASI-pilled takes are not ASI-pilled. Perhaps there needs to be a further gradation.)
Yeah, I share all of your worries in almost the same words. In a way I feel we shouldn't even wish for worlds where responsibility is taken away from us. It's better to aim for the opposite: becoming more capable of responsibility for ourselves.
Maybe the only principled path is to slow down AI and accelerate human intelligence. Preferably via technologies that can help everyone, like pharma / brain stimulation / brain-computer interfaces, rather than technologies that will create inequality, like designer babies.
I've mentioned this possibility to quite a few people. The usual response is we can't slow AI progress enough, IA will be too slow, blah blah. Quitters.
I see pharma, brain stimulation, brain-computer interfaces, and designer babies alike as sharing several (entwined but distinct) properties:
I'd like to see more of this attitude but also applied in directions breaking past those limitations. Tech can be multiplayer, infrastructural, like libraries, citations, fact-checking, Wikipedia. What comes next? Can we develop and distribute it quickly? (Community notes, epistemic scaffolding, knowledge commons, foresight tooling, coalition tech, bargaining bots, ...?)
Beautifully written! You frame the question well. It's one of my favorite questions. I've considered it a lot and have rarely heard answers along these lines (although I'm sure others have similar ideas):
I think what people miss is the expansion of both physical and mental possibilities in a world with a benevolent ASI. For starters, we needn't settle for one shared world.
What is your ideal world? You shall have it! You want discovery and self-determination? Even at risk of disaster and death? Very well!
I will have a different utopia, shared with those whose preferences are close enough for compromise.
We'll have as many as we want. Many of us will visit each others' utopias, if their rules allow it.
There are limitations in what we can each get, but they are small - I think near epsilon.
You'll say that something is missing if your challenge is created by an all-powerful ASI who could solve those problems for you if you but asked. How will you have your full sense of discovery, achievement, and self-determination? You won't love this, but with consideration I think you'll come around: we erase that knowledge from your memory, so that you can have the full experience of discovery and struggle and creating your own future.
We'll do it only with your consent. You'll protest that this is not really what you want, but I'll protest that my objection to dying for your form of dignity is stronger than your desire to inhabit the one true world in which struggle is real. Your discomfort lasts only until you take the memory operation. And it will be reversible. You set the exit conditions, or if you truly want to struggle and die, feel free.
This will allow for an unlimited range of challenges, simulated in as much soul-crushing detail as you care for, lasting for as short or long a time as you signed up for. Choose carefully, and try a little before you commit to a lot!
Where will these utopias exist? If you insist, we can terraform new worlds and put you in cold sleep until they're ready and we can get you there. I will be taking the fast route with easier travel to the many neighboring utopias: simulation.
If you think that an upload isn't you, we can keep your brain running in your body. If you can accept that a copy that proclaims "I am still me! I have every memory, every trace of doubt that I would still be me - and I am!" is actually you, we can dispense with the meat brain and perhaps we will run multiple of you in parallel, merging memories if they haven't diverged too far.
The possibilities are nearly endless.
We need to really stretch our imaginations before concluding that a benevolent god couldn't provide the best of all possible worlds.
Perhaps we're already in such a sim. I don't worry about this, because I think if we are the goal is probably to solve alignment, anyway. And perhaps have some fun along the way.
Related, but tangential: Imagining success is far harder than imagining failure. This may be a major blocker to convincing people we can all win big if we just cooperate on building ASI safely.
Edit: I realize this doesn't fully relieve your concerns. I don't think they can be relieved without you imposing your will on me. And I'd really rather you not, when I've got so much to gain. In exchange, I'll forgo imposing my will on anyone, too.
Something has to run the world, and I'd rather it be a benevolent god than a bunch of fallible humans. Having nobody be in charge so we can fight about it has been very far from optimal so far. Your historical examples have usually been fraught with fear and danger of death and sometimes worse.
I'd also expect the humans to arrive at a similar scheme if they don't screw it up. So human control of the future is mostly okay with me too.
I think the sense of possibility and helping shape the future is something that very few human beings have gotten much of, and it has played relatively small roles in the emotional lives of even the ones who have truly shaped the world. I think we'd actually get a lot more of that sense of agency and adventure by getting worlds of our own to shape, shared challenges to overcome, and adventure from exploring other people's challenges and uniquely designed worlds.
The main issue with permanent disempowerment (where the future of humanity gets some crumbs, compared to the AIs) is uplifting resources. All the little human games are human scale things (especially in the virtual worlds), but compute for bigger minds (for the little humans that want to eventually grow) can't be both an illusion and serve its essential purpose. A Moon-scale mind/world still needs the whole Moon (metaphorically, as a share of compute), and similarly for a star-scale mind/world. Then there's security of (your share of) compute into deep astronomical time, when the stuff scavenged around the galaxy cluster that remains accessible to you is running out globally. Or even earlier, once the first wave of expansion/colonization is done and the global resources no longer grow and start trading off each other.
The galaxy clusters without other grabby aliens in them (if that's statistically a thing) might also have better security into deep time, if their substrate-level governance (that maintains/defends compute on the side of the physical world) refuses to truly merge with Earth-originating one (not that the Earth-originating AIs are necessarily much better than the aliens in that respect from the perspective of the future of humanity, and no galaxy cluster accessible to the future of humanity is free from their dominance, under permanent disempowerment). The future of humanity might then lack the option to settle such galaxy clusters (before accelerating expansion of the universe makes them causally isolated from other galaxy clusters, and thus no longer at risk of encountering more alien substrates with a meaningful foothold).
I'm not sure I follow your thinking here:
if their substrate-level governance (that maintains/defends compute on the side of the physical world) refuses to truly merge with Earth-originating one
If there aren't grabby aliens, then all the AIs are earth-originating in this scenario? I'm assuming a sovereign ASI that maintains alignment with its distant clones. Do you mean cooperative aliens their ASIs?
I wasn't thinking that AIs got most resources in this scenario. I'm considering the "benevolent god" scenario the OP title implies: an ASI that's pretty much fully aligned to humans and our values. In that world, humans get first dibs on resources (although I think we'll want to share them with AIs).
Humans can grow and do bigger things in that scenario, too. But I don't think it's psychologically healthy to measure one's life by the scale of one's projects. Preparations for encountering aliens and their ASIs, compute security, etc, would be made by the servant-god by necessity, since it's far more competent than humans or independent AIs by necessity (I'm assuming a singleton scenario that allows us to avoid molochian competition).
Living in virtual also allows pretty dramatic subjective time expansion. For that and other reasons, I doubt baseline humans will still be around when we're worried about heat death issues; we'll likely all have decided to transcend in some way or another. Or if baseline humans are still around they'll have a lot of help from their transhuman kin as well as their servant-god.
Subjective time expansion is like algorithmic improvements for small LLMs, the result is still better with big LLMs (if they also get the algorithmic improvements), it's not a tradeoff. The scale of projects (and feasibility of the kinds of uplifting/"transcension" that need a lot of compute to maintain) is the crux for permanent disempowerment, which many people seem OK with. What I'm pointing out is that it puts hard constraints on some things you didn't mention, so your discussion doesn't make distinctions relevant to this crux.
an ASI that's pretty much fully aligned to humans and our values
The ambiguity is that an ASI that's as aligned with humans as humans are aligned with each other still probably takes almost all of the stuff. An ASI that gives away most of the things it secures is a much stranger beast, and I don't believe we are at all on track to get there on current trajectory, even though I think there's only about 30% chance of extinction. While in most other outcomes the future of humanity gets permanently disempowered, it probably retains autonomy and self-determination, because under technological maturity these things are approximately as cheap as not wiping out the future of humanity in the first place (and also mean approximately the same thing on the object level).
So with my model of alignment, it takes a much more cautions course of development to avert permanent disempowerment (even with a nontrivial probability), but on the current trajectory there's already a significant chance to get all the little prosaic utopias (with hard caps on growth, and uncertain future as it runs into astronomical time). Thus it's somewhat policy-relevant (in the crazy world that's OK with a 10-30% chance of extinction) if big projects and security into astronomical time are valued.
If their aren't grabby aliens, then all the AIs are earth-originating in this scenario?
I meant merging substrate-level governance with grabby aliens (in the galaxy clusters that do have them) would dissolve the concern (so that the merged substrate maintains everyone's property rights in the galaxy cluster compute, even as it shrinks eventually). It actually seems plausible that such merging would occur (essentially to avert the penalty of worse security), but it's not necessarily the case.
In contrast, one of my greatest fears for ASI is that it will freeze our values in place, so that human political systems never evolve past what is dictated by our current weird mish-mash of concepts.
As an example, our current thinking treats political boundaries as nearly sacred, non-negotiable facts of human existence. I claim this is an absolute absurdity. We should actively and energetically redraw boundaries so that they align better with real contours of political values. If half the country hates the other half, it's just obvious that you should draw a new boundary between them. Of course the result will not be perfect, since populations are not perfectly separable, but it's a cliche that you ought not let perfect be the enemy of the good.
Don't get me wrong, liberalism is a nice idea, it was a great improvement over what came before. But it was also corrupted and subverted to serve the ends of the powerful. Given how much progress has been made in every other sphere of human endeavor, it's anomalous to think that this cluster of ideas, mostly developed by British and French thinkers 300 years ago, is the perfect, ultimate form of human political philosophy.
It seems to me that this article somewhat overestimates democracy. Democratic institutions are simply mechanisms composed of people that take human behavior as input and produce policy decisions. As long as people's preferences and behavior have any significant influence on the decisions of the dictator Claude, he does not seem fundamentally worse.
Democratic institutions are simply mechanisms composed of people that take human behavior as input and produce policy decisions.
I think in democracy, we do not only see the side of preferences; it has a side beyond preferences. The kind of utilitarianism that maximizes utility may indeed be solved in theory under this ASI assumption, with nothing to find fault with. But I think democracy is politics; what we lose is the possibility and potential of political life and choice itself. For example, a herd of managed pigs can be given saunas, massages and so on, but pigs are managed pigs; we would not say we are doing politics with pigs, we are managing pigs. Why are we managing pigs? Because pigs do not constitute a real power constraint on us, and cannot fully understand what we are thinking. So we do not treat them according to the principles for humans. Of course, it cannot be denied that many people may feel satisfied.
Partly this is me grieving the coming loss of innocence. There are certain kinds of freedom people used to have that are gone now — hopping on a train and go to a new town; walking into a forest and picking out a patch of land to be your home; jumping on a boat, sail to a different country, wander right in; building your own house in whatever strange way you want. There was also a time when science was virgin soil. Maybe we’ll get some of these back, but maybe some of them are necessary sacrifices, or unavoidable features of progress.
I think games will be important for purpose in the future. There are some feelings like that of finding a genuinely new scientific discovery which will certainly be gone in the ASI world.
Imo, we should create games that capture the feeling of all of these experiences and let people experience them first hand. Ideally they're created today by people who experience these things first-hand and still remember them viscerally, so that they can be a first-party genuine recreation rather than a recreation from the history books. That will not make the experience real, but it would be as real as it gets, and we have a limited time to record them before AI takes over more and more meaningful things in our current daily lives.
There are situations where I’d rather die with dignity than live without.[3] [...]
there are some things at stake, distinct from whether we survive, and perhaps even in tension with it, which I nonetheless care about protecting.
[WASHINGTON] Head full of fantasies of dyin' like a martyr? [HAMILTON] Yes. [WASHINGTON] Dying is easy, young man. Living is harder. [HAMILTON]
we don't condone passive romanticized suicidality in this house
This strikes me as a core problem of AI alignment, rather than something lost amid the struggle to solve alignment.
We don't yet know how to build human-aligned AI. But I expect one property any human-aligned AI must have is that it actually listens to humans in some way. Some of the most promising partial proposals for safe superintelligence involve quite a bit of creativity in translating human input into AI behavior (I'm thinking in particular about iterated distillation and amplification here).
For any good proposal to scale up AI to a benevolent sovereign, I'd expect the proposer to offer strong technical and humanistic answers to the question: "How does the machine take human input, and what does that mean for humans' autonomy and self-determination?"
I appreciate you writing this and think you’re pointing at something important! I think a post-ASI slow-down is under-discussed and underrated, and that’s part of what I was trying to point at in this intermediate version of part of a larger as-yet unpublished essay series I’m working on.
Zvi recently coined a nice distinction between AGI-pilled and ASI-pilled: as more people are reckoning with the speed of progress, some are beginning to come around to the fact that AI could be superhuman at a huge range of tasks in a way that radically reshapes society, but not that it could leap us to the end of the tech tree shortly afterwards and start making nanotech and dyson spheres.
Hence, you see people saying things like “Because of AI we must rethink the entire notion of mathematical progress”, or “I have to make sure I amass lots of capital before most jobs get automated”: pretty crazy statements, and yet, still not quite grappling with how wild things could get in the next few years, let alone the next decade.
I feel a particular frustration here as someone who is both pretty ASI-pilled and pretty into liberal democracy. And when I say liberal democracy I don’t so much mean the specific political structure that seems efficient at our current tech level but rather the set of underlying moral commitments like non-domination and a government of the people — commitments which I’m aware might soon be a lot more expensive.
What to do? Well, on the one hand, there’s a growing body of legitimately AGI-pilled liberalism, pushing for things like pluralistic alignment and AI-assisted democracy, but mostly not really reckoning with the possibility of humans losing all meaningful leverage — that even if we could align AIs, we would still be facing an earth-shattering upheaval of our political economy. And on the other hand, the default position among the ASI-pilled seems to be that sooner or later we’re just going to have to hand over all power to Claude 8 Pantheon. Honestly, I find myself a little adrift.
De parabel der blinden, by Pieter Bruegel the Elder
These two factions are both correctly noticing that modern liberal democracies are wholly unequipped for what is coming. We can already see legislative processes struggling to keep up with the state of current AI — which, I must remind you, is the weakest it will ever be.
And the liberal democracy crowd is correctly noticing that actually there’s a lot you can do to shore this up against AGI — indeed, lots of ways AGI could help with AGI. It might even be able to help reverse some of the brutal erosion that liberal democracy has been suffering of late. They are also quite prudently continuing to observe that alignment is not only a technical problem but also a political and moral question about what values should have influence over the future.
Meanwhile, the Claude 8 Pantheon crowd is correctly noticing that it’s possible we’re headed for a reasonably fast takeoff (as in, less than a term of office) that completely flips the gameboard, and it’s a heck of a lot simpler to just put an aligned AI in charge rather than trying to retrofit deliberative processes. Liberalism and democracy are both pretty demanding, and we may not be in a position to make many demands of our solutions. Also, maybe aligned AGI would actually be more moral than us?
I could anthropologise more about the STEM high modernists not realising they’re high modernists and the humanities people who don’t like straight lines, but ultimately that’s just a way of dodging the harder and more important question, which is: what do I even want here? What is this ASI liberalism thing I’m trying to stand up for?
It’s not liberal values, like having people be safe from harm. I think even the people who want to hand over power to an ASI ASAP are kind of hoping said ASI will lock those things in. In fact, I think many of them believe that a fairly swift handover is in expectation the best way to safeguard those values.
It’s also not liberal institutions, like everybody voting for a local representative every N years or a (mostly) free market revealing the fair price of goods. These aren’t really core to what I care about, so much as mechanisms that enact it, given certain ambient constraints. And if those constraints change — as I suspect they will — then so should the mechanisms.[1]
What it is, I think, is a secret third thing getting lost in the middle — the bit where the government is a thing made up of people, and the people have power, and the moral right of the government to exercise power comes from the people in it.
Freedom of Speech, by Norman Rockwell
I think the best way to pin down this third thing is by looking at where it’s in tension with the others. Or rather, hopefully we can all agree that hollow zombified remnants of democratic institutions are not great, and that specific principles like voting structures are valuable for what they do, rather than for their ceremonial aesthetic. So let’s focus on how the process ends up in tension with individual values and outcomes.
Here’s one example: I do not like benevolent dictators, not because they might secretly be evil but because they have the capacity to arbitrarily infringe on people’s rights, whether they exercise it or not. There’s a long digression to be had about the fragility of consequentialism and the nature of decision procedures, but ultimately, I’m not a utilitarian.[2]
Relatedly, I think it’s very bad to manipulate or coerce people even in pursuit of saving the world. That might sound like an extremely lukewarm take but, well, unfortunately I think lots of people disagree. And to be fair, this conviction of mine does have some unpleasant consequences that I accept: if an idealised democracy collectively decided to blow itself up with me inside then I think that would be its right and I wouldn’t want to stop it. There are situations where I’d rather die with dignity than live without.[3]
I get all torn up about technocracy because I believe there are smarter ways to make society function but I also kind of like the fact that we do things the slow way where everyone is involved. And sure it’s not perfect — it’s all kinds of broken — but there’s something important there. I was actually a bit surprised when I realised this, but apparently I’m some kind of bleeding heart patriot.
And when I read stuff like Ian M. Banks’s Culture novels, where mankind is ruled over by machines of loving grace, I do get a bit of visceral revulsion. I do not want to put on the whispering earring. I do not want man to be abolished.
Still from (and rot13 spoiler for) Gur Gehzna Fubj
Of course, none of this is new — there’s a well-oiled subfield of research full of people who keep saying things like “it is important that the values we align AIs to are not decided by a small group of powerful people” and “the benefits of AI progress should be widely shared”. Unfortunately, it feels to me like many of the people saying this just don’t seem to get it — what they want to do about all this is, like, fund upskilling programmes and change what goes in the constitution and support development in third world countries and do more participatory democracy schemes instead of trying to prevent the potential oncoming apocalypse.
If you are worried about literally everybody dying in five-ish years and the light of human civilization winking out for good, it’s sometimes a bit hard to take such people seriously. “But my participatory democracy programme will help us navigate the challenges and opportunities of AI,” they say. “Sure thing buddy,” you reply.
(Lest you think me callous, part of what prompted this essay is a melancholy conversation about how much it sucks to feel yourself inadvertently developing a reflexive dismissal of smart and thoughtful people who are really trying unusually hard to do good in a changing world, whom you happen to disagree with about how well neural nets will generalise.)
It especially doesn’t help that these types of concern — equality and dignity and so on — are so widely recognised as important that it sometimes feels like they’re the default fig leaf that organisations pick up to Show They Care, in a way that sometimes feels like an intentional deflection from grappling with ASI.
I get the sense sometimes that people who have really stared into the void basically just feel alienated from people who have not done that. These void-gazers notice that they believe a lot of very weird things that appear to be frighteningly accurate. Democracy is hard and slow and everything is really unbelievably ten kinds of on fire. All these debates about legitimacy and pluralism are interesting preoccupations for people worried about mere AGI. Once you’re worried about ASI, you don’t really have the luxury. Politics is useful insofar as it gives a lever for influencing the world, but mostly it’s inconvenient that politicians are ‘waking up’ because now there’s a risk they’ll start doing things like wanting all this power for themselves.
Morning Sun, by Edward Hopper
It’s easy to take potshots, but I expect there is a very painful tradeoff between good outcomes and good processes, which bites so much harder if and when humans stop having anything useful to contribute. And all these other questions are somewhat moot if we’re all dead. Separately, I happen to think that if we mess these other questions up bad enough, we could still basically lose all value even if we’ve solved technical alignment, but the point I’m trying to make here is that there are some things at stake, distinct from whether we survive, and perhaps even in tension with it, which I nonetheless care about protecting.
Partly this is me grieving the coming loss of innocence. There are certain kinds of freedom people used to have that are gone now — hopping on a train and go to a new town; walking into a forest and picking out a patch of land to be your home; jumping on a boat, sail to a different country, wander right in; building your own house in whatever strange way you want. There was also a time when science was virgin soil. Maybe we’ll get some of these back, but maybe some of them are necessary sacrifices, or unavoidable features of progress.
And part of this is an extended apology for gradual disempowerment. One of the successes of the paper is that it swayed some smart people sceptical of takeover towards taking x-risk seriously. But it mostly didn’t ASI pill them, so their solutions still seem to me to be pretty unlikely to work, and in many cases I expect them to make the problem actively worse. Conversely, some ASI-pilled people were like “ah thank you for finally expressing what I’ve been thinking”, but plenty basically don’t buy it and are pushing in directions which I expect will make things worse.[4]
Positioned as I am with feet in both worlds, I’m a bit worried that the ASI crowd is writing off liberalism a bit because the liberalism crowd is writing off ASI, and vice versa.
And heck, I do think there’s something here worth fighting for.
Impression, Sunrise, by Claude Monet
Maybe we get worlds where we really are watched over by machines of loving grace who also tile the further edges of the universe with happy squiggles and the literal utility scores are through the roof, and future humans are a bit like pampered kittens — the world is pretty nice for them, but not because they have any meaningful leverage, liberty, or comprehension. This sounds great from a utilitarian perspective, but I do not like it one bit.
And I do not think it is the only way — I think there are worlds where we focus on massively increasing the coordination bandwidth between humans and more generally start by maxing out our own existing potential to be smarter, wiser, and better. They may not be likely or easy, but the fire is still burning. We could start by just getting a bit clearer about what people mean when they talk about the dangers of extreme power concentration, or what we want to align ASI to, and what that would even mean.
One nice thing about the existential risk movement of yore was that it was all pretty hypothetical and we could all have our different answers to thought experiments, but now we’re getting into the period where some people are actually going to have to decide whether to hit the big button that says “give the power to Claude so that it can try to save the lightcone from the evils of men’s hearts”. As it happens, I expect I am somewhat more sceptical about whether that will turn out well, but that’s not really the core of it. If I believed it was likely to work, I would still feel bad about it, and even if it all worked out and we found ourselves in the golden fields being watched over by the proverbial machines of loving grace, I would still feel like something had been lost along the way.
This, incidentally, will force us to ask some very difficult questions about what properties of those mechanisms we really care about, and separate the bugs from the features. For example, what are democratic representatives actually representative of? Would real communism be good if we could actually try it?
Technical digression for philosophy nerds: The precise term for this conception of liberty is ‘republicanism’. Carlsmith and Laine have characteristically thoughtful essays on the tension between liberalism and welfare which shaped my thinking here, but the notion of liberty they focus on is itself the more Benthamite conception grounded in non-interference. That said, Laine’s piece and Carlsmith’s broader series do grapple with non-domination in spirit.
Awkwardly I do not think we currently pass the bar for “idealised democracy”. By analogy, I support the principle of euthanasia, and have more mixed feelings about implementation. But the fact that we currently fall short of my vision of good democracy doesn’t lead me to feel that anything goes.
It’s part of the reason why I and others have been doing things like running workshops on post-AGI civilizational equilibria, but even there, it’s really an uphill battle to get people to fully grapple with the bit about post-AGI civilizational equilibria.