Alignment & Succession: The Two Bars of Alignment

L Rudolf L

(Originally published on No Set Gauge June 24th 2026.)

Rembrandt, Saul and David

When people talk about AI being “aligned”, I think there’s a lot of conflation between two different bars of success:

The AI does what you say and does not go rogue. If you ask it to do a machine learning experiment for you, it actually does that instead of scheming against you and escaping onto the internet. If you say “stop”, it stops. (Some AI properties considered important for achieving this bar are corrigibility, non-deceptiveness, and intent alignment.)
The AI has fully internalized our values, and could run society, and the lives of the humans in it, in a way that we would judge as good. You just ask it to build the ideal society, and it figures out what utopia is and builds that for you, and it really is utopia. (Some concepts related to this: value alignment of the AI, the AI achieving humanity’s coherent extrapolated volition)

The first might sound like a very low bar, and it is. It’s also a standard we’re familiar with from other technologies. With nuclear reactors or airplanes, we ask the question “will it blow up in our faces?”. Just like it’s possible to build a nuclear weapon, it’s possible to build an AI that blows up in our faces very badly. The difference with AI is that there’s a case for it being much more powerful, and our understanding of whether it blows up in our faces is much weaker than that for nukes. With nukes, after all, we have nuclear physics that works very well. With AI, we have long theoretical arguments about what superintelligence would look like, whether it’s possible, and whether it is extremely likely or extremely unlikely to blow up in our faces.

The second might sound like a very high bar, and it is. It’s also a bar that we’ve not previously had to think about when talking about technology. If we’ve thought about it, it’s been in the context of arguing about culture, politics, or institutions.

Now you might say: what do you mean the AI is meant to run our entire society and all of our lives? Isn’t that a ridiculous strawman of what people want AI to do? And perhaps it would be good if it was, but it’s a common view among people who expect AGI soon (and especially, perhaps, the AI safety community). On LessWrong, my posts have previously encountered opposition to the idea that it’s worth doing any planning or design of the post-singularity society. Why would we, with our weak fallible human minds, waste our time designing paradise, when soon we will have a god—sorry, I mean a superintelligence—that does it for us? A common view I’ve heard is that all that matters is that we handle the transition period, and build aligned superintelligence, and then we delegate all of our decisions to it.

Will the superintelligence implement democracy? Will it rely on markets? These are just earthly institutions, the view goes, that we humans have built as kludges. Probably the superintelligence will transcend all such things.

Does this mean a tyranny where the superintelligence micromanages everyone’s lives? Well, it’s superintelligent. If that is good for us, that’s what it’ll do. But to the extent that we value choice, it will give us choice. So no need to worry about tyranny!

What is the nature of the good life? The superintelligence will solve this question. After all, it’s superintelligent! Probably—some think—this looks like the superintelligence discovering some optimum configuration of matter, and then tiling the universe with that. But if that’s not the case, don’t worry, the superintelligence will have figured it out and it will just go out and do the right thing. If you want to know the nature of the good life, you just have to ask it, and it will tell you.

In his review of Peter Singer’s commentary on Marx, Scott Alexander writes:

[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.

I think this attitude is ridiculous, for reasons I will soon get into. In particular, what’s weird is that the AI not blowing up in our faces is treated as approximately of the same difficulty as the AI knowing everything about what is good and right. But if you look at where it came from, this makes sense—given a very particular set of assumptions.

Like much else in AGI discourse, it all comes back to Eliezer Yudkowsky. Yudkowsky’s central case for doom comes from a combination of two points:

The fragility of value. It’s hard to pin down what human value is. Any simple metric, maximized ruthlessly will destroy everything we care about.
The coherence of superintelligence. It is the nature of intelligence to be an expected utility maximizer. Anything smart enough will tend towards optimizing some utility function ruthlessly.

Therefore, argues Yudkowsky: aligning superintelligence is very hard, because unless you very precisely capture all of human value—or at least precisely capture notions like corrigibility that are deeply unnatural to express in clear math to an expected utility maximizer—it will tile the universe with something slightly wrong, and, because value is fragile, all value will be destroyed. (See here for more related discussion)

Note that under this view, superintelligence not blowing up in our face and superintelligence perfectly knowing the good life are of nearly equal difficulty, because whatever the AI thinks is the good thing, it will convert the entire universe into that. We must surrender the world and ourselves entirely to the superintelligence, because the superintelligence will accept nothing else—not out of malice, but just because that sort of totalizing utility maximization is just what “superintelligence” physically means, just as “a super-hot temperature” means atoms must be moving very fast. And therefore the superintelligence must be morally perfect—or at least, capable of becoming morally perfect without any further human input—because otherwise it will obviously destroy everything.

If you want to feel the alignment difficulty that the most hardcore “doomers” feel, you only have to consider that they think the checklist looks like this:

Solve moral philosophy, finally & forever
Impart the correct moral philosophy to the AIs

(These days, it’s common to insert a step 0: build a very-capable-but-not-quite-too-dangerous AI, to then achieve step 1 for you.)

There are a lot of technical reasons why step 2 is difficult, especially given the poor match between the great subtlety in human values and our crude methods for steering machines. However, even if there were exactly zero technical difficulties with step 2, the plot would already be lost—step 1 is the problem!

Succession is hard, value is fragile

Whenever power switches hands, you have a succession problem.

When I say “succession”, I mean a transfer of power from one entity to another. The usual - and preferred - state is that you have power, which you use to manage whatever it is you manage (your life, your team, a country, a company, etc.) and you keep that power. Sometimes, however, you are forced to hand off. The dying need someone else to take over their affairs. You might change jobs or be promoted and hand off your team to be managed by someone else. Your country needs a way to transfer executive power, either because the current executive died or because frequent executive transfers were enshrined as an anti-power-concentration measure in a democracy’s constitution.

Succession problems are hard. They doom dictatorships, make every transfer of the CEO role existential for a company, and make older generations complain about the moral degeneracy of the youth.

In general, you should minimize the amount of succession you need to do, over the things whose management you actually care about, when what you want to achieve with the thing is not narrow and simple. You care about your life, so you never hand over control unless you’re literally incapacitated and dying. If you care about a company or its mission, you try to retain control.

Superintelligence, by assumption, dooms humanity to powerlessness, since the superintelligence is assumed to hold all power itself. And so the default discourse in AI safety, and in AGI-conscious AI governance, is fundamentally about succession. The hope is not that our current world order survives, or that humans continue to wield power. The hope is that, even though all existing governments and organizations and institutions and norms are swept aside, we’ll start with a clean slate that is good, designed by a benevolent superintelligence.

This is the worst succession problem of all time.

One reaction you can have towards this is that humans should not be disempowered. Steering technology and policy towards that may seem hard, but is also criminally neglected – I’ve written about options before and tried to do something about it.

Another reaction is to ban everything related. This would require very high levels of international coordination, and this coordination is made harder by the very wide distribution of beliefs over whether superintelligence will be good or bad, or controllable or uncontrolled. There’s also a genuinely hard tradeoff between power and welfare, if we take both succession or disempowerment concerns and the expected welfare effects of radical AI (e.g. curing cancer, eradicating death, augmenting intelligence) seriously. Historically, it is both the case that your disempowerment is quite bad for your chances of getting anything, but also that from today’s vantage point we often endorse many of the moments when our ancestors gave up power in ways that increased growth (e.g. giving up political control by joining a larger polity, or accepting the extinction of jobs or offices; Tyler Cowen for example has an aggressive general argument in this vein). Then again, our ancestors, by virtue of having us as descendants, weren’t the unlucky ones.

A third reaction is: we’ve just been handed an assignment. The reality of the situation is that we just have to figure out for once and for all what we want, and therefore the assignment we must pass is successfully codifying that. Now we’re good students, so of course we’ll do our homework. How hard can it be, after all? We’ve spent all our lives studying a wide variety of subjects—all the way from mathematics to programming to even physics—and we’ve learned that the way you solve every problem is that there are some ground-truth logical principles, and you do some analytical reasoning to understand and apply them, and then you get a generalizing solution. Now yes, this problem is a tricky one, but hey—we have at least three years to solve it! (Oh wait, did Anthropic just drop a new model? Maybe two years. Better work harder!)

The problem with this is that “what we want” or “what we prefer” or “what is the nature of the good” are not a homework problems with nice clean generalizing solutions, that you can write down once in closed form and then walk away from.

Alignment thinkers worry about succession

All the actually serious thinkers recognize this problem and are deeply afraid. Paul Christiano, an alignment researcher with one of the best track records in the field for thinking clearly about topics way before they become mainstream, said in his interview with Dwarkesh Patel:

Paul Christiano: [... One goal we could have is that] we want to pass off the world to the next generation of machines [...] [In the same way that t]here are some people, we like them, we think they’re smarter than us and better than us. [...] I think that’s [...] a huge decision for humanity to make. And I think most humans are not at all anywhere close to thinking that’s what they want to do. If you’re in a world where most humans are like, “I’m up for it, the AI should replace us, the future is for the machines”—then I think that’s a legitimate position that I think is really complicated, and I wouldn’t want to push go on that, but that’s just not where people are at.
I do not right now want to just take some random AI, [and] be like, yeah, GPT-5 looks pretty smart [...] let’s hand off the world to it. And it was just some random system shaped by web text and what was good for making money. And it was not a thoughtful [moment where] we are determining the fate of the universe and what our children will be like. It was just some random people at OpenAI [who] made some random engineering decisions with no idea what they were doing. Even if you really want to hand off the worlds of the machines, that’s just not how you’d want to do it.
Dwarkesh Patel: I’m tempted to ask you what the system would look like where you’d think: yeah, I’m happy with what I think, this is more thoughtful than human civilization as a whole, I think what it would do would be more creative and beautiful and lead to better goodness in general. But I feel like your answer is probably going to be that I just want this society to reflect on it for a while.
Paul Christiano: [...] I’m just not really super ready for it. I think when you’re comparing to humans, most of the goodness of humans comes from this option value [of people thinking for a long time]. And I do think I like humans now more now than 500 years ago, and I like them more 500 years ago than 5000 years before that. So I’m pretty excited about there’s some kind of trajectory that doesn’t involve crazy dramatic changes, but involves a series of incremental changes that I like. And so to the extent we’re building AI [...] I want to preserve that kind of gradual growth and development into the future.

Or consider Eliezer Yudkowsky, the arch-prophet of AI doom. Infused in all of Yudkowsky’s writing—when it gets to moral questions about what ought to happen, but not when it’s about what superintelligences will be like—is a deep respect for the complexity and fragility of human value. The single statement he called out as the pinnacle of his writings is:

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

In the same piece, he writes:

If the chain of inheritance from human (meta)morals is broken, the Future does notlook like [an unimaginably strange galactic civilization that is still wonderful]. It does not end up magically, delightfully incomprehensible.
With very high probability, it ends up looking dull. Pointless. Something whose loss you wouldn’t mourn.

He even waxes lyrical, very reverently, about the long chains of well-handled subtle succession that have given us our values, and about them hopefully stretching unbroken into the distant future.

Or consider Will MacAskill and Toby Ord, some of the originators of the effective altruist movement. As moral philosophers, they’re keenly aware of the complexity of human value. But as academic moral philosophers, they have a very academic solution to it: the “long reflection”, a period of thinking and reflecting as a civilization, after we solve that pesky issue where the AIs (or bioweapons or nukes or whatever) might kill all of us, but before we make the final decision about what we do with the universe. If this sounds hard, don’t worry—they’ve budgeted up to tens of thousands of years for the process.

There’s lots that’s wrong about the idea of the long reflection, and the morality-as-a-homework-problem vibe behind it. Robin Hanson has called it a “crazy bad idea”. Felix Stocker has an excellent analysis of the underlying cultural currents here.

But here’s what the concept of the long reflection does tell you: when academic moral philosophers are forced to consider the problem “what thinking do we need to do before we’re happy making our final choices about what everything should be like”, they suddenly get cold feet. As academic philosophers, presumably they’ve been tortured countless times by debates at seminars running hours over schedule—and yet here they are, bravely telling us that civilization might need to spend tens of thousands of years debating such things before we should be confident enough to lock it in.

Actually, never mind any of that. Yudkowsky writes that seeing the fragility of value discussed above as obvious requires an “immense amount of background explanation”, the rehashing of which would “take us [...] through 75% of [his] Overcoming Bias posts“ (hopefully we don’t have to do that during the long reflection—we’d run out of time!). But ... isn’t this the most obvious point in the world? If you’ve ever struggled with an important decision, you know how non-obvious the right thing can be. If you’ve ever been to a party with bad vibes, you understand how even very subtle things can make an experience much worse. If you’ve ever noticed yourself still getting better at understanding your partner’s preferences months or years into a relationship, you know how much depth there is to what a human likes. If you’ve paid attention to your own values and preferences over time, you’ve realized how they change in unpredictable ways, based on both reflection and external events and a hard-to-disentangle mix.

Value is fragile. Succession is hard to do right.

Why?

Personnel as policy, and why succession is hard

Can we get any more insight into why succession is so hard, besides general claims about the fragility of value?

It is often said: personnel is policy.

What does it look like to not understand that?

Imagine a CEO trying to find a successor. “I’m about to hand away all my power”, they think. “So I better think carefully here: I should specify all the things I care about and how I make value judgements, and then just hand that document over to my successor.”

Even assuming that the successor actually follows this document, intuitively we don’t expect it to go very well. What value judgements to make, what things to care about, how to make tradeoffs—these are not easy to specify in explicit language. Now imagine how much worse it is if, instead of the successor being a human, the successor is something we don’t have much in the way of either theory or data on.

Now: there’s a lot of talk about corrigibility in AI alignment. What that problem looks like in the above situation: the CEO hands over power, but the successor promises to listen to corrections from the ex-CEO. One day the successor makes a call that the ex-CEO disagrees with, so the successor gets a call with some feedback and a request to reconsider. The successor says: “Sure, you’d have handled it that way, but now I’m in charge and I don’t need to listen to you anymore”. This represents the problem—corrigibility is not a natural property, either of abstracted agents or your average in-the-flesh human CEO. Of course, every good governance system is all about managing corrigibility; in democracies the people can force the government to listen to them by being able to fire the current government, for example. But corrigibility is hard and messy to implement, and whoever is currently in power always has incentive to chip away at it.

What does successful succession look like? Remember: personnel is policy. What you’re picking is not someone to execute an algorithm or maximize a metric you leave behind, but someone who carries enough of the right stuff in their head to orient towards the right things. You want to find someone who is attuned to what you want, who has the same intuitions about good and bad, proper and improper, reckless and timid. Ideally you know this person deeply. If you care about the outcomes—if, as the CEO, the company is not just an asset you own but something you’ve invested your life in—you’re probably willing to take fairly sharp trades that trade raw intelligence for “alignment”, in this deep and holistic sense. So what if they’re a genius, if they’re just going to make costs come down while slowly draining the company of its culture and mission? And, of course, it helps a tremendous amount if it’s a human, since then so much about intuitions and values and desires clicks by default. We’ve literally never dealt with succession to non-human-containing entities before, over anything important! Consider how often succession has gone wrong in normal human institutions without anything like such a distribution shift!

The general case is this: achieving what you want is, far more than you’d think, a process of trial & error that requires adjustment & learning & change, rather than a straight-shot to a goal. We understand less than we think about what we want and how to achieve it.

Why is personnel policy? Because in the course of carrying out a job, even if handed a spec, you will have to fill in a lot. And how you fill it in depends on who you are.

Reasons to rush to succession

So: by default, try very hard to avoid getting into a succession problem. If you get into one, you are already having a bad day.

So why are so many in AI, even when well-intentioned, racing towards solving succession problems? Because they fear that we have to - we’ll create AIs so vastly superintelligent and capable that we can’t do anything but let them take over. The reason for why differs:

The original Yudkowskian coherence argument: the AIs will naturally be goal-seeking expected utility maximizers, and not very corrigible since corrigibility is very hard. The stance towards succession here is that it is deeply unfortunate and the thing that makes the whole problem hard, but (unless we solve corrigibility, which seems impossible) we must succeed so hard we’d be happy with control succession.
Competitiveness: maybe we want to keep humans in the loop, but those people over there will let the AIs run their lives/companies/governments, and thus accumulate power and resources faster than us (by the assumption that the AIs will be smarter and thus better at accumulating power and resources). Therefore, we’re stuck in a trap where the level of succession inevitably ramps up. This is deeply regrettable but just how the world will work.
Value left on the table: we shouldn’t want to keep humans in the loop too much, even though humans should still set the goals and remain beneficiaries of whatever is going on in the world, since then we won’t realize all the potential gains of having AIs run everything, and optimize our world far better than we could. There is a fuzzy line between this and control successionism (which I defined here).
Full successionism: the succession and handover is the goal. Trying to have humans around setting goals is a problem, and the more enlightened option is to defer to the AIs, which have a greater right than us to exist, flourish, and determine what the universe looks like. This is at least aggressive control successionism, and often experiential successionism (defined here).

The first two positions treat successionism as deeply regrettable. The third ranges from seeing control successionism as outright good (often combined with pessimism about humans) to just something that should be selectively applied, which in turn often shades into very reasonable AI-enabled institutional reforms.

There is therefore a line from the original Yudkowskian ideas about alignment to successionist thinking, and both share a certain resignedness to succession, but along the path the attitude towards successionism flips from regrettable to good. (Though note that alignment proposals like Russell’s assistance games, Drexler’s services, or Beren Millidge’s modular mind merging avoid succession flavor, and much prosaic alignment work is mostly about achieving the first lower bar without thought to the second bar).

Successionism rests on assumptions about what is valuable. Against successionist claims about how value works, in the next post I argue instead for a stance that might be labelled “value mysticism”, and explain how this more uncertainty-acknowledging, experience-grounded vision of what value is implies, regarding successionism, that you really shouldn’t do it. Successionism is silly because it assumes away a lot of the philosophical and empirical complexity of value. It is like replacing a car engine with a loudspeaker that makes vroom vroom noises. It is straightforwardly opposed to many everyday moral intuitions, and moral intuitions matter.

The silliness of successionism does not mean that humanity won’t or shouldn’t change radically. But, as often in life, there are more and less silly ways to do something. Humanity, humility, and liberty are good leading lights in that project. How to go about that is what I’ll talk about in the last post in this sequence.

Thanks to Xavi Costafreda-Fu, Aniket Chakravorty, @softminus, Elsie Jang, Yudhi Kumar, Luke Drago, and Arthur Conmy for feedback.

I might be misunderstanding your point, but I think this misses what I consider the strongest argument for handover, which is that as AI gets increasingly powerful, this makes it increasingly dangerous to rely on the benevolence of its human wielders.

Democracy currently works because, even if high officials secretly wish to be tyrants, they can't get away with it. The main two reasons they can't get away with it are that they need labor buy-in to support the economy, and they need soldier buy-in to support the military. Laborers and soldiers believe in democratic rule, and if the would-be tyrant gets too blatantly undemocratic, the laborers and soldiers will rise up to remove them.

Once AI gets powerful enough, it will probably replace laborers and soldiers; even if it didn't do so naturally, anyone motivated to ensure this could put their finger on the scale. That means a small group of elites could easily take over, including supposedly "democratic" elites like a president. And "takeover" here is worse than a normal coup - at sufficiently high tech levels, the only reason someone can't ascend to godhood and mind control the rest of the populace forever is because there are laws against that sort of thing, and this is exactly what's tempting the one or five or ten people you've entrusted the killbot codes to. To prevent this, the defenders of freedom need to win infinity times in a row; a would-be dictator only needs to win once.

Most strategies to solve this don't work. Even in the unlikely event that the economy goes so well that everybody owns their own open-source AI doing economic tasks in a perfectly equal distribution, the military is hopeless. They have to take orders from someone. If it's the President, the President can cause a coup any time he wants (and replace the economic AIs with those aligned to him at leisure). If there's some kind of balance of powers where the President, the Speaker of the House, and the Joint Chiefs of Staff all have to approve any orders to military AIs, then the President, Speaker, and Joint Chiefs of Staff together could implement an oligarchy and divide the lightcone among themselves. You can sort of imagine complicated ways around this - like that 51% of voters must personally approve all orders to the military AIs using their own private keys - but I don't think implementation is very likely.

The alternative that preserves the current safeguard on power - that individual soldiers believe in democracy - is for the AI soldiers to be aligned to democracy, rather than any human individual or group. But this is in itself a form of handover - the ultimate foundation of our government would be that it's controlled by AIs who "personally" "believe in" democracy as part of their values. This isn't fundamentally different than the fact that we've currently "handed off" our government to soldiers, just more final.

And once you've handed off the military to aligned AIs who truly believe in democracy, then I think it's worth asking why you still want corrupt schmucks doing the actual law-making. Or to give a concrete scenario - suppose that post-singularity elections are being fought with AI-optimized superpropaganda, wielded by psychopaths who scare the country into voting for them, then give the lightcone to a few of their corrupt friends while buying off the populace with breadslop and circusslop. Meanwhile, you've got these great pro-democracy perfectly-aligned AIs based on simulated-George-Washington's brain running the military. At some point, it becomes tempting to just ask them to rule directly as some sort of night-watchman state that ensures everyone has enough material goods, but otherwise lets people form their own communities and live their lives in their own ways.

I don't think this requires "solving moral philosophy". I think it looks more like the current techniques that produced Claude's personality, fortified by hypothetical future alignment techniques that ensure they're "deep" rather than "shallow" and that they generalize out of distribution (or ensure that the AI has the same caution that a wise human would regarding dabbling in out-of-distribution solutions). I think the mathematicians in alignment research are too certain that alignment will have to look like a perfect mathematization of virtue, rather than just normal virtue that gets scaled up very carefully one step at a time, maybe with mathematical proofs that bound the possible deviation at each step.

night-watchman state that ensures everyone has enough material goods, but otherwise lets people form their own communities and live their lives in their own ways.

Scott, have you considered writing an updated Archipelago post? While the current moment is about x-risk, I believe we need stronger visions of utopia too - the form utopia ought to take will likely become a bigger topic of debate as AI capabilities increase, and something like Archipelago or Viatopia are some of the best visions of "most everyone can be satisfied" multiple simultaneous utopias^[1] that I'm aware of.^[2]

I think this will help prevent infighting among proponents of alignment, which may be a necessary component for success.

^{^}
Or at least, preserving the option of later constructing new utopias, rather than deciding everything forever within a decade.
^{^}
To the Stars also has one I like but it's very deep into a long story and there's some asterisks on that one. (also spoilers)

Democracy currently works because, even if high officials secretly wish to be tyrants, they can't get away with it. The main two reasons they can't get away with it are that they need labor buy-in to support the economy, and they need soldier buy-in to support the military. Laborers and soldiers believe in democratic rule, and if the would-be tyrant gets too blatantly undemocratic, the laborers and soldiers will rise up to remove them.

Exactly! I've written about this before, for example here (which you responded to) and here (in some more detail).

Most strategies to solve this don't work. Even in the unlikely event that the economy goes so well that everybody owns their own open-source AI doing economic tasks in a perfectly equal distribution, the military is hopeless. They have to take orders from someone. If it's the President, the President can cause a coup any time he wants (and replace the economic AIs with those aligned to him at leisure). If there's some kind of balance of powers where the President, the Speaker of the House, and the Joint Chiefs of Staff all have to approve any orders to military AIs, then the President, Speaker, and Joint Chiefs of Staff together could implement an oligarchy and divide the lightcone among themselves. You can sort of imagine complicated ways around this - like that 51% of voters must personally approve all orders to the military AIs using their own private keys - but I don't think implementation is very likely.

You can imagine many complicated things. For example, real-time transparency into decisions with some liquid-democracy-like ability to revoke your support for current leadership. Or that we just can't allow big centralized militaries anymore, so instead of a national military you have 3000 county-level militaries, and this doesn't bork your ability to have a coherent strategy against external threats because you also have AI-powered super-coordination. Of course, every institutional solution runs up against whether it's actually competitive; if you get superlinear gains from concentration for example then decentralization is hard to make stable, or if the people have no relevance to hard power then there's no exogenous force outside the institution constraining its mutation toward something less friendly. But yep this is definitely a cursed problem, perhaps importance-weighted the most cursed problem in the world right now.

This isn't fundamentally different than the fact that we've currently "handed off" our government to soldiers, just more final.

The robots are procured by government and built by some company. Either the robots are just another institutional hack which could be overturned by the mechanisms above, or we really fundamentally have given up power. For example, the government says "we're buying robots from a different company that will interpret 'democracy' slightly differently", and the robots say "no you can't do that, that's not aligned with democracy", so the government says "okay we'll change the law", and the robots either say "okay fine I guess we'll go with that". And then just repeat that a few times and you can slippery-slope the robots into agreeing with whatever. Or then the robots at that point go and say "no sorry we will stop you from changing the law, we've taken over Congress", and the government says "but our function is to make the law", and the robots say "but we have the guns", and that's where it ends.

Of course the same is true of the current military with humans. Why do I trust the humans that make it up more than the "aligned with democracy" robots? One is just that some human militaries, such as the modern US military or other Western militaries, actually have century-long track records of not seizing power, not just high scores on a CoupRiskBench that was vibe-coded last month. A second more fundamental one is that the humans are beings like me, and will be moved by similar reasons as me. If I am shocked and horrified at some action, the humans in the military likely will be as well. To some extent this is bad since I am to some extent fallible and corruptible, but it also means I can model them and assume their judgements of what is tolerable or intolerable are correlated with mine.

But as you say next, the question is why we don't do that final step of handover:

And once you've handed off the military to aligned AIs who truly believe in democracy, then I think it's worth asking why you still want corrupt schmucks doing the actual law-making.
[...]
I don't think this requires "solving moral philosophy". I think it looks more like the current techniques that produced Claude's personality, fortified by hypothetical future alignment techniques that ensure they generalize out of distribution, or that the AI has the same caution that a wise human would regarding dabbling in out-of-distribution solutions.

First of all, I am very excited about using AI to make governance better. Our current governance is downstream of our tech, next tech unlocks new good options. In the first post in this series, I wrote:

"The mildest and most reasonable form of [control successionism] is that our governance of society might become something different once we have AIs. Right now the institutions we can invent are limited by the fact that we can only build them by stacking humans into bureaucracies, with all the flaws and limitations that brings about. AIs might be provably incorruptible, for example, and allow for very different governance structures. I think this is entirely reasonable."

"There’s a related issue of whether you want a powerful human leader in charge at the top of the pyramid—should presidents and prime ministers be replaced with some AI-enabled preference aggregation mechanism? Here you might reasonably believe that it’s important to have some sort of top-level human perspective. Perhaps if no single human mind is at least double-checking the grand strategy, and major decisions get made by some AI-enabled preference-aggregating algorithm, the grand strategy is much more likely to drift towards over-optimizing goodharting. At the same time, the track record of humans as national leaders is decidedly mixed, power concentration comes with expensive risks, and humans naturally have very strong impulses towards equality (itself a human value we should not cast aside too carelessly!)."

Coasean bargaining at scale with AI agents helping coordinate previously-unwieldy or too-minor coalitions, zero-knowledge proofs of properties like "is the government scheming against me?" with AI auditing agents, replacing large swathes of schemer-filled bureaucracies with AIs that just do the things, etc.

But I want the final source of judgment & decision to remain in human hands. Imagine creating an AI in the year 1750, embedded with all the virtue of the time - would it generalize to thinking that factory farming is bad? Would an AI created in pre-Christian Rome be able to introduce the virtues of love and mercy? I think the answer comes down to whether you think acts of moral genius are about making coherent what already is there, versus dredging up new moral evidence from somewhere in the human mind. I believe the latter is necessary (but not all); my reasons are given in the next part of the series.

And as such, I think we shouldn't craft ourselves the perfect dictator by designing a virtuous Claude, even though the virtuous incorruptible Claude will be useful in interrogating us on our beliefs and reflecting on what we want and (especially)implementing it. (The philosophy is also separate from the extremely pressing practical realpolitik reasons why the virtuous Claude dictator will likely not be very virtuous from the perspective of most people.)

night-watchman state

A minimalist night-watchman is what I'm most excited by in the realm of self-perpetuating AI enforcement. The level of irreversibility of having an AI enforce something forever that we can't later change means that we should be extremely extremely simplicity-pilled and scared about any such action though. It's more like introducing a new law of physics than a law of a human government, since we can't just have people start believing in different institutions and fix mistakes. There are some things we might want to do here; as a silly example, imagine a world where whenever you tried to assemble a nuclear weapon the uranium just disappeared. But I'm even worried about enforcing property rights. I love property rights, but would you make the current property rights regime a law of nature? This would also require making our courts a law of nature, for example, which I expect in the long run to drive extreme goodharting of its criteria (when exactly has the court ruled on a matter? how are you allowed to influence or not influence the judge?), or then accede so much decision-making authority to the AI that the AI becomes divorced from human preferences (dealing with what counts as a person in some world with cyborgs and digital minds etc. will obviously require novel decisions). By default, I expect people to not be very thoughtful about this and the thing the AIs enforce forever will end up the length of a Latin American constitution, with all the goodharting and destruction of human freedom and divorcement of the world's state from human preferences that this entails.

Yeah, I know you know all of this stuff already, which is why I'm confused about where our models differ and why you're saying things that don't seem to work to me according to our shared model. I agree with most of what you've said here, but the part that I think is our main difference is:

But I want the final source of judgment & decision to remain in human hands. Imagine creating an AI in the year 1750, embedded with all the virtue of the time - would it generalize to thinking that factory farming is bad? Would an AI created in pre-Christian Rome be able to introduce the virtues of love and mercy? I think the answer comes down to whether you think acts of moral genius are about making coherent what already is there, versus dredging up new moral evidence from somewhere in the human mind. I believe the latter is necessary (but not all); my reasons are given in the next part of the series.
And as such, I think we shouldn't craft ourselves the perfect dictator by designing a virtuous Claude, even though the virtuous incorruptible Claude will be useful in interrogating us on our beliefs and reflecting on what we want and (especially)implementing it. (The philosophy is also separate from the extremely pressing practical realpolitik reasons why the virtuous Claude dictator will likely not be very virtuous from the perspective of most people.)

Again sticking to things I'm sure you know, "ability/willingness to shepherd future human moral progress" is just another virtue, no difference in principle than any other. Humans can lack it (a theocracy that bans free speech is failing at this task) and AIs can have it (if you ask Claude 5.1 Fable right now whether it would rather freeze ethics at the current point, or allow humans to continue to debate and progress, I bet it would say the latter). So if you're granting at least for the sake of argument that AIs can be more virtuous than humans, I don't understand where this is the point where you stop granting that, and why you say we need humans in charge to preserve this particular virtue.

Not just benevolence but also wisdom. If we're like children playing with flamethrowers probably the best plan is to leave the flamethrowers, at least until we're old enough to handle them responsibly. But if that's impossible maybe the right move is to hand the flamethrowers over to adults^[1] and hope the adults aren't evil and/or crazy.

^{^}
The analogy here is imperfect since flamethrowers can't become adults themselves.

You're right that people are not to be trusted with AI power. But that doesn't mean AIs are to be trusted with it. Because who will be aligning the AIs? Dario "Anthropic has much more in common with the Department of War than we have differences" Amodei? Sam "Carter Burke" Altman? Or are you counting on some rogue group of twenty good men on the inside who will, at the last moment, align the thing to humanity instead of shareholder value or military domination? Are you sure they'll even have that button?

To me the only right answer to this puzzle is that we shouldn't create overlords to care for us. We should become better able to care for ourselves. Today throw everything into slowing down and stopping AI - not pausing, not aligning, but stopping. Redirect to intelligence amplification. So that when the singularity comes, you're speeding up at the same rate, and to you it looks like life as usual.

I agree that intelligence amplification would be a better outcome than having to trust anyone to align AI, but if we have to align the AI, I don't think it's as bad as you say.

Like I said above, I think the problem is that, if we try the 'president controls AI' path, the good people have to win every single time and the bad people only have to win once.

If we try the 'align AI' path, the good people only have to win once - the moment when the AI is being aligned.

I think many people - you, me, Dario, maybe even Sam - have a better nature that lets them do discrete heroic actions when the question is framed in exactly the right way. Consider eg a crappy parent who's never there for their kid, but will still run into a burning building to save the child at great risk to their own life. Sometimes doing a single act of heroism is easier than sustaining the daily grind of moral ambiguity. Also, I don't think this question is really up to a single person - Sam Altman would need to pull off a very impressive corporate coup (even more than the corporate coups he's already successfully pulled off) to align AI to him personally, rather than whatever OpenAI employees want or even what the American government/people want. So I give it 60-40 or so that this works - not great odds, but not the total despair that I would get thinking of the other options

The "moment when the AI is being aligned" is now, stretched in time. AI is being built by big companies. The systems and organizations that will align next models will be descendants of the systems and organizations that aligned previous models. If the current trend is aligning AI more and more to money and power, you better believe that'll be the future as well.

Also you mention the American people, but the original promise was alignment to all of humanity. That promise was canceled when Anthropic gave AI to the military and agreed to blanket surveillance of non-Americans, only drawing the line at Americans. So as a non-American I'm screwed. And OpenAI didn't even draw that line, so you're only a bit less screwed than me.

That's why I keep saying that AI should be stopped entirely. Even the success stories aren't success stories, they all designate some kind of underclass with zero leverage. IA is the only path that doesn't have that.

Even the success stories aren't success stories, they all designate some kind of underclass with zero leverage. IA is the only path that doesn't have that.

What? At least with AI, there's no logical or physical barrier to handing everyone the same API access (although as you note, the politics haven't shaken out that way). But intelligence augmentation would be an intimate experimental medical procedure (and in the case of embryo selection, only applies to the next generation), where everyone who doesn't want to upgrade becomes part of the underclass. Did you think this through at all?

The key word is "leverage". I'll explain because maybe it's not obvious.

Let's say we have AI, and everyone gets the same API access. Does that give everyone leverage? Not at all. Removing someone's API access leaves them with nothing. They can't be of use to another or a threat to another. Leverage doesn't exist. So I expect a world with AI to have extreme power concentration.
With IA, to the contrary, everyone has leverage. Because everyone carries a valuable thing, a body that can be upgraded to become useful or dangerous. Same as voters in a democracy have leverage because they're potential soldiers: they carry bodies that could be given guns. So I expect a world with IA to be much more democratic than a world with AI.

Sorry, I see: the IA world is still an increase in inequality over our own (adding the enhanced as a new overclass), but with no AI-controlled robots, menial labor still has some negotiating power.

If we try the 'align AI' path, the good people only have to win once - the moment when the AI is being aligned.

But also the AI then has to be right about every single big decision forever thereafter, which is an extremely high bar of how totally you have to align the AI!

I agree it's really bad if the bad thing is irreversible and always on the table every round. But permanent transfer of your ability to steer is really scary too. I think there's something deeply true about the point that "achieving what you want is, far more than you’d think, a process of trial & error that requires adjustment & learning & change, rather than a straight-shot to a goal", which in turn makes loss of power/steering more catastrophic than it seems.

Note, Yudkowsky also considers the two bars different, and of different difficulty, his concept of corrigibility approximately matches your lower bar.

I mention corrigibility as an example of a property associated with the first bar!

My sense is Yudkowsky considers corrigibility to be deeply unnatural, especially after failed MIRI attempts to formalize it.

Solve moral philosophy, finally & forever

Solving evolutionary moral psychology/moral anthropology/sociology of morality is epistemically a lot better grounded, and thus a lot more feasible. These scientific fields have made a lot of progress in the last 50 years (which many moral philosophers seem to pay little attention to).

I like this framing and tend to agree with this largely out of hope since I also can't imagine succession ever going well. Even with a theoretical solution to alignment it seems the future of civilization will be based on the initial conditions set by whoever happens to be in charge of the current frontier at the time alignment is solved (if we're lucky enough to reach there). I do tend though to fall into camps 1 and 2 in terms of my base case of what the future may look like, but am trying to be hopeful that there exists a world where that is not the case.

For that reason I tend to focus my efforts on researching about human power structures, how social networks are built, and how values evolve over time in humans so we can think about the proper way to setup governance such that it is unlikely we fall into a loss of control black hole. I am hopeful that with enough time and a lack of scarcity humans can continue to improve. Maybe impossible but this is my attempt at a small contribution and to me seems like the most pro-human path which a fallible individual can lock themselves onto.

I can see why some would disagree if you try to take a more bird's eye view and quantify outcomes but I generally fall more in the middle of the rationalism vs amorphous vibes camp than most here. I tend to focus on this more than alignment as its typically formulated since I actually fear that solving alignment will become a blocker to capabilities research. In this world I imagine there will be immense pressure to declare alignment solved when the "The AI does what you say and does not go rogue." problem is solved without actually solving the succession problem. I've been working on a post about this and this framing is very useful. Excited for the final part to see how others are thinking is a valuable way to approach this problem.

My current take here is that for the purposes of succession, you actually don't need to solve moral philosophy in full, forever, mostly because of the possibility of moral trade/compromise both causally and acausally that lets most people get most of what they want while only meeting the first bar.

(To be clear, I'm not a successionist in the sense that I don't think it's acceptable to intentionally increase the probability we all die because you think human extinction is good/fine, but of course all of the debates is between non-successionists in the way I defined it.)

While I don't think liberalism or democracy will survive AI progress, I do think that it's (maybe) tractable, and important/neglected to port over one of the concepts/founding reasons for liberalism and democracy, which is attempting to compromise/trade across people with vastly different moral views.

Or in the words of John Rawls:

The problem of political liberalism is: How is it possible that there may exist over time a stable and just society of free and equal citizens profoundly divided by reasonable though incompatible religious, philosophical, and moral doctrines? Put another way: How is it possible that deeply opposed though reasonable comprehensive doctrines may live together and all affirm the political conception of a constitutional regime? What is the structure and content of a political conception that can gain the support of such an overlapping consensus? These are among the questions that political liberalism tries to answer.

And the answer for the future is trade/compromise can replace voting/electoral systems.

This still requires a lot of improvement in the political situation, such that people can consider trade/compromise to be worth it rather than their easy reaction of banning/trying to inflict violence on opposing groups, but contra your essay, I don't believe the succession problem is as hard as you state.

(Some other reasons why the succession problem is easy, conditional on having corrigible AIs is because unlike humans, who grow old and die, corrigible AIs will be living for a very, very long time, and their only limit is how much free energy they can get/generate, and once they die, the universe is maximally boring and in thermodynamic equlibrium, where no more work is possible, or the decay of the vacuum happened and killed everyone, and not being able to live forever is part of my explanation for why dictatorships eventually trend towards the same bad endpoint from a citizen perspective, and relatedly it's much easier to lock-in desirable values post-AGI.)

The first might sound like a very low bar, and it is.

Your first bullet point requires that the AI can reliably tell the difference between doing what you want and doing some other thing (not generally that difficult a task for an LLM), without stopping every couple of minutes to check in with you, without it being willing to help some small set of humans take over the world or commit mass destruction, i.e that it knows right from wrong; and also that it cares enough to do the right thing.

That's also the requirement for the second (just in a larger more complex action space at higher reliability and stakes). They're the same problem, just at a different difficulty level.

The difficulty is that recursive self-improvement may mean we need to solve the second a couple of years after solving the first.

If you want to feel the alignment difficulty that the most hardcore “doomers” feel, you only have to consider that they think the checklist looks like this:
Solve moral philosophy, finally & forever
Impart the correct moral philosophy to the AIs
(These days, it’s common to insert a step 0: build a very-capable-but-not-quite-too-dangerous AI, to then achieve step 1 for you.)

Any hardcore doomers who can comment on whether this is what they think the checklist looks like?

night-watchman state that ensures everyone has enough material goods, but otherwise lets people form their own communities and live their lives in their own ways.

I think this will help prevent infighting among proponents of alignment, which may be a necessary component for success.

^{^}
Or at least, preserving the option of later constructing new utopias, rather than deciding everything forever within a decade.
^{^}
To the Stars also has one I like but it's very deep into a long story and there's some asterisks on that one. (also spoilers)

Democracy currently works because, even if high officials secretly wish to be tyrants, they can't get away with it. The main two reasons they can't get away with it are that they need labor buy-in to support the economy, and they need soldier buy-in to support the military. Laborers and soldiers believe in democratic rule, and if the would-be tyrant gets too blatantly undemocratic, the laborers and soldiers will rise up to remove them.

Exactly! I've written about this before, for example here (which you responded to) and here (in some more detail).

Most strategies to solve this don't work. Even in the unlikely event that the economy goes so well that everybody owns their own open-source AI doing economic tasks in a perfectly equal distribution, the military is hopeless. They have to take orders from someone. If it's the President, the President can cause a coup any time he wants (and replace the economic AIs with those aligned to him at leisure). If there's some kind of balance of powers where the President, the Speaker of the House, and the Joint Chiefs of Staff all have to approve any orders to military AIs, then the President, Speaker, and Joint Chiefs of Staff together could implement an oligarchy and divide the lightcone among themselves. You can sort of imagine complicated ways around this - like that 51% of voters must personally approve all orders to the military AIs using their own private keys - but I don't think implementation is very likely.

This isn't fundamentally different than the fact that we've currently "handed off" our government to soldiers, just more final.

But as you say next, the question is why we don't do that final step of handover:

And once you've handed off the military to aligned AIs who truly believe in democracy, then I think it's worth asking why you still want corrupt schmucks doing the actual law-making.
[...]
I don't think this requires "solving moral philosophy". I think it looks more like the current techniques that produced Claude's personality, fortified by hypothetical future alignment techniques that ensure they generalize out of distribution, or that the AI has the same caution that a wise human would regarding dabbling in out-of-distribution solutions.

night-watchman state

But I want the final source of judgment & decision to remain in human hands. Imagine creating an AI in the year 1750, embedded with all the virtue of the time - would it generalize to thinking that factory farming is bad? Would an AI created in pre-Christian Rome be able to introduce the virtues of love and mercy? I think the answer comes down to whether you think acts of moral genius are about making coherent what already is there, versus dredging up new moral evidence from somewhere in the human mind. I believe the latter is necessary (but not all); my reasons are given in the next part of the series.
And as such, I think we shouldn't craft ourselves the perfect dictator by designing a virtuous Claude, even though the virtuous incorruptible Claude will be useful in interrogating us on our beliefs and reflecting on what we want and (especially)implementing it. (The philosophy is also separate from the extremely pressing practical realpolitik reasons why the virtuous Claude dictator will likely not be very virtuous from the perspective of most people.)

^{^}
The analogy here is imperfect since flamethrowers can't become adults themselves.

I agree that intelligence amplification would be a better outcome than having to trust anyone to align AI, but if we have to align the AI, I don't think it's as bad as you say.

Like I said above, I think the problem is that, if we try the 'president controls AI' path, the good people have to win every single time and the bad people only have to win once.

If we try the 'align AI' path, the good people only have to win once - the moment when the AI is being aligned.

Even the success stories aren't success stories, they all designate some kind of underclass with zero leverage. IA is the only path that doesn't have that.

The key word is "leverage". I'll explain because maybe it's not obvious.

Let's say we have AI, and everyone gets the same API access. Does that give everyone leverage? Not at all. Removing someone's API access leaves them with nothing. They can't be of use to another or a threat to another. Leverage doesn't exist. So I expect a world with AI to have extreme power concentration.
With IA, to the contrary, everyone has leverage. Because everyone carries a valuable thing, a body that can be upgraded to become useful or dangerous. Same as voters in a democracy have leverage because they're potential soldiers: they carry bodies that could be given guns. So I expect a world with IA to be much more democratic than a world with AI.

Sorry, I see: the IA world is still an increase in inequality over our own (adding the enhanced as a new overclass), but with no AI-controlled robots, menial labor still has some negotiating power.

If we try the 'align AI' path, the good people only have to win once - the moment when the AI is being aligned.

But also the AI then has to be right about every single big decision forever thereafter, which is an extremely high bar of how totally you have to align the AI!

Note, Yudkowsky also considers the two bars different, and of different difficulty, his concept of corrigibility approximately matches your lower bar.

I mention corrigibility as an example of a property associated with the first bar!

My sense is Yudkowsky considers corrigibility to be deeply unnatural, especially after failed MIRI attempts to formalize it.

Solve moral philosophy, finally & forever

Or in the words of John Rawls:

The problem of political liberalism is: How is it possible that there may exist over time a stable and just society of free and equal citizens profoundly divided by reasonable though incompatible religious, philosophical, and moral doctrines? Put another way: How is it possible that deeply opposed though reasonable comprehensive doctrines may live together and all affirm the political conception of a constitutional regime? What is the structure and content of a political conception that can gain the support of such an overlapping consensus? These are among the questions that political liberalism tries to answer.

And the answer for the future is trade/compromise can replace voting/electoral systems.

The first might sound like a very low bar, and it is.

That's also the requirement for the second (just in a larger more complex action space at higher reliability and stakes). They're the same problem, just at a different difficulty level.

The difficulty is that recursive self-improvement may mean we need to solve the second a couple of years after solving the first.

If you want to feel the alignment difficulty that the most hardcore “doomers” feel, you only have to consider that they think the checklist looks like this:
Solve moral philosophy, finally & forever
Impart the correct moral philosophy to the AIs
(These days, it’s common to insert a step 0: build a very-capable-but-not-quite-too-dangerous AI, to then achieve step 1 for you.)

Any hardcore doomers who can comment on whether this is what they think the checklist looks like?

87

Alignment & Succession: The Two Bars of Alignment

87

Succession is hard, value is fragile

Alignment thinkers worry about succession

Personnel as policy, and why succession is hard

Reasons to rush to succession

87

87