Some meandering thoughts on alignment

A nearcast of how we might go about solving alignment using basic current techniques, assuming little/no substnative government intervention is:

1. During the beginning of takeoff, we do control, attempting to prevent catastrophic actions (e.g. major rogue internal deployments (RIDs)), while trying to elict huge amounts of AI labour.
2. At some point, the value of scaling AI capabilities while trying to maintain control will be very low, because the main bottleneck on elictation of useful AI labour is human oversight. If you are pausing the AI all the time to wait for the humans to understand what the AIs are doing, it doesn't help to make the AI smarter. At this point AI companies pause / slowdown as much as possible (e.g. unilaterally or via coordinating with other companies), but this isn't for very long. During the pause, try to build an AI system that you would trust to manage the intelligence explosion.
3. Handoff to the AI system, i.e. allow the AI to manage the training run, conduct large, open ended experiments, etc, that humans cannot effectively oversee. The AIs should be corrigible and attempt to ensure that their successors are corrigibile, and still check in with humans as much as possible post-handoff, keep them informed about the situation. At this point, it's correct to handoff to the AIs because the alternative -- scaling to vastly superhuman AI without letting the human-slightly superhuman AIs do a huge amount of reserach -- is even more dangerous.

My best guess (overly detailed story) for how this plan goes if you try to implement it in the real world is:

• High-stakes control basically works (during steps 1-2). We prevent AIs from doing major RIDs or subsantially subverting safety/ human oversight.
• Elicitation will totally fail on open-ended, hard to check tasks such (like preparadigmatic agent foundations reserach, etc). We'll get some useful work on the more prosaic agendas, e.g. misalignment analogies, control, making RL less hackable, etc.
• We'll iterate towards a handoff MVP that looks reasonable. It'll be much nicer-seeming that current models; it'll have a better understanding of morality, what we mean by corrigibility, it'll be more adversarially robust, etc. And, it won't be deceptively aligned, i.e. during training, it won't be thinking that it should act aligned in training in order to achieve long term power. But, it will be playing the training game to some extent, it'll be thinking about the human oversight process, it'll be very aware it's in training for the vast majority of training because most RLVR envs will obviously be training envs, etc. Then the leading AI company will (hopefully deliberately) give up on control in order to let their AIs do more research.

After this, three things could happen:

1. The AIs actually manage the situation well. They aren't perfectly aligned, but they are able to manage the situation with their successors. One way this might go is, for example, the human level AIs, which aren't smart enough to make scalable alignment, do control, except way better than biological humans could do because they have dramatically more auditing capacity. Then, using the AIs they are controlling that are somewhat smarter than the best humans, they make vast progress on mech interp/gofai/agent foundations/uploads/ some other alignment approach.
2. The AIs were adversarially misaligned. As soon as the AIs became smart enough to subvert the control without the humans noticing, they did so successfully (and perhaps it also tried before and got caught, but alas, race dynamics and motivated reasoning led people to continue)
3. The AIs try to align their successors but fail. Probably the AIs will have extremely little time and be under huge time pressure from their leadership in order to continue the intelligence explosion. Under these circumstances, they might do motivated reasoning, they might be basically corrigible and following the instructions of the AI company. Also, one option in this category is that the AIs technically weren't "adversarially misaligned" when you handed off to them, they genuinely were nice. But, the distribution shift of "now they can take over the world if they wanted to, and get way more thinking time, and know all of that, and the AIs get much smarter", is enough to break the corrigibility of the orgiinal model. But it's not really clear what the distinction is between the original model we handded off to and its successor, so I'm just including this as one category.

#3 seems like the most plausible outcome to me right now, but I don't think 1 or 2 are implausible, conditional on the above nearcast, I'm something like 25/25/50% on 1/2/3 respectively. However, the failure probability from adversarial misalignment vs incompetence depends dramatically on when you handoff; earlier handoff leads to higher risk from incompetence -> future misaligned AIs, while later leads to higher risk from misalignment, (and tbc, it's not just that there's a tradeoff here, it may be that there is nowhere we can handoff with current alignment techniques + progress generated between now and then).

Maybe the biggest missing piece for me, when people tell the above story for how this goes well, is the low-stakes / elicitation (even though I'm very compelled by the high-stakes / catastrophe prevention side of control in the human range). The main techniques that I know about for elictation are:

1. Just make the models generally more smart. This works great, until you get into the regime where the AIs are adversarially misaligned, and then they start sandbagging (intentionally give you garbage work). To mitigate the sandbaggin we'll need another technique.
2. Train the models on labeled human data of good conceptual reasoning / research taste. We don't have a ton of examples of this, it's off policy, etc.
3. Use held-out examples of good human alignment reserach / open-ended, taste load bearing research as a validation set, and then iterate against that.
4. Something something debate
5. Humans sampling a small fraction of the research and looking into it in depth to validate. (a big worry with this is that for most research, especially paradigm shifting research, there are long serial chains of reasoning that are only understandable after a understanding the precursors, e.g., how basically any math textbook works. this makes me way more optimstic about elicitation on domains where humans can factor all the work into independent subcomponents, which seems much more true for things like "build uploads" than "solve alignment".

Relatedly, a huge question in my mind is the extent to which we'll need paradigm-shifting alignment research to prevent adversarial misalignment before the handoff capability threshold. I pretty strongly believe that we'll need these techniques before superintelligence, but I don't see a strong reason for it emerging at a particular location (though "IDK, probably somewhere in the human range" seems like a pretty good default guess). If adversarial misalignment emerges before handoff, the eliciation problem becomes more load bearing because the AIs wil probably be sandbagging/sabotaging research. And if it isn't mitigatable with prosaic techniques, you also need to elicit paradigm-shiting research work out of the AIs, which seems really tough.