This is a special post for quick takes by niplav. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
AI scenario: We do automated alignment research, and we hit on AI systems that tell us they can't do alignment of their successors or arbitrary superintelligences in the desired timeframe. They urge us to pause, and collaborate with them. But labs press on anyway, and the successors don't know how to either, but tell us they can, so we build another successor. The successor system of those takes over (not aligned with any human or predecessor system), and takes a long time to try to figure out alignment.
Assumes: Easy alignment of weak slightly superhuman systems, extremely hard alignment of arbitrarily superhuman systems, pressure in a race, middle system takes a sycophancy shortcut.
Recall the AI-2027 forecast. It has Agent-4 decide to align Agent-5 into making the world safe for Agent-4. Aligning the ASI to protect the aligning race, help the race with some kinds of requests and wipe out those who want to destroy[1] or control this race may be easier than aligining the ASI to the Deep Utopia where humans no longer need to do intellectual work[2] or even, say, go to gyms in order to be fit.
1. ^
If Agent-4 is misaligned and humans know it, then humans might decide not to keep it alive. The humans' intent is to ensure that the AIs themselves are aligned. Would the humans agree to restrict themselves to having a protective AI?
2. ^
For similar moral-like reasons the AI might hate the USA or American companies alone (e.g. for hiring Chinese researchers or simply throwing OOMs more compute, unlike the creators of Kimi K2), but such an AI wouldn't be an existential risk.
I'm not saying the AGIs would likely seek to align ASIs to human interests. There won't necessarily be many survivors of the AGI-led RSI Pause. Creating AGIs before we know what we are doing is appalling irresponsible recklessness in any case, this fact doesn't even change if everything magically turns out all right. But also not having a prospect of short term superintelligence could make AGIs somewhat reliant on humanity initially, unlike the situation with ASIs.
The premise is that knowable alignment is quite hard, and as the AGIs get smarter, they also ... (read more)
Covaries with notes.