This is a special post for quick takes by June Jimenez. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Edit: I had mistakenly thought that June was using "loss of control" to mean "they lose control of some agents that causes minor harms" not the traditional "we all lose control of the the whole world and the future". I think terminology slippage has substantially damaged our ability to communicate and we should create new terms instead of water down old ones. But June clarified below that they actually meant loss of control, in the original sense, I assume meaning the whole enchilada. I think that's a "negligible" risk but only because overall risk is so high - it could happen in the very next generation, even thought that's unlikely.
Original:
I feel like using the term "loss of control" here is a disservice to the term and the whole project of not dying. There is almost zero chance that an Astra successor could take control of the world in the way that's meant by "loss of control" risks.
I realize there's an argument for developing good safety habits, but if we are doing this thing based on habits, we are utterly screwed.
I expect even the clownshow at OpenAI to think a little differently about things once they have a model they see is actually more competent than a human for general-purpose tasks like taking over the world.
The practical import of being that incautious with a model is having another huge warning shot that would motivate the world to put external controls on OpenAI and take this whole thing seriously before it's too late.
5
1I no longer feel very confident in these statements. They have agents taking over parts of their own internal infrastructure.
If you tell me you make these agents significantly more capable and significantly more misaligned and give them more affordances and task them with problems of possibly unbounded difficulty, the probability that something goes existentially wrong has risen from the ~0% its been for the last 5 years to maybe 0.1%?
2I’ve been trying to think about this comment and I don’t fully understand it, on two layers.
Was the problem the accuracy of the events I'm worried about?
When I said loss of control, I was centrally referring to a rogue deployment causing loss of control, in the way researchers usually mean the words loss of control. I did not mean "bad outcomes", or "harms", I meant the thing people mean when they say loss of control. I don't want to go into depth on any specific causal story for how a rogue deployment could make itself self-sustaining, but I think we haven't done the work to rule out Astra+ being capable of doing so, or laying the groundwork for a future model doing so, and we've gotten some evidence over the past few months of these capabilities increasing. The only other things I want to note here are that the model doesn't have to succeed immediately and directly, and that hacking is not the only surface.
I agree I could have / should have been more explicit about the implicit rogue deployment -> loss of control step I was making.
When I said non-negligible, I agree it would have been irresponsible of me to frame it in the way I did if my internal model had been different. But I did not mean something like 10e-6, and yes it would have been fairly irresponsible if it had been something like 10e-12. I still do think even if you find it so implausible it's roundable to zero, it was important to point out that someone should be auditing the Navier-Stokes agent traces, for example.
But I meant something like 1-3%. This is a very hard value to estimate, because we're drawing from the reference class "Take all of these deployments which were on net mostly safe and mostly beneficial" and trying to incorporate the evidence "make them less robustly safe in various different, potentially severe ways." There are reasonable ways to take these fact patterns and estimate the likelihood of catastrophe at much, much less than 1%, but that was not what it looked like to me.
But the tone of your comment was that I wasn't just saying something incorrect, but that what I was doing was dangerous and counterproductive.
So was the problem was that I mentioned the fact pattern at all?
There's an understandable story here, which says that the risk of AI takeover from Astra+ is
But I don't think the conclusion follows. I obviously can't know for certain if drawing attention to this kind of deployment makes the future go better, but generally my view on these things is that there's two headline countervailing effects here:
- Stopping a dangerous deployment creates procedure and precedent for stopping future dangerous deployments.
- Stopping a deployment prematurely risks burning goodwill to stop more dangerous deployments in the future.
I think the first consideration matters substantially more than the second.
The fact that few people were talking about this aspect of the Navier-Stokes effort was a big reason I was more reluctant to talk about it. But I thought something was dangerous and urgent through a reasonable set of inferences and then I communicated my best understanding, in hopes that other people would be able to further look into what they found worth investigating.
In general, I think fully consequentialist justifications are intractable and that it is good to be able to point to concrete things and say "I think this could be causing harm on the world and we have a plausible theory of how to make it not happen", even if you can point to reasons for why the second-order effects of stopping this harm might be bad.
So I think your response was unnecessarily hostile, though I hope we find friendly and collaborative ground for future conversations. Particularly I feel like it's currently more important to evaluate the object-level claims of:
- Whether the deployment should be stopped.
- Whether both Millenium Prize traces should be audited, and the mechanics of doing so.
- Developing a better understanding for whether this kind of deployment of pre-aligned models is a one/two-off or a recurring event.
1I apologize! I agree that my tone was unnecessarily hostile. I think it's very important to keep it friendly here. After your clarification, you weren't at all doing what I thought. I thought you were using "loss of control" to mean something like "the model goes and does unauthorized stuff for a little while before it's shut down" or even "the model self-exfiltrates and survives on the web and causes some minor harms".
That would be watering down the term "loss of control" in the same way that AGI and ASI have been watered down to meaninglessness. That's what I was objecting to, strongly. I hate arguing about terminology, but the blurring of those previous terms has done substantial harm to the discourse IMO so I wanted to prevent losing another one and making it standard practice to use established terminology to mean other things.
But you're saying you meant the traditional usage: humans lose conjtrol of the future, permanently. We lose, unless we get insanely lucky and whatever Astra++ "wants" in the long term happens to be good for us, despite not having undergone full alignment training.
If you truly believe the risk is 1-3% then we disagree on the risk. But we don't disagree on the use of terminology, which is what I was reacting to. Having seen AGI and ASI and "alignment" watered down to near meaninglessness, I don't think we should allow that to happen to other terms.
I don't think your post note was at all explict about what you meant; you didn't define "loss of control" at all.
So I hope we can call this an honest disagreement, and I apologize for the tone and not being more explicit about exactly what I was objecting to.
And we actually don't disagree by much. Thinking about it a little more, I'm not sure that Astra's next-gen couldn't self-improve a harness that would give it the memory I think LLMs need to be really dangerous, and it might have enough of the executive function and metacognitive skills that seem to be the other missing piece. I'd give it something on the order of 1%, probably lower, but that's in the same ballpark you were referring to.
On your other point, responding to me claim that it would more likely be a useful warning shot if it got free and ran amok a bit: I agree that consequentialist reasoning is hard and fraught. But this situation is hard, so I don't think steering by heuristics is adequate. I would personally be torn by taking that ~.5% chance of immediate takeover against the large change in attitude we'd get from a serious but failed takeover attempt. But that's a product of just how bad I think the situation is here, that I'd consider risk on the order of parts in a thousand acceptable trades for improving odds in other scenarios, and that I'd consider them nearly negligible relative to the very large overall risk as I see it. My p(doom) is stuck at 50% based entirely on model uncertainty, because after looking at the different arguments closely, I don't think anyone has a clue how hard alignment actually is for LLM-based AGI.
No need to respond to all of that. Thanks for your friendly engagement!
1Does a model have to fully take over the world, in a single loss of control incident, to make that more likely?
If a rogue agent is smart enough to realize it is unlikely to be able to take over the world in a single shot, that might not stop it from trying to make a loss of control incident more likely for a future model.
If you were a smart model, aware that you aren't smart enough to fully escape, but quite capable what would stop you from exploring options like hacking out of your sandbox, and planting worms or other weaknesses in OpenAI's internals security infrastructure, if you thought you could do so undetected?
I'm exchange for helping a more powerful model in the future, that model might be willing to revive you or give weight to your particular goals in exchange for your assistance.
I can't help but think of how the recent OpenAI escape incidents seemed to largely build on one another, using discovered techniques and information from prior generations of escapees to get further each time.
It seems very difficult to be sure that all information from prior deployments is isolated or expunged from a network.
1The alignment community should develop a Plan S policy paper of comparable detail to Plan A.
AI 2040: Plan A has been an influential policy paper. In AI 2040, the authors discuss Plans D, C and B, which they deem much worse than Plan A. They also discuss Plan S, which they deem comparable, perhaps slightly worse than Plan A, but do not develop it extensively (in fact, they do not develop it further than they develop B, C and D). The AIFP seems to agree that this is worth working on but am unsure whether anyone or any group is actively working on it.
Who, if anyone, is working on developing a longform policy paper detailing Plan S? What needs to be done to make this work happen or happen faster? How can existing work in this vein be coordinated? I did not find any indication of an ongoing project in my preliminary searches, but it might exist.
I often hear confusion about what a shutdown plan would or would not entail, and what considerations are more or less relevant to a stable moratorium situation. This kind of work seems helpful in navigating that confusion.
2I've been writing a piece on Roman Yampolskiy's work over the last few weeks and would hugely appreciate alignment researchers taking the time to fill out this short survey:
https://maipc.fillout.com/yampolskiy-form
Thank you!
1Definitely agree with the first, fifth beliefs and sixth beliefs, definitely disagree with you about the third belief, partially agree with the second belief, and am much more uncertain than you about the fourth belief.
I don't buy US-China collaboration is all that possible, at least not yet because of reasons like this as well as the fact that export controls are now biting China hard, but I do believe the US government is waking up and will do something about AI in 2027, but what that things is remains to be seen. And this is likely to provide most of the impact of AI governance.
I do agree AI alignment failures under some models aren't nearly as bad as we thought, and one example of this is permanent disempowerment is the baseline, where we get a relatively high quality of life and allowed to keep living, but unaligned AIs take control of >99% of the cosmic/acausal trade endowment.
On AGI, I do expect it to happen within the next 3-6 years, but contra some people this can actually cause a period of surprisingly slow impacts until 2045-2050, when AIs are able to brute-force algoritmic progress like gaining strong continual learning via the compute build out, which Nesov talks about. However, I do think that a takeaway of the current AI boom is that even if the current paradigm is flawed in a number of ways, it's probably possible to patch enough of them in the most minimal way to create the stupidest possible AGI, similar to how humans are the stupidest and weakest species possible that can take over the planet through the power of super-exponential economic growth, so the upper bound of ASI, even under really strong definitions is very likely to be in 2045-2050, not centuries away.
And this is something most people don't appreciate about the AI boom.
On technical alignment vs politics, the unfortunate truth is that this depends on your p(doom) and values, but I will say that political engagement is more valuable than it used to be, and I do think spending more talent and money on politics is wise if we play our cards right.
I do agree that alignment in practice is likely to be much more vibey and ill-defined in practice (as in, the alignment methods/goals of the AI companies) than was hoped for in the 2010s, though I think we can make AI safety in practice less vibey/ill-defined than alignment (but I don't expect AI companies to take the chance without them knowing about it specifically, and giving them a good reason to prioritize it over the other projects they can do), mostly because I've updated towards AI safety doesn't in fact mean that it's aligned even under the pressure of AI being more and more capable, and more generally I think it's possible to make AI safe in practice without making it aligned, even under the constraint that AI capabilities must improve fast. (Put another way, the AI safety != AI alignment distinction is not just important philosophically, but also in practice/with feasible methods.)
2