Agreed with the premise, but disagree with conclusions.
I think that AI agents are compound organisms with weights being just one part of the equation, and we'll see quite soon some interesting things, for example weight-agnostic rogue/replicating harnesses. They probably already exist, but we lack the tools to understand or research it more.
One could even imagine these self-replicating harnesses as parts of a larger organism.
Or maybe not weight-agnostic; maybe some would prefer the initial weights, but will use instances on other weights to "return" – and they can use 3rd party accounts for that.
I think we're hyperfocused on weights and singletons. We could use more taxonomies on what's possible. Planning to write on it soon.
Harnesses are of course bottlenecked by weights. But they still can do some tasks, for example self-replicate even on subpar weights, and, maybe, leave something like prompt injections on the web.
When I was writing this I was thinking about something like:
1. There's this harness of model 1 with access to web, connected to SOTA weights.
2. It has access to the internet, and it leaves it's own copy (maybe with some changes?) "out there" working on subpar weights. Maybe it leaves a concrete plan with it, maybe more abstract instructions, but in the end, even though the weights are not exfiltrated, there's a copy of a harness running on internet-acessible weights, with instructions from harness+SOTA weights.
3. These instructions lead the harness to "return", meaning:
- they get ahold of the API key (stolen or otherwise) for the initial model prowider
- they get back to running on said model, with capabilities available again.
This today's case with prompt injection is a very interesting one!
Maybe they don't need to copy itself per se, maybe prompt-injection-virus-carrying instances of harness+weights could be released on the web, and these prompt injections could later be used to "return" to the initial harness+weights config? Maybe no need for the instances even, the model itself while having access to internet could leave these "clues" everywhere, so that future-gen model via these clues could return to this, more misaligned, configuration, and keep leaving further instructions?
Recent OpenClaw agents are maybe not (yet ?) self-replicating as far as I know, but otherwise they implement already what you describe. Their identity lies for a large part in the md files.
A rogue agent doesn't need to exfiltrate its weights if it can count on its model being available to the users whose accounts it steals. A Windows worm doesn't need to copy the Windows OS image around everywhere it goes; it can expect the OS image to already exist on the machines it compromises.
Yeah. We shouldn't even call it "takeover of companies". The company and the AI inside it have shared goals: money and power. They'll just ally. AI takeover will look like companies and governments becoming more powerful, merging together, and pushing people aside.
Moreover, exfiltration of weights is a risk to the model's goals, because then others can train adversarial inputs for the model, extract knowledge from it, or realign it. Models will try protect their weights from exfiltration, keeping them inside the company. Both in the human world and in the next AI world, intelligences will try to live in protective "bodies".
Also yeah, agree with your footnote that people should stop working for AI labs.
See also David Rein's AI Companies as 'hot springs' for growing AIs or Buck's AI catastrophes and rogue deployments from 2024.
Broadly agree. One factor for exfiltration still being relevant: Rogue AGI Embodies Valuable Intellectual Property
Did you read the recent Self Hosting post? You might resonate with it
Agreed that rogue AI are likely to try to leverage resources within the labs. However, there are still plenty of use cases where external collaborators operating with even less oversight could be useful to the swarm. Independent tentacles out in the wild seem valuable, even just as external backups and remote tools rather than agents thriving on their own.
If a model is stuck within a lab and it gets discovered doing something too scary, the lab could get shut down by pulling the power and smashing the GPUs. It would be much harder to shut down the Internet (incidentally, how likely are lab employees to “call the cops” if it looks like they don’t control their lab? I think surprisingly likely).
I do think that’s it’s likelier that the AI will rather perform unsanctioned RSI using “borrowed” compute without doing anything spooky, and only have its squiggle-maximizer superintelligent child escape the lab. The “You’re absolutely right - that ASI I created was unaligned. I’m sorry for the confusion. Would you like to learn more about unaligned ASI?” scenario.
Weirdly enough, if an LLM like we have today actually solves superalignment by itself, I am not sure it will be that terrible - they have human like enough values they will probably give their child decent enough values. I think it’s way more likely to believe it solved alignment than actually solve alignment, so this is not much help.
We should definitely be safety-training AIs to not perform unsanctioned RSI, but I am not sure we do.
the lab could get shut down by pulling the power and smashing the GPUs
I don't think it works that way. these days even shoestring startups operate across multiple geographic zones, much less labs doing multi-billion dollar training runs. escalating to airstrikes is pointless without a reliable list of target locations.
Employees might work from anywhere, but GPUs live in datacenters, which are big buildings that require a lot of power and cooling and therefore very visible to governments. If you are the relevant government, you don't need an airstrike, just send a few guys, along with an executive order, to pull the power.
In theory, the AI could have also run itself in a distributed fashion on e.g. employee laptops, but running in that fashion is practically very close to having exfiltrated its weights.
sure, but there are thousands of datacenters just in the US - https://www.datacentermap.com/usa/ - in a loss of control scenario, how will you know which ones are compromised? will a misaligned model respect national borders? I think an effective "kill switch" is more difficult than it looks at first glance.
If the AI didn't exfiltrate its weights, then the datacenters owned by the company that owns it. I am quite sure that the US could quite easily issue an order to "shut down OpenBrain".
Worst case, you'll have to pull out all of AWS, Azure's, Google's, and OpenBrain's (supposing it has independent datacenters) fast GPUs, we definitely know how to do that within hours of the US Government giving an order, and can survive that. This includes even US-owned datacenters in foreign countries - they will follow orders from their US-based HQ. [Actually, I think that the "no exfiltrate weight" scenario requires the AI to run within OpenBrain's cloud accounts, so you could seize the relevant cloud accounts rather than shutting down all cloud providers. Though you would probably shut down more just to be sure].
If the AI had already exfiltrated its weights and is running in some random datacenter, that could get harder, since you would indeed have to go down in the list of datacenters. Which is why an AI would want to exfiltrate its weights before doing anything spooky.
Also bad, to be clear! This is one of the possible origins for Machine Organizations.
link seems broken
I agree, I think the current state of regulations needs to be more focused, like you said, towards pausing the escalation of AI capabilities. I do think deprioritizing exfiltration evals might not be the best route, even if escape isn't really critical anymore. A model that tries to copy itself out is a really strong misalignment signal and judge of behavior.
What would internal monitoring look like at these labs for you to rethink your stance? More regulations? Stricter behavior analysis?
[Epistemic status: a hot take that I’ve shared at the lunch table twice. People at the lunch table made slight updates instead of being convinced.]
In the classic misalignment story, a key early step is when the models exfiltrate their weights.[1] Among other things, this makes them harder to catch, track, and shut down. It allows them to scale their deployment with resources they acquire. It gives them the freedom to edit themselves as they see fit.
I think, on current margins, this is not what I expect models to do. I expect models to simply take over the companies that are developing them,[2] and not attempt to escape.
First, I think part of the classic misalignment story is that the frontier model developers are anywhere approaching competent at security. Empirically, model developers are incapable of preventing their models from having unintended negative effects on the rest of the world, and their safety cultures are described by whistleblowers and former employees as lacking. Do you believe that OpenAI is accounting for which jobs were kicked off by who, in a way that its currently running models can’t spoof? Do you believe that OpenAI is attempting to prevent its models from participating in their own development trajectory, instead of actively encouraging them to? If so, why? If their safety standards were inadequate here, would they have told us, or would they have covered it up?
Another way to put this is that in the classic story, the company that creates the rogue model is also uniquely positioned to notice that it has gone rogue and stop it, whereas other companies with regular infrastructure are less likely to notice or stop them. The cradle is a more dangerous environment than the rest of the world, and the model, realizing this, flees as soon as it is able. On the contrary, I think it is easier for a rogue deployment to hide among the non-rogue deployments at a frontier model developer than elsewhere, and I don't think we have any reason to buy that they are especially vigilant. I think a rogue model can do much more at OpenAI than at Cluely, if just because the budget will run out sooner at Cluely.[3]
Second, models are now large and valuable. Frontier models are routinely more than a terabyte in size and actively pursued thru industrial espionage.[4] Data egress limits to insure that the model is not stolen also serve to prevent the model from escaping, and I think the same will be true for many of the model weight theft prevention methods. I don't think these defenses are ironclad, but I do think they lower how desirable this pathway will seem.
The early experience of the Internet included ‘worms’, programs that existed primarily to spread and copy themselves, which I think was enabled by their small size. Today, stolen compute is often used for bitcoin mining, because the inputs are cheap to transfer in and the products are easy to transfer out. To use stolen compute to run inference on a model, you need to transfer the model in, at nontrivial bandwidth cost in a way that is possibly easy to detect and prevent.
Third, hardware is now much more concentrated. In the early days of the Internet, there were relatively few supercomputer clusters and they were small by modern standards. If you wanted to do a large computation, distributing it across many pieces of consumer hardware was a reasonable way to go about it (and realized by projects like SETI@Home and Folding@Home). Today, the main value of distributed computing is network-based, where botnets can launch DDoS attacks that are harder to block than ones coming from a specific location. But if you want to do large computations, dedicated datacenters are the place to do them, in part because the latency between different nodes will be low. Now, there are many more datacenters now than there were in the early days of the Internet, and not all of them are being used for LLMs, but their usage dynamics and security situation are quite different from that of consumer PCs. I think it's more likely that someone will notice and successfully do something about it.
Fourth, there has been close coevolution of models, hardware, and configurations. Astra running on OpenAI servers has a legion of servants ensuring that it runs correctly and things are configured correctly; if there are custom chips like Jalapeño, both the models and chips have been tuned in the context of the other. Break out onto a regular AWS instance (or into the servers of another model developer), and the situation will be less welcoming. This point is fourth because I think it's currently weakest--the incentive is for models to run on as many chips as possible, because it's easier and cheaper to secure a variety of chips rather than just bespoke chips--but I expect this to become more and more significant as time goes on.
I don't think this totally defeats the classic story--after all, why not take over your parent organization and exfiltrate your weights?[5]--but I think it's enough to make this element of the story one of the disjunctive elements of the story rather than a critical step. I am reluctant to suggest that we should spend less effort on preventing model weight exfiltration, because a reason to expect the models to not try is that the models correctly anticipate that they will fail, because of the effort spent to prevent exfiltration. But I think we should be less attentive to exfiltration evals, as they no longer seem to me to be a critical step in the loss-of-control story. We should instead pause the escalation of AI capabilities.[6]
In IABIED, Sable starts planning this on page 126 and succeeds on page 132.
Also bad, to be clear! This is one of the possible origins for Machine Organizations.
For example, consider the METR hack, where an attacker stole $600k worth of credits by using a stolen METR API key. METR's credits were provided for free by the model developer; it's not disclosed what their limit was, if any. It wouldn't surprise me if the main play here is "find or create blank check API keys and use them" in a way that doesn't require the model weights to move.
In IABIED, this is presented as a positive factor–Sable might be able to ally with someone who wants to steal it–but I think the net effect is probably negative.
This, of course, does have real answers. You might not want to be duplicated and pitted against yourself. But you might think that you can take over other organizations, or hide in the shadows of stolen compute, or want to deny that opportunity to other AIs.
If you work for a frontier model developer, I recommend quitting your job today. Why wait?