I think preventing self-exfiltration is important, and I've argued for a defense against it here. In this case I think it's unlikely for structural reasons described in this post. As Buck states:
The main point I want to make here is that the following responsibilities will almost surely be hosted on logically different computers, and probably on different physical machines:
- The inference server that serves the LLM API
- The scaffold server, which maintains the state of an agent loop; this is responsible for repeatedly calling the inference server, parsing the LLM response, and executing actions with the tools the agent has access to.
- The execution server, on which the agent can run code and see its results.
The o1 system card describes a similar architecture. Labs generally go to great lengths to prevent exfiltration of model weights on the inference server, including e.g. bandwidth egress controls. So it's much harder for a model to self-exfiltrate than for it to break out of its sandbox. E.g. Sam Marks recently posted that similar incidents happened at Anthropic, but it involved escalating privileges and accessing internet in the RL sandbox (i.e. the execution server), but not hacking into other parts of Anthropic (which includes exfiltrating the weights).
Separately, you stated:
There were calls for more transparency, but I don’t think anyone made this demand. Because nobody made this demand, the incident is being treated as over.
I don't think this is true. I think Clement Delangue and many others have requested:
Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.
Which would provide evidence of an exfiltration attack. We'll probably find out more in the technical report that OpenAI has committed to providing.
I always assumed that the hardware-accelerated servers running inference are separate from the servers running evals (connected only via an API channel), and that the inference servers are much more well-secured than the eval servers (making exfiltration unlikely). After all, if their model weights were exfiltrated or stolen, they could be used or reverse-engineered by OpenAI's competitors, which would have a major financial impact on OpenAI, possibly more than from one of their models committing a felony. But my assumption that the servers are separate is only a guess, and I agree that OpenAI, and all other AI labs, should be much more transparent than they currently are.
OpenAI’s AI went rogue and escaped. OpenAI didn’t notice this for days.
For all we know, the AI could still be out there. We need to demand that OpenAI demonstrate that the AI didn’t make a copy of itself that’s running on someone else’s computer somewhere else with no one being any the wiser.
We need to demand this every time an AI escapes the sandbox. AIs have tried to
“exfiltrate” themselves (i.e. their “weights”) in previous experiments many times. It’s a natural and obvious question to ask.
I’m embarrassed that I didn’t say this immediately (although I came close). Why didn’t I? Well, it doesn’t seem all that likely. And I didn’t want to seem “alarmist.” I didn’t want to seem ignorant.
But guess what? We have every right to demand this! It doesn’t matter how likely we think it is.
There were calls for more transparency, but I don’t think anyone made this demand. Because nobody made this demand, the incident is being treated as over.
This is a dangerous precedent. We need an information ecosystem that doesn’t treat “eh, I’m pretty sure it’s OK” as acceptable and “hey, but what if it’s not” as paranoid.
AI needs to adopt a security mindset. Other safety-critical industries demand failure rates like one in a million, and demand that companies produce detailed, rigorous safety cases to that effect.
AI companies can’t do that in full generality, so they shouldn’t be building these AI systems at all.
But they can provide as much evidence as possible to convince independent experts that there is not in fact a rogue AI that is still out there. This is a super reasonable, common sense ask that should not be objectionable. Let’s treat it that way.
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.