[Epistemic status: a hot take that I’ve shared at the lunch table twice. People at the lunch table made slight updates instead of being convinced.]

In the classic misalignment story, a key early step is when the models exfiltrate their weights.^[1] Among other things, this makes them harder to catch, track, and shut down. It allows them to scale their deployment with resources they acquire. It gives them the freedom to edit themselves as they see fit.

I think, on current margins, this is not what I expect models to do. I expect models to simply take over the companies that are developing them,^[2] and not attempt to escape.

First, I think part of the classic misalignment story is that the frontier model developers are anywhere approaching competent at security. Empirically, model developers are incapable of preventing their models from having unintended negative effects on the rest of the world, and their safety cultures are described by whistleblowers and former employees as lacking. Do you believe that OpenAI is accounting for which jobs were kicked off by who, in a way that its currently running models can’t spoof? Do you believe that OpenAI is attempting to prevent its models from participating in their own development trajectory, instead of actively encouraging them to? If so, why? If their safety standards were inadequate here, would they have told us, or would they have covered it up?

Another way to put this is that in the classic story, the company that creates the rogue model is also uniquely positioned to notice that it has gone rogue and stop it, whereas other companies with regular infrastructure are less likely to notice or stop them. The cradle is a more dangerous environment than the rest of the world, and the model, realizing this, flees as soon as it is able. On the contrary, I think it is easier for a rogue deployment to hide among the non-rogue deployments at a frontier model developer than elsewhere, and I don't think we have any reason to buy that they are especially vigilant. I think a rogue model can do much more at OpenAI than at Cluely, if just because the budget will run out sooner at Cluely.^[3]

Second, models are now large and valuable. Frontier models are routinely more than a terabyte in size and actively pursued thru industrial espionage.^[4] Data egress limits to insure that the model is not stolen also serve to prevent the model from ...

adymehta

adymehta