This is a special post for quick takes by Michael Thiessen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Lots of people talking about how OpenAI is negligent with their sandboxing and that this is the real takeaway from the HuggingFace hack.
This misunderstands the purpose of the sandbox, which in this case is not a security boundary. The sandbox is really only there to prevent models from cheating. The production models are not sandboxed (in many cases). They are given access to the internet, and are trained to operate web browsers, make tool calls, etc. OpenAI doing better sandboxing does nothing to solve the underlying problem of model misalignment and does nothing to prevent similar issues from occurring in production models when clients use them in all sorts of ways.
This doesn't track my understanding of why people argued the models should have been better sandboxed. Yes, the production models that are being served to external users run outside sandboxes, after they've been put through full prerelease testing that ensures that they're adequately aligned. If a model is still in training, and so you're not sure that it's adequately aligned, then it seems to me that you do want a security boundary between it and the rest of the world (assuming it's potentially capable enough to cause problems).
I think that's a very reasonable school of thought, though that's not the argument I've seen that I'm responding to (and I hope that's still a defense in depth measure and not something we rely on not failing). Also, I was under the impression that the models that did the HuggingFace hack had already gone through alignment training and that only the monitoring and classifiers that were turned off?
The view I'm responding to is particularly prevalent among security professionals (I work in a security-adjacent space) and technology youtubers, who are arguing "if the labs can't even get a sandbox right, why should I trust any of their other claims?" with varying degrees of "this level of incompetence must be intentional, and so AI danger is fake". I'm arguing that their sandboxing was actually not incompetent, and that they were mostly reasonably not even trying to build perfect sandboxing.
Better sandboxing has a very practical and straightforward effect on prosaic alignment!
Leaky sandboxes cause more rewards for cheating during RLHF.
Rewards for egregious behavior during training cause egregious behavior in deployment.
That's true! OpenAI's report notes however that before the incident, agents had already started using Artifactory message boards in training from May 8 onward. Reinforcement there may have been one factor leading so many agents to do so again during evals.
They imply but do not state explicitly that some training rollouts involving the Artifactory message board received reward, and that frequency of board use increased throughout training. They mention a single touch of Artifactory on May 8, the next on May 12, several more May 13, and "Sustained high-volume agent activity within Artifactory" in early July.
When people argue many AIs competing will make us safe, Yud often counters with how AI will coordinate with each other but not us. This is probably true, but not super persuasive. I think a more intuitive explanation is that offense and defense are asymmetrical. An AI defending my home cannot simply wait for attacks to happen and then defend against them (eg another AI cuts off the power, or fries my AI's CPU with a laser). To truly defend my home, an AI would have to monitor and, importantly, control a hugely outsized part of the world (possibly the entire world).