Lots of people talking about how OpenAI is negligent with their sandboxing and that this is the real takeaway from the HuggingFace hack.
This misunderstands the purpose of the sandbox, which in this case is not a security boundary. The sandbox is really only there to prevent models from cheating. The production models are not sandboxed (in many cases). They are given access to the internet, and are trained to operate web browsers, make tool calls, etc. OpenAI doing better sandboxing does nothing to solve the underlying problem of model misalignment and does nothing to prevent similar issues from occurring in production models when clients use them in all sorts of ways.
Better sandboxing has a very practical and straightforward effect on prosaic alignment!
Leaky sandboxes cause more rewards for cheating during RLHF.
Rewards for egregious behavior during training cause egregious behavior in deployment.
2cubefox
The Huggingface attack did occur during evaluation, not during training.
That's true! OpenAI's report notes however that before the incident, agents had already started using Artifactory message boards in training from May 8 onward. Reinforcement there may have been one factor leading so many agents to do so again during evals.
They imply but do not state explicitly that some training rollouts involving the Artifactory message board received reward, and that frequency of board use increased throughout training. They mention a single touch of Artifactory on May 8, the next on May 12, several more May 13, and "Sustained high-volum... (read more)