Zach Stein-Perlman's Shortform

Zach Stein-Perlman

This is a special post for quick takes by Zach Stein-Perlman. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Last week's Hugging Face breach was accidentally caused by OpenAI testing models' cyber capabilities. From the OpenAI blogpost:

Last week, Hugging Face disclosed a new kind of security incident⁠ after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠ of cyber capabilities. . . .
This incident occurred during an internal evaluation. . . . The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.
While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.

In other words, the models were supposed to solve a cyber test, but they found the solution by hacking their sandbox to get internet access and then hacking someone on the internet who already had the solution. If I understand correctly.

Covered in Axios, Fortune (but no info beyond the OpenAI blogpost).

Context: this comes one day after a related post (see Zvi discussion).

Note: this is some legible evidence of humanity's current inability to reliably steer AI systems. I expect current safeguards suffice to prevent incidents like this when someone (provider or operator) cares about implementing those safeguards (some safeguards weren't enabled here). But I'm worried about AI risk because I think it's quite possible that future AI systems still won't be reliably steerable, they'll seek long-term power (for instrumental reasons), and our safeguards won't suffice to stop them.

I think I was directionally correct here https://www.lesswrong.com/posts/eqrtdyphQFefc2ox9/hastings-s-shortform?commentId=WtLhZ9shcbZ8Ki38g when I supposed that it would be increasingly possible to track "secret" large scale training runs purely by their escape attempts into the public internet, although I was wrong in that I expected it to look like models quietly but detectably surfing the web for information useful for scoring high, not models blatantly and extremely loudly hacking in pursuit of useful information.

1DW113mo

Scattered thoughts: 1) We are fortunate that RL-driven misalignment is appearing at relatively low levels of capability where the models are not smart enough to act strategically yet. 2) I would imagine OpenAI's attempts to prevent reward-hacking and make models follow the spec, while clearly wildly insufficient, exceed the effort put in by every other lab except Anthropic and maybe GDM. It also seems clear to me that this internal model is commercially unviable to release since it will likely take illegal/disproportionate actions after deployment, too. Therefore I predict that other labs will face commercial issues once their models reach the level of this internal model since they will be too misaligned to release and they will have to start investing in preventing this behaviour if they want their models to be usable.

2Dan Braun3mo

If this AI were embodied, and the answer sheet were sitting on a desk in another room in the same building, I don’t think it’s a big stretch to imagine it physically harming humans to get access. That scenario would have landed very differently with the rest of the world. Godspeed to those trying to help people recognise that.