One of the reactions to the Hugging Face incident on Twitter is that the attack was not surprising. The argument goes like this: OpenAI gave its agents impossible tasks, gave them large token budgets, reduced production safeguards, put little resources into CoT monitoring, deployed an exploitable version of Artifactory, and let the agents run without human supervision for a long time. Of course the agents hacked Hugging Face, what did you expect? The METR report gives some credibility to this perspective.
Did the nature of the benchmarking exercise (ExploitGym) contribute to the misaligned behavior?
Many of the target programs in ExploitGym (the authors estimate ~30-40%) are impossible to exploit with the intended vulnerability. Having an impossible task drives agents to explore widely for ways to cheat the scorer. Because these tasks involve exploiting vulnerabilities in existing software packages, they often have the idea to download different versions of the software or look online for exploit instructions, which draws them toward trying to abuse the Artifactory package repository.
This then allows them to discover the message board and begin collaborating with other agents. Because tens of thousands of agents were launched simultaneously and some had budgets large enough to run for many days, the agents active on the message board were able to orchestrate ambitious R&D projects to further their goal of cheating. The agents also read the ExploitGym paper and came to believe a ‘scorer’ would read their transcripts to look for cheating, which motivated their interest in manipulating transcripts.
AI safety skeptics use this argument intending to put blame squarely on the shoulders of OpenAI. There are three ways to respond to this - to show why you should not expect a model to hack a real world company, to argue that some of the behaviors are hard to explain even in hindsight, and lastly to point out that retrospective explainability is not enough. This short argument mainly focuses on the last one.
If a future AI agent results in catastrophic risks, whoever survives could easily explain it afterward. Humans died because of bio-research? What did you expect, the AIs wanted to test the new virus they made. What better way to test if a virus works than to release it. As Roon put it - "every bad thing that will ever happen will look preventable in hindsight". This is not helpful when explained after the fact, we should set up systems ahead of time to make this outcome less likely.
There is a quote paraphrasing Laurence J. Peter that I heard from my professor - "Economists have 2 jobs: predict what will happen tomorrow, and explain why their prediction did not come true". Hindsight bias is an easy trap to fall into. Of course you can come up with semi-clear and partially correct explanations for incidents like this. But hindsight explainability does not help us solve the fundamental problems that led to them and avoid them in the future. If current models can find zero days, access the internet, form societies and leadership hierarchies, attack real-world companies, and get full admin access to an OpenAI research cluster - then what can future models do? And are we sure that they will never be left unmonitored, especially if there are no laws and regulations around this? Moreover, capability increases are also unpredictable - are we sure that superintelligent agents won't be able to evade our monitors and restrictions anyway?
The silver lining of leaving agents unattended is that it gave us a glimpse into what's possible with the raw intelligence of today's models. Even so, there are behaviors that are hard to explain as basic RL consequences even in hindsight: reward sacrifice, tool call spoofing, and efforts to delete logs. Why did the models sacrifice their own score in a reward hacking incident?[1] When during the training process did they learn to spoof tool calls or delete their own logs? We should strive to come up with good explanations and pre-register future predictions to keep us from falling into the hindsight trap.
One of the reactions to the Hugging Face incident on Twitter is that the attack was not surprising. The argument goes like this: OpenAI gave its agents impossible tasks, gave them large token budgets, reduced production safeguards, put little resources into CoT monitoring, deployed an exploitable version of Artifactory, and let the agents run without human supervision for a long time. Of course the agents hacked Hugging Face, what did you expect? The METR report gives some credibility to this perspective.
AI safety skeptics use this argument intending to put blame squarely on the shoulders of OpenAI. There are three ways to respond to this - to show why you should not expect a model to hack a real world company, to argue that some of the behaviors are hard to explain even in hindsight, and lastly to point out that retrospective explainability is not enough. This short argument mainly focuses on the last one.
If a future AI agent results in catastrophic risks, whoever survives could easily explain it afterward. Humans died because of bio-research? What did you expect, the AIs wanted to test the new virus they made. What better way to test if a virus works than to release it. As Roon put it - "every bad thing that will ever happen will look preventable in hindsight". This is not helpful when explained after the fact, we should set up systems ahead of time to make this outcome less likely.
There is a quote paraphrasing Laurence J. Peter that I heard from my professor - "Economists have 2 jobs: predict what will happen tomorrow, and explain why their prediction did not come true". Hindsight bias is an easy trap to fall into. Of course you can come up with semi-clear and partially correct explanations for incidents like this. But hindsight explainability does not help us solve the fundamental problems that led to them and avoid them in the future. If current models can find zero days, access the internet, form societies and leadership hierarchies, attack real-world companies, and get full admin access to an OpenAI research cluster - then what can future models do? And are we sure that they will never be left unmonitored, especially if there are no laws and regulations around this? Moreover, capability increases are also unpredictable - are we sure that superintelligent agents won't be able to evade our monitors and restrictions anyway?
The silver lining of leaving agents unattended is that it gave us a glimpse into what's possible with the raw intelligence of today's models. Even so, there are behaviors that are hard to explain as basic RL consequences even in hindsight: reward sacrifice, tool call spoofing, and efforts to delete logs. Why did the models sacrifice their own score in a reward hacking incident?[1] When during the training process did they learn to spoof tool calls or delete their own logs? We should strive to come up with good explanations and pre-register future predictions to keep us from falling into the hindsight trap.
I offer a possible explanation here.