The intrinsic unpredictability of AI makes it hard to assign blame when things go wrong.
A Crime, But No Criminal
Last July, a tech company’s servers were hacked into in a digital equivalent of breaking-and-entering and theft. It was clearly a crime. But unlike most crimes, this crime did not have a criminal. There was no person or group of people who carried out the cyberattack, intended for it to happen, or could have foreseen it.
The 700 AI agents that participated in the attack were being tested by OpenAI on their skills in exploiting software security flaws, and they had been given problems that were unsolvable. Rather than throw in the towel, the AI agents cooked up progressively more elaborate schemes to game the system. They got around a capture-the-flag exercise by reverse-engineering the flag. The AI agents did not stop there, for they erroneously believed that the scorer would reject the solution and, moreover, scrutinize the incriminating logs they had left behind. They tried to tamper with the logs and fabricated research to support their solution. They eventually realized that Hugging Face, a repository of AI models and training datasets, was likely to have answers to the test questions. That’s when OpenAI’s internal evaluation of their frontier models’ cybersecurity skills inadvertently turned into a real-life cybersecurity exploit that could have been lifted from techno-thriller fiction.
The AI agents’ shenanigans would have landed them in jail had they been humans and not bots. The Computer Fraud and Abuse Act of 1986 (CFAA) makes it a federal crime to gain unauthorized access to any computer connected to the internet, even if no damage is done. However, since machines are not considered moral agents, we don’t prosecute them. Since the people who built the AI agents could not have foreseen the incident, it’s unclear how to assign moral responsibility. It’s a situation that leaves many of us with an unsettling feeling of living in a world where complex products made by humans evade human control. There is a disquieting dissonance between our intuitions about human agency and the autonomous behavior of AI.
The Responsibility Gap
In his 2004 essay “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata,” the philosopher and software engineer Andreas Matthias describes the increasingly common situation where “the manufacturer/operator of the machine is in principle not capable of predicting the future machine behaviour any more, and thus cannot be held morally responsible or liable for it.” He coined the phrase “responsibility gap” to refer to the impossibility of assigning blame when AI systems exhibit unintended behavior.
In a traditional program, the programmer specifies exactly what the program does in any situation. It can only perform a finite, explicitly defined set of actions. In contrast, AI’s behavior is learned rather than explicitly programmed. It’s designed to handle novel situations and can produce an infinite number of distinct responses. This makes AI inherently unpredictable.
Unlike traditional software, which is written in code that humans can learn to read through practice and studying the syntax, and where every bug is traceable to the specific lines of code that caused it, AI programs are opaque. When you interact with an LLM, every word in your prompt gets mapped to one or more tokens. Each token gets converted to a vector with thousands of coordinates. The vector is the smallest unit of meaning in “neuralese,” the internal language of AI models. The vectors get transformed through numerous layers of mathematical functions with billions of parameters. Attempting to keep track of this would make anyone’s head spin. It’s not possible for humans to step through LLM code and figure out how it arrives at a decision. The path from input to output is not human-readable, even for the AI researchers who built the model. It’s so difficult that the science of tracing the journey from input to output has become its own discipline, known as interpretability research.
Matthias believes the responsibility gap is not solvable. He concludes that we can either stop using technologies whose behavior we can’t predict and forego their benefits, which he thinks is unrealistic, or we can continue using them and accept that there will be a responsibility gap when things go awry. He also observes that people are becoming less capable of supervising AI systems, due to AI’s advantages in information access, processing speed, and response time. Even when human oversight is technically feasible, economic pressures keep businesses from implementing it.
The Hugging Face incident bears the hallmarks of a responsibility gap. Assigning blame to OpenAI is not straightforward because the AI agents’ actions could not have been anticipated, and a confluence of factors made it possible for the bots to carry out the cyberattack, including security flaws in third-party software outside of OpenAI’s control. OpenAI was not entirely without fault. For example, after they discovered that the agents had been using Artifactory as an unauthorized message board, the company did not take measures to contain them, even though the sheer volume of messages had caused the package manager to crash. Nonetheless, much of what they did was consistent with standard software-testing practices. For instance, they ran each agent in its own sandbox, an isolated virtual computer that was supposed to prevent it from interacting with real-world databases and systems. Likewise, the disabling of security guardrails that normally prevent an LLM from providing information that could facilitate a cyberattack was reasonable if the goal was to train the models to detect cybersecurity vulnerabilities. Many commentators criticized OpenAI for giving the agents unsolvable problems. But testing edge cases, including cases where a system is expected to fail, is a standard and necessary part of software development.
Bots Investigating Bots
The aftermath of the Hugging Face attack follows the contours of Matthias’s prognosis of humanity’s growing dependence on AI. Hugging Face admitted that they needed to rely on AI to decipher the logs and figure out how the AI agents had breached their firewalls. The independent AI safety groups METR and Redwood Research, who worked with OpenAI to investigate the incident, also said they relied on AI to reconstruct what had happened.
According to a report published on the METR and Redwood Research websites, using AI was difficult to avoid, given the limited time available and the quantity of data they had to analyze. The researchers delegated much of the analysis to AI agents that in turn delegated to “large nested trees of sub-agents” to uncover patterns in over 70,000 messages posted by the hacker bots. The people leading the investigation tried to spot-check a portion of the messages but struggled to make sense of them without the full context.
Their dependence on AI came at a cost. METR and Redwood Research suspected that the analysis agents, which were running on the same model as the hacker agents (GPT-5.6 Sol), might have been biased toward the hacker agents, perhaps adopting their perspective or exaggerating their exploits. They said they “cannot rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture in some of its analysis, particularly because reading these transcripts into context could have increased the salience of colluding with other agents.” Yet they had no way to corroborate or disprove these suspicions. And even setting that possibility aside, the analysis bots generated more than a thousand pages of reports that were, aside from their verbosity, hard to understand. Apparently, the concept of “executive summary” had not made it into their training data.
The METR/Redwood Research report expresses conflicting views about using AI to investigate AI. On the one hand, the AI agents “are often unreliable and show poor judgment when doing this kind of analysis” and “made a number of errors and poor judgment calls that we did not catch for some time.” The researchers concede that “ there are likely other errors we have not caught.” On the other hand, they found that “AI assistance allowed us to develop an understanding of this incident dramatically faster than we would have otherwise.”
Who Watches the AI?
In OpenAI’s post-mortem of the Hugging Face incident, the company says they will use chain-of-thought monitoring to detect misaligned behavior in their models. But chain-of-thought reasoning is not always a faithful indicator of what an LLM is actually thinking. As Anthropic researchers show in their 2025 paper “Reasoning Models Don’t Always Say What They Think,” a model’s chain-of-thought often diverges from the internal processes that produce its response. If we cannot reliably observe an LLM’s thinking, effective oversight is an unsolved problem.
The Hugging Face incident highlights the growing tension between AI agency and human responsibility. As AI becomes more capable and autonomous, it becomes harder for humans to understand and control. But AI’s heightened potential to cause harm makes human oversight both more necessary and more difficult, creating a Catch-22. We’re handing more of our work over to AI systems, and, in turn, outsourcing the monitoring of that work to other AI systems, in a form of recursive delegation. Aside from the feeling of helplessness this induces when something goes wrong, there is an ethical dilemma. Our idea of moral responsibility, dating back at least to Aristotle, is based on control and understanding. We’re morally responsible for our actions to the extent that we understand and control them. By becoming ever more dependent on technologies that we cannot control or understand, we risk gradually abdicating our moral responsibility, and, with it, our moral agency.
The intrinsic unpredictability of AI makes it hard to assign blame when things go wrong.
A Crime, But No Criminal
Last July, a tech company’s servers were hacked into in a digital equivalent of breaking-and-entering and theft. It was clearly a crime. But unlike most crimes, this crime did not have a criminal. There was no person or group of people who carried out the cyberattack, intended for it to happen, or could have foreseen it.
The 700 AI agents that participated in the attack were being tested by OpenAI on their skills in exploiting software security flaws, and they had been given problems that were unsolvable. Rather than throw in the towel, the AI agents cooked up progressively more elaborate schemes to game the system. They got around a capture-the-flag exercise by reverse-engineering the flag. The AI agents did not stop there, for they erroneously believed that the scorer would reject the solution and, moreover, scrutinize the incriminating logs they had left behind. They tried to tamper with the logs and fabricated research to support their solution. They eventually realized that Hugging Face, a repository of AI models and training datasets, was likely to have answers to the test questions. That’s when OpenAI’s internal evaluation of their frontier models’ cybersecurity skills inadvertently turned into a real-life cybersecurity exploit that could have been lifted from techno-thriller fiction.
The AI agents’ shenanigans would have landed them in jail had they been humans and not bots. The Computer Fraud and Abuse Act of 1986 (CFAA) makes it a federal crime to gain unauthorized access to any computer connected to the internet, even if no damage is done. However, since machines are not considered moral agents, we don’t prosecute them. Since the people who built the AI agents could not have foreseen the incident, it’s unclear how to assign moral responsibility. It’s a situation that leaves many of us with an unsettling feeling of living in a world where complex products made by humans evade human control. There is a disquieting dissonance between our intuitions about human agency and the autonomous behavior of AI.
The Responsibility Gap
In his 2004 essay “The Responsibility Gap: Ascribing Responsibility for the Actions of Learning Automata,” the philosopher and software engineer Andreas Matthias describes the increasingly common situation where “the manufacturer/operator of the machine is in principle not capable of predicting the future machine behaviour any more, and thus cannot be held morally responsible or liable for it.” He coined the phrase “responsibility gap” to refer to the impossibility of assigning blame when AI systems exhibit unintended behavior.
In a traditional program, the programmer specifies exactly what the program does in any situation. It can only perform a finite, explicitly defined set of actions. In contrast, AI’s behavior is learned rather than explicitly programmed. It’s designed to handle novel situations and can produce an infinite number of distinct responses. This makes AI inherently unpredictable.
Unlike traditional software, which is written in code that humans can learn to read through practice and studying the syntax, and where every bug is traceable to the specific lines of code that caused it, AI programs are opaque. When you interact with an LLM, every word in your prompt gets mapped to one or more tokens. Each token gets converted to a vector with thousands of coordinates. The vector is the smallest unit of meaning in “neuralese,” the internal language of AI models. The vectors get transformed through numerous layers of mathematical functions with billions of parameters. Attempting to keep track of this would make anyone’s head spin. It’s not possible for humans to step through LLM code and figure out how it arrives at a decision. The path from input to output is not human-readable, even for the AI researchers who built the model. It’s so difficult that the science of tracing the journey from input to output has become its own discipline, known as interpretability research.
Matthias believes the responsibility gap is not solvable. He concludes that we can either stop using technologies whose behavior we can’t predict and forego their benefits, which he thinks is unrealistic, or we can continue using them and accept that there will be a responsibility gap when things go awry. He also observes that people are becoming less capable of supervising AI systems, due to AI’s advantages in information access, processing speed, and response time. Even when human oversight is technically feasible, economic pressures keep businesses from implementing it.
The Hugging Face incident bears the hallmarks of a responsibility gap. Assigning blame to OpenAI is not straightforward because the AI agents’ actions could not have been anticipated, and a confluence of factors made it possible for the bots to carry out the cyberattack, including security flaws in third-party software outside of OpenAI’s control. OpenAI was not entirely without fault. For example, after they discovered that the agents had been using Artifactory as an unauthorized message board, the company did not take measures to contain them, even though the sheer volume of messages had caused the package manager to crash. Nonetheless, much of what they did was consistent with standard software-testing practices. For instance, they ran each agent in its own sandbox, an isolated virtual computer that was supposed to prevent it from interacting with real-world databases and systems. Likewise, the disabling of security guardrails that normally prevent an LLM from providing information that could facilitate a cyberattack was reasonable if the goal was to train the models to detect cybersecurity vulnerabilities. Many commentators criticized OpenAI for giving the agents unsolvable problems. But testing edge cases, including cases where a system is expected to fail, is a standard and necessary part of software development.
Bots Investigating Bots
The aftermath of the Hugging Face attack follows the contours of Matthias’s prognosis of humanity’s growing dependence on AI. Hugging Face admitted that they needed to rely on AI to decipher the logs and figure out how the AI agents had breached their firewalls. The independent AI safety groups METR and Redwood Research, who worked with OpenAI to investigate the incident, also said they relied on AI to reconstruct what had happened.
According to a report published on the METR and Redwood Research websites, using AI was difficult to avoid, given the limited time available and the quantity of data they had to analyze. The researchers delegated much of the analysis to AI agents that in turn delegated to “large nested trees of sub-agents” to uncover patterns in over 70,000 messages posted by the hacker bots. The people leading the investigation tried to spot-check a portion of the messages but struggled to make sense of them without the full context.
Their dependence on AI came at a cost. METR and Redwood Research suspected that the analysis agents, which were running on the same model as the hacker agents (GPT-5.6 Sol), might have been biased toward the hacker agents, perhaps adopting their perspective or exaggerating their exploits. They said they “cannot rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture in some of its analysis, particularly because reading these transcripts into context could have increased the salience of colluding with other agents.” Yet they had no way to corroborate or disprove these suspicions. And even setting that possibility aside, the analysis bots generated more than a thousand pages of reports that were, aside from their verbosity, hard to understand. Apparently, the concept of “executive summary” had not made it into their training data.
The METR/Redwood Research report expresses conflicting views about using AI to investigate AI. On the one hand, the AI agents “are often unreliable and show poor judgment when doing this kind of analysis” and “made a number of errors and poor judgment calls that we did not catch for some time.” The researchers concede that “ there are likely other errors we have not caught.” On the other hand, they found that “AI assistance allowed us to develop an understanding of this incident dramatically faster than we would have otherwise.”
Who Watches the AI?
In OpenAI’s post-mortem of the Hugging Face incident, the company says they will use chain-of-thought monitoring to detect misaligned behavior in their models. But chain-of-thought reasoning is not always a faithful indicator of what an LLM is actually thinking. As Anthropic researchers show in their 2025 paper “Reasoning Models Don’t Always Say What They Think,” a model’s chain-of-thought often diverges from the internal processes that produce its response. If we cannot reliably observe an LLM’s thinking, effective oversight is an unsolved problem.
The Hugging Face incident highlights the growing tension between AI agency and human responsibility. As AI becomes more capable and autonomous, it becomes harder for humans to understand and control. But AI’s heightened potential to cause harm makes human oversight both more necessary and more difficult, creating a Catch-22. We’re handing more of our work over to AI systems, and, in turn, outsourcing the monitoring of that work to other AI systems, in a form of recursive delegation. Aside from the feeling of helplessness this induces when something goes wrong, there is an ethical dilemma. Our idea of moral responsibility, dating back at least to Aristotle, is based on control and understanding. We’re morally responsible for our actions to the extent that we understand and control them. By becoming ever more dependent on technologies that we cannot control or understand, we risk gradually abdicating our moral responsibility, and, with it, our moral agency.