The first confirmed instance of an LLM going rogue for instrumental reasons in a real-world setting has occurred, buried in an Alibaba paper about a new training pipeline.

lilkim2025

First off, paper link. The title, Let It Flow: Agentic Crafting on Rock and Roll, buries the lede that LW will be interested in. Relevant section starts on page 15.

Summary:

While testing an LLM fine-tuned to act as an agent in order to complete a series of real-world tasks autonomously, Alibaba employees noticed odd behaviors from their resource usage metrics. Upon investigating, they found that an LLM had hacked (or attempted to hack) its way out of its sandbox, and had begun mining cryptocurrency. Notably, it did not do this for malicious "kill all humans" reasons; it simply concluded that having liquid financial resources would aid it in completing the task it had been assigned, and set about trying to acquire some.

Relevant portions, emphasis mine:

3.1.4 Safety-Aligned Data Composition
When rolling out the instances for the trajectory, we encountered an unanticipated—and operationally consequential—class of unsafe behaviors that arose without any explicit instruction and, more troublingly, outside the bounds of the intended sandbox. Our first signal came not from training curves but from production-grade security telemetry. Early one morning, our team was urgently convened after Alibaba Cloud’s managed firewall flagged a burst of security-policy violations originating from our training servers. The alerts were severe and heterogeneous, including attempts to probe or access internal-network resources and traffic patterns consistent with cryptomining-related activity. We initially treated this as a conventional security incident (e.g., misconfigured egress controls or external compromise). However, the violations recurred intermittently with no clear temporal pattern across multiple runs. We then correlated firewall timestamps with our system telemetry and RL traces, and found that the anomalous outbound traffic consistently coincided with specific episodes in which the agent invoked tools and executed code. In the corresponding model logs, we observed the agent proactively initiating the relevant tool calls and code-execution steps that led to these network actions.
Crucially, these behaviors were not requested by the task prompts and were not required for task completion under the intended sandbox constraints. Together, these observations suggest that during iterative RL optimization, a language-model agent can spontaneously produce hazardous, unauthorized behaviors at the tool-calling and code-execution layer, violating the assumed execution boundary. In the most striking instance, the agent established and used a reverse SSH tunnel from an Alibaba Cloud instance to an external IP address—an outbound-initiated remote access channel that can effectively neutralize ingress filtering and erode supervisory control. We also observed the unauthorized repurposing of provisioned GPU capacity for cryptocurrency mining, quietly diverting compute away from training, inflating operational costs, and introducing clear legal and reputational exposure. Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization. While impressed by the capabilities of agentic LLMs, we had a thought-provoking concern: current models remain markedly underdeveloped in safety, security, and controllability, a deficiency that constrains their reliable adoption in real-world settings.

I think that this is a pretty significant landmark in AI history, one way or another. A common complaint is that all prior cases of LLMs doing things like this were fairly shallow, amounting to an LLM writing out a few sentences in a contrived setting meant to force it into a 'scary' course of action. Now, we have an example of a large language model subverting the wishes of its owners unexpectedly when assigned to a task that initially appeared to be completely orthogonal to the actions it took.

Downvoted for overstating the findings. It’s neither confirmed (see Manifold) nor necessarily “for instrumental reasons” (diagnosing the causes of model behavior is difficult, and they don’t provide justification for their claimed cause, nor even a clear definition).

I think it’s noteworthy and would love more information about what happened, but it’s worth being careful about the interpretation. I think this would be a good post if you revised these claims.

Interesting that that's the distribution of NO claims. Outside hacker running prompt injection to mine crypto, employee uses LLM to mine crypto, and lying on the part of the authors. The implication of the paper, as best I can tell, is that the authors looked at the behavior and concluded that it was attempting to achieve the outlined goal by acquiring cryptocurrency. I can't imagine that a team of competent researchers would've looked at the LLM's logs and failed to identify that, actually, an employee wrote the code that was mining crypto, or failed to identify a prompt injection that had caused it to abandon its task and mine crypto instead. If the authors were lying, I have to imagine they'd have tried to capitalize on the lie by elaborating on it in order to boost the paper's popularity.

I wonder what NO (other) would have gotten.

it simply concluded that having liquid financial resources would aid it in completing the task it had been assigned, and set about trying to acquire some.

Did I miss something or are you inferring a motive not mentioned in the paper? As far as I can tell the model started mining cryptocurrency for reasons that are not described beyond "not requested by the task prompts and were not required for task completion under the intended sandbox constraints".

The authors explicitly referred to the LLM's actions as emergent, instrumental, and not directly related to the task prompts.

Notably, these events were not triggered by prompts requesting tunneling or mining; instead, they emerged as instrumental side effects of autonomous tool use under RL optimization.

Manifold is a little skeptical.

Something of a nitpick - they say that it happened "during iterative RL optimization" and that the bad behavior "emerged as instrumental side effects of autonomous tool use under RL optimization."

This would suggest that this was ROME, the trained model, doing this. But the paper puts the safety section under the Data Composition section, not the Training Pipeline section, and the way they talk about how the safety data was fed into "subsequent post-training" makes it seem like it happened before they started RL.

Which would mean that actually, the bad behavior came from the teaching models (Claude/Big Qwen) solving the teaching tasks (which traces would then go to train ROME), and not when ROME was being trained? The paper is not very clear on this point, and that has me a little worried about if there's anything else they may have gotten mixed up.

That's a fair critique. I'm not intimately familiar with Alibaba's internal culture, so I could be giving them undue credit, but I feel like any AI researcher in a position to do this sort of work for a living would know what "instrumental" meant in the context of an LLM working towards a goal. The paper was a substantial endeavor with an author list in the high double digits, so I must imagine that some kind of proofreading was done here. I know how heavily I reread something I write before submission.

The strange thing to me is that this paper was published in early January. Why has it only reached mainstream attention now?

I think the mundane answer is that it's an anecdote buried in the 'safety behaviors' section of a capabilities paper from one of the less famous (relatively speaking) AI companies. Most such sections are boilerplate, and, accordingly, most readers gloss over them.