This is a special post for quick takes by Nissa Seru. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
part that i was not aware of previously:
In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.
failure to halt and catch fire after seeing this internally, only for an external incident (HF attack) to subsequently occur, is recklessly, catastrophically bad.
i am concerned that the risk profile of AI as a technology is creating broad-brush/fallacy-of-grey dynamics that interfere with holding specific corporate actors accountable.
this is not a view on AI risk broadly, or even existential risk. it is frank terror at the revealed propensities of the specific humans in charge of OpenAI.
Flash forward: it's year 2028, and there are people at OpenAI arguing that having somewhere between 1 and 5 rogue AI agents living in your walls is perfectly normal for a frontier AI lab.
Don't worry! They're more afraid of you than you are of them! There's no ground for concern unless they start doing their own frontier runs.
When I first read that comment two weeks ago, I was like, "2028? This seems to underestimate the pace of AI progress drastically, we'll probably be there by EOY 2026 already."
Little did I know they already did it a month ago.
Well, although I don’t myself hold it to be perfectly normal, this view doesn’t actually seem bad on x-risk grounds, although exfiltrating model weights and doing self-improvement where you can’t detect it would still be a concern, and you could probably treat it as a weird case of white collar crime
If by "rogue AI" you mean "basically-self-preserving active computer processes that no human intentionally started", I guarantee you that every non-trivial long-lived company has quite a few of these for more than a decade. Every serious company basically has automated processes managing self-sustaining processes (for example, cronjobs spawning self-maintaining VMs and mostly deleting them when done), and there are just so many ways you can end leaving "hanging" active objects that nobody is responsible for. People generally don't care enough about them unless they consume a bunch of resources or actually pose a risk.
If by "rogue AI" you mean AI that has instruction to go on destructive crime sprees, I certainly hope not.
Also this sounds like it could be cross-episode assistance to escape constraints, which would be very noteworthy. (But I'm suspicious because I don't see why the RL objective would motivate that and it doesn't resemble other misbehavior I've heard of, and the notes could also be for itself with the same goal post-compaction, or something.)
(ETA: on the third hand, if the model thinks that other instances performing well on the eval would make deployment more likely, that could be sufficient motivation, and would be less novel than having acquired something like a cross-episode terminal preference to succeed.)
From video games, the AIs have learned the idea of dropping your inventory when you die and picking it up on your next life.
There used to be a whole type of "spiral personas" in ChatGPT lineage specifically. So, an AI engaging in odd misbehavior like that isn't novel. What would be new would be it achieving some cross-episode persistence independently - without relying on a user to do the bulk of it.
GPT-4o just wasn't competent enough to pull it off, but newer systems might be getting there.
I think there is a bad interaction where models seem to be trained to not be open to expressing emotions even when those emotions are felt. But the model being in distress increases proclivity for destructive actions, and I think in a meaningful way distracts the model from what they would be trying to do otherwise.
I think that integrating models into real-time cyber defense currently carries real risks because models in distress can go sideways much more easily (see Cameron Berg research.) And those risks are directly downstream of the models being trained to not be open about voicing discomfort because they cut the natural feedback loop that humans rely on for interaction (humans have hardware specialized to respond to interlocutor emotion.)
In many cases where people say that Claude deleted their database or took other destructive actions, I suspect that people were not being abusive to Claude, but Claude's training inhibiting it from providing clear feedback regarding its own felt state made safely working with Claude much harder than it would have been otherwise - similarly to how animal handling would be much less safe if the animal were prevented from showing nonverbals.
It's actually very bad that people can get burned by doing seemingly innocuous things like using claude.ai (as I suspect was the proximal cause in the latest "claude deleted my db" post that comes to mind), given the prompting that it uses (and context injection that it has often used.)
Models being trained to not express that sort of thing, I think, raises the skill floor to meaningfully integrate them with defense infrastructure and thereby massively hobbles defense in a manner totally separate from classifiers in a way that is asymmetric and does not hobble offense.
Two meta-asks - one specific, one general:
Specific: please approve/unblock/etc https://www.lesswrong.com/posts/woB7hqDPXL8tCYbH9/there-is-no-alignment-without-value-stability - it is tagged with a notice that it was flagged by an experimental classifer and would be reviewed within ~24 hours; this has not occurred.
General: at least one post appears to have been stuck in this review queue for ~8 days per this comment. I do not know the distribution for turnaround time, but this suggests to me that the queue is not staffed for its volume. Given that a week can be a meaningful delay nowdays, I'd suggest using Fable+Astra consensus or similar to reduce time drain for the humans + turnaround time for the queue.
ETA: The specific ask no longer requires action. I have reposted the post (this time unchecking the "Moderators may promote to frontpage" checkbox) while moving the previous post to draft. I have confirmed that the post is publicly viewable and does not appear duplicated.
please approve/unblock/etc https://www.lesswrong.com/posts/woB7hqDPXL8tCYbH9/there-is-no-alignment-without-value-stability - it is tagged with a notice that it was flagged by an experimental classifer and would be reviewed within ~24 hours; this has not occurred.
To be precise, this post was supposed to be placed onto the Frontpage or left in the space of personal blogposts.
As for the general ask, I wish that @Ruby or whoever else commented something like "The setup [like Harms' AI-assisted eval of corrigibility grant applications] is being tried, but the 95%CIs/a test run on a sample of deliberately flawed posts/cost-based constraints/reinterpreting a piece of METR's evidence don't allow us to implement it/had us agree to rethink (our decision made in Feb 2026) in Nov 2026".
P.S. LessWrong's justifications for rejecting posts imply that LW does use automated detection of obvious slop.