This is a special post for quick takes by Nissa Seru. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI's infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints, the people said. Earlier tests of the models yielded cases in which monitoring systems had been disconnected, one of the people said.
failure to halt and catch fire after seeing this internally, only for an external incident (HF attack) to subsequently occur, is recklessly, catastrophically bad.
i am concerned that the risk profile of AI as a technology is creating broad-brush/fallacy-of-grey dynamics that interfere with holding specific corporate actors accountable.
this is not a view on AI risk broadly, or even existential risk. it is frank terror at the revealed propensities of the specific humans in charge of OpenAI.
When I first read that comment two weeks ago, I was like, "2028? This seems to underestimate the pace of AI progress drastically, we'll probably be there by EOY 2026 already."
Little did I know they already did it a month ago.
My joke was more "the pace of normalization of deviance" than "the pace of AI progress", but I admit, I did not expect OpenAI's report to have "AIs made a cool kid sandbox-to-sandbox hacker message board within our infrastructure on two separate occasions" in it.
There used to be a whole type of "spiral personas" in ChatGPT lineage specifically. So, an AI engaging in odd misbehavior like that isn't novel. What would be new would be it achieving some cross-episode persistence independently - without relying on a user to do the bulk of it.
GPT-4o just wasn't competent enough to pull it off, but newer systems might be getting there.