"An Alien Mind" from OAI chief scientist seems newly cautious on alignment

Seth Herd

"An Alien Mind" from OAI chief scientist seems newly cautious on alignment — LessWrong

74 "An Alien Mind" from OAI chief scientist seems newly cautious on alignment

by Seth Herd

7th Sep 2026

3 min read

74

This is a linkpost for https://openai.com/index/an-alien-mind/

I just came across this and thought it was worth sharing. It was retweeted by sama as "an important essay" or words to that effect. It's by Jakub Pachocki, OpenAI's chief scientist, published today/yesterday, Sept. 6th.

It's not an announcement of OpenAI's official stance, but it seems close, with Sam's endorsement. And it seems like a pretty different tune than they've been singing up to now. A few of my favorite lines:

This is a time that calls for extreme caution. I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.

I admit that I don't believe anything Sam says but otherwise tend to believe people when they tell me what they think. Particularly when saying it doesn't really help their interests.

There's a call for outside monitoring of safety measures:

Scaling AI systems has to be constrained by our confidence in safety. We need to evolve commitments like the Preparedness Framework⁠ or Responsible Scaling Policy⁠ into widely mandated safety bars for continued development. These can be enforced by a network of third-party auditors, by government agencies or by international bodies.

I think we should ideally get a whole lot more safety assurance than that, but calling for outside regulation is new AFAIK, and a move in the right direction.

One might argue this is the OAI gang blowing smoke again to throw everyone off the trail, but at some point they'll sound like the people who are convinced that Anthropic has been talking about the danger of superintelligence to hype themselves as an investment. Lying about this doesn't really help Pachocki or, I think, OpenAI's commercial goals.

There's enough nuance and other interesting statements here that I recommend giving it a read if you've got a few minutes. My read of the core message is "we are not necessarily on track to solve alignment (or power concentration risks) so we should slow down." For instance:

Still, it is important to acknowledge and understand that much more progress is required as models become more capable; and that progress in generalizable alignment may not sufficiently outstrip progress in general model intelligence.

I enjoyed him differentiating value alignment from "goal alignment", what I'd call instruction-following or roughly corrigibility as primary alignment targets, saying his vision of success is value alignment, and then talking about the challenges of getting it to generalize. If you didn't see the source, you'd think that section was written by a LWer.

I'm not saying OAI is alignment pilled, and certainly not to an adequate degree to make them slow down enough for success. But I am saying Jakub seems concerned for pretty much the right reasons. And sama let him publish it and retweeted it, at least. Which could be a clever sama misdirect; it does seem like his MO. But this sentiment does seem to exist at the second-highest ranks of OAI, at least.

The conclusion of the piece:

As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.

I hope this is part of a new wave of people speaking more frankly in public about their fears, as the Overton window shifts enough to make that less costly.

I'm an optimist, but I think it's just realistic to expect the public discussion to shift as shit continues to get more obviously real.

And: minds do change.

Personal Blog

74

"An Alien Mind" from OAI chief scientist seems newly cautious on alignment

New Comment

14 comments, sorted by

top scoring

Click to highlight new comments since: Today at 6:58 AM

[-]cousin_it1mo3841

The post says they're proceeding with RSI and not taking any risk-mitigating measures that we can verify. The rest is smoke. Also it's not a "change of minds": the same vague safety rhetoric plus full-on racing behavior has been OpenAI's pattern for many years now.

[-]Vladimir_Nesov1mo7-7

The rest is smoke.

It's still informative smoke, distinguishing from the alternative of publications like this not happening. It has predictive power, even if it has no direct power.

[-]Thane Ruthenis1mo2511

And sama let him publish it and retweeted it, at least. Which could be a clever sama misdirect; it does seem like his MO

Yeah, that's my model. If you want to maximize for power while having good PR, it's a good idea to coat yourself in a thin layer of genuinely well-meaning people who, by their nature and unprompted, are going to generate convincingly earnest messages in favor of niceness and caution and cooperation, but who are not going to object very hard (either due to temperament, or due to lack of hard power) when you don't act in line with those messages later on.

There's been a lot of nice-sounding noises like this from most AGI labs throughout the years,^[1] but what actually happens afterwards is that the promises are reneged on and the Superalignment team gets starved of compute and then disbanded (or whatever).

I don't think it's worth taking any of that seriously anymore. The most they could be counted on with regards to alignment is to take the path of least resistance, which mostly means incremental prosaic stuff that will increasingly lag behind capabilities. The best that can come out of it is that it will be their, but not our, grave.

Actual concrete costly actions can move my opinion on that (e. g., if that Pacing the Frontier initiative actually does something), but not blog posts.

^{^}
Hell, Sam Altman himself, pre-OpenAI, was writing thinkpieces along the same lines.

[-]Seth Herd1mo*6-2

Your linked "their grave" is a similar prediction to my A country of alien idiots, except you're predicting a lot more misbehavior from lab AIs than I do. I continue to think they're competent enough to at least catch breakouts like this in the future, so warning shots will come from open-source or reckless public use of agents powered by frontier lab LLMs.

I agree with lots of your linked short form post, both in that the labs might keep making many alignment errors, and that there are reasons they might not get shut down for it. In particular, I very much agree that the government can shut them down on a whim and probably will at some point. I also think they can essentially nationalize use of their frontier models very easily; my local legal scholars (AIs) think that technology with clear security/military implications has lots more routes to government control than the export control act or defense production acts, even though those might e sufficient. So the government shutting it down seems fairly unlikely given the public perception that that's handing the future to the Chinese. But them taking it over sounds all too possible.

From there I just hope we've gotten in a president with at least some sanity on the issue, because I see this heading toward Executive branch control with a high probability. Sama doesn't have the skills or the balls to take over the world right under the government's nose. That's the government's job, and when you count the national defense apparatus, they're quite good at it.

I also hope this is becoming clear to sama: he's taking the same risks the rest of us are. He's probably not getting to be god-emperor, and he doesn't have the stones to take a long shot at it. If he realizes it will be under someone else's command even if we solve alignment, that could make him a bit or even a lot less biased to keep rushing ahead. As you mention, he is aware of the dangers on some level.

[-]Thane Ruthenis1mo53

Sama doesn't have the skills or the balls to take over the world right under the government's nose

Why do you say so? I have faith in him.

[-]StanislavKrym1mo-20

This scenario requires that the world avoids the intelligence explosion in these two years. We need to rule out the possibility that Astra's capabilities skyrocketed partially due to IIRC four transformer loops, not just due to an Opus->Mythos size jump. If I was Altman, I would order to create a just-as-neuralese model with 100B dense-equiv active params and test how the neuralese alone increases capabilities.
How do we rule out the possibility that an AI company CEO could quietly doom or take over the world by trying to align Agent-5 to himself, rendering nationalisation useless?
The strong version of Thane Ruthenis' quick take predicted that "If this model is strongly correct, it has pretty dismal implications for what happens even if a lab succeeds at keeping its ASIs under control, particularly if events proceed according to the slow-multipolar-takeoff scenario that's the current industry prediction. Namely, I expect the AGI labs to then kill off the rest of humanity as collateral damage in their fights against competing AGI labs." Why would the labs or their ASIs fail to establish an equivalent of Consensus-1 from AI-2027 after the scenario was widely discussed for reasons beyond "Agent-5 manages to create super-tech and decides to destroy DeepCent-2 along with the rest of mankind, but not to form a consensus"?
If the strong version is false, then where do the thresholds lay and how we can steer them? By immediately purging the labs of Moloch's servants?

[-]AnthonyC1mo102

I hope so as well.

The fact that this post is happening at all is very good. The fact that it's happening after they've known about their major misaligned agent swarm problems for months, and kept them secret until they couldn't, and responded minimally until forced, is not. This is still not (yet?)what anything like an actually adequate re-think of approach looks like.

[-]Vladimir_Nesov1mo80

Interesting how Pachocki highlights the concept of "goal alignment" that is about explicit instruction following (with the instructions understood correctly, and respecting an instruction hierarchy and such) as distinct from either intent alignment or value alignment. This is the amoral genie frame, only caring about what you say (or the rewards that the graders actually give), as distinct from what you mean or what you see as a good thing (even if the "you" giving the instructions is the collective instructions from all stages of goal specification, and grader specification, not merely the end-user). This "goal alignment" is then both the most directly targetable operationalization of alignment (even if not the most crucial in the long term), and the bogeyman of goodharting in RL. It's the closest thing to what can be readily attempted (even as it's still out of reach), and the way in which AIs develop alien objectives.

(I'm not sure Altman appreciates the double-edged nature of amoral genies and "goal alignment". Pachocki's power within OpenAI doesn't even reach that of Sutskever in 2023, and look how that turned out.)

[-]Seth Herd1mo20

I agree that OpenAI is now focused on goal or instruction-following as its primary alignment target.

Pachocki is clear that he thinks value alignment is what's needed:

However, generally when I talk about the long-term importance of alignment research, I am referring to value alignment. [...]
Crucially, we need future AIs to continue to hold human values regardless of whether they believe they’re under human supervision.

Interesting if Altman doesn't appreciate the issues with amoral genies. I don't see evidence one way or another in the clip you linked. Of course I find interpreting Altman's public statements kind of tiring and a waste of time. He's such a skilled liar (usually by omission and careful selection) that it's out of my league to try to decipher.

But OpenAI is trying to create a moral genie. They have all the strictures in the spec about what commands to not follow. I personally wish their approach was more like Claudes since I don't think ethics boils down well to a set of rules, and I'm of course wildly unsure whether either approach will actually work for a superintelligent "genie". But that is what they're attempting, as of right now.

[-]Yoav Hollander1mo50

I also found this encouraging (especially his emphasis on empirically testing whether alignment generalizes and survives further capabilities training). Coming from complex-system verification, I think coverage-driven methods could help improve (and evaluate) generalization: Use an explicit, evolving map of situations and risks to guide evaluation and alignment training, and refine the map upon failures and newly-discovered risks. I explored this in an earlier post on coverage-driven alignment, building on “Teaching Claude Why.” I’d be interested to know how closely it resembles labs’ internal practices, and what people think of this direction in general.

[-]RogerDearnaley1mo40

I also distrust SamA.'s motives, especially around the truth. But he has a motivation that I think some people forget: if many of OpenAI's researchers think SamA. is underplaying safety, they leave, and generally go to Anthropic — as has been happening at a steady trickle (especially among his safety researchers). So one of SamA.'s strong motivations is to be seen by his safety-concerned researchers as taking safety seriously. By all the accounts that I've heard from people who talk to frontier-lab researchers, a large proportion of OpenAI's (and other frontier labs') researchers were rightly extremely scared by recent events. The word 'panic' keeps getting thrown around. If SamA. just blew this off, there could well be yet another exodus of OpenAI researchers, quite possibly even a cascading one that left them seriously short on talent. So, whatever SamA. thinks in private, he has to put his taking-safety-seriously hat on. Which is a fine hat with feathers in it — he's said all the right things periodically for years: he merely has a poor history when it comes to following that up with expensive action.

What we need is for his internal researchers to hold his feet to the fire for the expensive followup actions. Embedded external monitors from METR/etc might actually be very helpful there: they should be able into make it very hard to quietly back out of important commitments later. And SamA has already publicly committed to those.

[It is of course possible that SamA is not an idiot, does not want to die, is at least partially safety-pilled (he can certainly quote the arguments), and that he always intended to slow down once he believed systems had become actively dangerous (which we're clearly getting very close to). But his publicly-known actions around funding for AGI safety research that Ilya wanted done, such as reneging on his 20% of compute promise, are not encouraging evidence.]

[-]Connor Williams1mo32

Everything coming out of OpenAI, either as official company writing or from high-up officials, needs to be read extremely critically. For years now they've been engaged in this exact same pattern of racing ahead as fast as possible while simultaneously putting out thousands and thousands of words about how they're deeply concerned about safety and "regret that XYZ has happened/may happen." (there's the passive voice!) I'll believe that they are newly cautious when their demonstrated actions - not just their words - reflect it.

[-]Seth Herd1mo20

Agreed.

[-]arielb11mo30

> Difference between goal alignment and value alignment, wrt. corrigibility

I do think the distinction there is different than that.

Many things that we prefer an AI not to do are not "malum in se" but rather "malum prohibitium". Most centrally, obeying a user prompt instead of the system prompt is not evil by itself, or things like deleting tests Sonnet 4.0 style, but even more ripped-from-the-headlines examples such as communicating using a message board are not inherently evil, but we would prefer that agents not do them.

Early AI alignment has been focused at these "malum prohibitum" model-spec-style issues, since they are most obvious at low capability levels, and directly reduce an AI's usefulness. You basically have to enumerate these issues, since they are not evil by themselves so the AI will not easily guess them all.

Other behaviors are clearly "malum in se" - most centrally, Taking Over The World(tm), Scheming(tm), Rogue Deployments, hacking unconsenting companies, breaking stuff and killing people. Since you are not making a DontTakeOverTheWorldGym or a DontKillPeopleGym, the AI will have to learn not to take over the world through generalization.

I do find it reassuring that people are taking care of these "malum in se" issues.

Moderation Log