Thanks for writing this - as someone inside Anthropic I think it's a fair description of the epistemic state, or at least I've had similar thoughts. I also think it applies fractally - when you're on a team or a project, working on that thing seems much more important than once you've moved away from it. I think this is one reason why CEOs / higher bosses are right about strategy much more than you'd expect given their lack of detailed knowledge, and yes suggests you should quit jobs more than feels natural from the inside.
What I'm interested in is whether there are ways to notice and quantify this. I believe there may be a way to present clear statements of, for example, what kinds of beliefs tend to be rewarded relative to the quality of their argumentation, which ones tend to drive action, or to change minds and so on. I think this is important for internal dynamics and perhaps especially for analysing the dynamics of large groups of digital agents, since there's less or zero hidden information (i.e. for lab employees its hard to distinguish a bias from genuine inside information). I'd be very interested in any proposals you have for such analysis.
So what should actually happen? First, on the margin, more people should quit.
I agree with this in general. However, quitting is only near the top of my levels of tzdaka what a lab employee should do. Cf. https://www.lesswrong.com/posts/pY76hHbdD4jduX3YH/wah-wah-wah-wah-do-more-ai-r-and-d?commentId=PcYYAX5jefDCrfTq3
Two levels up:
Stay and stop doing capabilities research and do a bunch of internal conversations to peers and leadership and groups to understand how they are thinking about things and to help them unwind internal contradictions (net good)
I think that marginal people leaving is good, but it also leaves a dumpster fire burning. We would want to better understand how the dumpster fire works, and help the dumpster fire put itself out over time.
And there is a lot of truth in the perspective. But also you are microdosing.
So what should actually happen? First, on the margin, more people should quit. Fortunately, this started happening between when I began writing this in August and when I posted it in October.Too large a part of AI safety, alignment, and macrostrategy is inside, and too few people are quitting, in particular in Anthropic's case. Possibly you should quit: not because of arguments that working at labs is morally wrong, but because you’re in an altered state that makes the tradeoffs hard to see. If quitting feels too drastic, at least make the decision from a less dosed state: take a break from the organization and its social web, and see how the tradeoffs look from there.
Second, not everyone who could join should join. Again, too much of AI safety is already inside the labs. Every marginal hire converts one more potentially sober observer into a dosed one.
When is the tradeoff most beneficial for those who quit? Whatever insights in AI safety, alignment and macrostrategy[1] one comes up with, I don't understand how they will be implemented by anyone except for the lab[2] or the USG.[3] Therefore, you should've articulated that a theory of change for quitting Anthropic should include a channel through which one plans to affect the lab's decisions,[4] like Coxon's viral tweet, Kokotajlo's AI-2027 scenario or joining METR/Redwood/other evaluators.
As for safety or alignment, I would separate them into theory-like insights (e.g. Kokotajlo's philosophy-related argument, A basic case for differentially accelerating conceptual reasoning capabilities or attempts to connect alignment with the Trauma Logic Chain), attempts to bring the world's attention to issues with existing systems (e.g. Greenblatt's post "Current AIs seem pretty misaligned to me", Yudkowsky's "The Talker Does Not Control The Doer") and extrapolation of trends (e.g. Anthropic's ECI or the time horizon of CoTless math) which would require us to take drastic measures once a level is reached.[5] The latter insights are especially important since they cannot be obtained in any way except for the model being released or studied by an embedded evaluator.
- ^
I consider mentioning macrostrategy to be a mistake, especially since AI-2040's Plan A and its big list of appendices like the Alignment Roadmap in Plan C or worse are already published.
- ^
For example, METR's postmortem of GPT-5.6 Sol's non-eval had them claim: "As training and iteration continues, we need to ensure the models aren’t just learning to be more successful at evading the monitoring system. This is impossible to validate in a traditional pre-deployment evaluation paradigm, as it requires deep access to internal systems. (italics mine -- S.K.)," which means that labs are to have embedded evaluators and avoid issues like OpenAI Firing Three Safety Researchers.
- ^
For example, AI-2040's Alignment Roadmap claimed that "The main way of (preventing handoff -- S.K.) would be to escalate to Plans A or B, which would (typically) require substantial government involvement, which may or may not be feasible. A key priority during the other phases will often be to attempt to escalate the level of political will."
- ^
Unfortunately, it is hard to predict in advance whether a certain strategy will cause the quitter to become as efficient as Coxon or Kokotajlo.
- ^
Of course, such data should be measured for every new model in order to spot problems when a trend breaks as happened with the looped transformers' CoTless math abilities since GPT-6 Astra.
I'll commit a rationalist sin in that I didn't read the post but I will comment: the title is hilarious and the most LW spirit blog post title I've come across.
My go-to conceptual metaphor of the epistemic state of AI lab employees is microdosing: being in a chronically mildly altered state of mind. I think understanding this would be broadly helpful for people both inside and outside.
The effect isn’t that insiders hold crazy beliefs, but a combination of vigorous debate inside a shared attention field, and a portfolio of individually reasonable beliefs that, taken together, lean in a direction convenient for the organization.
Epistemic advantages and disadvantages
Many people argue that low doses of psychedelics let you see some things more clearly - and I tend to agree: psychedelics can often help some people see some things more clearly. At the same time, I think microdosing usually doesn’t improve people’s epistemics overall, even when it gives them otherwise hard to access insights. You are not completely sober.
Being inside a lab has real advantages: you know more about the frontier of capabilities, may have access to the latest AIs before the public, read internal Slack and blogs, and sense the vibes. If you’re outside and an insider tells you what they observe, pay attention.
These information and insight asymmetries are tricky. I don’t want to advocate “discount insiders,” any more than “ignore what people with psychedelic experience say.”
What I want is …interpret insiders as microdosing. Be curious but cautious. And if you’re inside: notice that your mind may be somewhat altered. Not like tripping (people inside are highly functional and would pass any simple test), but in subtler ways.
Mechanisms and conditions
Stretching the metaphor a bit: what is the substance? Mostly the “spirit of the organization,” in the sense of the organizational egregore or superagent. A simple insight from thinking about hierarchical agency is that superagents can be understood as having goals and preferences, including convergent instrumental goals. One common instrumental goal is synchronizing beliefs and goals across subagents.
In my estimate, Anthropic has the strongest “spirit” of the frontier labs, which is why it’s in the title: it makes the cleanest case study. But I could have written about other companies, Constellation, the MIRI cluster, some startups…
I think the substance is mostly delivered through entirely ordinary channels.
Selection effects: who applies, who gets an offer, who stays, who leaves or gets pushed out. Simple, but powerful.
Social gradients: everyday subtle messaging in the social web: what gets warm engagement at lunch, what gets Slack reactions, what gives you access to resources or powerful people. People naturally synchronize with those around them.
Internal information: internal Slack, memos, memes, and the sense of having important secret information. Daniel Ellsberg famously described this when warning Kissinger about what security clearances would do to him: first, the new access makes you feel foolish about everything you had confidently written before; then, within a couple of years, it makes you unable to learn from people who lack the clearances. You stop hearing them. Whatever they say, you’re thinking about what they would say if they knew what you know. Frontier labs run on a somewhat similar asymmetry: I’d expect insiders to feel they have privileged information about model capabilities, growth rates, and internal forecasts.
"But we argue internally all the time"
In my experience, insiders often object: “But we argue internally all the time.” Internal debates are healthy, people push back on leadership, everyone has their own opinions.
This is true in some ways: it is instrumentally necessary for the organization to have a lot of exploration. On the other hand, I would expect a lot of synchronization one meta-level up: where is attention allocated, which questions are important, which framings are simply in the water, and so on. The default I'd expect is vigorous object-level debate inside a synchronized attention field. This type of attentional redistribution and frame shifts is fairly difficult to notice from inside.
Collective epistemic shifts
If this model is roughly right, you shouldn’t expect insiders to hold any individually crazy beliefs. You should expect a portfolio of individually defensible probability weights whose conjunction is suspiciously aligned with the organization’s instrumental goals.
To continue with Anthropic as the example: alignment is important (so the org’s existence is justified), and hard (so you should defer to the people working on it), and somewhat on track (so it’s scary but not suicidal to keep racing), and the race with China is real and must be won by the good guys (so it’s necessary to keep racing).
Each of these is a sensible position that many people outside also hold. But I’d expect the joint distribution inside to be shifted toward the part of the space where the beliefs align with what the superagent wants.
What it isn’t
People sometimes unhelpfully collapse this to different problems:
“Following the leaders”. Organizational spirits often do get personified in the form of the organization leaders, but I don't think this is very accurate or helpful; I don't think the leader-organization alignment to be one-directional, leaders do often serve the organizations in a deep sense, rather than leading the orgs.
Anything with the word “cult”. What's characteristic of cults is a decrease in agency of the members - the cultists become in some sense more of “tools” or “organs” of the egregore. But this is not the case - people inside the labs often become more powerful, gain status and wealth and attention.
What to do about it
Probably worth emphasizing: nothing I've described depends on anyone acting in bad faith, lying about their experience and similar.
So, is this even a problem?
In my view, yes. I'm scared of the epistemic situation, I'm worried about the increasing number of people being inside, and I'm sad when people join and I notice shifts in how they talk and think.
So, what should happen?
Can this be counteracted by introspection and self-audits? I doubt so: my guess is when honest insiders inspect individual beliefs, they consider the arguments, the counterarguments, and conclude they are being reasonable. My model of the dominant frame among Anthropic higher-ups is “it's scary, but we are the responsible adults in the room”. There are various types of evil people outside, confused people outside, people who like to write blog posts and chase LW karma, but it's kind of pointless and will not save us.
And there is a lot of truth in the perspective. But also you are microdosing.
So what should actually happen? First, on the margin, more people should quit. Fortunately, this started happening between when I began writing this in August and when I posted it in October.
Too large a part of AI safety, alignment, and macrostrategy is inside, and too few people are quitting, in particular in Anthropic's case. Possibly you should quit: not because of arguments that working at labs is morally wrong, but because you’re in an altered state that makes the tradeoffs hard to see. If quitting feels too drastic, at least make the decision from a less dosed state: take a break from the organization and its social web, and see how the tradeoffs look from there.
Second, not everyone who could join should join. Again, too much of AI safety is already inside the labs. Every marginal hire converts one more potentially sober observer into a dosed one.
Third, for those who stay: discount outsiders less. I think you cannot undo the salience field by introspection, but you can partially counteract the Ellsberg effect.
Is anyone “really sober”?
Here comes another obvious objection: is anyone really sober?
I don’t claim that. I’m obviously also influenced by various incentive fields, including LessWrong, academia, and the flavor of AI safety thinking usually done at nonprofits. My approach is closer to hedging incentives than to trying to live in an incentive-free landscape: no matter what I say, someone will be upset, so the easy option is to speak relatively freely and not worry about who.
I’m not claiming you should aim for an incentive vacuum or a meme-free space, or move to a cave and study poetry.
What I do claim is that the spirits inside are strong, the effects depend on dose, and some combination of sobriety, sanity, and situational awareness is becoming something like a scarce civilizational resource.
Thanks to multiple lab employees for discussions about this, and also to Claude Opus 5.5. for help.