This is a special post for quick takes by Thomas Kwa. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Anthropic will have better alignment that cause its models to incur less x-risk at a given capability level than other labs
However, Anthropic will cause much more direct x-risk than other labs, because Anthropic is ruthless and believes that the game theory means they need Claude Mythos 7.5 to cause 10% or 20% x-risk in order to prevent other companies from causing even more x-risk
If they severely underestimate risks, this could turn into 50%+ x-risk
This seems like exactly the danger to worry about. It's a perfect recipe for badly underestimating the risks of RSI.
The motivation to underestimate p(misalignment doom) within Anthropic and particularly for the founders is quite strong. (Same goes for other orgs obv.)
Combine that with the complexity of the situation, which gives plenty of ways to think about the question, allowing one's motivations to help choose comfortable angles - each legitimate as far as it goes (because full answers aren't available).
Then add Anthropic as an epistemic environment. Double-counting effects produce strong social/intellectual affiliation biases even in humans who are trying to be rational.
This is exactly where motivated reasoning effects are strongest in experiments.
I wrote about this and reviewed the empirical science of these effects in Motivated reasoning, confirmation bias, and AI risk theory but I didn't specifically address orgs. They do fit my analysis of groups, though.
This seems worth a full post but I don't know how quickly I can do it...
I feel compelled to add: antipathy seems quite likely to amplify the in-group bias. This is why I think yelling at Anthropic is such a bad idea, even if you're right about what you're trying to convey.
The motivation to underestimate p(misalignment doom) within Anthropic and particularly for the founders is quite strong. (Same goes for other orgs obv.)
Yes. Even leaving aside financial motivations, lab employees and founders are heavily selected to be outliers in at least two different ways:
They are far more likely to believe that true superhuman intelligence is possible in the next few years than the average person.
They are unusually likely to believe—given elementary knowledge of how AIs are constructed—that building a superintelligence has a high... (read more)
This models the period when any sufficiently capable AI is able to take over the world but none can secure the world. In the second model I relax this assumption to something more realistic, where a misaligned AI has power equal to that of their company adjusted by some factor