Something that I've continually found surprising about AI Safety as opposed to working at Nvidia / Apple is that frontier labs (now potentially multi-trillion dollar valuation for profit tech companies) are not treated like you'd treat any other multi-trillion dollar valuation for profit tech companies by default.

I've made this point before as it comes up most often with Anthropic:

I’m confused about much of the discussion on this post being about whether Anthropic has done “net good”.

The post is very specifically a deep dive into the fact that Anthropic, like any other company, should not have their leadership’s statements taken at face value. IMO this is a completely unrealistic way to treat companies in any field, and it’s a bit frustrating to see the rationalist presumption of good faith extended over and over by default in contexts where it’s so incredibly exploitable.

Again this is not a specific criticism of Anthropic, if a new lab starts tomorrow promising to build Safe SuperIntelligence for example, we should not assume that we can trust all their leadership’s statements until they’ve mislead people publicly a few times and someone has a deep dive comprehensively documenting it.

but wanted to note that I think it applies across labs, for example, OpenAI (and Anthropic) both released a batch of misalignment incidents today. Sydney correctly pointed out that this is the third time they've dumped these incidents on a Friday:

Kai (OpenAI) then responded:

This is obviously intentional, and is standard practice for large companies with comms teams who want to bury bad news:

While it's technically true that "there could be innocent good faith explanations" and "corporations aren't a monolith and have diverse views internally", these are completely unreasonable defaults to have about large tech companies.

It often seems extremely taboo to assume that anyone is operating out of anything other than good faith or some other variant of mistake theory. This does not feel like a serious way to approach dealing with large tech companies and IMO it is constantly leveraged against people in AI Safety when they interact with frontier labs.

The Ten People On The Inside likely should prioritize things that capabilities teams aren't going to do anyway and people considering joining labs with similar motivations should likely think about whether they're sure they're going to be able to do work that wouldn't otherwise be done for capabilities reasons. The thing that's top of mind here is reward hacking, which is increasingly becoming a capabilities problem.^[1] To the extent research is done here, it seems pretty important to avoid the failure mode of "repeatedly iterate on environments and observable behavior"^[2] which is the kind of thing I would expect by default for capabilities reasons.

1. ^{^}

   A good example of research on reward hacking that I don't think would've necessarily gotten done "by default for capabilities reasons" is inoculation prompting and similar investigations

2. ^{^}

   This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training

I’ve seen the sentiment a few times that we don’t need solutions to the issues raised in Automating Alignment Is Harder Than You Think^[1] , i.e. something like scalable oversight / weak-to-strong generalization. Has anyone made this case before? I’ve heard it often from people inside (and even outside) of labs, and my impression is that it’s load bearing for research prioritization with respect to which research directions to pursue and how to allocate researcher time.

1. ^{^}

   Or similar issues raised in https://joecarlsmith.com/2025/04/30/can-we-safely-automate-alignment-research/#5-evaluation-failures or https://openai.com/index/introducing-superalignment/