There are some conditions under which the correct thing is to whistleblow and leave, rather than keep trying to make improvements on the margin. For instance, if your job becomes "get good benchmarks on alignment" while ignoring clear signs of actual misalignment or hidden reasoning, or if you become aware of a coverup of a warning shot, then staying there means complicity in existential risk.
Recent developments underscore that anyone "working on safety/alignment" at OpenAI is at best disempowered to reduce existential risk, and at worst culpable for increasing it; and the best thing they can do right now is whistleblow. Furthermore, MATS should loudly refuse to work with OpenAI mentors. (Which is why I commented this here despite it being not a direct response to your post.)
This is not an endorsement of other labs. I find Anthropic's public statements on their alignment plan and their level of caution to be inadequate, and I think that their broken promise not to push the frontier forward has caused other companies to act more recklessly. I think that the Alphabet governance of DeepMind undermines what safety culture it had (and while we're talking about how to leave, I strongly support TurnTrout's actions as a model). And we all know how much worse the lesser labs are.
But some people in this community still say with a straight face that working at OpenAI on alignment could be helpful, and today is a great occasion to proclaim how clearly that has been falsified. (Not least because the majority of x-risk-pilled members of OpenAI have ended up leaving the company, individually or en masse, for reasons that I suspect are undisclosed because of legal threats.)
1
I think most AI safety researchers are unhelpful, but "warning shots" is a terrible argument for telling them to quit. Interestingly, "warning shots" have also been used as an argument against engaging in political advocacy, countered for similar reasons as yours. Warning shots are something we should focus on preparing a response to, if needed, not try to make more likely (directly or through intentional inaction).
The main issues with safety research for me are externalities:
- Capabilities: "safety" gets fuzzily defined to include just making the systems more powerful.
- Profit: making an AI system more aligned, even in the best case, is essentially making it more controllable, which increases its market value, which fuels race dynamics.
- Legibility: safety problems exist on a continuum of obvious to hidden. Fixing the obvious problems first is a recipe for getting bitten hard by the hidden problems later. It's also what market pressures incentivize.
Some lines of safety research, like agent foundations, sidestep these, at the cost of being hard to justify in terms of short term returns or even theory of change. If the economic landscape were different, such as by an enforced pause with strict conditions for advancement (externally verified safety proofs and the like), then I can imagine meaningful safety work happening at scale.
As of right now, the core issue is power. The AI labs have too much of it. You're either helping them get more, pushing back, or irrelevant. Pausing isn't just to "buy time," it's to change the conditions in which AI exists.
4I am obviously biased as a member of OpenAI's alignment team, but the following is a position I've held for a very long time: you should strongly suspect any argument of the form "you should make the situation worse in the present so it can be better in the future". People that try to do this usually succeed in the former but not in the latter.
If you have the skills to be a safety researcher in frontier lab then there are many great options of employment for you. If are not feeling that your work matters and you're making a positive impact, that's a good reason to look for other options. But if you feel that you are making a positive impact in terms of making AI models safer today, but somehow it would is better if they were less safe, then that's highly unlikely to work out. It is also extremely risky, since indeed we cannot control how harmful the next "warning shot" would be.
I realize it does not look like this from the outside, but at OpenAI it does not feel at all that safety or alignment people are disempowered. My sense is that the "warning shots" have been heard loud and clear by people across all teams in the company. But I will not try to defend this claim or argue about it here. I do not expect people from outside to take our word for it, but hope and expect that we will show improvement with deeds rather than words.
I am also not claiming that OpenAI or other frontier labs are the best place to have positive impact as a safety researcher. There is no generic answer here that applies to each person. I would just urge people to choose a position where they are making measurable positive impact in the near term (e.g. O(1)years) horizon, since the future beyond that is extremely uncertain.
1
To distill the more relevant part of my other comment: I agree with you that "worse is better" is a bad category of argument, but there's a better argument in this direction.
I believe that a significant fraction of "prosaic safety work" at the frontier labs has had the effect of hiding misalignment in publicly released versions rather than making the models more robustly aligned.
For instance, I expect that the prosaic alignment improvements from Sydney to the final GPT-4 were surface-level only (as evidenced by later failures like GPT-4o), and if the alignment researchers had refused to play whack-a-mole at the time, then commercial progress would have been properly halted.
3I think the hypothesis is worth seriously considering. I have more complicated thoughts I'm trying to write up at the moment in a long-form post.
One crude way to reason about the benefits of prosaic alignment would be to study Anthropic, which ranks at the top in terms of resource spend on prosaic alignment, and compare this to OpenAI, which has approximately the same level of capabilities with its strongest models but spends much less (possibly zero?) on prosaic alignment.
Comparing Ant vs OAI, I make the following observations
- Despite a larger investment in alignment, both Anthropic and OpenAI had "models escaping the sandbox" incidents recently. (It's true that OpenAI's incident resulted in much more severe consequences. This is maybe some evidence that Claude behaved in a relatively more aligned way than the internal-HPIM GPT.)
- Ryan Greenblatt does not seem to distinguish heavily between Claude and GPT in his discussion of misalignment in recent AIs and the impression I get is that both are similarly misaligned.
- I think this is consistent with what other third parties report in terms of alignment evals, e.g. METR's frontier risk report, and (a quick eyeball of) this report by Transluce on concerning behaviour in mental health contexts
- More generally, both labs have a tendency to announce that "our models are the most aligned to date" and then fail to anticipate ways in which the public finds their models to be misaligned. I take this as evidence that "alignment audits" are substantially lagging behind novel ways in which misalignment emerges from training.
- Separately, it seems plausible to me that eval awareness and general situational awareness has sufficiently contaminated both models that the alignment audits are no longer good proxy measures of alignment
Generally, while Anthropic has some wins, this does not seem like an order-of-magnitude difference vs OpenAI. A notable exception is that Anthropic's model APIs seem substantially more jailbreak-resistant than OpenAI's, e.g. in terms of resisting and refusing universal jailbreaks.
Note that I've selected evidence above which seems maximally characteristic of "prosaic alignment" in particular.
- If we expand the scope to "prosaic safety" and even more generally "security mindset" then I think the evidence (and calculus) would look different.
- Certain kinds of prosaic safety, e.g. cybersecurity, seem much more likely to be doing real good and seem to have lower downside risk of concealing misalignment in the next generation of models
I'll also note that "prosaic safety is net bad" is sufficient but not necessary for "should safety researchers quite labs", i.e. if prosaic safety is doing some good, but doing something else could be even better, then safety researchers might want to quit anyway, similar to Joe Benton leaving Anthropic and joining METR
1The new name doesn’t do the job. I think a less-defensive version of the addendum should be moved to the top of the post, or significant changes made to the introduction clarifying that you’re only addressing one narrow concern.
The problem as I see it is that the ‘surge of support’ for people leaving labs is not mostly founded on this argument, and instead on the others which you have flagged as out of scope (which is reasonable, but should be signaled more strongly earlier).
The current title (where you added ‘re warning shots’) could be read as ‘In light of recent warning shots, should lab employees leave?’ Indeed this, and other similarly misleading readings, are more natural than the reading you seem to intend, which is more like ‘there’s this one particular warning shot argument I see sometimes that doesn’t go through’ (a point I locally agree with you on).
1To attempt a slightly finer-grained distinction between different kinds of work, here are some examples of work that seems fairly robustly positive to me:
- Ensuring transcripts are logged / backed up somewhere secure
- Interpretability
- Ensuring that model training keeps CoT monitorable
- Advocating for access for third parties to do risk assessment
- Model organisms
- (maybe, depending on details?) Async monitoring
OTOH, shallow alignment training and many parts of the AI control agenda do seem more fraught (or at the very least have potentially serious downsides as well as upsides).
1Even if true, the nonprofit funding influx might help us do very compute-intensive experiments outside frontier AI companies!
1
I've recently heard a surge of support for an old argument: AI safety researchers should not work at frontier AI companies because this reduces the likelihood of non-lethal warning shots, and we need warning shots to build support for an AI pause/slow-down. This argument has several components:
I think this argument has some merit. I expect that RLHF++ will probably be insufficient to align TED-AI circa-2028 and such systems will be terrifically difficult to monitor or control. However, I think there are also significant weaknesses to this argument. Here are some countervailing points to consider:
Overall, I am cautiously optimistic about working on certain types of safety research at frontier AI companies, particularly if a coordinated AI slowdown (e.g., Plan A) occurs. I nevertheless feel highly uncertain about the "Alignment MVP" strategy in light of recent containment failures and alignment results. I think this merits serious consideration.
Addendum: I won't rehash the extensive debate over whether AI "safety" researchers working at frontier AI companies are causing harm via other channels, such as:
These are legitimate concerns, but largely irrelevant to the topic of this post.
Disclosure: I'm the CEO of MATS, which trains AI safety researchers, including with mentors at frontier AI companies, and a meaningful fraction of our alumni end up working at those companies. If the argument I'm responding to is right, a good chunk of what MATS has done might be counterproductive, so I have an obvious incentive to find it wrong. I've tried to steelman it anyway and I'd appreciate feedback.