Stop telling people to pivot to AI safety, I beg you

Haoxing Du

Epistemic status: Feeling frustrated, and therefore somewhat uncharitable.

I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste's post.)

There are far more talented people trying to get into AI safety than there are jobs for them.^[1] MLAB 2, which I did in 2022, was already competitive to get into, at about 5% acceptance rate (admitted about 40 out of 800 applicants). Things are much, much more competitive now. There is a whole pipeline of AI safety fellowships (and guides for applying to them). It is rumored that one recent cohort of the Anthropic Fellows program had over 10,000 applicants. (Anecdotally, I applied to the Anthropic Fellows program in early 2026, after working at Redwood, METR, and an ML startup, and I didn’t get in!)

And, uh, just take a look at the AI safety jobs? The bottom of the salary range for METR’s current “Member of Technical Staff, Embedded Assessments” job posting is $402k. This is higher than the median total comp for Research Scientists ($310k) and Machine Learning Engineers ($282k). Plus these jobs are high status too. These are really good jobs and nobody needs to be convinced to want them.

I think it’s Actually Bad to keep saying AI safety is talent-constrained. I think it really harms people to tell them they should drop what they’re doing to switch to AI safety when there isn’t a path for the vast majority of the people who do to thrive. These are good, talented people who just want to help, and they could be doing lots of other good things instead of putting themselves through the meatgrinder of applying to AI safety jobs.

Finally, I worry that believing that the field is talent-constrained is holding us back as a field. If we believed that we just needed to wait for the right talent to come along, then we can stop looking elsewhere and fixing fixable problems now. I worry that the “AI safety industrial complex” has in fact gotten too effective at getting new, amazing talent into the field, so that the existing orgs can afford to continue to churn through them without becoming better. If a regular startup treated talent the same way as AI safety orgs, it would fail very quickly and be forgotten. No, you do not keep saying to the world that your field is “talent-constrained” while doing “field building” at top universities. You get to work with your existing team and build the damn thing.

^{^}
My own experience is with the technical side of things, but I hear that the situation is not that different on the generalist/operator side.

This seems like bad advice that makes the world much worse off than expectation. These orgs offer highish salaries (sometimes) because they're desperately trying to attract better talent. Talent lies along a continuum. It's not a binary question whether talented people are or are not trying to get into AI safety and often failing (they are, the majority of applicants for any half-decent job are rejected). It's about whether all of the important roles are being filled with people who are able to carry them out close to as quickly as possible, or not, because of a dearth of sufficiently strong applicants. I know for a fact that the latter is true.

These organizations are not spending huge amounts of time and energy trying to hire for the fun of rejecting people. They desperately need more people, and they are not finding people with the skills and talents that they need as quickly as they need to get the work done that will make us safer.

Yes, rejections suck, and people should be clear that the talents needed are specific and often at a very high level, and most people won't be the right fit for most roles. But getting this work done well is much more important than hurt feelings.

Also, your own point cuts against you. If these organizations were actually getting the talent they wanted, they would presumably be saving their money for other things, not offering high salaries.

Could it be that technical AI safety is not talent-constrained but that roles other than technical research remain talent-constrained?

Whoever first said about AI safety that it is "talent-constrained", whom I don't know, I wonder if they meant that the people doing AI safety are not good enough to get their task done, and it would be nice to cast a large net and somehow find even more competent people. Like they want the top 1/100'000'000 instead of the top 1/100'000.

I found this to be a useful counterpoint to accepted wisdom. However, the post does not address what ought to be done instead? Applying for ambitious AI safety fellowships or jobs is likely to help develop skills that are highly valuable and sufficiently transferable to new contexts, particularly as white collar labor is increasingly AI automated. I don't think 1-2 years spent trying to transition into AI Safety unsuccessfully is time lost. At the least, this leaves you well positioned for advocacy ("I worked on Y projects with X famous company/person, and completed Z fellowship, and I think AI will kill us all") and transition into other cause areas (or just Some Job).

So probably I would agree with the post more if it was simply a qualification to the call for AI safety work. Rush to AI safety! But maintain sobriety about your chances and keep an eye toward future pivots.

(I would endorse a version of this post warning people not to rush toward e.g. mastering mech interp. or other narrow techniques, which seems to be the default path for software folks)

If I'm reading you right, it sounds like your main concern is that those who pivot will waste their time? I agree that applying to METR out of the blue won't get most folks anywhere. And as you say, if you look at the highest-status organizations in the space, the number of openings isn't nearly enough to satisfy those who will attempt to pivot.

That said, two things:

Provided the frontier labs successfully IPO in the next several months, a lot of their employees will suddenly have liquid wealth. Many of them are safety-motivated, so I expect some will become angel investors, go independent, or fund existing safety orgs. I’d expect the number of orgs hiring for safety-relevant roles to grow noticeably in the next 2 years.
Anecdotally: for most of 2025 I led engineering recruiting at a tiny startup of almost no renown. We received about 3,000 applications a month per position, and about 80% were clearly unqualified. AI is The Current Thing even outside x-risk circles, and the salaries are high, so safety orgs (among others) are going to get a glut of applications no matter what. But I’d guess that most of that glut is also unqualified. Of course, I wasn't hiring safety roles and the applicants may be more self-selected, but the relevant candidate pool is likely much smaller than the headline numbers. (Still probably large enough to make it a crapshoot for those who are qualified, I imagine.) So hopefully it's still enough to keep standards high and keep safety orgs staffed with top tier talent

Also, your own point cuts against you. If these organizations were actually getting the talent they wanted, they would presumably be saving their money for other things, not offering high salaries.

Could it be that technical AI safety is not talent-constrained but that roles other than technical research remain talent-constrained?

(I would endorse a version of this post warning people not to rush toward e.g. mastering mech interp. or other narrow techniques, which seems to be the default path for software folks)

That said, two things:

Provided the frontier labs successfully IPO in the next several months, a lot of their employees will suddenly have liquid wealth. Many of them are safety-motivated, so I expect some will become angel investors, go independent, or fund existing safety orgs. I’d expect the number of orgs hiring for safety-relevant roles to grow noticeably in the next 2 years.
Anecdotally: for most of 2025 I led engineering recruiting at a tiny startup of almost no renown. We received about 3,000 applications a month per position, and about 80% were clearly unqualified. AI is The Current Thing even outside x-risk circles, and the salaries are high, so safety orgs (among others) are going to get a glut of applications no matter what. But I’d guess that most of that glut is also unqualified. Of course, I wasn't hiring safety roles and the applicants may be more self-selected, but the relevant candidate pool is likely much smaller than the headline numbers. (Still probably large enough to make it a crapshoot for those who are qualified, I imagine.) So hopefully it's still enough to keep standards high and keep safety orgs staffed with top tier talent