come work on dangerous capability mitigations at Anthropic

[-]Zach Stein-Perlman2mo2614

I believe that improving Anthropic's ability to prevent misuse via API decreases P(doom) much less than e.g. working on risks from misalignment. There are lots of factors pointing in this direction: misuse poses less risk than misalignment, misuse via API is much more on track to be solved (or the solution is much more on track to be cheap) than misalignment, and solving misuse-via-API only prevents a small fraction of misuse risk since there's also misuse-via-weights (from open-weights models, which are some 6-12 months behind the frontier, or stolen models, which is important since by default powerful models will be stolen). Edit: also note that most misuse-via-API risk comes from non-Anthropic models, and preventing misuse at Anthropic likely won't transfer to other companies, whereas work on misalignment at Anthropic—not to mention work outside of a frontier AI company—might well improve safety at several companies.

I think most people who could do this could do something even better, such as averting risks from misalignment.

[-]Dave Orr2mo8-2

I 100% endorse working on alignment and agree that it's super important.

We do think that misuse mitigations at Anthropic can help improve things generally though race-to-the-top dynamics, and I can attest that while at GDM I was meaningfully influenced by things that Anthropic did.

[-]Neel Nanda2mo30

Will Anthropic continue to publicly share research behind its safeguards, and say what kinds of safeguards they're using? If so, I think there's clear ways to spread this work to other labs

[-]Dave Orr2mo50

We certainly plan to!

[-]ryan_greenblatt2mo40

From my perspective, the main case for working on safeguards is transfer to misalignment risk. (Both in terms of transferring of the methods and the policy/politics that this work maybe helps with transferring.)

I don't have a strong view about how much marginal work on the safeguards team transfers to misalignment risk, but I think the case for substantial transfer (e.g. comparable to that of working on the alignment science team doing somewhat thoughtful work) is reasonable, especially if the person is explicitly focused on transferring to misalignment or is working on developing directly misalignment relevant safeguards.

By default, I'd guess that people interested in mitigating misalignment risk should by default (as in, in the absence of a strong comparative adavantage or particularly exciting relevant project) prefer working on the alignment science team at Anthropic (just comparing options at Anthropic) or something else that is more directly targeted at misalignment risk.

[-]Neel Nanda2mo170

I worked somewhat with Dave while he was at GDM, and can happily endorse him as good competent, sincerely caring about x-risk and a great person to work with! (Very sad to see you go 😢 )

[-]Dave Orr2mo74

I'm new to Anthropic myself, leading the Safeguards team. I joined a few weeks ago, inspired by the mission and the opportunity. I'm really worried about the world as AI continues to get more powerful, and I couldn't pass up the chance to help if I could.

I was previously at GDM working on similar problems (miss you all!), but the chance to help drive the safety agenda at Anthropic as we transition to a new scarier world felt too important to miss.

So far everything at Anthropic except my new commute is amazing, but most of all the feeling of the mission is intense and awesome. Also the level of transparency inside the company is astounding for a company this size (not that it's big compared to many others).

Obviously there could be some honeymoon effect here but honestly I'm having a lot of fun, and I honestly think Safeguards (along with Alignment Science) makes a real different in safety for the world.

LESSWRONG
LW

LESSWRONG
LW

31

come work on dangerous capability mitigations at Anthropic

31

31