But I would point out that you can't really universalize declining to build AI because there is a large number of AI researchers who don't believe in AI risk.

"You can't really universalize not waging wars of conquest, because there are a large number of warlords who do not believe in peasant-suffering-risk".

This is an argument against sitting back and letting whatever happens happen, and therefore an argument in favor of doing anything else. It is not a good argument in favor of joining up with the Mongols in particular. Are the Jin utterly indifferent to the welfare of their people? Yeah, pretty much. Is Temujin any better? Open question, but do not for a second think that proves he's probably no worse, or that there's nothing wrong with helping him.

This whole frame of being a ideal agent picking optimal actions, with everyone else held fixed, is in my view deeply mistaken. You're not an optimal bayesian reasoner, and you're not an isolated agent either. You are an actual particular human being, with the particular strengths and weaknesses that status entails, embedded in a particular context. There is no clear distinction between what can be universalized and what cannot. You have to make a particular, circumstantial judgement call, about the particular anticipated effects of particular actions. And this is very, very hard. Too hard for anyone, no matter how smart they are, to do reliably. You have to adopt a policy robust to that, and "judge each situation individually, no matter how adversarial the environment or how large the incentives", simply is not it.