Irrespective of everything else, I think the kerfuffle over Anthropic/DOD shows that half-measures are tactically inferior to principled stands.

If Anthropic had come out of the gate with "our product is not a weapon, and we will not pursue any military contracts", they would have vastly more options right now, and much better PR. Instead, they leaned heavily towards the neoconservative position on wars like the one in Eastern Europe, happily working with the U.S. military industrial complex there. Likewise, Amodei's statements about China indicate support for military use in a hypothetical conflict against the PRC. Their own statements begin with some variant of "we are enthusiastic about working with the military".

"I do not want to build weapons" has a lot more public support and legal optionality behind it than "I want a veto against the elected government's monopoly on force", which is the message that the U.S. government and the overwhelming majority of Anthropic's critics appear to have taken from their statements and actions thus far. Whether or not this is their actual position, the fact that many people sincerely believe that this is their position^[1] was an entirely preventable outcome.

I should note that this is not a political position that I am taking - maybe you believe that U.S. interventions A and B are just but conflict C isn't. I certainly do. But once you have built something that you identify to the public and to the government as a powerful, dangerous weapon, a binary switch flips, and everyone with an interest in determining who is allowed to control powerful, dangerous weapons has an interest in you. There's no such thing as a halfway crook; the benefits of being a pacifist^[2] and the benefits of being a military contractor are mutually exclusive.

Also, in the long run, assuming you accept the notion that the endgame here is some variety of "build God", being willing to participate in killing people in any capacity will vastly decrease the number of people who trust you enough to let you do it. If the "I would never hurt anyone, for any reason" people are on the verge of building God, then even if you don't share their values, you might not be as willing to risk a nuclear war in order to stop them.

1. ^{^}

   Some people allege insincerity, but as many of Anthropic's strongest supporters have pointed out, this fight is incredibly expensive and inconvenient for the U.S. government. They would not be fighting it if they thought it was avoidable.

2. ^{^}

   And they exist! A sincere commitment to never involving oneself in matters of war is something many great researchers have successfully made, and it insulates you from all of the power struggles that take place within that domain. There are benefits to being not-a-threat and not-a-prize.