Manhattan Project was quite a unique (at least the most well-known) case when a group of famous scientists succeeded to persuade the government that particular technology is a very big threat if created by the wrong group of people. Suppose somehow the group of famous scientists succeed to persuade the US government that AGI is an existential risk. Or, (maybe that can be simpler to explain to everyone), is that even aligned AGI, but aligned with the interests and values of the wrong group of people, is not exactly what you want. Now the question - would this decrease the AGI existential risk, or not?

I can see the following pros and contras:

Pro 1: significantly more focus on alignment research.
Pro 2: Talking about the "wrong group of people" danger, I think a democratic government is a safer bet than for example a private company. 
Pro 3: Most of the big players in AI are US companies. Now there is a danger of the "AI arms race" between them, which is a potential threat due to the rush and risky decisions. US government can potentially monitor IT companies' progress which will give additional protection from risky decisions in the AI race.

Contra 1: Potentially "AI arms race" can start between countries (US-China?). Would it be worse than the current competition between companies?

Which are other pros and contras? What is the net effect, in your opinion? If the net effect is obviously beneficial, who would be the scientists so famous that they can actually persuade the US government as Einstein and Scillard did in 1939? 

New Answer
Ask Related Question
New Comment

1 Answers sorted by

I'm not convinced that the government wouldn't be the riskier approach. Their incentives often don't really lie with the stated missions of the agencies in charge. The official purposes of government are more aligned with us, but the actual purposes are not aligned in many cases. For profit corporations, on the other hand, are forced by the profit motive to have substantially more inherent alignment (though it can still be pretty low.). Governments are likely to mandate solutions that won't or can't work.

That said, the corporations are only aligned at all with humanity by virtue of being full of humans, and relying on human customers.

In general, when it isn't clear what government should do, it is usually better that it doesn't act at all (though not in every case.).

That said, the actual Manhattan project was almost completely separate from outsiders, and the government could very well try out a whole bunch of alignment strategies in general, while releasing those that are likely to work for further scrutiny at the later stages. This would be especially useful for slow takeoff scenarios.

Many would argue that an AI arms race is already developing between US companies and China.

(Note: I am quite skeptical of near future AI danger. I clearly believe in slow or no takeoff.)