the CEOs of AI companies have been filtered to not be people who get that
I'd like to go on a bit of a tangent since I don't recall seeing this line of thinking here on LW: are we sure that this is the case, that they think that what they are doing is reasonably safe? This seems to me to be stranger than the case where they know it's not safe, and perhaps not safe at all. What I mean is that they surely are very motivated and they have strong incentives in pursuing AGI, but they don't strike me as... stupid/incompetent?
Maybe they think that this is for the ... (read more)
There is also an efficiency component to the energy expenditure of making social bids. Not only it can take someone more energy to make the same (perceived) bid, but the same bid can have different effects based on social skillfullness and a bunch of other things. This can be seen as a parameter in the dynamic equilibrium that can produce the feedback loop you mention.
Some counterpoints.
I'm not sure the economics really allow for long pauses, you can't really pause indefinitely and then move on once you've done your safety homework, but let's set it aside for a moment.
Suppose that OpenAI and Ant are in front, and they commit to pausing frontier training. The best response from tier 2 labs could be to catch the (perceived) frontier and only then pace. But where exactly is it? You'd likely get to a situation where a multitude of players are clustered around it, all of them quite certain to be a bit behind.
At that point I ... (read more)