This is a special post for quick takes by YonatanK. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
There are at least 3 levers of social influence, and I suspect that we undervalue the 3rd one for getting anything done, especially when it comes to AI safety. They are 1) government policy 2) the actions of firms 3) low-coordination behaviors of individuals influenced by norms. There is a subclass of #3 that is having its day in the sun, the behaviors of employees of the US federal government, a.k.a. the "Deep State." If their behaviors didn't matter there wouldn't be a perceived need by the Trump administration to purge them en masse (and replace with loyalists, or not). But if government employees' low-coordination individual choices matter, then so can the choices of members of the general population.
States and firms are modern instruments, whereas (at least if you trust some of the accounts from Dawn of Everything) for about 100,000 years the more organic form of coordination was all humans had, and it worked surprisingly well (for example, people could and did travel long distances and count on receiving shelter from strangers).
As already stated, we rely on norm-observance in government employees performing their duties, and in everyone else to more or less comply with laws in functioning welfare states, but traditional norm enforcement is weakened by liberal laissez fair values. But if one believes (a la Suleyman's The Coming Wave) that AI undermines the liberal welfare state, which is likely to be captured by powerful AI firms, then one shouldn't discount norm-enforced resistance emerging to fill the void for an increasingly disenfranchised population.
It is therefore a mistake to treat the race dynamics of AI development between firms and nation-states as an inevitable force pointing in only one direction. Given a critical mass of people recognizing that AI is bad for them, low-coordination resistance is possible, despite the absence of democratic policy-making.
On the flip side, this also suggests a tipping point where AI economic disruption becomes extremely violent, between powerful government-capturing firms wishing to maintain control and general populations resisting. Thus we should consider the existence of a hidden race between would-be powerful government-capturing firms and a would-be resistant population.