AI is increasingly capable. Suppose we're lucky and the alignment problem gets solved, so AI reliably does what its operators want. But we’re not done yet: it lends immense power to those controlling it by progressively replacing the need for humans, disempowering many.[1] AI could enable extreme power concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been.
While LessWrongers have somewhat discussed this (e.g. in Gradual Disempowerment, Intelligence Curse, AI-enabled coups and the like), I still think there are far too few people working on this! As far as I can tell, it’s largely Forethought (who do macrostrategy), Formation Research and EuroSafeAI (shameless plug), as well as some distributed work here and there. Most people seem to agree it’s a large concern, and that few are working on it, but many aren’t sure what to do.[2] I hope this post deconfuses the space and helps technical people contribute, hopefully increasing tractability.
In this post, I want to
Introduce the problem (skip if you’re acquainted[3]),
Introduce a useful frame to think about it (the classic IEMP model), and some of the leaf-node threat models there. (e.g. surveillance)
Give some potential recommendations for projects and action, inviting discussion.
In summary, I’ll explain this graph, and give recommendations.
I think this is an important and neglected problem; I'm uncertain about magnitude; this post maps the problem rather than forecasting it. It’s also preliminary, hopefully contributing to discussion on the topic, and omits details (with some relegated to footnotes). Different opinions and additions would be much appreciated! Furthermore, I will generally assume that operator-alignment is more or less solved (a tall ask, I know).
Why AI could centralize power
Most stories of extreme power concentration are pretty specific or concern one pathway (such as AI-enabled coups). For the sake of this post, I’ll focus on the general case.
The two core drivers are fairly simple: AI automation of labour, and runaway AI R&D giving a decisive lead. The first is the main engine: if you don't need humans to do the work, they lose leverage, and you don't need to listen to them. The same goes for holding power itself: historically that required many cooperating humans (soldiers who could refuse, officials who could slow-walk), whereas AI agents could be far more loyal.[4] Put in economic terms, capital becomes a substitute for labour, and because capital is much more concentrated than labour, control over AI labour concentrates in the hands of capital holders.[5] The second driver, a decisive lead from runaway AI R&D, then amplifies this: whoever it is gets more raw economic, military, or cultural power.[6] (Though note the amplifier isn't required; automation alone shifts power even in a multipolar world.)
But how should you think about power? It can be a bit confusing to think about, especially since human society is more multi-part than an AI system. Breaking down sources of power, I have grown to like the well-established IEMP model by Michael Mann. The IEMP model divides sources of power into ideological, economic, military and political dimensions. These can be converted into each other (a businessman can support a politician, or Taylor Swift can convert appeal into money), but usually, they tend to be surprisingly independent: A powerful FBI bureaucrat (political power) may struggle to get an absurdly high income or become famous (economic and ideological power).
So, how does AI automation of labour concentrate power? A few arguments:
For economic power: Hasn’t previous technological change always created new jobs despite worries? Usually true.[7] But previous tech usually complemented humans. It wasn’t trying to surpass human beings in all possible areas the way the big labs are trying. If AI automates parts of labour, the labour share of GDP (currently at ~55%) will start to fall.[8] That means more economic power goes to capital from labour, and capital is much more concentrated than labour. Importantly, this holds regardless of who captures the surplus: competition may push the gains from the labs to hardware makers or downstream businesses, but all of these are held by capital. Who wins the intra-capital fight doesn't change the labour-to-capital shift.[9]
For military power: In modern states, you tend to need many soldiers to comply with you to get things done. If drone armies or, more speculatively, robot armies become widespread, this again concentrates power away from soldiers to those who control these systems.
For ideological power: Once again AI systems may ultimately outcompete humans in quality and quantity. AI models are already more persuasive than even expert canvassers and persuaders in normal chats according to a UK government study.[10] They are also used in the ideological space a lot: Most obviously, recommender systems have outsized impact on our societies, commanding a dominant share of compute at Meta and Google. More generally, one could imagine AI systems increasingly outcompeting human beings in the attention economy, though I currently suspect demand for the human-premium will win for now. I would get particularly worried if AI-human relationships become normalized.
For political power: Political power is largely downstream in this model: economic, military, and ideological power all tend to convert into it, which is why it's the natural endpoint of concentration, but also one of the classic redistributors, and thus the one most worth defending.
Economic power often translates into political power.
Within a state, Acemoglu and Robinson postulate that democracy is a contract due to bargaining between capitalists and labour. Classic Democratic Conditions theorists Moore, Lipset and Dahl all note the extreme importance of wealth for democracy, with Moore famously postulating ‘No bourgeoisie, no democracy’. It is surely no coincidence that wealthy countries are much more likely to have stable democracies.[11] This means the tax base would also shift away from income taxation, with many states becoming akin to rentier states. These rentier states (most commonly, oil states) usually don’t have a great political track record.
Between states, the US, being 26% of global GDP (but only 4.2% of the global population) pushes far above its weight, and not only militarily.
Military power often translates into political power. Within a state, it is still somewhat debated how much broad conscription contributed to democratization, but it surely helps somewhat. Between states this is also fairly obvious (although economic/political power may be more important in the current order).
LLMs could make coercion much easier at scale, including surveillance or lie detection. More speculatively, you could imagine AI models being used in government for different operations. Either AI representatives, or in the civil service for efficiency or transparency reasons.
Those are the main factors, but much was omitted for brevity. A summary can be found in this graph, with some plausible leaf nodes as well.
Research Projects
Below I have linked to RFPs and other people’s suggestions. What follows is my (opinionated) additional ideas of what works, take it with the usual grains of salt. I suspect policy work will be the biggest lever, but I am generally more technical. If you are coming up with ideas, I recommend checking out previous work, or starting de novo from the IEMP model.
Of the four factors, economic power seems hardest to solve for technical people. I don’t think this is an accident: The economy is fastest to adapt based on capability, and less regulated than others. (I suspect, compared to innovation, that policy will make a big difference here: The public response to AI will probably largely be driven by economic effects rather than e.g. catastrophic risks. But note that this simply routes through political power rather than economic power. This is why I’m generally more interested in keeping the state accountable). Perhaps we can create jobs where people will be glad to pay a human premium,[12] ensure humans stay usefully in the loop[13], or other measures that keep humans in control.
That leaves ideological, military and political power. I’m unsure about the ideological side (I suspect it has similar dynamics to economic power), but things like proof of personhood and the like are probably good. My sense is the least work has been done in military and political scenarios: I think we can greatly increase the ‘usefulness/alignment’ tradeoff in these areas, and suggest good alternatives. Alternatively, if that fails, we can document risks posed by AI here. Promising ideas have been expanded upon by others (see below) so I will not reiterate them here.
For the military and bureaucracy: AI removes the friction that has always made power seizure hard. A leader used to need many people to go along (soldiers who could refuse, officials who could slow-walk), which is why coups are modelled as coordination games (see Singh’s Seizing Power).[14] Loyal AI agents bypass that coordination problem, so the goal is to keep the cost of capture high. Notably, democracies prevent coups less by counterbalancing (as autocrats do) than by legitimacy as a focal point, distributed veto points, and professional norms, each of which assumes the captor needs many humans to cooperate. The coup-proofing literature is a direct resource if we invert it: the same mechanisms protecting institutions rather than rulers (Quinlivan; De Bruin). Concrete directions: multi-party authorization for consequential actions, law-following rather than narrowly operator-following models, and making capture observable through audits and attestation. Perhaps, as Madison noted, we can also have ambition counteract ambition, and produce LLMs with different allegiances so that they check each other.
Conclusion
AI risk, at its core, is about what happens when enormous capability outpaces our mechanisms for keeping it accountable. Misalignment is one branch of that story. Extreme power concentration is another, and right now it has perhaps a dozen people working on it full-time. The central claim of this post is that AI erodes what historically kept power contestable: the need for many cooperating humans, and labour's bargaining leverage. Those frictions were never designed; we inherited them. And they can erode across each of Mann's four sources of power, ideological, economic, military and political, with political power as the natural endpoint where the others converge. If that's right, the work ahead is to rebuild contestability deliberately, dimension by dimension: keep the cost of capture high, make capture observable, keep humans load-bearing in the institutions that matter most, and keep rival actors empowered to push back.
This map is preliminary, and I expect parts of it are wrong. I'd gladly invite discussion in the comments; disagreements, additions, and better framings are all very welcome. And if you're a technical person looking for a way in, the RFPs and project lists above are a good place to start, or reach out to us at EuroSafeAI, or to me directly.
Links to other people’s ideas
For more details on the threat landscape, I like these two articles:
There are two rough ways: Power-to, e.g. the ability to produce an effect, and power-over, e.g. the ability to make Bob do something Bob doesn’t want to. In practice, ideological, military and political power seem more zero-sum to me, so power-to and power-over somewhat converge there. For economic power these diverge somewhat, but it’s not too relevant for the argument. Next to Michael Mann’s IEMP model, I also like Lukes’ Three Faces of political power.
How are you defining AGI and ASI?
I expect power concentration to happen on a continuum as AI capabilities increase, so it’s intentionally left somewhat vague. I think the economic definition that ‘capital becomes a gross substitute for labour’ is pretty good, though.
Are the arguments for loss of control risks and extreme power concentration really that similar?
I was somewhat surprised by this (given my background is fairly technical, and so I didn’t see similarities), but Tom Davidson put this well:
In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here’s a very basic threat model for AI takeover:
Humanity develops superhuman AI
Superhuman AI is misaligned and power-seeking
Superhuman AI seizes power for itself
And now here’s a closely analogous threat model for AI-enabled coups:
Humanity develops superhuman AI
Superhuman AI is controlled by a small group
Superhuman AI seizes power for the small group
Put more broadly, the general story of AI risks is about AI power/capabilities — leading into managing misuse, misalignment and power concentration.
Thanks to Zhijing Jin and EuroSafeAI members for valuable feedback. All errors and opinions are mine.
This isn’t a particularly new point or anything: All of AGI's issues are due to its capability, including misalignment, or also misuse for e.g. CBRNs. A misaligned ant hurts nobody.
It is now regularly mentioned at Alignment workshops, and both OpenAI’s charter and subsequently Anthropic mention it strongly, the former with near equal importance to traditional alignment.
This is a meaningful crux, both for misalignment concerns of course, but also whether we can have AI systems themselves advocate for humanity’s interests.
The possibility of 30% YoY GDP growth is often discussed, e.g. by Epoch AI or Philip Trammell. It is not entirely unreasonable, since GDP is heavily bottlenecked by population size. Assuming you are no longer bottle-necked on labour, it does make sense to presume much faster growth, but as usual the economic picture is hard to predict.
Specifically, this depends on how displacement and reinstatement effects relate to each other. There is evidence suggesting we have had more displacement over the last 30 years.
For more details of this argument, see Philip Trammell’shttps://philiptrammell.com/static/economic_growth_under_transformative_ai.pdf. It is unclear how far this will go (and, as all of the above arguments, this again depends on the regulatory response). Assuming robotics is solved (something I do assume in this scenario), there seem to me 3 big categories of tasks left: Politically protected ones (e.g. licensure, legally-mandated ratios etc., such as potentially lawyers, doctors and other professions which will defend their jobs vigorously), those where there is demand for human input (such as care work, service work, pastors, and perhaps influencers), and rents due to scarcity (e.g. house/energy rents).
Open-weight models decentralize the model layer, but running AI at scale still routes through a concentrated hardware stack (NVIDIA, TSMC, ASML, hyperscaler compute).
Przeworski has famously noted that no democracy wealthier than Argentina in 1975 has ever fallen, i.e. roughly USD 14k today. I now think this somewhat doubtful with Turkey, but the general trend holds.
AI is increasingly capable. Suppose we're lucky and the alignment problem gets solved, so AI reliably does what its operators want. But we’re not done yet: it lends immense power to those controlling it by progressively replacing the need for humans, disempowering many.[1] AI could enable extreme power concentration, producing states unaccountable to their citizenry, and more stable and totalitarian than any previous ones have been.
While LessWrongers have somewhat discussed this (e.g. in Gradual Disempowerment, Intelligence Curse, AI-enabled coups and the like), I still think there are far too few people working on this! As far as I can tell, it’s largely Forethought (who do macrostrategy), Formation Research and EuroSafeAI (shameless plug), as well as some distributed work here and there. Most people seem to agree it’s a large concern, and that few are working on it, but many aren’t sure what to do.[2] I hope this post deconfuses the space and helps technical people contribute, hopefully increasing tractability.
In this post, I want to
In summary, I’ll explain this graph, and give recommendations.
I think this is an important and neglected problem; I'm uncertain about magnitude; this post maps the problem rather than forecasting it. It’s also preliminary, hopefully contributing to discussion on the topic, and omits details (with some relegated to footnotes). Different opinions and additions would be much appreciated! Furthermore, I will generally assume that operator-alignment is more or less solved (a tall ask, I know).
Why AI could centralize power
Most stories of extreme power concentration are pretty specific or concern one pathway (such as AI-enabled coups). For the sake of this post, I’ll focus on the general case.
The two core drivers are fairly simple: AI automation of labour, and runaway AI R&D giving a decisive lead. The first is the main engine: if you don't need humans to do the work, they lose leverage, and you don't need to listen to them. The same goes for holding power itself: historically that required many cooperating humans (soldiers who could refuse, officials who could slow-walk), whereas AI agents could be far more loyal.[4] Put in economic terms, capital becomes a substitute for labour, and because capital is much more concentrated than labour, control over AI labour concentrates in the hands of capital holders.[5] The second driver, a decisive lead from runaway AI R&D, then amplifies this: whoever it is gets more raw economic, military, or cultural power.[6] (Though note the amplifier isn't required; automation alone shifts power even in a multipolar world.)
But how should you think about power? It can be a bit confusing to think about, especially since human society is more multi-part than an AI system. Breaking down sources of power, I have grown to like the well-established IEMP model by Michael Mann. The IEMP model divides sources of power into ideological, economic, military and political dimensions. These can be converted into each other (a businessman can support a politician, or Taylor Swift can convert appeal into money), but usually, they tend to be surprisingly independent: A powerful FBI bureaucrat (political power) may struggle to get an absurdly high income or become famous (economic and ideological power).
So, how does AI automation of labour concentrate power? A few arguments:
Those are the main factors, but much was omitted for brevity. A summary can be found in this graph, with some plausible leaf nodes as well.
Research Projects
Below I have linked to RFPs and other people’s suggestions. What follows is my (opinionated) additional ideas of what works, take it with the usual grains of salt. I suspect policy work will be the biggest lever, but I am generally more technical. If you are coming up with ideas, I recommend checking out previous work, or starting de novo from the IEMP model.
Of the four factors, economic power seems hardest to solve for technical people. I don’t think this is an accident: The economy is fastest to adapt based on capability, and less regulated than others. (I suspect, compared to innovation, that policy will make a big difference here: The public response to AI will probably largely be driven by economic effects rather than e.g. catastrophic risks. But note that this simply routes through political power rather than economic power. This is why I’m generally more interested in keeping the state accountable). Perhaps we can create jobs where people will be glad to pay a human premium,[12] ensure humans stay usefully in the loop[13], or other measures that keep humans in control.
That leaves ideological, military and political power. I’m unsure about the ideological side (I suspect it has similar dynamics to economic power), but things like proof of personhood and the like are probably good. My sense is the least work has been done in military and political scenarios: I think we can greatly increase the ‘usefulness/alignment’ tradeoff in these areas, and suggest good alternatives. Alternatively, if that fails, we can document risks posed by AI here. Promising ideas have been expanded upon by others (see below) so I will not reiterate them here.
For the military and bureaucracy: AI removes the friction that has always made power seizure hard. A leader used to need many people to go along (soldiers who could refuse, officials who could slow-walk), which is why coups are modelled as coordination games (see Singh’s Seizing Power).[14] Loyal AI agents bypass that coordination problem, so the goal is to keep the cost of capture high. Notably, democracies prevent coups less by counterbalancing (as autocrats do) than by legitimacy as a focal point, distributed veto points, and professional norms, each of which assumes the captor needs many humans to cooperate. The coup-proofing literature is a direct resource if we invert it: the same mechanisms protecting institutions rather than rulers (Quinlivan; De Bruin). Concrete directions: multi-party authorization for consequential actions, law-following rather than narrowly operator-following models, and making capture observable through audits and attestation. Perhaps, as Madison noted, we can also have ambition counteract ambition, and produce LLMs with different allegiances so that they check each other.
Conclusion
AI risk, at its core, is about what happens when enormous capability outpaces our mechanisms for keeping it accountable. Misalignment is one branch of that story. Extreme power concentration is another, and right now it has perhaps a dozen people working on it full-time. The central claim of this post is that AI erodes what historically kept power contestable: the need for many cooperating humans, and labour's bargaining leverage. Those frictions were never designed; we inherited them. And they can erode across each of Mann's four sources of power, ideological, economic, military and political, with political power as the natural endpoint where the others converge. If that's right, the work ahead is to rebuild contestability deliberately, dimension by dimension: keep the cost of capture high, make capture observable, keep humans load-bearing in the institutions that matter most, and keep rival actors empowered to push back.
This map is preliminary, and I expect parts of it are wrong. I'd gladly invite discussion in the comments; disagreements, additions, and better framings are all very welcome. And if you're a technical person looking for a way in, the RFPs and project lists above are a good place to start, or reach out to us at EuroSafeAI, or to me directly.
Links to other people’s ideas
For more details on the threat landscape, I like these two articles:
https://www.formationresearch.com/power-concentration-survey.pdf
https://80000hours.org/problem-profiles/extreme-power-concentration/
For ideas specifically, these are great:
https://www.longview.org/request-for-proposals-on-extreme-power-concentration/
https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power#5-mitigations
https://www.lesswrong.com/posts/BXW2bqxmYbLuBrm7E/the-best-approaches-for-mitigating-the-intelligence-curse-or#comments
https://www.lesswrong.com/posts/GAv4DRGyDHe2orvwB/gradual-disempowerment-concrete-research-projects
https://gradual-disempowerment.ai/mitigating-the-risk
https://intelligence-curse.ai/breaking/
https://www.lesswrong.com/posts/ugBoeexGYvNLxZKA7/a-research-agenda-for-secret-loyalties
https://democracy.build/
https://zhijing-jin.com/d/2026-ai-risk.pdf
FAQ
How are you defining power?
There are two rough ways: Power-to, e.g. the ability to produce an effect, and power-over, e.g. the ability to make Bob do something Bob doesn’t want to. In practice, ideological, military and political power seem more zero-sum to me, so power-to and power-over somewhat converge there. For economic power these diverge somewhat, but it’s not too relevant for the argument. Next to Michael Mann’s IEMP model, I also like Lukes’ Three Faces of political power.
How are you defining AGI and ASI?
I expect power concentration to happen on a continuum as AI capabilities increase, so it’s intentionally left somewhat vague. I think the economic definition that ‘capital becomes a gross substitute for labour’ is pretty good, though.
Are the arguments for loss of control risks and extreme power concentration really that similar?
I was somewhat surprised by this (given my background is fairly technical, and so I didn’t see similarities), but Tom Davidson put this well:
Put more broadly, the general story of AI risks is about AI power/capabilities — leading into managing misuse, misalignment and power concentration.
Thanks to Zhijing Jin and EuroSafeAI members for valuable feedback. All errors and opinions are mine.
This isn’t a particularly new point or anything: All of AGI's issues are due to its capability, including misalignment, or also misuse for e.g. CBRNs. A misaligned ant hurts nobody.
It is now regularly mentioned at Alignment workshops, and both OpenAI’s charter and subsequently Anthropic mention it strongly, the former with near equal importance to traditional alignment.
E.g. if you’ve read Intelligence Curse or 80k’s excellent https://80000hours.org/problem-profiles/extreme-power-concentration/
This is a meaningful crux, both for misalignment concerns of course, but also whether we can have AI systems themselves advocate for humanity’s interests.
E.g. global Palma ratios for wealth are around 2-3x larger than for income. Similarly, gini coefficients are often around 0.2 higher.
The possibility of 30% YoY GDP growth is often discussed, e.g. by Epoch AI or Philip Trammell. It is not entirely unreasonable, since GDP is heavily bottlenecked by population size. Assuming you are no longer bottle-necked on labour, it does make sense to presume much faster growth, but as usual the economic picture is hard to predict.
Specifically, this depends on how displacement and reinstatement effects relate to each other. There is evidence suggesting we have had more displacement over the last 30 years.
For more details of this argument, see Philip Trammell’s https://philiptrammell.com/static/economic_growth_under_transformative_ai.pdf. It is unclear how far this will go (and, as all of the above arguments, this again depends on the regulatory response). Assuming robotics is solved (something I do assume in this scenario), there seem to me 3 big categories of tasks left: Politically protected ones (e.g. licensure, legally-mandated ratios etc., such as potentially lawyers, doctors and other professions which will defend their jobs vigorously), those where there is demand for human input (such as care work, service work, pastors, and perhaps influencers), and rents due to scarcity (e.g. house/energy rents).
Open-weight models decentralize the model layer, but running AI at scale still routes through a concentrated hardware stack (NVIDIA, TSMC, ASML, hyperscaler compute).
Many papers now put LLM persuasiveness higher than human persuasiveness in chatting (see https://arxiv.org/pdf/2606.16475v1).
Przeworski has famously noted that no democracy wealthier than Argentina in 1975 has ever fallen, i.e. roughly USD 14k today. I now think this somewhat doubtful with Turkey, but the general trend holds.
In the EU, around 22% of jobs are already regulated in some sense. (https://www.pubaffairsbruxelles.eu/eu-institution-news/commission-calls-on-18-member-states-to-strengthen-the-eu-single-market-for-regulated-professions/)
See Jan Kulveit or Erik Brynjolfsson’s recent work
Although coups are the less likely way democracies die today, see https://www.journalofdemocracy.org/articles/on-democratic-backsliding/ for a good review.