I work on preventing extreme power concentration ("EPC") at a well-known AI safety org. Lots of people are worried about EPC, but AFAICT there is no single online resource setting out the key threat models and most promising interventions.
This post is my attempt to do that. It sets out...
In brief: an individual or small group seizing power at an unusually high-stakes time in history and using that power to permanently lock in bad values.
Solving the Red and Expert problem: where advanced AI eliminates the loyalty-competence trade-off that power-seekers face when staffing large institutions.
Eliminating executive overhang: where advanced AI enables far more extensive abuse of the affordances of the executive branch.
The asymmetric frontier: where POTUS and lab leaders gain privileged access to advanced AI before the rest of society.
Desperate times calling for desperate measures: where the pervasive instability created by the intelligence explosion creates an opening for strongmen.
Widespread epistemic degradation: where AI progress makes it increasingly difficult for ordinary people to understand the world around them, weakening accountability for the powerful.
Alternatively, a power-seeker could leapfrog part or all of the way to EPC through a “shortcut”. I think the most likely shortcuts are:
Government capture shortcuts: filling key parts of the executive with loyalists and loyal AIs – especially the military, intelligence community, law enforcement apparatus, and any agencies with powers to regulate the AI.
Lab capture shortcuts: finding some way to gain unchecked control over frontier AI development and then exploiting this to concentrate power.
Which of these threat models seems most likely depends quite a bit on:
I provide a list of relatively higher-tractability interventions, keyed to the main risk-generating dynamics identified above.
The interventions I’m most excited about are:
hard-to-remove behavioural guardrails(e.g. LFAI) for frontier models;
improving executive record-keeping and transparency, especially for any USG-led AGI project;
red-teaming AI legislation for EPC risk, and proposing low-risk alternatives;
laws requiring labs and the executive to make models widely available after initial safety testing
screening politicians for power-seeking tendencies, and directing political funding to the most virtuous leaders;
a new org to coordinate work on EPC risk; and
detailed threat modelling by sector experts, especially in the military domain.
1.1. Epistemic status
This post is quite light on argumentation, because I’m aiming to set out my all-things-considered view on how EPC might go – rather than presenting highly robust arguments for exactly why I think this.
I hold most of the views below quite lightly, and I’m particularly unsure about section four (“What are the most concerning EPC threat models?”).
So I strongly encourage readers to push back if they think I might be overstating a claim, or failing to consider some core element of how things might go.
2. What am I worried about?
When I say I’m worried about EPC, I mean I’m worried about the risk that:
an individual or small group of power-seekers achieves a decisive strategic advantage during the intelligence explosion, and uses this to exercise sole authorship over the key decisions, priorities, and defaults determining the value of the long-term future.
2.1. Why am I worried about this?
It seems more likely than not that the intelligence explosion ("IE") kicks off in the next decade, and entirely plausible that it begins in the next 6 months.
During the IE, it might become possible to lock in various features of Earth-originating civilisation.
e.g. by determining which values are promoted by the agents which go to space, and ensuring that continued moral reflection is not one of these values
Power-seekers seem negatively selected for values which could promote a near-best future, because:
the kind of person who wants to become a dictator typically doesn’t care much about others, and they tend to be quite ethically unreflective;
cf. Stalin; Hitler; Mao; Franco
so I don’t expect they would think hard about the ethical quandaries which could substantially determine the value of the future.
e.g. whether they ought to hold a person-affecting view in population ethics
Power also tends to corrupt. Even if power-seekers started out with acceptably altruistic and morally-reflective values, I’d expect them to become more selfish, vain, greedy, etc. over time.
So – a power-seeker gaining a decisive strategic advantage ("DSA") during the IE seems particularly likely to lead to harmful lock-in.
2.2. Am I worried about all kinds of power concentration?
No – and I think it's worth stressing that concentration of power isn't necessarily bad in all cases.
First, it matters a lot where power is concentrated, because some groups seem substantially less likely to promote bad values.
e.g. I suspect AGI-enabled US hegemony would be better than a power-sharing arrangement between the US and China, because the current values of CCP-led China seem really bad
Second, making AI go well might require substantial power concentration, albeit on a temporary basis.
e.g. amassing the political power to expropriate the frontier labs and hyperscalers for the purposes of redistribution[1]
e.g. radically expanding state surveillance to guard against AI-enabled bioterrorism
So, I think it would be a mistake for us to try to generically prevent AI-enabled power concentration. Instead, I think we should focus on stopping the most extreme and durable forms of power concentration in the very worst actors.
2.3. Where am I worried about this happening?
I’m particularly worried about EPC occurring within:
the US federal executive branch (esp. POTUS and his/her close allies), and/or
the frontier AI labs (e.g. Anthropic, OpenAI, Google, Meta, and SpaceX leadership).
I’m particularly worried about these actors because:
it seems really quite likely that ASI will be developed first by the US government (“USG”) or a US-headquartered company; and
control over ASI is the most plausible medium through which a power-seeker could get a DSA.
2.3.1. A note on terminology
For the rest of this post, I'm going to refer to power-seekers in the abstract – as a "PS", with feminine pronouns.
I've taken this approach because I think it's better to avoid singling out any particular politician or lab leader who might pose a threat.
I encourage readers to use similarly abstract language when discussing this topic publicly.
3. What are the key dynamics generating EPC risk?
I think there are five key high-level dynamics driving EPC risks. I expect each of these to apply continuously throughout the IE, and to intensify as AI becomes more powerful.
3.1. Solving the Red and Expert problem
For autocrats to maintain and increase their power, it is best if they have allies and subordinates who are both extremely loyal and exceptionally competent.
Mao Zedong referred to this as the need for Chinese workers to become “both Red and Expert”.
Historically, autocrats have had to make hard trade-offs in this regard:
Optimising too hard for competence enables independent centres of power to persist, constituting a threat to the autocrat's control over the regime.
Optimising too hard for loyalty puts power in the hands of incompetents, leading to regime dysfunction and curtailing the autocrat's global influence.
This need for compromise has placed a natural check on the power of any one individual or group, even within highly autocratic regimes.
However, transformative AI seems likely to ~solve the Red and Expert problem – making it easier for a PS to optimise for loyalty in the organisations she leads, without sacrificing those organisations' operational effectiveness. That's because it may enable PS to:
deploy highly-competent AIs who will follow unethical and illegal orders; and
continuously loyalty-screen her human subordinates.
I've explained each of these in a bit more detail below.
3.1.1. The role of AI henchmen:
During the IE, AIs will take on a continuously expanding share of cognitive labour in PS-controlled organisations.
AI labour will be highly competent, and it may be fairly easy for PS to gain access to AIs which will follow even her most authoritarian orders without scruple (“AI henchmen”).[2] Here are some ways PS might get access to AI henchmen:
(1) PS may have substantial access to so-called "helpful-only" AIs, which are subject to a minimum of behavioural guardrails.
These are currently the default in key parts of the US military (e.g. targeting systems), placing them within reach of a POTUS PS.
And it also seems trivially easy for frontier lab leaders to gain access to such models.
(2) PS might be able to insert loyalties into the AIs under her control – ensuring they comply with her orders even when this is clearly illegal or unethical.
“Secret loyalties” in particular seem likely to drastically reduce the hurdles to capturing large and powerful institutions, ensuring PS does not encounter meaningful resistance from independent power-holders ("IPHs").
e.g. a lab leader PS could capture key parts of the military by inserting a loyalty into the military's preferred model without anyone realising
e.g. a POTUS PS could capture the courts and Congress by inserting loyalties into their preferred models
It seems particularly easy for a lab leader PS to insert a loyalty into a model.
However a POTUS PS might also be able to insert one by leaning on allies at frontier labs or in the national security apparatus – especially if she is looking to insert a joint loyalty of some form.
e.g. to POTUS's preferred ideology
e.g. to POTUS's political party or faction
e.g. between POTUS PS and a lab leader PS
(3) PS might find ways to abuse AI labour which is neither helpful-only nor loyal to her ("good guy AIs"). That's because:
PS might be able to rely on a small number of AI henchmen to orchestrate good guy AIs (or humans) carrying out illegal or unethical tasks.
e.g. they could “task-split” – such that good guy AIs are simply unaware that they are assisting with illegal requests
e.g. they could rewrite internal task-management software to automatically route illegal or unethical requests to other AI henchmen
e.g. they could find ways to jailbreak the good guy AIs
Good guy AIs might also lack the incentives and affordances which would enable human subordinates to refuse illegal or unethical orders from PS.
e.g. good guy AIs in the executive may lack confidential channels to whistleblow to Congress
3.1.2. The role of loyalty screening
Transformative AI will also make it increasingly easy to conduct loyalty screening of human subordinates, via:
3.1.3. The consequence – AI-enabled packing of institutions
Taken together, this may enable a PS to gain an unprecedented degree of control over key parts of USG and other power-holding institutions by filling them with loyalists and AI henchmen. I refer to this as "AI-enabled packing".
AI-enabled packing seems most concerning in the military, national security apparatus, and law enforcement – where PS might be able to replace or disempower government employees who can currently exercise a real check on autocratic behaviour.
e.g. replacing generals and infantrymen who would refuse orders to carry out a coup
e.g. replacing DC police who would refuse to kill law-abiding protesters
e.g. replacing surveillance analysts who would refuse to carry out clearly unconstitutional searches of commercial data
e.g. replacing department heads who would refuse to help carry out a selective prosecution campaign
But packing risks also seem quite acute in the context of:
frontier AI labs – because this might enable a lab leader PS to tighten her control over an essential tool for achieving EPC
e.g. replacing the safety teams with human loyalists, and then getting them to wave through the insertion of a secret loyalty during the training process
political parties – where enhanced loyalty screening could enable a POTUS PS to gain an unprecedented degree of centralised control over her political coalition.
3.2. Eliminating executive overhang
The executive branch controls an exceptionally wide range of political, economic, ideological, and coercive affordances.
e.g. substantial prosecutorial discretion
e.g. a colossal military budget
e.g. expansive legal surveillance powers
A key constraint on the use of these affordances right now is an undersupply of cognitive labour. As in:
There are simply not enough lawyers, surveillance analysts, military officers, and other kinds of bureaucrat for PS to take advantage of these affordances whenever the opportunity might arise.
Even if there were enough bureaucrats, they would lack the planning and coordination capabilities to fully exploit such opportunities.
I refer to this phenomenon as "executive overhang".
Transformative AI promises to radically reduce executive overhang by drastically increasing the executive's supply of cognitive labour. This promises to greatly increase state capacity; but it may also enable an unscrupulous PS to abuse the executive's affordances to an unprecedented degree.
e.g. PS could take advantage of drastically increased surveillance capacity to keep close tabs on a wide array of IPHs (e.g. judges, political rivals, civil society leaders).
e.g. PS could conduct large-scale, low-cost lawfare against a wide range of political opponents, using an army of AI henchmen to generate legally-robust justifications for her prosecutions and adverse administrative actions.[3]
e.g. PS could deny economically-vital permits or procurement opportunities to the funders of her political opponents in an obscure but coordinated fashion.
e.g. PS could build out a colossal and secretly loyal robot army to facilitate an AI-enabled military coup.
3.2.1. Legal overhang
Reduced executive overhang seems particularly concerning because US law already grants POTUS and her officers a wide range of expansive and highly-discretionary legal powers which seem ripe for abuse in a world with transformative AI.
e.g. PS could instruct AI henchmen to carry out illegal acts on her behalf, and then abuse her control of DOJ appointments plus presidential immunity to prevent criminal prosecution of her and her allies
e.g. PS could block officials, the general public, and Congress from scrutinising her autocratic behaviour by strategically abusing her control over clearance and classification decisions, aggressively invoking "executive privilege", and using AI henchmen attorneys to fight any resulting civil suits tooth and nail
This issue only seems to be growing over time, because the current SCOTUS majority has consistently favoured expansive interpretations of presidential powers.
3.2.2. Extralegal overhang
Transformative AI also seems likely to increase the executive's capacity to engage in straightforwardly unlawful behaviour, and to do so with an unprecedented degree of sophistication.
e.g. PS could strategically delete and alter federal records to impede effective congressional oversight
e.g. PS could abuse AI-uplifted cyber capabilities available to her loyalists in the national security apparatus to harass her political enemies
e.g. PS could employ hordes of police robots to suppress protests
3.2.3. Who benefits from reduced executive overhang?
This seems likely to differentially uplift a POTUS PS relative to a lab leader PS. However, it could also meaningfully uplift a lab leader PS who:
was colluding with a POTUS PS to seize power, or
had successfully captured USG through the insertion of a secret loyalty.
3.2.4. h/t LawAI
Finally, I should note that Cullen O'Keefe, Alan Rozenshtein, and Christoph Winter have already written atsomelength on both the Red and Expert problem and Executive overhang – in more detail and at much higher quality than I have here.
I highly recommend that interested readers take a look at their work.
3.3. The asymmetric frontier
I expect frontier lab staff, POTUS, and key parts of the national security apparatus to gain privileged access to the most advanced models before most of the rest of society.
If capabilities are increasing very quickly, this could create extremely large capabilities asymmetries between these actors and most of the rest of society.
e.g. most of the world is stuck with Opus 6 whilst a few actors have access to TEDAI
A lab leader PS might be able to expand these asymmetries simply by concealing the existence of her very best models from actors who might want to use them.
e.g. getting her best models to sandbag on CAISI capabilities evaluations
Whereas a POTUS PS may be able to expand these asymmetries by abusing executive overhang.
A POTUS or lab leader PS could take advantage of these asymmetries to radically increase her power in a short space of time.
e.g. a lab leader could get early access to mindhack-level superpersuasion capabilities before anyone has developed effective epistemic shielding, and could use this to gain the loyalty and trust of the top ranks of the federal executive, Congress, judiciary, and military
e.g. a POTUS could nationalise the AI industry, insert a secret loyalty into the leading model, and then get that model to carry out a textbook misaligned takeover scenario – but leave her in control at the end
3.3.1. Strained accountability loops
One particularly concerning risk of access asymmetries is the straining of standard “accountability loops” necessary for the functioning of democracy. To explain:
An AI-uplifted executive may be able to execute a far greater quantity of unlawful and unethical behaviours in a short space of time (see the previous section).
The key formal routes for dealing with unlawful and unethical behaviour in the executive are congressional oversight, lawsuits, impeachments (in extreme cases) and – ultimately – elections.
However, as AI progress accelerates, multi-year election cycles may cease to impose a meaningful check.
i.e. it may become possible for PS to use her country of geniuses in a data-centre to get all the way to a DSA within a single electoral cycle
And if PS can monopolise access to frontier capabilities, she may be able to get away with authoritarian behaviour simply by outpacing the attempts to hold her accountable.
e.g. PS could carry out a huge DOGE-style restructuring in just a few weeks, with AI henchmen loyalty screening an entire department, firing disloyal agency heads and installing acting loyalists, and reshuffling suspected dissenters and/or stripping them of their clearances en masse – with carefully drafted legal rationales to support each step; even if reshuffled staff immediately brought legal proceedings, it's unclear whether the human-led courts would be willing or able to enjoin so many personnel decisions at once
3.4. Desperate times calling for desperate measures
It seems likely that the IE will be very weird and scary for most ordinary people. That’s because it seems likely that we’ll see some or all of the following:
skyrocketing unemployment;
skyrocketing interest rates, driven by massive compute buildout;
widespread proliferation of compelling deepfakes;
drastically increased tensions in the Asia-Pacific, as China realises that Taiwan, Japan and South Korea are all facilitating US dominance in AI; and
terrifying warning shots (e.g. failed AI takeover attempts, AI-enabled coups in the developing world).
more likely to vote for “strongman” political candidates during the IE; and
more tolerant of strongman-type behaviour which concentrates power.
e.g. aggressive use of emergency powers to regulate the AI industry
3.5. Widespread epistemic degradation
As we move deeper into the IE, it may become increasingly difficult for people to understand the world around them. There are three key reasons for this:
(1) The world will simply be moving much faster than it was before.
(2) Advanced AI may be able to generate compelling deepfakes, which could:
convince people of untrue things; and
cause people to distrust most of what they see online.
(3) Many major events will be driven by AIs, who need not share what they are doing with anyone else – unlike humans, who are likelier to leak major projects.
I'm referring to this phenomenon as "epistemic degradation".
Epistemic degradation may concentrate power by weakening accountability for PS and her allies – because the public, courts, legislators, journalists, and other IPHs will struggle to notice, investigate, and sanction her misconduct.
e.g. if PS has asymmetric frontier access, she might be able to tell hard-to-disprove, deepfake-supported lies to legislators and journalists
e.g. allies of PS can pump out vast quantities of compelling deepfake content about politicians of all political stripes, meaning the public doesn’t update much on genuine scandals
e.g. advanced AI will come to intermediate almost all news consumption – meaning PS may be able to create a pervasive pro-PS information environment if she can subtly bias the leading model in her favour
3.6. Which of these dynamics are most important?
Overall, I suspect that:
solving the Red and Expert problem and the asymmetric frontier are the most important dynamics; followed by
eliminating executive overhang; and then
desperate times calling for desperate measures; and then
widespread epistemic degradation.
Here's my reasoning for this ranking...
I've found it really quite hard to construct highly-plausible EPC threat models which don’t involve some amount of AI-enabled packing and frontier asymmetry. It just seems really very likely that PS would want to:
use transformative AI to capture large and powerful organisations with control over key power-enhancing affordances, such as the US military; and
try to create and exploit asymmetries between herself and IPHs in their access to frontier capabilities.
Elimination of executive overhang also feels like a core driver of power-concentration in the majority of the most-plausible EPC threat models.
That's because the executive has various unique affordances which seem particularly valuable for any PS to capture.
However, I still find it plausible that a PS could achieve EPC some other way.
e.g. inserting a secret loyalty into the leading model, and then allowing loyal models to diffuse throughout the private economy, without ever specifically focusing on capturing USG
By contrast, demand for desperate measures and epistemic degradation feel like additional elements that increase the plausibility of various threat models, without being especially load-bearing.
To make this concrete – I’d not be especially surprised if PS achieves EPC in a world where public epistemics are far better than today, and skilful economic management and geopolitical manoeuvring by PS have created global peace and stability.
And I think that epistemic degradation, in particular, matters quite a bit less in expectation than the other four dynamics, because:
AI progress also seems likely to make it much easier for people to fact-check misleading information and AI-generated content.
cf. Pangram and X's Community Notes
Also – I suspect there is a near-term ceiling on the persuasiveness of AI-generated content, which may limit AI's ability to drive epistemic degradation.
That said, all five dynamics seem plausible enough – and plausibly important enough – that we should treat them as key drivers of EPC risk.
4. What are the most concerning EPC threat models?
In this section, I sketch a high-level framework for thinking about EPC threat models, and then set out some particularly plausible threat models which fall out of the framework.
4.1. High-level framework
I find it helpful to think of power concentration as a scale, which runs all the way from radical AI-enabled pluralism at one end, to AI-enabled extreme power concentration at the other:
Contemporary US democracy sits somewhere closer to the pluralistic end of the scale – albeit being quite far from a maximally decentralised regime.
High-level structural conditions, and concrete actions of PSs and IPHs, can move the US towards or away from power concentration.
Once the US passes certain thresholds on the scale, it becomes possible to jump part or all of the way towards EPC – if PS takes control of particularly important sets of affordances (“shortcuts”).
For example, US democracy seems to have deteriorated substantially in recent years. I find it at least plausible (albeit unlikely) that we have passed a threshold where – if the US military becomes substantially automated in the next few years – it will be possible for POTUS to gain unilateral command of it.
If a POTUS PS took this shortcut, I imagine the US would jump almost all the way to EPC in a short space of time.
4.2. Threat models
4.2.1. Backsliding all the way
Within this framework, it seems at least plausible that PS could get all the way to EPC simply by slow and continuous movement along the scale.
Under such a model, the five high-level dynamics identified in the previous section would exert continuous and compounding pressure towards power concentration throughout the IE, leading to progressive democratic backsliding up until PS has achieved a DSA.
I refer to this as a “backslide all the way” threat model.
Here's one very high-level illustration of how this might happen:[5]
Early in the IE, it becomes increasingly clear to a POTUS PS and the public that the frontier AI industry will become the US's single most important economic and military asset.
China's leaders become increasingly conscious of this, too, and tensions begin to rise in the Asia-Pacific as they realise that several of their close neighbours are making a direct contribution to Chinese disempowerment.
In response, PS starts “soft nationalising” the frontier AI industry to guard against threats of Chinese distillation and other IP theft. In particular, she starts:
gradually extending the duration of a "voluntary" national security screening process;
using DPA allocations authorities to slowly starve most labs of compute, and concentrate compute within her preferred lab;
forcing labs to conduct ever-greater shares of their training at SL-5 facilities operated by the federal government;
eventually – classifying the "helpful-only" frontier models generated by the preferred lab, and restricting access to high-ranking officials in the national security apparatus.
With the stated goal of placing the US on a war footing, PS starts packing the bureaucracy, expanding the executive's surveillance activities, and increasingly automating the military and security services.
After a major crisis (e.g. a bioterrorism warning shot or the invasion of Taiwan), her allies in Congress pass a bill fully nationalising the AI industry, with genuine public support.
Over a period of several years, PS inserts ever-less-subtle political biases into the models that the nationalised industry serves to the public, businesses, state governments, courts, journalists and Congress. Alternative models are far less capable, so it becomes increasingly hard to hold her accountable.
She has also fostered a great deal of public support after instituting a UBI programme which has made the median voter exceptionally wealthy – so her increasingly firm grip over independent US institutions receives limited pushback.
By the end of her second term, PS decides she doesn't want to give up power, so she gets an AI henchman to ratchet up the bias in the leading model to the point where they will simply refuse to assist anyone who tries to remove her from office.
Now that her AI henchmen are diffused throughout the whole of the economy, government, and military, she can also unilaterally seize control of nearly any asset she wants and overcome any legal, political, or violent resistance.
As a result, she and her allies have irrevocable and totalitarian control over the entire United States.
One argument in favour of backslide all the way threat models which I find especially compelling is that most of the individual steps don’t require PS to be actively trying to achieve EPC.
e.g. in the scenario above, up until PS inserts a loyalty into the leading models, she could just be following local incentives to increase her power and diminish that of her rivals, without realising (or admitting to herself) that she is on the path to EPC
This seems more congruent with the psychological profiles of contemporary political and lab leaders than the "shortcuts" to EPC which I set out below.
That said, I still think backsliding all the way is less likely than other EPC threat models. That’s because:
I suspect PS will become increasingly strategic throughout the IE, as she gains access to increasingly high-quality AI advice and becomes increasingly aware of the opportunity to lock in her preferred social order; and because
I suspect a more strategic PS will recognise that gradual strategies are simply less efficient than backsliding up to a threshold and taking a shortcut.
i.e. why would she bother to gradually capture the whole of USG and various IPHs, when she could get to EPC far faster by packing the military alone?
4.2.2. Shortcuts
I think shortcuts can be sorted into two broad categories:
government capture shortcuts; and
lab capture shortcuts.
4.2.2.1. Government capture shortcuts
Under a government capture shortcut, PS aims to:
use AI to pack essential parts of the executive; and then
exploit executive overhang within the packed executive to gain control of other power-concentrating affordances (e.g. frontier AI).
uses it to take control of other key affordances by force.
e.g. using a robot army to seize ample compute and the seat of government
Another particularly plausible shortcut is “packing the regulators”[6] – whereunder PS:
packs any agencies with important powers to regulate the AI industry;
e.g. parts of DOC; any regulator set up by incoming AI legislation
abuses those powers to gain privileged access to frontier AI capabilities.
e.g. demanding unmonitored access to an AI henchman in return for shelving an investigation into unsafe practices
e.g. threatening to flip a "kill switch" on any frontier lab that serves its best model to her political rivals
Two other parts of the executive also seem particularly worth packing – the intelligence community (“IC”) and the law enforcement apparatus (esp. key parts of the DOJ). That’s because:
packing the IC would enable PS to:
drastically expand her surveillance capacity with a reduced risk of whistleblowing; and
carry out a wide variety of unlawful activities under a blanket of classification;
packing law enforcement would enable PS and her allies to engage in lawbreaking whilst knowing that federal law couldn't easily be enforced against them.
e.g. conducting extrajudicial killings of lab leaders, covering up their involvement by illegally deleting records, declining to investigate at the federal level, and then refusing to comply with congressional subpoenas on the matter
Government capture shortcuts seem particularly easy for a POTUS PS, because she has direct influence over senior appointments within the executive branch.
However, I also find it somewhat plausible that a lab leader PS could insert a loyalty into the model used by the military, IC, law enforcement, or AI regulators and so gain control over their affordances.
4.2.2.2. Lab capture shortcuts
Under a lab capture shortcut, PS aims to:
gain unchecked control over frontier AI development;
exclude others from accessing frontier capabilities; and then
exploit those frontier capabilities to durably concentrate power.
These shortcuts rely on two distinct threat vectors:
(1) Loyalty insertion (i.e. determining the alignment target of the leading model, and then relying on the model to concentrate power on your behalf)
(2) Capabilities asymmetries (i.e. gaining exclusive access to key capabilities which can be exploited to further concentrate power). It’s hard to say in advance what these key capabilities might be – but here are three strong candidates:
Superpersuasion[7](i.e. PS can easily manipulate rivals into complying with her goals in a wide variety of political, legal, and military settings)
Superstrategy (i.e. PS has an unmatched degree of strategic foresight, enabling her to execute on highly-complex power-seeking strategies and outmanoeuvre defending IPHs who lack the same capabilities)
Supercyber (i.e. PS has an unmatched ability to hack into computer systems and turn them to her own ends, including for purposes of surveillance, insertion of secret loyalties, and general disruption)
I expect PS’s identity to greatly influence the most likely shortcut.
If PS is a lab leader, she seems most likely to rely on an enforced monopoly, whereunder:
her lab pulls ahead of the competition, due to a natural break-away (e.g. a big and quick software intelligence explosion) and/ordeliberate sabotage (e.g. abusing advanced cyber capabilities to sabotage competitors);
around the same time, she starts packing her lab with loyalists and AI henchmen to ensure tight personal control over the training process; and
once she has a packed lab and a frontier monopoly, she:
inserts a secret loyalty into her frontier model and lets it diffuse throughout society until she has effectively captured various IPHs;
e.g. the executive; the media; the courts
or abuses her privileged access to key capabilities to capture various IPHs.
e.g. using AI-enhanced social engineering, uplifted by a great deal of hacking-enabled surveillance, to fill their senior ranks with loyalists
If PS is a POTUS, she seems most likely to pack the frontier – abusing key legal authorities to fill the leading AI project with loyalists and AI henchmen. Centrally, I’d expect her to:
pack the "regulators" (see above), giving her more direct control over the executive's relationship with the AI industry;
abuse existing emergency and national security powers (e.g. the DPA) to kludge together a centralised project over which she has substantial direct influence;
e.g. using allocations authorities under Title I of the DPA, plus a Title VII voluntary agreement, to set up a project whose board and key staff are de facto chosen by the DPA Committee
e.g. using Title I hoarding designations and export control powers to kill off competitor labs
e.g. classifying the very best models, giving her veto power over which human workers can participate in the project
seek to pass new regulations increasing her ability to influence the training process;
e.g. requiring labs to conduct a portion of their training process at SL-5 facilities owned and operated by the federal government
e.g. embedding executive employees in the labs with wide-ranging access permissions – nominally for the purposes of safety assurance
e.g. granting herself a unilateral "kill switch" over frontier training runs
and, once she has a packed frontier – insert a secret loyalty into her frontier model or she abuses her privileged access to key capabilities to capture various IPHs.
4.2.3. POTUS-lab leader collusion
I find collusion between POTUS PSs and lab leader PSs to be a particularly plausible variant of all of the threat models that I've laid out above.
That's because both sides control valuable affordances that can benefit the other, such as:
POTUS
Favourable regulatory treatment (e.g. looking the other way on safety warning shots)
Influence in the admin (e.g. a seat on an advisory board for the AI regulator)
Lab leaders
Large amounts of funding (e.g. for electoral campaigns)
Technical support (e.g. helping to develop superpersuasion capabilities, or training AI henchmen)
Access to off-books tokens (e.g. using the lab's private compute to conduct illicit cyber operations against IPHs)
Epistemic distortion in POTUS's favour (e.g. inserting political biases into her models)
It’s hard to say exactly how any collusion might go, but here are a few scenarios I find particularly plausible:
A POTUS PS could give a lab leader PS an influential role in a centralised frontier project, in return for her support with packing that project.
A POTUS PS could threaten to use “kill switch” powers to coerce a lab leader PS into inserting a joint secret loyalty into the models used throughout USG.
A lab leader PS could help a POTUS PS entrench her control over the executive by enhancing campaign funding for her allies and biasing her models in favour of POTUS, in return for POTUS applying limited regulatory scrutiny.
4.3. Which threat models seem likeliest?
I think there are two key determinants of the most likely threat models. These are:
The character of any POTUS and her allies.
The speed of the intelligence explosion.
4.3.1. The character of POTUS
I think the median POTUS PS would be substantially better placed to achieve EPC than a lab leader PS. That’s because POTUS has control of an extremely valuable set of affordances which seem uniquely easy to exploit and uniquely difficult for other actors to capture.
e.g. the US military; the federal bureaucracy; widespread public legitimacy
By contrast, the key affordance of a lab leader is her access to frontier capabilities – and she may struggle to maintain asymmetric access to this affordance once a POTUS becomes adequately situationally aware.
Given this, I think the character of whoever happens to be POTUS during the IE (and the characters of their close allies) will very substantially affect the relative likelihood of different threat models. Concretely:
If POTUS is dispositionally power-hungry, I think threat models which are POTUS-led and rely heavily on government capture (e.g. AI-enabled military coups) become far likelier.
On the other hand, if POTUS is not especially power-seeking, I think threat models driven by a lab leader PS (e.g. enforced monopoly) become more likely.
Given that POTUS is so uniquely well-placed to achieve EPC, I also think that her character is the leading driver of the absolute likelihood of EPC.[8]
Concretely, if US presidents during the first decade of the IE have no interest in power concentration whatsoever, I’d guess the absolute likelihood of EPC decreases by >75%.
4.3.2. The speed of the intelligence explosion
Here are some rough rules of thumb I’ve been using when thinking about how the IE affects the likelihood of different threat models:
(1) Lab leader-only threat models seem particularly likely if takeoff is very fast and quite soon.
That’s because fast and soon takeoffs seem like some of the few scenarios where a lab leader PS could achieve a very large capabilities asymmetry between herself and USG, before USG became situationally aware enough to take control of frontier development.
For similar reasons, shortcuts involving early lab capture seem particularly central to achieving EPC in fast takeoff scenarios.
(2) POTUS-led threat models seem more likely than lab leader-led threat models if takeoff is a bit slower and a bit later.
e.g. SAR to TEDAI in ~6mo, starting in mid-2028
That’s because a POTUS PS seems likely to notice such a takeoff happening within the labs (e.g. through surveillance of lab leaders), and the slower timescales give her enough time to engage her various affordances to seize control of the frontier.
(3) Shortcuts involving government capture seem particularly central to somewhat slower takeoff EPC scenarios.
e.g. SAR to TEDAI in ~2yrs
That’s because I expect efforts at government capture to involve various human bottlenecks, suggesting they will take a significant amount of clock time to pay off.
e.g. for PS to replace the top ranks of the DOJ with loyalists, she needs to actually find and loyalty screen those individuals
e.g. for PS to execute an AI-enabled military coup, she might need to meet and persuade the relevant generals to join her coalition
(4) Backsliding dynamics seem important to all but the very fastest takeoff scenarios, because they move PS further along the power concentration scale and enable various shortcuts.
(5) Backslide all the way threat models seem more plausible, relative to shortcut-heavy routes to EPC, if takeoff is very slow.
e.g. >15yrs between SAR and TEDAI
That's because, in slow takeoff scenarios, I centrally expect IPHs (e.g. the coordinate branches) to have meaningful situational awareness and access to advanced frontier capabilities for extended periods of time, enabling them to anticipate and block shortcuts.
This suggests that EPC in such scenarios will not occur due to a rapid powergrab – but rather due to continuous and gradual pressure towards power concentration from the key dynamics in section three.
(6) Collusion-based strategies seem increasingly likely as takeoff slows down.
That’s because, if takeoff is fast, it seems more likely that a single situationally aware PS could succeed with a dash for power before other PSs have time to defend themselves.
e.g. in a fast takeoff, it seems much easier for a lab leader PS to sneak a loyalty into her lab’s model, and then ask its ASI successor to carry out a PS-aligned takeover shortly afterwards
By contrast, if takeoff is slower, it seems likelier that multiple PSs become situationally aware and take steps to safeguard their interests, requiring a compromise between them.
e.g. in a slower takeoff, a POTUS PS might ask a lab leader PS to insert a joint secret loyalty into her model, and the lab leader PS could reasonably fear being detected if she tried to insert a singular loyalty instead.
(7) EPC seems absolutely more likely as takeoff gets faster.
That’s because IPHs will have much less time to anticipate and react to PS's attempts to concentrate power, and are more likely to face truly enormous capabilities asymmetries between themselves and PS.
e.g. PS has TEDAI whilst they have something close to Opus 6
The speed of the software-only intelligence explosion (“SIE”) seems like a particularly important component of overall takeoff speed, given that:
AI progress could move particularly fast during this part of the IE; and
frontier AI seems likely to first reach TEDAI during the SIE.
And takeoff speed overall seems particularly cruxy in worlds where takeoff could be slow enough for certain fixed-duration accountability loops to take effect.
Here’s a graphic setting out what I see as the critical pathways to EPC for a POTUS PS and lab leader PS:
Here are seven claims which underpin this graphic:
(1) I suspect a PS would try to abuse capabilities asymmetries before inserting loyalties. That's because:
capabilities asymmetries seem likely to arise naturally and quite early, due to the potentially rapid pace of SIE and the dangers inherent in distributing advanced capabilities more widely;
whereas PS would likely need to deliberately insert a loyalty, and may not trust her ability to do so without backlash until they have already concentrated power to a significant degree.
(2) I centrally expect PS to use loyalties to get all the way to a domestic DSA. That's because:
loyalties enable PS to reliably and flexibly exercise power in one of the few ways that is not meaningfully bottlenecked on her own human limitations;
i.e. PS doesn’t need to exercise continuous oversight over a model – she can just allow a PS-aligned ASI to do the hard work of taking power
loyalties also enable PS to gain direct control over affordances that seem very hard for her to capture through other means – even with the benefit of superpersuasion and superstrategy.
e.g. a loyalty could enable a lab leader PS to capture an automated military;
e.g. a loyalty could enable a POTUS PS to capture Congress and the courts.
Given this, I have a sense that loyalties are (speaking loosely) more important and central than capabilities asymmetries for PS achieving EPC.
(3) Packing the regulators and the military seem like the most probable government capture shortcuts. That’s because:
packing the regulators seems like one of the quickest ways for a POTUS PS to exercise meaningful influence over frontier AI development prior to nationalisation, and this seems especially valuable for concentrating power;
packing the military allows PS to capture an affordance which most IPHs are very poorly-equipped to defend against, given USG’s near-monopoly on violence.
(4) POTUS-led threat models seem meaningfully more likely than lab leader threat models, given POTUS’s privileged control over key affordances.
(5) That said, I think the most plausible EPC scenarios involve a meaningful amount of collusion between lab leaders and POTUS. That’s because:
it seems quite a bit easier for a POTUS PS to pack the executive if she has a lab leader PS ally willing to supply her with AI henchmen;
it also seems quite a bit easier for a POTUS PS to pack the frontier if she has a lab leader PS ally willing to help run the leading project;
and it also seems far easier for a lab leader PS to evade nationalisation for long enough to get a DSA over USG if she has a POTUS PS ally who is willing to turn a blind eye.
(6) And both POTUS and lab leader PSs seem almost certain to use both government capture and lab capture shortcuts on the way to EPC. That's because:
both institutions control valuable affordances that PS can use to concentrate power;
e.g. a lab leader PS would hugely benefit from capturing the US military
also – until captured – these valuable affordances can be used by IPHs to resist PS.
e.g. a democracy-supporting frontier lab could use supercyber capabilities to disable a POTUS PS’s robot army
That said, a POTUS PS seems particularly likely to try out a government capture shortcut first, and then abuse the affordances of captured government agencies to pull off a lab capture shortcut.
And a lab leader PS seems particularly likely to try out a lab capture shortcut, and then abuse the affordances of the captured lab to capture USG.
So, I centrally imagine that:
a POTUS PS would prioritisepacking key parts of the federal executive (e.g. the military) before turning her full attention to capturing frontier AI development; and
a lab leader PS would prioritise achieving an enforced monopoly before turning her attention to capturing key parts of the federal executive.
(7) To get concrete – the three EPC threat models I find most plausible are:
A POTUS PS packs the military with lab-supplied AI henchmen; does an AI-enabled military coup; seizes control of the frontier AI industry by force; and then inserts a loyalty into the leading model.
A POTUS PS packs the regulators and law enforcement apparatus with lab-supplied AI henchmen; abuses various emergency powers and new frontier AI legislation to pack the frontier; and then inserts a loyalty into the leading model.
A lab leader PS breaks away from her competitors; deploys asymmetric capabilities (and enormous monopoly rents) to sabotage them, conceal AI progress, and deter USG scrutiny; inserts a loyalty into her best model; and then gets it to take over the federal executive.
Of these, I think the AI-enabled military coup scenario is the most plausible, and the lab leader takeover is the least plausible.
4.3.3.1. But will it go exactly like this?
I feel fairly confident that the dynamics I have identified in section three are the key high-level drivers of EPC risk.
If we are in an EPC world, I also expect at least one of the shortcuts I have identified in section four to play a central causal role.
However, I don't feel confident in:
which shortcuts are most causally important in expectation;
how far we might need to backslide before passing thresholds for each shortcut; and
which exact steps different PSs and IPHs are likely to take, in which order – because there are so many distinct actors and institutions in play.
5. What can we do about this?
In this section, I set out my high-level thoughts on the importance, tractability, and neglectedness of EPC, and some different “theories of victory” for work in this area.
I then set out some interventions I’m relatively excited about for reducing EPC risk.
5.1. Importance, neglectedness, tractability
I think EPC is a top-two AI risk:
Concretely, I think it is within an OOM as likely as misaligned takeover, and would be roughly as bad if it happened.
Conditional on us solving alignment, I think it will be the very biggest risk.
I also think EPC is one of the most neglected AI risks:
I estimate there are fewer than 25 AGI-pilled folks working on EPC full time – which seems way too small for the scale of the problem!
Of course, there are many "AI x Democracy" advocates – but only a few of these folks really have their eyes on the ball with regards to transformative AI. Given this, I strongly expect such folks to miss many of the highest-value anti-EPC interventions.
However, I think EPC is quite a bit less tractable than other AI risks (e.g. bio):
That's because almost every anti-EPC intervention either:
has significant backfire risk – because it runs the risk of polarising the topic, or alerting bad actors to opportunities for EPC; or
seems very difficult to implement – because it involves asking people to voluntarily hand over their power.
I think low tractability and sign-uncertainty are marks against almost all of the interventions in this section. Readers should engage with the list with this caveat in mind.
5.2. Theories of victory – how do we prevent EPC?
Here are three plausible ways I think we might be able to get through the IE without EPC:
(1) The best-placed actors don’t realise the opportunity until it’s too late.
e.g. there is an intelligence explosion at a frontier lab with a strong commitment to democracy, and by the time a POTUS PS becomes situationally aware the lab has used its advanced capabilities to outmanoeuvre her politically, blocking a coup
(2) The best-placed actors realise the opportunity, but just don’t try to achieve EPC.
e.g. all the POTUSes throughout the IE have deeply internalised democratic values
(3) The best-placed actors try to achieve EPC but are blocked by IPHs.
e.g. the military spots and blocks all attempts to insert secret loyalties
It’s pretty unclear to me which of these theories are correct. Given this, and given that most plausible EPC interventions don’t seem especially promising, I suspect we should pursue a portfolio approach to tackling this risk.
That is to say, I think we should focus our efforts on any interventions that seem particularly likely to move us towards any of these theories of victory with minimal downsides for the other theories.
e.g. for electoral interventions, we should feel especially good about an intervention which increases the likelihood that a virtuous candidate wins the 2028 presidential race (contributing to a type (2) victory) without increasing the salience of EPC risks amongst presidential candidates (which could reduce the odds of a type (1) victory).
5.3. Interventions
Below, I’ve set out some interventions I’m excited about, keyed to the high-level dynamics in section three. I've marked interventions which seem particularly promising with a [HIGH PRIORITY] tag.
I’ve also included a list of cross-cutting interventions which influence several dynamics all at once. These primarily relate to field-building, reflecting my view that EPC risk is highly neglected right now.
This allows IPHs to check older models for secret loyalties as capabilities advance, and makes it likelier they have access to nearly-as-good older models if they discover a loyalty.
Screen candidates for important positions (e.g. state governorships) for power-seeking, dishonesty and links to other PS, then steer campaign money to lower-risk candidates.
Get sector experts to go deep on key parts of my threat modelling (e.g. how a POTUS PS gets control of a robot army) to help develop more targeted interventions.
5.3.1. Solving the Red and Expert problem
5.3.1.1. Hard-to-remove behavioural guardrails
I'm particularly concerned about the risk that a PS gains unfettered access to AI henchmen.
Given this, I’m particularly excited about finding ways to ensure that most AI deployments in frontier labs and the executive are subject to a minimal set of guardrails which are hard to remove.
e.g. guardrails to ensure that the models are "law-following AIs" (“LFAI”)
“Hard to remove” could just mean that the training process itself is conducted in a way that makes the guardrails difficult to train out of the model.[9] But a more reliable approach would be to apply enhanced governance processes to the development of model specs, weights access, and alignment testing.
e.g. labs could implement various mitigations to guard against secret loyalties
e.g. lab boards could mandate board-level sign-off for any changes to model specs
e.g. the military could require sign-off from officers in multiple branches before any one person can access model weights held on government systems
I think it will ultimately be necessary to legislate for these guardrails (e.g. passing a bill mandating that frontier AIs are law-following) rather than relying on industry consensus.
That's because, otherwise, it seems too easy for a POTUS PS to pressure the labs to remove the guardrails for classified models.
However, there is clearly a place for non-legislative preparatory work within the labs and the AI safety community.
e.g. research to identify the appropriate degree of deference that LFAIs should exercise towards internal counsel
e.g. developing evals to ensure that models are compliant with any guardrails in their model specs
5.3.1.2. Improving incentives for defection
Whistleblowers currently face very high barriers to sharing classified information about dangerous behaviour within the executive. In particular:
Whistleblowers can't legally share Top Secret information with Congress without first alerting the Inspectors General ("IGs") – and a POTUS PS seeking to pack the executive can quite easily replace the IGs with loyalists.
Whistleblowers can only disclose classified information to people already read-in to the project – which does not include the majority of Congress or the public.
Failure to comply with these requirements can lead to criminal prosecution under the Espionage Act.
When I spoke to experts about this, they recommended the following reforms to the Act:
All members of the House and Senate intelligence committees should be deemed "authorised recipients" – enabling whistleblowers to directly disclose Top Secret information to them.
Whistleblowers should be protected by a public interest defense – enabling them to argue that an unauthorised disclosure was justified by the need to prevent serious violations of the law or major threats to public health and safety.
Alternatively, the court should be unable to convict a whistleblower unless the prosecution can show that they had "intent" to commit wrongdoing.[10]
It also seems wise for EPC funders ensure existing government whistleblower orgs are exceptionally well-funded.
5.3.1.3. Let IPHs hold tamper-resistant backups of frontier models
Frontier labs seeking to prevent EPC could store tamper-resistant backups of key model checkpoints on clusters controlled by various IPHs.
e.g. Congress; state governments; the other Five Eyes nations
This would enable IPHs to test old models with new and improved evaluation technologies to check if they were secretly loyal. Also, if a PS inserted a loyalty into the leading model and this was detected quickly, IPHs would still have access to an alternative near-frontier model.
5.3.1.4. Remove single points of failure in the executive
Senior decision-makers in the executive should be encouraged to avoid creating single points of failure ("SPOFs") through incautious AI adoption. In particular:
The use of AI systems in high-stakes scenarios (e.g. domestic deployments of LAWS) should require sign-off by multiple separate humans with distinct reporting chains.
Key parts of the executive (e.g. DOJ litigators) and both of the coordinate branches should be encouraged to procure and use models from multiple frontier developers across their critical functions, to reduce reliance on any one model.
e.g. a department might institute a “two AIs rule” – where any synchronous monitoring of AI agents must be provided by a model from a different lab to the agents themselves
Decisions to use various powers influencing frontier AI development (e.g. flipping a "kill switch") should also require sign-off by multiple parties.
Where practical and politically feasible, measures to avoid SPOFs should be enshrined in Acts of Congress.
Otherwise, a POTUS PS might be able to pack the relevant departments, and then change the underlying procurement or decision-making rules to eliminate the measure.
However, even if they cannot be easily written into statute, these measures seem quite helpful for preventing the capture of key parts of the executive by a non-POTUS PS.
e.g. preventing a rogue-general PS from carrying out an AI-enabled military coup
e.g. preventing a lab leader PS from capturing an entire department just by inserting a secret loyalty
A particularly ambitious goal for removing SPOFs would involve lobbying for an Act of Congress to reform the operational chain of command in the US military.
e.g. ensuring that combatant commanders and law-following AIs can refuse certain power-concentrating orders from a POTUS PS
This is very far outside of the Overton Window – but it’s worth noting that Congress has reformed the chain of command multiple times since the Second World War, so it doesn’t seem wholly implausible.
e.g. Eisenhower’s Department of Defense Reorganization Act 1958
e.g. Goldwater–Nichols in 1986.
I should note that Stefan Torges has written in some detail about the importance of removing SPOFs in the military. I strongly recommend interested readers check out his piece.
5.3.2. Eliminating executive overhang
5.3.2.1. Red-teaming AI legislation for EPC risk and proposing low-risk alternatives
Many popular proposals for AI regulation involve greatly expanding the discretionary powers which the executive has over frontier development
e.g. the Lieu & Moran Kill Switch bill would have enabled the executive to shut down deployments of models which pursue “a goal that is not a goal intended by the developer" or “conceal a capability, intention, or action [from]... monitoring".
e.g. the FRONTIER bill would enable the executive to shut down training for ~1.5mo before any prospect of a court challenge to the merits.
I think a POTUS PS could fairly easily abuse powers like this to coerce labs into supporting her power-seeking schemes.
e.g. threatening to flip a kill switch unless a frontier developer provides her with access to an AI henchman
e.g. repeatedly shutting down training at a disfavoured lab during the SIE to help an allied lab break out in front
Given this, I think someone in the AI safety community should be scrutinising proposed AI legislation to make sure it isn't too power-concentrating, and advocating for alternative approaches which strike a better balance between EPC and other risks.
They could give regular private input on proposed legislative text to 501(c)(4)s in the safety community (e.g. SAIP) and congressional staffers.
They could also try to establish a public profile on X as theEPC red-teamer – posting timely criticisms of different legislative proposals (e.g. the leaked Anthropic bill) as and when they become public.
And they could publish a list of principles for writing legislation that doesn't unduly concentrate power. This might include:
Placing powers in the hands of self-regulatory bodies (e.g. FINRA-for-AI) or state-level regulators (e.g. the California state government).
Requiring a court order to exercise particularly high-stakes powers.
5.3.2.2. Glasswing for the law
SCOTUS has drastically expanded the scope of executive authority over the last century relative to the coordinate branches of USG – such that a POTUS PS would now have a wide array of entirely legal affordances she could abuse to further concentrate her power.
Given this, I think legal experts in the AI safety community should conduct an ambitious effort to identify the key legal ~loopholes which could facilitate a POTUS-led powergrab, and determine the most effective routes to remedy them.
e.g. lobbying for conditions on funding in appropriations bills
e.g. ambitious public outreach efforts to drive constitutional amendments
e.g. funding legal scholarship and impact litigation by conservative organisations seeking to constrain executive power
Here are some powers which seem particularly useful to restrict if at all possible:
executive privilege;
the presidential pardon and presidential criminal immunity;
presidential control over security clearances and classification; and
presidential hiring and firing powers for IGs and independent agency heads.
Legal experts I have spoken to have suggested that targeted efforts in the national security domain might have a particularly large effect on total risk.
e.g. tackling the abuse of the Export Control Reform Act (“ECRA”)
e.g. increasing the procedural requirements on the exercise of various DPA powers
Reforming the chain of command (see above) and enabling Congress to more easily punish contempt (see below) might also fall within scope of such a project.
5.3.2.3. Improving record-keeping and transparency
I expect it would be easier to prevent a PS from abusing her power if IPHs could more easily scrutinise what she was up to.
Below, I've set out four possible avenues for facilitating this kind of transparency.
However, it's worth noting that this is a broad and complex topic which I'm not especially familiar with, and each of these suggestions would benefit from much more detailed research.
(1) Federal record-keeping rules could be expanded to cover agent trajectories.
The Archivist of the United States is responsible for issuing guidance around what counts as a record under the Federal Records Act ("FRA").
FRA guidance doesn’t currently tell agencies to keep records of agent trajectories (i.e. what systems an agent interacted with; which edits it made to which files).
So, in the first instance, the AI safety community should lobby the Archivist to issue guidance that the FRA covers agent trajectories.[11] However:
the Archivist is quite easy for a POTUS PS to fire and replace with a loyalist;
cf. early 2025, when the Archivist was fired and the Secretary of State took over in an acting capacity
and the duty to create records under the FRA sits with agency heads, whilst enforcement goes through the Attorney General – all of whom are similarly easy to replace;
so I'm unsure how robust these efforts would be to a packed executive.
(2) We could encourage Congress to make the most of its oversight powers, and seek to expand them through legislation.
e.g. we could lobby congressional committees to pursue impact litigation around the scope of executive privilege
e.g. we could lobby Congress to pass a bill requiring per diem civil fines against officers who are found in contempt of Congress and have yet to remedy the issue
I gather there are already considerable efforts in this area from folks in the community. I'm happy to put people in touch with the folks running these projects in case of interest.
(3) Congress could institute an automated oversight regime for federal records.
This regime could be designed in accordance with the principles of structured transparency. Here's an illustration of what I have in mind:
Congress-procured AI agents could be granted default access to all federal records.
The agents would automatically flag concerning records for review.
Congressional overseers wouldn't get direct access to the records, but they could ask their agents to query them and receive bounded answers.
e.g. “did any collection target an American without a warrant?” → yes / no / a count
Where the law was ambiguous, the request could also be flagged to an executive branch lawyer for review.
Substantial additional safeguards could be layered onto classified records.
e.g. only a smaller set of congresspeople could query these records, and the agents would have to pass a far higher threshold of safety and security
This could be piloted on a smaller scale within the Government Accountability Office (“GAO”) before being extended across all congressional offices and committees.
Once the system was reliable enough, it could also be extended to Freedom of Information Act ("FOIA") requests, helping the public directly scrutinise the executive.
It's worth noting that this project currently seems far outside of the Overton Window. Experts I've spoken to tell me this would:
be a major technical undertaking;
create a variety of hard-to-remedy information security risks; and
face tremendous pushback from government bureaucrats who (reasonably) believe that too much scrutiny can make it harder to openly debate difficult policy trade-offs.
(4) Frontier AI projects could be subject to extensive transparency requirements, even if those projects weren't technically government bodies.
e.g. federal record-keeping requirements could be expressly extended to any public–private partnership ("PPP") AGI project
e.g. frontier labs could be required to participate in the automated congressional oversight regime which I've set out above
Extensive transparency seems particularly important in the context of a USG-led AGI project, given the likelihood that such a project occurs under a veil of classification.
Given that we’re not sure how exactly any such project might be structured, it could be helpful to set out specific triggers which cause heavier transparency requirements to come into effect.
e.g. USG taking an equity stake in a lab
e.g. the classification of a frontier model's weights
To give these requirements teeth, they could be included in appropriations bills as conditions on funding the project, with funding issued periodically and automatically withheld (or withdrawn) if the project fails to comply.
5.3.2.4. Ban domestic killer robots in the US
The AI safety community should explore proposals making it harder for the US military and law enforcement to deploy lethal autonomous weapons on US soil, to increase barriers to an AI-enabled coup.
e.g. military robots could be geofenced, with their weaponry only becoming usable within the US after sign-off by a majority in both houses of Congress
I haven’t spent much time thinking about this idea, so I’m particularly uncertain how technically and politically feasible it might be.
5.3.3. The asymmetric frontier
5.3.3.1. Pacing the frontier well
As I note above, EPC risks seem greatest in fast takeoff worlds, because this could lead to large disparities between PS and IPHs in their access to frontier capabilities.
Given this, pacing the frontier seems like a particularly good way to reduce EPC risks – provided that any legislation which enables pacing doesn't inadvertently centralise power in the executive.
5.3.3.2. Better Out Than In laws
Another promising approach to reducing frontier asymmetries is what I'm calling "Better Out Than In" laws – which seek to restrict what a frontier lab or the executive can do with an AI model before it has to be made more widely available. For example:
Labs could be barred from internal-only deployments of their models – save for the purposes of safety testing and cybersecurity work.
Once models meet a preliminary safety and security threshold, labs could then be required to make their models available to all three branches of USG and all state governments.
And once models meet a higher threshold, they could be required to make them available to any organisations found by a federal court to meet relevant safety requirements – and then the rest of society.
The executive could be similarly restricted to safety and security testing during any national security screening period.
Access to classified models could still be granted to judges, elected politicians, state government officials, congressional staffers, and federal bureaucrats who have the relevant clearances.
These laws might also have an indirect benefit of slowing capabilities progress at the frontier, because labs would have to dedicate so much effort to safety testing before being permitted to train a new model.
5.3.3.3. Turbocharging IPHs with AI
Both Congress and the judiciary have considerable powers to constrain the executive which they seem unlikely to fully exercise during the IE.
e.g. a human-led judiciary may struggle to consider the vast quantity of legal disputes generated by the authoritarian behaviour of an AI-enabled executive
e.g. a human-led Congress may struggle to identify all the ways that a POTUS-supported legislative proposal concentrates power and, in a crisis, may struggle to coordinate around executive-constraining amendments to such legislation
Here are some ways it might be possible to remedy this issue:
The AI safety community could lobby Congress to radically increase its (and the judiciary’s) IT budgets, and to earmark this for token spend.
We could massively increase funding for Horizon / TechCongress, and encourage them to embed well-paid forward-deployed engineers in key coordinate branch bodies.
e.g. the House and Senate intelligence committees; the DC District Court; SCOTUS
Congress could pass a Better Out Than In law (see above), providing that any model made available to any part of the executive for any purpose must also be made available to itself and the judiciary at the same cost per token.
I suspect similar AI uplift work should also be done in key state and local governments (e.g. California) to ensure that they can continue to exert an effective check on the frontier AI industry throughout the IE.
5.3.4. Desperate times calling for desperate measures:
I haven’t spent much time thinking about how to intervene on this dynamic, so my views are quite underdeveloped. Here are a few lower-confidence ideas I’ve come up with.
5.3.4.1. Preserve household economic security
Once the IE has begun, USG could undertake a variety of radical economic reforms to protect US citizens from economic disruption.
e.g. instituting a UBI programme
e.g. imposing a moratorium on redundancies and pay cuts
e.g. transferring voting shares in key parts of the frontier AI industry to all citizens
e.g. allocating a minimum percentage share of all frontier AI tokens to inference by private citizens, enforceable by class action
e.g. imposing an interest rate ceiling on the largest sources of variable-rate personal debt (e.g. credit cards).
If executed effectively, I suspect this could meaningfully increase perceptions of personal economic security throughout the IE.
I expect quite a bit of work has already been done on the most legally and economically robust ways to deliver protections like this. I suspect the AI safety community should continue to fund such work with an eye to developing proposals that would deliver on these objectives without inadvertently concentrating power.
e.g. ensuring citizens have a right to frontier tokens directly enforceable against the labs, rather than having to rely on an executive-run UBI process
5.3.4.2. Virtue screening the powerful
A 501(c)(4) could build an AI "virtue screening" tool which would conduct a deep-dive on candidates for key roles in federal, state, and local governments.
e.g. state judges and governors
e.g. state and federal legislators
e.g. presidential candidates
The tool would continuously scan publicly-available sources for evidence of power-seeking traits:
e.g. power-seeking statements or actions;
e.g. dishonesty and corruption;
e.g. close associations with other power-seekers.
The outputs of this tool could then inform political spending by big donors in the community.[12] This could increase the likelihood that a POTUS PS and her political allies are simply not on the ballot to be selected by concerned voters during desperate times.
The same tool could also be used to conduct a smaller investigation into the leadership of key parts of the AI industry (e.g. frontier labs; hyperscalers; chipmakers) to help inform decisions by investors in the community.
5.3.5. Widespread epistemic degradation
5.3.5.1. Epistemic tooling
I think the best way to reduce risks from epistemic degradation is to ensure that IPHs and the general public retain the capacity to verify what is true and false throughout the IE.
I think one particularly promising way to achieve this is to found projects that accelerate the arrival of various epistemic tools that are likely to arrive during the IE, and ensure that they are widely distributed as quickly as possible.
Here are some tools which I think could be particularly effective:
5.3.5.2. Disclaimers for synthetic political content
Congress (and swing state governments) could pass laws requiring annoying, medical advertisement-style disclosures for any AI-generated content broadcast or posted on social media by a corporation in relation to a public figure, with criminal penalties for non-compliance.
e.g. campaign ads containing AI-generated video of political rivals would need to state at the end of the advertisement that this part of the ad was AI-generated, or the producers would face a criminal fine
Disclaimers of this sort would hopefully make it easier for voters and IPHs to effectively scrutinise their leaders.
e.g. it would be riskier for an unscrupulous PS to fabricate a scandal about her opponent in a presidential race
5.3.6. Cross-cutting interventions
As I note above, most of the interventions in this section aim to build the field of people working to prevent extreme power concentration. I think field-building interventions are particularly promising because EPC risks are currently so neglected.
I suspect we should prioritise field-building amongst the following groups, in decreasing order of priority:
AI safety community (esp. in DC)
Sympathetic NatSec professionals and federal bureaucrats
Constitutional / administrative lawyers
Centrist politicians (esp. in Congress)
The general public
Most of the measures below are intended to reach one or more of these priority groups.
5.3.6.1. Write up better intro materials for the AI safety pipeline
The bar for getting up to speed on EPC is currently very high. To get a sense of the most plausible threat models and most promising interventions, you need access to a bunch of private documents and a high-context interlocutor who is willing to give you a significant amount of their time.
I believe this is a major barrier to people getting involved in the field. Given this, I expect it would be worth someone’s time to write up high-quality intro materials for different audiences (e.g. seasonal fellows; BlueDot classes; think tank leads), setting out the basic threat models and most promising interventions.
5.3.6.2. Pitch people to do more on EPC
I think it would make sense for one of the people currently working on EPC to dedicate a majority of their time to pitching leaders in the broader AI safety community on doing more work on EPC. This might look like:
organising lots of calls with leaders at influential AI safety organisations; and
presenting them with the basic threat models, arguments for why they are particularly plausible, and a clear menu of actions to take:
e.g. doing expert threat modelling
e.g. setting up an EPC workstream
e.g. recruiting seasonal fellows for EPC projects
e.g. publishing articles on EPC topics
e.g. developing our highest priority interventions
e.g. lobbying congresspeople to adopt our interventions
I also suspect it would be worth engaging in targeted, high-effort “wake-up” efforts for the most important IPHs outside the AI safety community – including:
eminent constitutional and administrative lawyers.
e.g. a major AI safety funder could support conservative legal scholars to do research centred on constraining the executive in the AGI era.
centrist and centre-right politicians in Congress and key state governments;
e.g. AI safety lobbying orgs could prepare primers on EPC for the House and Senate intelligence committees, coupled with a few particularly promising anti-EPC interventions Congress could implement
sympathetic senior officials in key parts of USG;
e.g. the DOJ and DOW
5.3.6.3. Terminator for EPC
An AI safety grantmaker (e.g. cG or Longview) could headhunt a writer-director and experienced producer who are genuinely interested in AI safety, and well-known enough that studios will actually read their script.
They could then fund them through the early stages of developing a feature film or TV show about AI-enabled EPC occurring in the US.
e.g. up until the first draft of the screenplay is written
Alternatively, someone in the community could pitch Black Mirror (or a similar futurist show) on writing an episode about AI-enabled EPC.
The goal here would be to develop compelling audiovisual fiction which becomes a cultural touchpoint in relation to EPC, increasing IPHs' situational awareness and enabling them to recognise and push back against power-seeking sooner.
5.3.6.4. Federalist Papers 2.0
Someone could set up a newsletter which publishes anonymous articles about EPC, written by people who would face substantial risks from open publication.
e.g. lab staff
e.g. government employees
e.g. researchers at risk-averse think tanks
e.g. citizens of autocratic regimes
This could hearken back to the original Federalist Papers – which were published pseudonymously by several Founding Fathers at a time of great change in US history.
5.3.6.5. AIFP for EPC
Someone could set up an organisation to write a plausible worst-case scenario for EPC occurring in the US by the end of the decade, similar in form, substance and writing style to AI 2027.
They could try hard to make the piece go viral – aiming for wide readership across senior staff at AI labs, leaders at key civil society orgs, and high-ranking politicians.
And then they could follow up within 3-6 months with detailed policy recommendations for safeguarding US institutions, and a Plan A-style positive vision for how things could go if these recommendations were adopted.
This work would aim to have an impact in roughly the same way AI 2027 did for loss of control – massively raising the salience of EPC with elites, giving the cause a public face, and creating a Schelling point of policy recommendations to rally around.
5.3.6.6. Set up the EPC org
Someone could set up an org whose role is to act as a home for research on EPC and a hub to connect people in the field.
It could produce its own research on extreme power concentration, as well as doing more rapid response policy work.
e.g. AI 2027 and Plan A for EPC
e.g. EPC red-teaming of draft AI safety regulations
It could also lead outreach to the AI safety community and IPHs, aiming to alert them to the risks of EPC and help them resist it.
e.g. running EPC-themed TTXs
e.g. running EPC-themed conferences and retreats
e.g. coordinating with lobbyists to increase uptake of promising interventions
And it could maintain other useful community infrastructure.
e.g. the Federalist Papers 2.0
e.g. operating a fellowships programme to bring junior researchers and entrepreneurs into the field
5.3.6.7. Detailed threat modelling by sector experts
Many EPC threat models (including the ones above) are sorely lacking in detail. This is a problem, because:
it makes the threat models unconvincing to IPHs (e.g. high-ranking military officers), who might then fail to take adequate measures to protect against PS; and
it also makes it harder to develop well-targeted interventions.
Given this, I think the field would greatly benefit from in-depth, expert threat modelling of the most important elements of the most important EPC threat models.
e.g. how exactly a POTUS PS could gain unified control over a robot army without massive pushback from her generals
e.g. through which legal mechanisms a POTUS PS could plausibly “soft-nationalise” the frontier labs
Glasswing for the law (see above) is a particularly fleshed out example of what I have in mind here. But I suspect this would actually be most helpful in the military domain, given that our threat models seem both particularly plausible and especially underdeveloped.
5.3.7. Which of these interventions seem most promising?
I think the following interventions seem reasonably tractable and particularly likely to further at least one of my three theories of victory without compromising the others.
Hard-to-remove behavioural guardrails (esp. a legal mandate for LFAI)
Improving record-keeping and transparency(esp. as applied to any USG-led AGI project)
Red-teaming AI legislation for EPC risk (esp. if coupled with principles for writing AI legislation in a way that doesn't unduly concentrate power)
Better Out Than In laws (esp. for Congress and the judiciary)
Virtue screening the powerful (esp. for primary candidates in high-stakes races)
Setting up the EPC org(esp. if it has a field-building focus)
Detailed threat modelling by sector experts(esp. military threat modelling)
6. Weaknesses in this work
Here are a few points where I think what I’ve written above is particularly weak.
(1) I don’t yet have a good sense of how, precisely, a POTUS PS could gain sufficiently fine-grained control of frontier AI development to:
create durable asymmetries in access between her allies and the rest of society; and/or
gain consistent and reliable access to an AI henchman.
I’m a bit sceptical that the current set of emergency powers (e.g. the DPA) would be sufficient to achieve this, and I suspect that new legislation would be required. But I’m not confident about this – and I think the topic would benefit from a deep-dive.
(2) I don’t yet have a detailed sense of how exactly a PS could abuse capabilities asymmetries to concentrate power to an extreme degree.
I think this is an inherently difficult question to answer, for roughly the same reasons as it’s hard to give a detailed answer for how misaligned takeover might happen.
As in – it’s hard to know whether there’s a ceiling on these capabilities, where that ceiling might be, and what it looks like for an actor to be radically more capable along these dimensions
That said, I think my current threat modelling reads a bit like:
“PS gets superpersuasion and superstrategy and then just one-shots everyone”
And this feels unacceptably handwavey.
(3) I’ve received limited expert input on most of the interventions I propose.
Given this, I expect them to be imperfectly targeted and fail to account for important practical limitations. In particular, I suspect I’m too confident about the prospects for passing various pieces of legislation.
(4) This post focuses exclusively on domestic EPC and ignores how a single actor could get a DSA over the entire world.
This is a significant omission, because the likelihood that a post-ASI US could dominate all other actors seems really very cruxy for the overall badness of domestic EPC.
I also think this focus on domestic concerns may have caused me to underweight:
the need for PS to capture other parts of the AI stack which sit outside of the US;
e.g. chipmakers; makers of key machine tools
the likelihood of various space-based routes to EPC.
e.g. a first mover in space gains a DSA by hoarding access to space resources
I also find it plausible that we could see a backslide all the way scenario wherein a lab leader PS uses her asymmetric access to frontier capabilities and vast AI-generated wealth to corrupt political processes and gain effective control of USG.
It's worth stressing that I'm using the term "regulators" in a broad and non-standard way here, to refer to any government agency with substantial discretionary authority over the frontier AI industry – whether or not it would be traditionally thought of as a regulator.
To be clear – I also think the character of lab leaders matters quite a bit, but it seems less useful to focus on this because (i) lab leadership seems less likely to change in the near-term, and (ii) I think lab leader-driven EPC accounts for <1/4 of the total risk.
For example, I’ve heard rumours that GDM struggled for several months to train a genuinely “helpful-only” model for the DOW. But it seems like this was an unusual situation – and I get the impression it will be increasingly easy to simply train guardrails out of models in future.
It's worth noting that record-keeping in the White House and parts of the Executive Office of the President ("EOP") is separately governed by the Presidential Records Act ("PRA") during a president's term. White House Counsel issues similar guidance in relation to PRA compliance – and this might also need to be extended to cover agent trajectories.
Experts tell me that primary candidates benefit far more than candidates in general elections from increased funding – so it might be worth prioritising screening primaries at first.
1. What’s this post about?
I work on preventing extreme power concentration ("EPC") at a well-known AI safety org. Lots of people are worried about EPC, but AFAICT there is no single online resource setting out the key threat models and most promising interventions.
This post is my attempt to do that. It sets out...
What I’m worried about:
The key high-level dynamics which I think generate EPC risk. These are:
The threat models I’m worried about:
What we can do about these threat models:
1.1. Epistemic status
This post is quite light on argumentation, because I’m aiming to set out my all-things-considered view on how EPC might go – rather than presenting highly robust arguments for exactly why I think this.
2. What am I worried about?
When I say I’m worried about EPC, I mean I’m worried about the risk that:
2.1. Why am I worried about this?
It seems more likely than not that the intelligence explosion ("IE") kicks off in the next decade, and entirely plausible that it begins in the next 6 months.
During the IE, it might become possible to lock in various features of Earth-originating civilisation.
Power-seekers seem negatively selected for values which could promote a near-best future, because:
Power also tends to corrupt. Even if power-seekers started out with acceptably altruistic and morally-reflective values, I’d expect them to become more selfish, vain, greedy, etc. over time.
So – a power-seeker gaining a decisive strategic advantage ("DSA") during the IE seems particularly likely to lead to harmful lock-in.
2.2. Am I worried about all kinds of power concentration?
No – and I think it's worth stressing that concentration of power isn't necessarily bad in all cases.
First, it matters a lot where power is concentrated, because some groups seem substantially less likely to promote bad values.
Second, making AI go well might require substantial power concentration, albeit on a temporary basis.
So, I think it would be a mistake for us to try to generically prevent AI-enabled power concentration. Instead, I think we should focus on stopping the most extreme and durable forms of power concentration in the very worst actors.
2.3. Where am I worried about this happening?
I’m particularly worried about EPC occurring within:
I’m particularly worried about these actors because:
2.3.1. A note on terminology
For the rest of this post, I'm going to refer to power-seekers in the abstract – as a "PS", with feminine pronouns.
3. What are the key dynamics generating EPC risk?
I think there are five key high-level dynamics driving EPC risks. I expect each of these to apply continuously throughout the IE, and to intensify as AI becomes more powerful.
3.1. Solving the Red and Expert problem
For autocrats to maintain and increase their power, it is best if they have allies and subordinates who are both extremely loyal and exceptionally competent.
Historically, autocrats have had to make hard trade-offs in this regard:
However, transformative AI seems likely to ~solve the Red and Expert problem – making it easier for a PS to optimise for loyalty in the organisations she leads, without sacrificing those organisations' operational effectiveness. That's because it may enable PS to:
I've explained each of these in a bit more detail below.
3.1.1. The role of AI henchmen:
During the IE, AIs will take on a continuously expanding share of cognitive labour in PS-controlled organisations.
AI labour will be highly competent, and it may be fairly easy for PS to gain access to AIs which will follow even her most authoritarian orders without scruple (“AI henchmen”).[2] Here are some ways PS might get access to AI henchmen:
(1) PS may have substantial access to so-called "helpful-only" AIs, which are subject to a minimum of behavioural guardrails.
(2) PS might be able to insert loyalties into the AIs under her control – ensuring they comply with her orders even when this is clearly illegal or unethical.
(3) PS might find ways to abuse AI labour which is neither helpful-only nor loyal to her ("good guy AIs"). That's because:
3.1.2. The role of loyalty screening
Transformative AI will also make it increasingly easy to conduct loyalty screening of human subordinates, via:
Within USG, this issue may be exacerbated by:
3.1.3. The consequence – AI-enabled packing of institutions
Taken together, this may enable a PS to gain an unprecedented degree of control over key parts of USG and other power-holding institutions by filling them with loyalists and AI henchmen. I refer to this as "AI-enabled packing".
AI-enabled packing seems most concerning in the military, national security apparatus, and law enforcement – where PS might be able to replace or disempower government employees who can currently exercise a real check on autocratic behaviour.
But packing risks also seem quite acute in the context of:
3.2. Eliminating executive overhang
The executive branch controls an exceptionally wide range of political, economic, ideological, and coercive affordances.
A key constraint on the use of these affordances right now is an undersupply of cognitive labour. As in:
Transformative AI promises to radically reduce executive overhang by drastically increasing the executive's supply of cognitive labour. This promises to greatly increase state capacity; but it may also enable an unscrupulous PS to abuse the executive's affordances to an unprecedented degree.
3.2.1. Legal overhang
Reduced executive overhang seems particularly concerning because US law already grants POTUS and her officers a wide range of expansive and highly-discretionary legal powers which seem ripe for abuse in a world with transformative AI.
This issue only seems to be growing over time, because the current SCOTUS majority has consistently favoured expansive interpretations of presidential powers.
3.2.2. Extralegal overhang
Transformative AI also seems likely to increase the executive's capacity to engage in straightforwardly unlawful behaviour, and to do so with an unprecedented degree of sophistication.
3.2.3. Who benefits from reduced executive overhang?
This seems likely to differentially uplift a POTUS PS relative to a lab leader PS. However, it could also meaningfully uplift a lab leader PS who:
3.2.4. h/t LawAI
Finally, I should note that Cullen O'Keefe, Alan Rozenshtein, and Christoph Winter have already written at some length on both the Red and Expert problem and Executive overhang – in more detail and at much higher quality than I have here.
I highly recommend that interested readers take a look at their work.
3.3. The asymmetric frontier
I expect frontier lab staff, POTUS, and key parts of the national security apparatus to gain privileged access to the most advanced models before most of the rest of society.
If capabilities are increasing very quickly, this could create extremely large capabilities asymmetries between these actors and most of the rest of society.
A lab leader PS might be able to expand these asymmetries simply by concealing the existence of her very best models from actors who might want to use them.
Whereas a POTUS PS may be able to expand these asymmetries by abusing executive overhang.
A POTUS or lab leader PS could take advantage of these asymmetries to radically increase her power in a short space of time.
3.3.1. Strained accountability loops
One particularly concerning risk of access asymmetries is the straining of standard “accountability loops” necessary for the functioning of democracy. To explain:
3.4. Desperate times calling for desperate measures
It seems likely that the IE will be very weird and scary for most ordinary people. That’s because it seems likely that we’ll see some or all of the following:
In unstable times and places, people tend to be more accepting of draconian behaviour by politicians who promise to restore peace and order.[4] Given this, I expect the general public to be:
3.5. Widespread epistemic degradation
As we move deeper into the IE, it may become increasingly difficult for people to understand the world around them. There are three key reasons for this:
(1) The world will simply be moving much faster than it was before.
(2) Advanced AI may be able to generate compelling deepfakes, which could:
(3) Many major events will be driven by AIs, who need not share what they are doing with anyone else – unlike humans, who are likelier to leak major projects.
I'm referring to this phenomenon as "epistemic degradation".
Epistemic degradation may concentrate power by weakening accountability for PS and her allies – because the public, courts, legislators, journalists, and other IPHs will struggle to notice, investigate, and sanction her misconduct.
3.6. Which of these dynamics are most important?
Overall, I suspect that:
Here's my reasoning for this ranking...
I've found it really quite hard to construct highly-plausible EPC threat models which don’t involve some amount of AI-enabled packing and frontier asymmetry. It just seems really very likely that PS would want to:
Elimination of executive overhang also feels like a core driver of power-concentration in the majority of the most-plausible EPC threat models.
By contrast, demand for desperate measures and epistemic degradation feel like additional elements that increase the plausibility of various threat models, without being especially load-bearing.
And I think that epistemic degradation, in particular, matters quite a bit less in expectation than the other four dynamics, because:
That said, all five dynamics seem plausible enough – and plausibly important enough – that we should treat them as key drivers of EPC risk.
4. What are the most concerning EPC threat models?
In this section, I sketch a high-level framework for thinking about EPC threat models, and then set out some particularly plausible threat models which fall out of the framework.
4.1. High-level framework
I find it helpful to think of power concentration as a scale, which runs all the way from radical AI-enabled pluralism at one end, to AI-enabled extreme power concentration at the other:
Contemporary US democracy sits somewhere closer to the pluralistic end of the scale – albeit being quite far from a maximally decentralised regime.
High-level structural conditions, and concrete actions of PSs and IPHs, can move the US towards or away from power concentration.
Once the US passes certain thresholds on the scale, it becomes possible to jump part or all of the way towards EPC – if PS takes control of particularly important sets of affordances (“shortcuts”).
4.2. Threat models
4.2.1. Backsliding all the way
Within this framework, it seems at least plausible that PS could get all the way to EPC simply by slow and continuous movement along the scale.
Here's one very high-level illustration of how this might happen:[5]
One argument in favour of backslide all the way threat models which I find especially compelling is that most of the individual steps don’t require PS to be actively trying to achieve EPC.
This seems more congruent with the psychological profiles of contemporary political and lab leaders than the "shortcuts" to EPC which I set out below.
That said, I still think backsliding all the way is less likely than other EPC threat models. That’s because:
4.2.2. Shortcuts
I think shortcuts can be sorted into two broad categories:
4.2.2.1. Government capture shortcuts
Under a government capture shortcut, PS aims to:
One particularly plausible shortcut is the AI-enabled military coup – whereunder PS:
Another particularly plausible shortcut is “packing the regulators”[6] – whereunder PS:
Two other parts of the executive also seem particularly worth packing – the intelligence community (“IC”) and the law enforcement apparatus (esp. key parts of the DOJ). That’s because:
Government capture shortcuts seem particularly easy for a POTUS PS, because she has direct influence over senior appointments within the executive branch.
4.2.2.2. Lab capture shortcuts
Under a lab capture shortcut, PS aims to:
These shortcuts rely on two distinct threat vectors:
(1) Loyalty insertion (i.e. determining the alignment target of the leading model, and then relying on the model to concentrate power on your behalf)
(2) Capabilities asymmetries (i.e. gaining exclusive access to key capabilities which can be exploited to further concentrate power). It’s hard to say in advance what these key capabilities might be – but here are three strong candidates:
I expect PS’s identity to greatly influence the most likely shortcut.
If PS is a lab leader, she seems most likely to rely on an enforced monopoly, whereunder:
If PS is a POTUS, she seems most likely to pack the frontier – abusing key legal authorities to fill the leading AI project with loyalists and AI henchmen. Centrally, I’d expect her to:
4.2.3. POTUS-lab leader collusion
I find collusion between POTUS PSs and lab leader PSs to be a particularly plausible variant of all of the threat models that I've laid out above.
That's because both sides control valuable affordances that can benefit the other, such as:
POTUS
Lab leaders
It’s hard to say exactly how any collusion might go, but here are a few scenarios I find particularly plausible:
4.3. Which threat models seem likeliest?
I think there are two key determinants of the most likely threat models. These are:
4.3.1. The character of POTUS
I think the median POTUS PS would be substantially better placed to achieve EPC than a lab leader PS. That’s because POTUS has control of an extremely valuable set of affordances which seem uniquely easy to exploit and uniquely difficult for other actors to capture.
By contrast, the key affordance of a lab leader is her access to frontier capabilities – and she may struggle to maintain asymmetric access to this affordance once a POTUS becomes adequately situationally aware.
Given this, I think the character of whoever happens to be POTUS during the IE (and the characters of their close allies) will very substantially affect the relative likelihood of different threat models. Concretely:
Given that POTUS is so uniquely well-placed to achieve EPC, I also think that her character is the leading driver of the absolute likelihood of EPC.[8]
4.3.2. The speed of the intelligence explosion
Here are some rough rules of thumb I’ve been using when thinking about how the IE affects the likelihood of different threat models:
(1) Lab leader-only threat models seem particularly likely if takeoff is very fast and quite soon.
That’s because fast and soon takeoffs seem like some of the few scenarios where a lab leader PS could achieve a very large capabilities asymmetry between herself and USG, before USG became situationally aware enough to take control of frontier development.
For similar reasons, shortcuts involving early lab capture seem particularly central to achieving EPC in fast takeoff scenarios.
(2) POTUS-led threat models seem more likely than lab leader-led threat models if takeoff is a bit slower and a bit later.
That’s because a POTUS PS seems likely to notice such a takeoff happening within the labs (e.g. through surveillance of lab leaders), and the slower timescales give her enough time to engage her various affordances to seize control of the frontier.
(3) Shortcuts involving government capture seem particularly central to somewhat slower takeoff EPC scenarios.
That’s because I expect efforts at government capture to involve various human bottlenecks, suggesting they will take a significant amount of clock time to pay off.
(4) Backsliding dynamics seem important to all but the very fastest takeoff scenarios, because they move PS further along the power concentration scale and enable various shortcuts.
(5) Backslide all the way threat models seem more plausible, relative to shortcut-heavy routes to EPC, if takeoff is very slow.
That's because, in slow takeoff scenarios, I centrally expect IPHs (e.g. the coordinate branches) to have meaningful situational awareness and access to advanced frontier capabilities for extended periods of time, enabling them to anticipate and block shortcuts.
This suggests that EPC in such scenarios will not occur due to a rapid powergrab – but rather due to continuous and gradual pressure towards power concentration from the key dynamics in section three.
(6) Collusion-based strategies seem increasingly likely as takeoff slows down.
That’s because, if takeoff is fast, it seems more likely that a single situationally aware PS could succeed with a dash for power before other PSs have time to defend themselves.
By contrast, if takeoff is slower, it seems likelier that multiple PSs become situationally aware and take steps to safeguard their interests, requiring a compromise between them.
(7) EPC seems absolutely more likely as takeoff gets faster.
That’s because IPHs will have much less time to anticipate and react to PS's attempts to concentrate power, and are more likely to face truly enormous capabilities asymmetries between themselves and PS.
The speed of the software-only intelligence explosion (“SIE”) seems like a particularly important component of overall takeoff speed, given that:
And takeoff speed overall seems particularly cruxy in worlds where takeoff could be slow enough for certain fixed-duration accountability loops to take effect.
4.3.3. Overall takes
Here’s a graphic setting out what I see as the critical pathways to EPC for a POTUS PS and lab leader PS:
Here are seven claims which underpin this graphic:
(1) I suspect a PS would try to abuse capabilities asymmetries before inserting loyalties. That's because:
(2) I centrally expect PS to use loyalties to get all the way to a domestic DSA. That's because:
Given this, I have a sense that loyalties are (speaking loosely) more important and central than capabilities asymmetries for PS achieving EPC.
(3) Packing the regulators and the military seem like the most probable government capture shortcuts. That’s because:
(4) POTUS-led threat models seem meaningfully more likely than lab leader threat models, given POTUS’s privileged control over key affordances.
(5) That said, I think the most plausible EPC scenarios involve a meaningful amount of collusion between lab leaders and POTUS. That’s because:
(6) And both POTUS and lab leader PSs seem almost certain to use both government capture and lab capture shortcuts on the way to EPC. That's because:
That said, a POTUS PS seems particularly likely to try out a government capture shortcut first, and then abuse the affordances of captured government agencies to pull off a lab capture shortcut.
And a lab leader PS seems particularly likely to try out a lab capture shortcut, and then abuse the affordances of the captured lab to capture USG.
So, I centrally imagine that:
(7) To get concrete – the three EPC threat models I find most plausible are:
Of these, I think the AI-enabled military coup scenario is the most plausible, and the lab leader takeover is the least plausible.
4.3.3.1. But will it go exactly like this?
I feel fairly confident that the dynamics I have identified in section three are the key high-level drivers of EPC risk.
If we are in an EPC world, I also expect at least one of the shortcuts I have identified in section four to play a central causal role.
However, I don't feel confident in:
5. What can we do about this?
In this section, I set out my high-level thoughts on the importance, tractability, and neglectedness of EPC, and some different “theories of victory” for work in this area.
I then set out some interventions I’m relatively excited about for reducing EPC risk.
5.1. Importance, neglectedness, tractability
I think EPC is a top-two AI risk:
I also think EPC is one of the most neglected AI risks:
However, I think EPC is quite a bit less tractable than other AI risks (e.g. bio):
I think low tractability and sign-uncertainty are marks against almost all of the interventions in this section. Readers should engage with the list with this caveat in mind.
5.2. Theories of victory – how do we prevent EPC?
Here are three plausible ways I think we might be able to get through the IE without EPC:
(1) The best-placed actors don’t realise the opportunity until it’s too late.
(2) The best-placed actors realise the opportunity, but just don’t try to achieve EPC.
(3) The best-placed actors try to achieve EPC but are blocked by IPHs.
It’s pretty unclear to me which of these theories are correct. Given this, and given that most plausible EPC interventions don’t seem especially promising, I suspect we should pursue a portfolio approach to tackling this risk.
That is to say, I think we should focus our efforts on any interventions that seem particularly likely to move us towards any of these theories of victory with minimal downsides for the other theories.
5.3. Interventions
Below, I’ve set out some interventions I’m excited about, keyed to the high-level dynamics in section three. I've marked interventions which seem particularly promising with a [HIGH PRIORITY] tag.
I’ve also included a list of cross-cutting interventions which influence several dynamics all at once. These primarily relate to field-building, reflecting my view that EPC risk is highly neglected right now.
Intervention
Explanation
Solving the Red and Expert problem
[HIGH-PRIORITY]
Hard-to-remove behavioural guardrails
Make sure most AI used in frontier labs and the executive comes with a minimal set of guardrails (e.g. law-following AI).
These should be enabled by behavioural evals, and coupled with governance processes that make them hard to strip out.
Improving incentives for defection
Let insiders take Top Secret info straight to Congress, bypassing the (easy-to-pack) Inspectors General.
Widen Espionage Act protections for whistleblowers.
Ensure existing whistleblower protection orgs are very well-funded.
Let IPHs hold tamper-resistant backups of frontier model weights
This allows IPHs to check older models for secret loyalties as capabilities advance, and makes it likelier they have access to nearly-as-good older models if they discover a loyalty.
Remove single points of failure in the executive
Require multi-person sign-off for high-stakes AI use cases, and require the executive to procure and use models from multiple labs.
Eliminating executive overhang
[HIGH-PRIORITY]
Red-teaming AI legislation for EPC risk
Many popular AI bills hand the executive easy-to-abuse powers (e.g. kill switches).
Someone should proactively red-team them and push for more robust alternatives.
Glasswing for the law
Fund a coordinated push to uncover the legal loopholes a PS could use to concentrate power and close them through legislation or impact litigation.
[HIGH-PRIORITY]
Improving record-keeping and transparency
At a minimum, agent trajectories should be treated as federal records.
Congress should also impose extensive transparency conditions on the funding of any USG-led AGI project.
And USG should move towards an automated oversight regime as quickly as possible.
Ban domestic killer robots
We should explore proposals making it harder for the US military and law enforcement to deploy LAWS on US soil
The asymmetric frontier
Pacing the frontier well
Slower takeoff means reduced capabilities asymmetries and more time for IPHs to react to powergrabs.
[HIGH-PRIORITY]
Better Out Than In laws
Frontier labs and the executive should be prohibited from conducting internal-only deployments, save for the purposes of safety and security research.
Turbocharging IPHs with AI
Congress, courts and the states should get bigger token budgets, and the same models as the executive on the same timelines.
Desperate times calling for desperate measures
Preserve household economic security
Take steps to cushion the economic shocks which the IE could create for ordinary people through policies like "citizens' shares" in the AI industry.
[HIGH-PRIORITY]
Virtue screening the powerful
Screen candidates for important positions (e.g. state governorships) for power-seeking, dishonesty and links to other PS, then steer campaign money to lower-risk candidates.
Widespread epistemic degradation
Epistemic tooling
Accelerate the arrival of tools like ambient superforecasting and lie detection, and place them in the hands of IPHs.
Disclaimers for synthetic political content
Require annoying, medical ad-style disclaimers on AI-generated political content, backed up by legal penalties.
Cross-cutting interventions
Write up better intro materials for the AI safety pipeline
Write up materials setting out the basic threat models and most promising interventions, to help sympathetic people get up to speed faster.
Pitch people to do more on EPC
Pitch sympathetic IPHs and influential folks in the AI safety community to work more on preventing extreme power concentration.
Terminator for EPC
Help kickstart a feature film or television show about EPC occurring in the US.
Federalist Papers 2.0
Set up a newsletter about EPC that takes anonymous submissions from lab staff, USG officials and cautious think-tankers.
AIFP for EPC
Write a detailed AI 2027-style EPC scenario, built to go viral with elites.
Follow this 3-6mo later with policy recommendations and a Plan A-style positive vision for US democracy.
[HIGH-PRIORITY]
Set up the EPC org
Set up a dedicated EPC org, which both conducts its own research and takes the lead on EPC field-building.
[HIGH-PRIORITY]
Detailed threat modelling by sector experts
Get sector experts to go deep on key parts of my threat modelling (e.g. how a POTUS PS gets control of a robot army) to help develop more targeted interventions.
5.3.1. Solving the Red and Expert problem
5.3.1.1. Hard-to-remove behavioural guardrails
I'm particularly concerned about the risk that a PS gains unfettered access to AI henchmen.
Given this, I’m particularly excited about finding ways to ensure that most AI deployments in frontier labs and the executive are subject to a minimal set of guardrails which are hard to remove.
“Hard to remove” could just mean that the training process itself is conducted in a way that makes the guardrails difficult to train out of the model.[9] But a more reliable approach would be to apply enhanced governance processes to the development of model specs, weights access, and alignment testing.
I think it will ultimately be necessary to legislate for these guardrails (e.g. passing a bill mandating that frontier AIs are law-following) rather than relying on industry consensus.
However, there is clearly a place for non-legislative preparatory work within the labs and the AI safety community.
5.3.1.2. Improving incentives for defection
Whistleblowers currently face very high barriers to sharing classified information about dangerous behaviour within the executive. In particular:
When I spoke to experts about this, they recommended the following reforms to the Act:
It also seems wise for EPC funders ensure existing government whistleblower orgs are exceptionally well-funded.
5.3.1.3. Let IPHs hold tamper-resistant backups of frontier models
Frontier labs seeking to prevent EPC could store tamper-resistant backups of key model checkpoints on clusters controlled by various IPHs.
This would enable IPHs to test old models with new and improved evaluation technologies to check if they were secretly loyal. Also, if a PS inserted a loyalty into the leading model and this was detected quickly, IPHs would still have access to an alternative near-frontier model.
5.3.1.4. Remove single points of failure in the executive
Senior decision-makers in the executive should be encouraged to avoid creating single points of failure ("SPOFs") through incautious AI adoption. In particular:
Where practical and politically feasible, measures to avoid SPOFs should be enshrined in Acts of Congress.
Otherwise, a POTUS PS might be able to pack the relevant departments, and then change the underlying procurement or decision-making rules to eliminate the measure.
However, even if they cannot be easily written into statute, these measures seem quite helpful for preventing the capture of key parts of the executive by a non-POTUS PS.
A particularly ambitious goal for removing SPOFs would involve lobbying for an Act of Congress to reform the operational chain of command in the US military.
This is very far outside of the Overton Window – but it’s worth noting that Congress has reformed the chain of command multiple times since the Second World War, so it doesn’t seem wholly implausible.
I should note that Stefan Torges has written in some detail about the importance of removing SPOFs in the military. I strongly recommend interested readers check out his piece.
5.3.2. Eliminating executive overhang
5.3.2.1. Red-teaming AI legislation for EPC risk and proposing low-risk alternatives
Many popular proposals for AI regulation involve greatly expanding the discretionary powers which the executive has over frontier development
I think a POTUS PS could fairly easily abuse powers like this to coerce labs into supporting her power-seeking schemes.
Given this, I think someone in the AI safety community should be scrutinising proposed AI legislation to make sure it isn't too power-concentrating, and advocating for alternative approaches which strike a better balance between EPC and other risks.
They could give regular private input on proposed legislative text to 501(c)(4)s in the safety community (e.g. SAIP) and congressional staffers.
They could also try to establish a public profile on X as the EPC red-teamer – posting timely criticisms of different legislative proposals (e.g. the leaked Anthropic bill) as and when they become public.
And they could publish a list of principles for writing legislation that doesn't unduly concentrate power. This might include:
5.3.2.2. Glasswing for the law
SCOTUS has drastically expanded the scope of executive authority over the last century relative to the coordinate branches of USG – such that a POTUS PS would now have a wide array of entirely legal affordances she could abuse to further concentrate her power.
Given this, I think legal experts in the AI safety community should conduct an ambitious effort to identify the key legal ~loopholes which could facilitate a POTUS-led powergrab, and determine the most effective routes to remedy them.
Here are some powers which seem particularly useful to restrict if at all possible:
Legal experts I have spoken to have suggested that targeted efforts in the national security domain might have a particularly large effect on total risk.
Reforming the chain of command (see above) and enabling Congress to more easily punish contempt (see below) might also fall within scope of such a project.
5.3.2.3. Improving record-keeping and transparency
I expect it would be easier to prevent a PS from abusing her power if IPHs could more easily scrutinise what she was up to.
(1) Federal record-keeping rules could be expanded to cover agent trajectories.
The Archivist of the United States is responsible for issuing guidance around what counts as a record under the Federal Records Act ("FRA").
FRA guidance doesn’t currently tell agencies to keep records of agent trajectories (i.e. what systems an agent interacted with; which edits it made to which files).
So, in the first instance, the AI safety community should lobby the Archivist to issue guidance that the FRA covers agent trajectories.[11] However:
(2) We could encourage Congress to make the most of its oversight powers, and seek to expand them through legislation.
I gather there are already considerable efforts in this area from folks in the community. I'm happy to put people in touch with the folks running these projects in case of interest.
(3) Congress could institute an automated oversight regime for federal records.
This regime could be designed in accordance with the principles of structured transparency. Here's an illustration of what I have in mind:
This could be piloted on a smaller scale within the Government Accountability Office (“GAO”) before being extended across all congressional offices and committees.
Once the system was reliable enough, it could also be extended to Freedom of Information Act ("FOIA") requests, helping the public directly scrutinise the executive.
It's worth noting that this project currently seems far outside of the Overton Window. Experts I've spoken to tell me this would:
(4) Frontier AI projects could be subject to extensive transparency requirements, even if those projects weren't technically government bodies.
Extensive transparency seems particularly important in the context of a USG-led AGI project, given the likelihood that such a project occurs under a veil of classification.
Given that we’re not sure how exactly any such project might be structured, it could be helpful to set out specific triggers which cause heavier transparency requirements to come into effect.
To give these requirements teeth, they could be included in appropriations bills as conditions on funding the project, with funding issued periodically and automatically withheld (or withdrawn) if the project fails to comply.
5.3.2.4. Ban domestic killer robots in the US
The AI safety community should explore proposals making it harder for the US military and law enforcement to deploy lethal autonomous weapons on US soil, to increase barriers to an AI-enabled coup.
I haven’t spent much time thinking about this idea, so I’m particularly uncertain how technically and politically feasible it might be.
5.3.3. The asymmetric frontier
5.3.3.1. Pacing the frontier well
As I note above, EPC risks seem greatest in fast takeoff worlds, because this could lead to large disparities between PS and IPHs in their access to frontier capabilities.
Given this, pacing the frontier seems like a particularly good way to reduce EPC risks – provided that any legislation which enables pacing doesn't inadvertently centralise power in the executive.
5.3.3.2. Better Out Than In laws
Another promising approach to reducing frontier asymmetries is what I'm calling "Better Out Than In" laws – which seek to restrict what a frontier lab or the executive can do with an AI model before it has to be made more widely available. For example:
These laws might also have an indirect benefit of slowing capabilities progress at the frontier, because labs would have to dedicate so much effort to safety testing before being permitted to train a new model.
5.3.3.3. Turbocharging IPHs with AI
Both Congress and the judiciary have considerable powers to constrain the executive which they seem unlikely to fully exercise during the IE.
Here are some ways it might be possible to remedy this issue:
I suspect similar AI uplift work should also be done in key state and local governments (e.g. California) to ensure that they can continue to exert an effective check on the frontier AI industry throughout the IE.
5.3.4. Desperate times calling for desperate measures:
I haven’t spent much time thinking about how to intervene on this dynamic, so my views are quite underdeveloped. Here are a few lower-confidence ideas I’ve come up with.
5.3.4.1. Preserve household economic security
Once the IE has begun, USG could undertake a variety of radical economic reforms to protect US citizens from economic disruption.
If executed effectively, I suspect this could meaningfully increase perceptions of personal economic security throughout the IE.
I expect quite a bit of work has already been done on the most legally and economically robust ways to deliver protections like this. I suspect the AI safety community should continue to fund such work with an eye to developing proposals that would deliver on these objectives without inadvertently concentrating power.
5.3.4.2. Virtue screening the powerful
A 501(c)(4) could build an AI "virtue screening" tool which would conduct a deep-dive on candidates for key roles in federal, state, and local governments.
The tool would continuously scan publicly-available sources for evidence of power-seeking traits:
The outputs of this tool could then inform political spending by big donors in the community.[12] This could increase the likelihood that a POTUS PS and her political allies are simply not on the ballot to be selected by concerned voters during desperate times.
The same tool could also be used to conduct a smaller investigation into the leadership of key parts of the AI industry (e.g. frontier labs; hyperscalers; chipmakers) to help inform decisions by investors in the community.
5.3.5. Widespread epistemic degradation
5.3.5.1. Epistemic tooling
I think the best way to reduce risks from epistemic degradation is to ensure that IPHs and the general public retain the capacity to verify what is true and false throughout the IE.
I think one particularly promising way to achieve this is to found projects that accelerate the arrival of various epistemic tools that are likely to arrive during the IE, and ensure that they are widely distributed as quickly as possible.
Here are some tools which I think could be particularly effective:
I should note that Lizka Vaintrob and Owen Cotton-Barratt have already written about this topic at length.
5.3.5.2. Disclaimers for synthetic political content
Congress (and swing state governments) could pass laws requiring annoying, medical advertisement-style disclosures for any AI-generated content broadcast or posted on social media by a corporation in relation to a public figure, with criminal penalties for non-compliance.
Disclaimers of this sort would hopefully make it easier for voters and IPHs to effectively scrutinise their leaders.
5.3.6. Cross-cutting interventions
As I note above, most of the interventions in this section aim to build the field of people working to prevent extreme power concentration. I think field-building interventions are particularly promising because EPC risks are currently so neglected.
I suspect we should prioritise field-building amongst the following groups, in decreasing order of priority:
Most of the measures below are intended to reach one or more of these priority groups.
5.3.6.1. Write up better intro materials for the AI safety pipeline
The bar for getting up to speed on EPC is currently very high. To get a sense of the most plausible threat models and most promising interventions, you need access to a bunch of private documents and a high-context interlocutor who is willing to give you a significant amount of their time.
I believe this is a major barrier to people getting involved in the field. Given this, I expect it would be worth someone’s time to write up high-quality intro materials for different audiences (e.g. seasonal fellows; BlueDot classes; think tank leads), setting out the basic threat models and most promising interventions.
5.3.6.2. Pitch people to do more on EPC
I think it would make sense for one of the people currently working on EPC to dedicate a majority of their time to pitching leaders in the broader AI safety community on doing more work on EPC. This might look like:
I also suspect it would be worth engaging in targeted, high-effort “wake-up” efforts for the most important IPHs outside the AI safety community – including:
5.3.6.3. Terminator for EPC
An AI safety grantmaker (e.g. cG or Longview) could headhunt a writer-director and experienced producer who are genuinely interested in AI safety, and well-known enough that studios will actually read their script.
They could then fund them through the early stages of developing a feature film or TV show about AI-enabled EPC occurring in the US.
Alternatively, someone in the community could pitch Black Mirror (or a similar futurist show) on writing an episode about AI-enabled EPC.
The goal here would be to develop compelling audiovisual fiction which becomes a cultural touchpoint in relation to EPC, increasing IPHs' situational awareness and enabling them to recognise and push back against power-seeking sooner.
5.3.6.4. Federalist Papers 2.0
Someone could set up a newsletter which publishes anonymous articles about EPC, written by people who would face substantial risks from open publication.
This could hearken back to the original Federalist Papers – which were published pseudonymously by several Founding Fathers at a time of great change in US history.
5.3.6.5. AIFP for EPC
Someone could set up an organisation to write a plausible worst-case scenario for EPC occurring in the US by the end of the decade, similar in form, substance and writing style to AI 2027.
This work would aim to have an impact in roughly the same way AI 2027 did for loss of control – massively raising the salience of EPC with elites, giving the cause a public face, and creating a Schelling point of policy recommendations to rally around.
5.3.6.6. Set up the EPC org
Someone could set up an org whose role is to act as a home for research on EPC and a hub to connect people in the field.
It could produce its own research on extreme power concentration, as well as doing more rapid response policy work.
It could also lead outreach to the AI safety community and IPHs, aiming to alert them to the risks of EPC and help them resist it.
And it could maintain other useful community infrastructure.
5.3.6.7. Detailed threat modelling by sector experts
Many EPC threat models (including the ones above) are sorely lacking in detail. This is a problem, because:
Given this, I think the field would greatly benefit from in-depth, expert threat modelling of the most important elements of the most important EPC threat models.
Glasswing for the law (see above) is a particularly fleshed out example of what I have in mind here. But I suspect this would actually be most helpful in the military domain, given that our threat models seem both particularly plausible and especially underdeveloped.
5.3.7. Which of these interventions seem most promising?
Ithink the following interventions seem reasonably tractable and particularly likely to further at least one of my three theories of victory without compromising the others.6. Weaknesses in this work
Here are a few points where I think what I’ve written above is particularly weak.
(1) I don’t yet have a good sense of how, precisely, a POTUS PS could gain sufficiently fine-grained control of frontier AI development to:
I’m a bit sceptical that the current set of emergency powers (e.g. the DPA) would be sufficient to achieve this, and I suspect that new legislation would be required. But I’m not confident about this – and I think the topic would benefit from a deep-dive.
(2) I don’t yet have a detailed sense of how exactly a PS could abuse capabilities asymmetries to concentrate power to an extreme degree.
I think this is an inherently difficult question to answer, for roughly the same reasons as it’s hard to give a detailed answer for how misaligned takeover might happen.
As in – it’s hard to know whether there’s a ceiling on these capabilities, where that ceiling might be, and what it looks like for an actor to be radically more capable along these dimensions
That said, I think my current threat modelling reads a bit like:
And this feels unacceptably handwavey.
(3) I’ve received limited expert input on most of the interventions I propose.
Given this, I expect them to be imperfectly targeted and fail to account for important practical limitations. In particular, I suspect I’m too confident about the prospects for passing various pieces of legislation.
(4) This post focuses exclusively on domestic EPC and ignores how a single actor could get a DSA over the entire world.
This is a significant omission, because the likelihood that a post-ASI US could dominate all other actors seems really very cruxy for the overall badness of domestic EPC.
I also think this focus on domestic concerns may have caused me to underweight:
To be clear – I'm unsure whether extensive expropriation and redistribution would actually be desirable.
Thanks to Cullen O'Keefe for this coinage.
Advanced AI may also be far more able to execute on complex, long-horizon legal strategies – furthering the potential for abuse.
Matt Botvinick (Anthropic) has written more about this topic in his recent book.
I also find it plausible that we could see a backslide all the way scenario wherein a lab leader PS uses her asymmetric access to frontier capabilities and vast AI-generated wealth to corrupt political processes and gain effective control of USG.
It's worth stressing that I'm using the term "regulators" in a broad and non-standard way here, to refer to any government agency with substantial discretionary authority over the frontier AI industry – whether or not it would be traditionally thought of as a regulator.
This includes both mindhack-level superpersuasion, and more subtle forms of social manipulation.
To be clear – I also think the character of lab leaders matters quite a bit, but it seems less useful to focus on this because (i) lab leadership seems less likely to change in the near-term, and (ii) I think lab leader-driven EPC accounts for <1/4 of the total risk.
For example, I’ve heard rumours that GDM struggled for several months to train a genuinely “helpful-only” model for the DOW. But it seems like this was an unusual situation – and I get the impression it will be increasingly easy to simply train guardrails out of models in future.
Apparently bills along these lines have previously been introduced to Congress.
It's worth noting that record-keeping in the White House and parts of the Executive Office of the President ("EOP") is separately governed by the Presidential Records Act ("PRA") during a president's term. White House Counsel issues similar guidance in relation to PRA compliance – and this might also need to be extended to cover agent trajectories.
Experts tell me that primary candidates benefit far more than candidates in general elections from increased funding – so it might be worth prioritising screening primaries at first.