an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. — Good, 1965, Speculations concerning the first ultraintelligent machine.
Since the 2000s, a few researchers (recently increasingly many) have taken seriously the prospect that some kind of intelligence feedback mechanism could kick in around the time that AI becomes competent enough to contribute to its ‘own’ (or rather, its successors’) development.
I’m not going to address the resulting prospect of speedup here, which I think is real, though perhaps surprisingly modest.[1] (If, like some of my friends, you think I’m underweighting this prospect, you should probably be even more concerned about concentration of influence.[2])
Rather, this note is briefly making the rather simple (but I think rather overlooked) observation: plausibly automated AI R&D would enable a (perhaps severe) concentration of influence… and this could be the most important effect to be paying attention to.
It’s a fairly obvious point once made. A more thorough analysis might shed light on how likely this is, the most important consequences, and any leading indicators to pay attention to. I’ll only make some starting gestures toward these.
R&D automation…: fewer human participants means fewer whistleblowers, less internal scrutiny, less governance and decisionmaking robustness. More concentration of that influence. That could mean more single points of failure. It’s famously difficult to maintain a conspiracy of more than one or two people: ‘two can keep a secret if one is dead’, as they say! And besides conspiracy, compared to larger teams, individuals and small groups may be far more susceptible to capture, coercion, corruption, or plain foolishness and rash decisionmaking.
Whose influence?
To be clear, you don’t need to be concerned about any specific people currently in positions of particular influence to think that concentration could be very impactful (and concerning).
The wisest supreme leader is still a single point of failure[3]. Especially during a period of tumult, leaders can be displaced and reporting structures revolutionised. Concentration of influence — even the prospect of that concentration — simultaneously raises the incentives for corruption and power struggle (while raising the stakes for the rest of us). Meanwhile, power corrupts and a centralised decisionmaker can rarely account for all the relevant considerations, even if they mean to. In AI, we’ve already seen company leadership falling out over differences of opinion, fighting it out in the boardroom and in court, and even clashing with governments. Turf war over the means of production of AI (which may increasingly equal the means of production full stop).
The point is that losing checks and balances can be a problem whoever is at the top[4]. As in politics, so in the heights of industry — which, today, and increasingly, firmly includes AI.
Suffice to say that I imagine the kinds of roles in the AI oversight chain which might be (nominally, initially) preserved longest are senior research directors, company executives, and closely-involved regulatory or executive government officials.
Perhaps more important: whose influence lost? Directly, the skilled human labour which today is involved in researching, engineering, and deploying frontier AI systems. Via them, their wider research networks, social ties, and any societal oversight they might have provided (e.g. whistleblowing to journalists or authorities). Plausibly also the marketers, account managers, even lawyers and others who interface with what a company is doing with its AI products, and their networks. (If not themselves replaced, they’d lose collegial understanding and influence via the replaced researcher-implementers). In short: everyone else.
Sketched using claude.ai
Concentration without (overt) malice
Why shed the researcher-implementers, the deployment engineers, and others (or sideline them from the important ‘real’ responsibilities)? It doesn’t have to be a power-grab, at first (or even at all) — it’s just good business! Human employees (especially software experts) are incredibly expensive compared with AI.[5]
Beyond this, analogously to areas where AI already surpasses humans (like chess), human participants may positively get in the way, slowing and compromising what would not only be a cheaper, but a more effective AI-only workflow.
A company might choose to instead eat this cost — perhaps it prefers to exhibit loyalty to current staff, acknowledges concerns around erosion of human oversight, or is inherently conservative in that way. But in a competitive environment (either commercially or in strategic and security terms), that may be tantamount to inviting defeat or irrelevance, if competitors are charging ahead.
At some point shareholders and other stakeholders might think it irresponsible not to embrace a replacement AI workforce. Ultimately this could add up to far fewer people having insight into what’s going on or a say in its direction. Even if you can’t (or don’t want to) immediately fire them, you might deliberately or even passively sideline your human employees as more and more work is carried out by machines.
Influence over what?
Notably, board, investors, auditors, journalists, and regulators (who already often suffer an information scarcity) — and the broader society they represent — could also be substantially cut out if logging, analysis, PR, and incident reporting are automated… This might only require the usual levels of corporate paltering rather than special malice. It might even be prosocially-motivated if the relevant actors are (justifiably or not) concerned about a power-grab by government. (It could of course also be deliberately selfish.)
Conversely, a government or regulator which strong-arms its way into the midst of the development process might itself become the locus of concentrated influence. (This in turn might be superficially or genuinely motivated by national security concerns.) The more automated a firm, the easier it may be to seize, either overtly or through more subtle channels. When employees are critical pieces in a production process, that’s a taller order.[6]
Sketched using claude.ai
I’m not suggesting any of this leads (immediately) to influence over all the world (though if you do take over the world, please consider my friend Owen’s advice for what to do if you find yourself in that situation). Nor even over an entire economy or state. But minimally, this looks like highly concentrated influence over a frontier AI company: presently a rare, increasingly lucrative, and burgeoningly politically-influential machine.
Activity at that frontier — the decisionmaking by leading AI developers — stands a chance of being among the most economically, politically, and strategically important activity in the coming years. If the cat is out of the bag with AI, that frontier, steered wisely, might be an important part of a societal resilience and readiness process (compare the recent Project Glasswing in cybersecurity). Used foolishly or even maliciously, the frontier might degrade societal capacity right at a time when it could be most important.
Influence… or power?
To repeat: I don’t think it’s given (or even particularly likely) that the starting possessors of a hypothetical concentration of influence would remain in control. From the point of view of a rogue AI, for example, control over a frontier AI company might be a delicious opportunity for expansion, defence, and societal dependency — ideal routes to escalated un-unpluggability. Likewise from the point of view of a power-hungry politician or corporate psychopath. Notice that corruptibility and coercibility are perhaps more plausible with fewer hands: it’s less practical to threaten, buy off, or muscle out a whole team or company than a single person.
So if this hypothetical automation-driven concentration of influence is a concentration of power, I don’t expect the ultimate wielder to be worthy of it. This marks a true difference of opinion with some others I know, who might think it more likely that, say, some current AI company leader could both accrue and keep hold of this level of power, and also use it wisely.
The obvious paths to hard power via AI are in surveillance — newly scalable with AI analysts scrutinising as many people’s movements as desired — and in development and automated production of flexible, high-range tools of coercion like weaponised drones. Other kinds of power might come from positions of economic leverage or narrative and propaganda dominance, conceivable with advanced-enough AI and without societal defences. That's one point at which concentration could become entrenchment of power.
Concretely, what might be different?
Let’s really quickly gesture at two cases which might be substantially different in a world where AI production is largely automated.
Imagine a near future: frontier AI company revenues are in the 100s of $billions or $trillions, AI services are used in civil services, executive decisionmaking, faith leadership, … That’s a lot of points of leverage for subtle manipulation. Suppose a leader (selfishly or under pressure) intends to inject secret loyalties into their products. If they’re sole controller without checks and balances, that could be trivial, with obvious concerning consequences. Without automation-replacement, there are perhaps dozens or hundreds of people with reasonable oversight per major release, and several people even for any given small change. That’s a far more difficult system to twist.
Alternatively, consider a company on the brink of truly autonomous, general, human-surpassing AI: a huge responsibility. Many ambitious but underwrought alignment targets have been suggested for such a project, many of them foolish, perhaps fatally so. A leader hubristic without serious checks might withdraw from critique, double down (perhaps sycophantically egged on by AI toadies), and drive through a ruinous agenda. Society at large might be none the wiser (or at least insufficiently alert and able to intervene). A leader willingly taking risks (perhaps with a selfish or misguided vision of unfathomable rewards) would be even more of a problem. On the other hand, a leadership more dependent on staff and teams may be subject to more psychological and cognitive support, scrutiny, moral encouragement, and so on. A literal healthcheck. Of course, human teams can be subject to concerning groupthink (not unheard of at AI development companies!), but the more off-base a suggestion is (ethically or pragmatically), the more likely it is that the collective intelligence of the team corrects it.
Any number of scenarios can be considered; rarely do they appear more promising from a societal perspective if a single or a few leaders are insulated from scrutiny and critique.
What’s the outlook?
It looks very unclear to me. Plausibly it’s in the balance and decisions and conversations now will shape how this plays out!
There are some reasons to think that (near-) automation of AI production could instead produce equal or even improved societal oversight.
For one, the sometimes unintuitive effect of automation is to increase per-worker productivity, making it more worthwhile to bring more humans into the process (even on myopic financial grounds). Historically this effect has anyway often presaged a sharp decline once automation reaches a sufficient level. This might make concentration actually more difficult for a time. I don’t expect this to play out for very long in AI frontier research[7], but I’m somewhat uncertain about this.
Directed sensibly, a glut of researcher-grade AI worker-equivalents could be tasked with analysis, scrutiny, decisionmaking robustness, even auditing and whistleblowing. There’s nothing in principle preventing this, for any given level of trustability in our AI. Perhaps we’ll end up bringing more people into the process in the increasingly important, complementary roles of oversight, decisionmaking, and direction-setting. These folks could be made more effective with AI tools and assistance. That’s a decision to be made.
As I suggested earlier, leaders perhaps soon faced with these temptations may balk, either at the moral prospect… or at the pragmatic prospect of being subsequently replaced, captured, or corrupted in the ways I hinted at.
Perhaps it’s a tenuous balancing act. Internally to a given AI producer, we might want to guarantee a large enough ‘in the loop’ workforce that we can trust internal deliberation and whistleblowing and so on to keep things on the rails. Outside that, we might want to avoid no-holds-barred proliferation of AI-producing capability — but we probably want at least a few developers to keep each other in check and reduce monopoly or kingmaker dynamics.
I don’t expect any such concentrations of influence to play out overnight. Loss of control is a process, not a moment. AI development organisations — and society at large — should be paying attention and having the necessary conversations. Much to look out for, and perhaps much to look forward to!
I would note that we usually need some concentrated executive and representative power in various roles in society — but we want those roles to be sufficiently monitored, and we want the processes which move people into (and out of) them to be sufficient to produce worthy selections.
This ‘humans expensive, AI cheap’ point is worth taking some care over. Contemporary frontier AI training is getting more expensive by the year. Running the best AI can be increasingly expensive, because ‘thinking harder’ is an effective way to scale capabilities on current margins. But once a capability is unlocked at the frontier, it typically becomes rapidly and radically cheaper, due to ongoing rises in compute efficiency and the ability to distill long thoughts into quicker reflexes. Further, you only need to train AI to do something once, and then it can be copied and run as many times as you want, it doesn’t need to sleep or take family time or get sick etc.
One reason is that this research does not scale very well in parallel, so returns diminish steeply per worker — and we’re already imagining AI that can substantially slot in as a cheaper replacement worker in most cases anyway.
There’s a lot of talk about automated AI R&D and the like. It’s been discussed since at least 1965 when statistician I.J. Good coined the term ‘intelligence explosion’:
Since the 2000s, a few researchers (recently increasingly many) have taken seriously the prospect that some kind of intelligence feedback mechanism could kick in around the time that AI becomes competent enough to contribute to its ‘own’ (or rather, its successors’) development.
This is increasingly recognised as a prospect we should take seriously. It’s been discussed somewhat furtively by leading AI developers for many years, but in late 2025 it became a public talking point. Sam Altman, CEO of OpenAI, recently set a target of fully automated AI-improving-AI by 2028. Jack Clark, co-founder of Anthropic, writes ‘AI systems are about to start building themselves’.
I’m not going to address the resulting prospect of speedup here, which I think is real, though perhaps surprisingly modest.[1] (If, like some of my friends, you think I’m underweighting this prospect, you should probably be even more concerned about concentration of influence.[2])
Rather, this note is briefly making the rather simple (but I think rather overlooked) observation: plausibly automated AI R&D would enable a (perhaps severe) concentration of influence… and this could be the most important effect to be paying attention to.
It’s a fairly obvious point once made. A more thorough analysis might shed light on how likely this is, the most important consequences, and any leading indicators to pay attention to. I’ll only make some starting gestures toward these.
As I presaged in Engineering a Safer World,
Whose influence?
To be clear, you don’t need to be concerned about any specific people currently in positions of particular influence to think that concentration could be very impactful (and concerning).
The wisest supreme leader is still a single point of failure[3]. Especially during a period of tumult, leaders can be displaced and reporting structures revolutionised. Concentration of influence — even the prospect of that concentration — simultaneously raises the incentives for corruption and power struggle (while raising the stakes for the rest of us). Meanwhile, power corrupts and a centralised decisionmaker can rarely account for all the relevant considerations, even if they mean to. In AI, we’ve already seen company leadership falling out over differences of opinion, fighting it out in the boardroom and in court, and even clashing with governments. Turf war over the means of production of AI (which may increasingly equal the means of production full stop).
The point is that losing checks and balances can be a problem whoever is at the top[4]. As in politics, so in the heights of industry — which, today, and increasingly, firmly includes AI.
Suffice to say that I imagine the kinds of roles in the AI oversight chain which might be (nominally, initially) preserved longest are senior research directors, company executives, and closely-involved regulatory or executive government officials.
Perhaps more important: whose influence lost? Directly, the skilled human labour which today is involved in researching, engineering, and deploying frontier AI systems. Via them, their wider research networks, social ties, and any societal oversight they might have provided (e.g. whistleblowing to journalists or authorities). Plausibly also the marketers, account managers, even lawyers and others who interface with what a company is doing with its AI products, and their networks. (If not themselves replaced, they’d lose collegial understanding and influence via the replaced researcher-implementers). In short: everyone else.
Sketched using claude.ai
Concentration without (overt) malice
Why shed the researcher-implementers, the deployment engineers, and others (or sideline them from the important ‘real’ responsibilities)? It doesn’t have to be a power-grab, at first (or even at all) — it’s just good business! Human employees (especially software experts) are incredibly expensive compared with AI.[5]
Beyond this, analogously to areas where AI already surpasses humans (like chess), human participants may positively get in the way, slowing and compromising what would not only be a cheaper, but a more effective AI-only workflow.
A company might choose to instead eat this cost — perhaps it prefers to exhibit loyalty to current staff, acknowledges concerns around erosion of human oversight, or is inherently conservative in that way. But in a competitive environment (either commercially or in strategic and security terms), that may be tantamount to inviting defeat or irrelevance, if competitors are charging ahead.
At some point shareholders and other stakeholders might think it irresponsible not to embrace a replacement AI workforce. Ultimately this could add up to far fewer people having insight into what’s going on or a say in its direction. Even if you can’t (or don’t want to) immediately fire them, you might deliberately or even passively sideline your human employees as more and more work is carried out by machines.
Influence over what?
Notably, board, investors, auditors, journalists, and regulators (who already often suffer an information scarcity) — and the broader society they represent — could also be substantially cut out if logging, analysis, PR, and incident reporting are automated… This might only require the usual levels of corporate paltering rather than special malice. It might even be prosocially-motivated if the relevant actors are (justifiably or not) concerned about a power-grab by government. (It could of course also be deliberately selfish.)
Conversely, a government or regulator which strong-arms its way into the midst of the development process might itself become the locus of concentrated influence. (This in turn might be superficially or genuinely motivated by national security concerns.) The more automated a firm, the easier it may be to seize, either overtly or through more subtle channels. When employees are critical pieces in a production process, that’s a taller order.[6]
Sketched using claude.ai
I’m not suggesting any of this leads (immediately) to influence over all the world (though if you do take over the world, please consider my friend Owen’s advice for what to do if you find yourself in that situation). Nor even over an entire economy or state. But minimally, this looks like highly concentrated influence over a frontier AI company: presently a rare, increasingly lucrative, and burgeoningly politically-influential machine.
Activity at that frontier — the decisionmaking by leading AI developers — stands a chance of being among the most economically, politically, and strategically important activity in the coming years. If the cat is out of the bag with AI, that frontier, steered wisely, might be an important part of a societal resilience and readiness process (compare the recent Project Glasswing in cybersecurity). Used foolishly or even maliciously, the frontier might degrade societal capacity right at a time when it could be most important.
Influence… or power?
To repeat: I don’t think it’s given (or even particularly likely) that the starting possessors of a hypothetical concentration of influence would remain in control. From the point of view of a rogue AI, for example, control over a frontier AI company might be a delicious opportunity for expansion, defence, and societal dependency — ideal routes to escalated un-unpluggability. Likewise from the point of view of a power-hungry politician or corporate psychopath. Notice that corruptibility and coercibility are perhaps more plausible with fewer hands: it’s less practical to threaten, buy off, or muscle out a whole team or company than a single person.
So if this hypothetical automation-driven concentration of influence is a concentration of power, I don’t expect the ultimate wielder to be worthy of it. This marks a true difference of opinion with some others I know, who might think it more likely that, say, some current AI company leader could both accrue and keep hold of this level of power, and also use it wisely.
The obvious paths to hard power via AI are in surveillance — newly scalable with AI analysts scrutinising as many people’s movements as desired — and in development and automated production of flexible, high-range tools of coercion like weaponised drones. Other kinds of power might come from positions of economic leverage or narrative and propaganda dominance, conceivable with advanced-enough AI and without societal defences. That's one point at which concentration could become entrenchment of power.
Concretely, what might be different?
Let’s really quickly gesture at two cases which might be substantially different in a world where AI production is largely automated.
Imagine a near future: frontier AI company revenues are in the 100s of $billions or $trillions, AI services are used in civil services, executive decisionmaking, faith leadership, … That’s a lot of points of leverage for subtle manipulation. Suppose a leader (selfishly or under pressure) intends to inject secret loyalties into their products. If they’re sole controller without checks and balances, that could be trivial, with obvious concerning consequences. Without automation-replacement, there are perhaps dozens or hundreds of people with reasonable oversight per major release, and several people even for any given small change. That’s a far more difficult system to twist.
Alternatively, consider a company on the brink of truly autonomous, general, human-surpassing AI: a huge responsibility. Many ambitious but underwrought alignment targets have been suggested for such a project, many of them foolish, perhaps fatally so. A leader hubristic without serious checks might withdraw from critique, double down (perhaps sycophantically egged on by AI toadies), and drive through a ruinous agenda. Society at large might be none the wiser (or at least insufficiently alert and able to intervene). A leader willingly taking risks (perhaps with a selfish or misguided vision of unfathomable rewards) would be even more of a problem. On the other hand, a leadership more dependent on staff and teams may be subject to more psychological and cognitive support, scrutiny, moral encouragement, and so on. A literal healthcheck. Of course, human teams can be subject to concerning groupthink (not unheard of at AI development companies!), but the more off-base a suggestion is (ethically or pragmatically), the more likely it is that the collective intelligence of the team corrects it.
Any number of scenarios can be considered; rarely do they appear more promising from a societal perspective if a single or a few leaders are insulated from scrutiny and critique.
What’s the outlook?
It looks very unclear to me. Plausibly it’s in the balance and decisions and conversations now will shape how this plays out!
There are some reasons to think that (near-) automation of AI production could instead produce equal or even improved societal oversight.
For one, the sometimes unintuitive effect of automation is to increase per-worker productivity, making it more worthwhile to bring more humans into the process (even on myopic financial grounds). Historically this effect has anyway often presaged a sharp decline once automation reaches a sufficient level. This might make concentration actually more difficult for a time. I don’t expect this to play out for very long in AI frontier research[7], but I’m somewhat uncertain about this.
Directed sensibly, a glut of researcher-grade AI worker-equivalents could be tasked with analysis, scrutiny, decisionmaking robustness, even auditing and whistleblowing. There’s nothing in principle preventing this, for any given level of trustability in our AI. Perhaps we’ll end up bringing more people into the process in the increasingly important, complementary roles of oversight, decisionmaking, and direction-setting. These folks could be made more effective with AI tools and assistance. That’s a decision to be made.
As I suggested earlier, leaders perhaps soon faced with these temptations may balk, either at the moral prospect… or at the pragmatic prospect of being subsequently replaced, captured, or corrupted in the ways I hinted at.
Perhaps it’s a tenuous balancing act. Internally to a given AI producer, we might want to guarantee a large enough ‘in the loop’ workforce that we can trust internal deliberation and whistleblowing and so on to keep things on the rails. Outside that, we might want to avoid no-holds-barred proliferation of AI-producing capability — but we probably want at least a few developers to keep each other in check and reduce monopoly or kingmaker dynamics.
I don’t expect any such concentrations of influence to play out overnight. Loss of control is a process, not a moment. AI development organisations — and society at large — should be paying attention and having the necessary conversations. Much to look out for, and perhaps much to look forward to!
Task-relevant data and compute look like the more biting bottlenecks, and research taste accrues mainly through expensive experimentation.
(unless you expect such accelerated singularity that influence prior is all that matters)
And may I be forgiven for slandering the current crop of potential supreme leaders as not very wise
I would note that we usually need some concentrated executive and representative power in various roles in society — but we want those roles to be sufficiently monitored, and we want the processes which move people into (and out of) them to be sufficient to produce worthy selections.
This ‘humans expensive, AI cheap’ point is worth taking some care over. Contemporary frontier AI training is getting more expensive by the year. Running the best AI can be increasingly expensive, because ‘thinking harder’ is an effective way to scale capabilities on current margins. But once a capability is unlocked at the frontier, it typically becomes rapidly and radically cheaper, due to ongoing rises in compute efficiency and the ability to distill long thoughts into quicker reflexes. Further, you only need to train AI to do something once, and then it can be copied and run as many times as you want, it doesn’t need to sleep or take family time or get sick etc.
For example, witness the periodic furore among tech company employees when their work is used toward military ends (latest).
One reason is that this research does not scale very well in parallel, so returns diminish steeply per worker — and we’re already imagining AI that can substantially slot in as a cheaper replacement worker in most cases anyway.