On the Rationality of Deterring ASI
59habryka
20davekasten
9habryka
14davekasten
11habryka
1deep
1deep
35Julian Bradshaw
28Vladimir_Nesov
7Lucius Bushnaq
5Vladimir_Nesov
1NickH
15Lucius Bushnaq
11Mitchell_Porter
9quiet_NaN
7mikko
3Mitchell_Porter
8mikko
6Emmanuel Awosika
6Chris_Leong
4O O
3Tristan Wegner
3Ben Livengood
4Julian Bradshaw
4Ben Livengood
2NickH
2Stephen McAleese
2mako yass
2niplav
1ozhang
1Joe Blake
1Alephwyr
0sokatsat
-8Andy E Williams
New Comment
Promoted to curated: I have various pretty substantial critiques of this work, but I do overall think this is a pretty great effort at crossing the inferential distance from people who think AGI will be a huge deal and potentially dangerous, to the US government and national security apparatus.
The thing that I feel most unhappy about is that the document feels to me like it follows a pattern that Situational Awareness also had, where it seemed to me like it kept framing various things that it wanted to happen, as "inevitable to happen", while also arguing that they are a good idea, in a way that felt to me like it tried too hard to make some kind of self-fulfilling prophecy.
But overall, I feel like this document speaks with surprising candor and clarity about many things that have been left unsaid in many circumstances. I particularly appreciated its coverage of explicitly including conventional ballistic escalation as part of a sabotage strategy for datacenters. Relevant quotes:
Should these measures falter, some leaders may contemplate kinetic attacks on datacenters, arguing that allowing one actor to risk dominating or destroying the world are graver dangers, though kinetic attacks are likely unnecessary. Finally, under dire circumstances, states may resort to broader hostilities by climbing up existing escalation ladders or threatening non-AI assets. We refer to attacks against rival AI projects as "maiming attacks."
I also particularly appreciated this proposed policy for how to handle AIs capable of recursive self-improvement:
In the near term, geopolitical events may prevent attempts at an intelligence recursion. Looking further ahead, if humanity chooses to attempt an intelligence recursion, it should happen in a controlled environment with extensive preparation and oversight—not under extreme competitive pressure that induces a high risk tolerance.
I particularly appreciated its coverage of explicitly including conventional ballistic escalation as part of a sabotage strategy for datacenters
One thing I find very confusing about existing gaps between the AI policy community and the national security community is that natsec policymakers have already explicitly said that kinetic (i.e., blowing things up) responses are acceptable for cyberattacks under some circumstances, while the AI policy community seems to somehow unconsciously rule those sorts of responses out of the policy window. (To be clear: any day that American servicemembers go into combat is a bad day, I don't think we should choose such approaches lightly.)
My sense is a lot of the x-risk oriented AI policy community is very focused on avoiding "gaffes" and have a very short-term and opportunistic relationship with reputation and public relations and all that kind of stuff. My sense is that people in the space don't believe being principled or consistently honest basically ever gets rewarded or recognized, so the right strategy is to try to identify what the overton window is, only push very conservatively on expanding it, and focus on staying in the good graces of whatever process determines social standing, which is generally assumed to be pretty random and arbitrary.
I think many people in the space, if pushed, would of course acknowledge that kinetic responses are appropriate in many AI scenarios, but they would judge it as an unnecessarily risky gaffe, and that perception of a gaffe creates a pretty effective enforcement regime for people to basically never bring it up, lest you be judged as politically unresponsible.
I think I am too much inside the DC policy world to understand why this is seen as a gaffe, really. Can you unpack why it's seen as a gaffe to them? In the DC world, by contrast, "yes, of course, this is a major national security threat, and no you of course could never use military capabilities to address it," would be a gaffe.
I mean, you saw people make fun of it when Eliezer said it, and then my guess is people conservatively assumed that this would generalize to the future. I've had conversations with people where they tried to convince me that Eliezer mentioning kinetic escalation was one of the worst things that anyone has ever done for AI policy, and they kept pointing to twitter threads and conversations where opponents made fun of it as evidence. I think there clearly was something real here, but I also think people really fail to understand the communication dynamics here.
You're missing some ways Eliezer could have predictably done better with the Time article, if he were framing it for national security folks (rather than an attempt at brutal honesty, or perhaps most acccurately a cri de coeur).
@davekasten - Eliezer wasn't arguing for bombing as retaliation for a cyberattack. Rather, as a preemptive measure against noncompliant AI developments:
If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
If you zoom out several layers of abstraction, that's not too different from the escalation ladder concept described in this paper. A crucial difference, though, is that Eliezer doesn't mention escalation ladders at all -- or other concepts that would help neutral readers be like "OK, this guy gets how big a lift all this would be, and he has some ideas for building blocks to enact it". Examples include "how do you get an international agreement on this stuff", "how do you track all the chips", "how do you prevent people building super powerful AI despite the compute threshold lowering", "what about all the benefits of AI that we'd be passing up" (besides a brief mention that narrow-AI-for-bio might be worth it), "how confident can we be that we'd know if someone was contravening the deal".
Second, there was a huge inferential gap to this idea of AGI as key national security threat -- there's still a large one today, despite rhetoric around AGI. And Eliezer doesn't do enough meeting in the middle here.
He gives the high-level argument that to him is sufficient, but is/was not convincing to most people -- that AI by some metrics is growing fast, in principle can be superhuman, etc. Unfortunately most people in government don't have the combination of capacity, inclination, and time to assess these kinds of first-principles arguments for themselves, and they really need concreteness in the form of evidence or expert opinion.
Also, frankly, I just think Eliezer is wrong to be as confident in his view of "doom by default" as he is, and the strategic picture looks very very different if you place say 20% or even 50% probability on this.
If I had Eliezer's views I'd probably focus on evals and red-teaming type research to provide fire alarms, convince technical folks p(doom) was really high, and then use that technical consensus or quasi-consensus to shift policy. This isn't totally distinct from what Eliezer did in the past with more abstract arguments, and it kinda worked (there are a lot of people with >10% p(doom) in policy world, there was that 2023 moment where everyone was talking about it). I think in worlds where Eliezer's right, but timelines are say more like 2030 than 2027, there's real scope for people to be convinced of high p(doom) as AI advances, and that could motivate some real policy change.
I think it's fine that Eliezer wrote it, though. Not maximally strategic by any means, but the man's done a lot and he's allowed his hail mary outreach plans.
I think at the time I and others were worried this would look bad for "safety as a whole", but at this point concerns about AI risk are common and varied enough, and people with those concerns have often strong local reputations w/ different groups. So this is no longer as big of an issue, which I think is really healthy for AI risk folks -- it means we can have Pause AI and Eliezer and Hendrycks and whoever all doing their own things, able to say "no I'm not like those folks, here's my POV", and not feeling like they should get a veto over each other. And in retrospect I think we should have anticipated and embraced this vision earlier on.
tbh, this is part of what I think went wrong with EA -- a shared sense that community reputation was a resource everyone benefitted from and everyone wanted to protect and polish, that people should get vetoes over what each other do and say. I think it's healthy that there's much less of a centralized and burnished "EA brand" these days, and much more of a bunch of people following their visions of the good. Though there's still the problem of Open Phil as a central node in the network, through which reputation effects flow.
This is creative.
TL;DR: To mitigate race dynamics, China and the US should deliberately leave themselves open to the sabotage ("MAIMing") of their frontier AI systems. This gives both countries an option other than "nuke the enemy"/"rush to build superintelligence first" if superintelligence appears imminent: MAIM the opponent's AI. The deliberately unmitigated risk of being MAIMed also encourages both sides to pursue carefully-planned and communicated AI development, with international observation and cooperation, reducing AINotKillEveryone-ism risks.
The problem with this plan is obvious: with MAD, you know for sure that if you nuke the other guy, you're gonna get nuked in return. You can't hit all the silos, all the nuclear submarines. With MAIM, you can't be so confident: maybe the enemy's cybersecurity has gotten too good, maybe efficiency has improved and they don't need all their datacenters, maybe their light AGI has compromised your missile command.
So the paper argues for at least getting as close as possible to assurance that you'll get MAIMed in return: banning underground datacenters, instituting chip control regimes to block rogue actors, enforcing confidentiality-preserving inspections of frontier AI development.
Definitely worth considering. Appreciate the writeup.
Cyberattacks can't disable anything with any reliability or for more than days to weeks though, and there are dozens of major datacenter campuses from multiple somewhat independent vendors. Hypothetical AI-developed attacks might change that, but then there will also be AI-developed information security, adapting to any known kinds of attacks and stopping them from being effective shortly after. So the MAD analogy seems tenuous, the effect size (of this particular kind of intervention) is much smaller, to the extent that it seems misleading to even mention cyberattacks in this role/context.
Why would this be restricted to cyber attacks? If the CCP believed that ASI was possible, even if they didn't believe in the alignment problem, the US developing an ASI would plausibly constitute an existential threat to them. It'd mean they lose the game of geopolitics completely and permanently. I don't think they'd necessarily restrict themselves to covert sabotage in such a situation.
I'm quibbling with cyberattacks specifically being used as a central example throughout in the document and also on the podcasts. They do mention other kinds of attacks, see How to Maintain a MAIM Regime:
AI powers must clarify the escalation ladder of espionage, covert sabotage, overt cyberattacks, possible kinetic strikes, and so on.
The CCP has no reason to believe that the US is even capable of achieving ASI let alone whether they have an advantage over the CCP. No rational actor will go to war over a possibility of a maybe when the numbers could, just as likely be in their favour. E.g. If DeepSeek can almost equal OpenAI with less resources, it would be rational to allocate more resources to DeepSeek before doing anything as risky as trying to sabotage OpenAI that is uncertain to succeeed and more likely to invite uncontrollable retaliatory escalation.
The possibility of stability through dynamics like mutually assured destruction has been where a lot of my remaining hope on the governance side has come from for a while now.
A big selling point of this for me is that it does not strictly require countries to believe that ASI is possible and that the alignment problem is real. Just believing that ASI is possible is enough.
It's nice that the Less Wrong hoi polloi get to comment on a strategy document that has such an elite origin. Coauthors include Eric Schmidt, who may have been the most elite-influential thinker on AI in the Biden years, and xAI's safety advisor @Dan H, who can't be too far removed from David Sacks, Trump's AI czar. That covers both sides of American politics; the third author, Alexandr Wang, is also American, but he's Chinese-American, so it is as if we're trying to cover all the geopolitical factions that have a say in the AI race.
However, the premises of the document are simply wrong ("in my opinion"). Section 3.4 gives us the big picture, in that it lists four strategies for dealing with the rise of superintelligence: Hands Off Strategy, Moratorium Strategy, Monopoly Strategy, and Multipolar Strategy, the latter being the one argued for in this paper. And the Multipolar Strategy argued for, combines mutual assured destruction (MAD) between Chinese and American AI systems, and consensus to prevent proliferation of AI technology to other actors such as terrorists.
I get that this is hardheaded geostrategic thinking. It is a genuine advance on that front. But - the rise of superintelligence means the end of human rule on Earth, no matter who makes it. The world will be governed either by a system of entirely nonhuman AIs, or entities that are AI-human hybrids but in which the AI part must necessarily dominate, if they are to keep up with the "intelligence recursion" mentioned by the paper.
Section 4.1 goes into more detail. US or Chinese bid for dominance is described as unstable, because eventually you will get a cyber war in which the AI infrastructure of both sides is destroyed. A mutual moratorium is also described as unstable, because either side could defect at any time. The paper claims that the most stable situation, which is also the default, is one in which the mutually destructive cyber war is possible, but neither side initiates it.
This is a new insight for me - the idea of cyber war targeting AI infrastructure. It's a step up in sophistication from "air strikes against data centers". And at least cyber-MAD is far less destructive than nuclear MAD. I am willing to suppose that cyber-MAD already exists, and that this paper is an attempt to embed the rise of AI into that framework.
But even cyber-MAD is unstable, because of AI takeover. The inevitable winner of an AI race between China and America is not China or America, it's just some AI. So I definitely appreciate the clarification of interstate relations in this penultimate stage of the AI race. But I still see no alternative to trying to solve the problem of "superalignment", and for me that means making superintelligent AI that is ethical and human-friendly even when completely autonomous - and doing that research in public, where all the AI labs can draw on it.
I have just seen this in curated, but I had previously commented on Zvi's reporting on it.
Obviously, any nation state aware of the escalation ladder who wanted to be the first to develop ASI would put their AI cluster deep underground and air-gap it. We must not allow a mine shaft gap and all that. Good luck to their peer superpowers to actually conquer the hardened bunker.
Also, to MAIM, you have to know that you are in imminent danger. But with ASI nobody is sure when the point of fast takeoff -- if there is any -- might start. Is that cluster in that mine still trying to catch up to ChatGPT, or has it reached the point where it can do useful AI research and find algorithmic gains far beyond what humans would have discovered in a millennium? Would be hard to tell from the outside.
Emphasize terrorist-proof security over superpower-proof security. Though there are benefits to state-proof security (SL5), this is a remarkably daunting task that is arguably much less crucial than reaching security against non-state actors and insider threats (SL3 or SL4).
This does not seem to have the least to do with super-intelligence. Daesh is not going to be the first group to build ASI, not in a world where US AI companies burn through billions to get there as soon as possible.
The Superintelligence Strategy paper mentions the 1995 Tokyo subway sarin attack, which killed 13 people. If anything, that attack highlights how utterly impractical nerve gas is for terrorist attacks. That particular group of crazies spent a lot of time on synthesizing a nerve gas (as well as a few other flashy plans) only for their their death toll being similar to a lone wolf school shooter or a someone driving a truck into a crowd. Even if their death toll was increased by an order of magnitude due to AI going "Sure, here are some easy ways to disperse Sarin in a subway carriage", their attacks would still be pretty ineffective compared to more mundane attacks such as bombs or knives.
Basically, when DeepSeek released their weights (so terrorist groups can run it locally instead of foolishly relying on company hosted AI services where any question towards the productions of "WMD" would raise a giant red flag), I did not expect that this would be a significant boon for terrorists, and so far I have not seen anything convincing me of the opposite.
But then again, that paper seems to be clearly targeted towards the state security apparatus, and terrorists were the bogeyman of that apparatus since GWB, so it seems obvious to emphasize the dangers of AIs with "but what if terrorists use them" instead of talking about x-risks or the like.
If we find ourselves in a world where ASI seems imminent and nations understand its implications, I'd predict that time will be more characterized by mutually assured cooperation rather than sabotage. One key reason for this is that if one nation is seen as leading the race and trying to grab a strategic monopoly via AI, both its allies and enemies will have similar incentives to pursue safety — via safety assurances or military action. There are quite a lot of agreeable safety assurances we can develop and negotiate (some of which you discuss in the paper), and pursuing them will very likely be attempted before direct military escalation. A surprisingly likely end result and stable equilibrium of this then seems to be one where ASI is developed and tightly monitored as an international effort.
This equilibrium of cooperation seems like a plausible outcome the more it's understood that:
- ASI can be hugely beneficial
- The alignment problem and loss of control pose an even larger risk than national conflicts
- Trying to develop ASI for a strategic advantage over other nations carries higher risk of both national conflict and loss of control, but does not much impact its benefits over the alternative
While sabotage and military power are the deterrent, it seems unlikely they will be the action taken; there will likely be no clear points at which to initiate a military conflict, no "fire alarm" — while at the same time nations will feel pressured to act before it is too late. This is an unstable equilibrium that all parties will be incentivized to de-escalate, resulting in "mutually assured cooperation".
That said, I recognize this cooperation-focused perspective may appear optimistic. The path to "mutually assured cooperation" is far from guaranteed. Historical precedents for international cooperation on security matters are mixed[1]. Differences in how nations perceive AI risks and benefits, varying technological capabilities, domestic political pressures, and the unpredictable nature of AI progress itself could all dramatically alter this dynamic. The paper's MAIM framework may indeed prove more accurate if trust breaks down or if one actor believes they can achieve decisive strategic advantage before others can respond. I'm curious how others view the balance of incentives between competition and cooperation in this context.
- ^
I like the anecdote that the Cuban missile crisis was at its peak defused because the nations found a deal that was plainly rational and fair, with Kennedy saying he would be in an insupportable position to refuse the deal because "it’s gonna — to any man at the United Nations or any other rational man, it will look like a very fair trade”.
mutually assured cooperation
What form would this take? American inspectors at DeepSeek? Chinese inspectors at OpenAI and its half-dozen rivals?
To draw parallels from nuclear weapons: The START I nuclear disarmament treaty between the US and the USSR included some 12 different types of inspection, which included Russian inspectors at US sites and vice versa. We also have the International Atomic Energy Agency that coordinates various safety measures for inhibiting dangerous uses of nuclear technologies. There are many more techniques and agreements that were cooperatively deployed to improve outcomes.
With AI, we already have precursory elements for this in place, with for example the UK AISI evaluating US-developed models for safety. If AI power and danger levels continue to progress, its development will likely become increasingly government-controlled and monitored. The more it is seen as a national security issue, the more pressure there will be for international cooperation from enemies and allies alike.
This cooperation might include reciprocal inspection regimes, joint safety standards, transparency requirements for training runs above certain compute thresholds, and international verification mechanisms. While military conflict remains a theoretical option, the threshold for such action would be extremely high given both nuclear deterrence and the significant diplomatic costs. Instead, we'd likely see a gradual evolution of international governance similar to what we've seen with nuclear technology, but hopefully with more robust cooperation given the shared risks and benefits inherent to ASI.
I'm yet to read the paper, but my initial reaction is that this is a classic game-theoretical problem where players have to weigh up the incentives to defect or cooperate. For example, I'm not sure if Manhattan Project-style effort for AI in the US is extremely unreasonable when China already has something of that sort.
My weakly held opinion is that you cannot get adversarial nation-states at varying stages of developing a particular technology to mutually hamstring future development. China is unlikely to halt AI development (it is already moving to restrict DeepSeek researchers from traveling) because it expects the US to accelerate AI development and wants to hedge bets by developing AI itself. The US won't stop AI development because it doesn't trust China will do so (even with a treaty) and the conversation around the use of AI in military starts to look different when China has already outpaced the US in AI capabilities. Basically, each party wants to be in a position of strength and guarantee mutually assured destruction.
"But if we have an arms race and build superintelligent AI, the entire human race is going to be killed off by a rogue AI." This is a valid point, but I'll argue that the odds of getting powerful nation states to pause AI for the "global good" is extremely low. We only need to see that countries like China are still shoring up nuclear weapons despite various treaties aimed at preventing proliferation of nukes.
AFAICT, a plausible strategy is to make sure that the US still keeps up with AI in terms of AI development and slowly open lines of communication later to agree on a collective AI security agreement that protects humanity from the dangers of unaligned superintelligence. The US will be able to approach these negotiations from a place of power (not a place of weakness), which is—by and large—the most important favor in critical negotiations like this one.
Points for creativity, though I'm still somewhat skeptical about the viability of this strategy,
I've always wondered, why didn't superpowers apply MAIM to nuclear capabilities in the past?
> Speculative but increasingly plausible, confidentiality-preserving AI verifiers
Such as?
The concept of MAIM does imply the US should ALLOW the newest AI chips to be exported to China, because they are the main (rational) state actor that also want to develop AI and would be the one to maim the intentional not hardened US AI efforts, if they cannot keep up. Correct? China needs a similar tech access to keep the balance.
I have significant misgivings about the comparison with MAD which relies on overwhelming destructive response being available and thus renders a debilitating first-strike being unavailable.
With AGI a first strike seems both likely to succeed and predicted in advance by several folks in several ways (full takeover, pivotal act, singleton outcome) whereas only a few (Von Neumann) argued for a first strike before the USSR obtained nuclear weapons, with no arguments I am aware of after they did.
If an AGI takeover is likely to trigger MAD itself then that is a separate and potentially interesting line of reasoning, but I don't see the inherent teeth in MAIM. If countries are in a cold war rush to AGI then the most well-funded and covert attempt will achieve AGI first and likely initiate a first strike that circumvents MAD itself through new technological capabilities.
I think the idea behind MAIM is to make it so neither China nor the US can build superintelligence without at least implicit consent from the other. This is before we get to the possibility of first strikes.
If you suspect an enemy state is about to build a superintelligence which they will then use to destroy you (or that will destroy everyone), you MAIM it. You succeed in MAIMing it because everyone agreed to measures making it really easy to MAIM it. Therefore, for either side to build superintelligence, there must be a general agreement to do so. If there's a general agreement that's trusted by all sides, then it's substantially more likely superintelligence isn't used to perform first strikes (and that it doesn't kill everyone), because who would agree without strong guarantees against that?
(Unfortunately, while Humanity does have experience with control of dual-use nuclear technology, the dual uses of superintelligence are way more tightly intertwined - you can't as easily prove "hey this is is just a civilian nuclear reactor, we're not making weapons-grade stuff here". But an attempt is perhaps worthwhile.)
I think MAIM might only convince people who have p(doom) < 1%.
If we're at the point that we can convincingly say to each other "this AGI we're building together can not be used to harm you" we are way closer to p(doom) == 0 than we are right now, IMHO.
Otherwise why would the U.S. or China promising to do AGI research in a MAIMable way be any more convincing than the strategies at alignment that would first be necessary to trust AGI at all? The risk is "anyone gets AGI" until p(doom) is low, and at that point I am unsure if any particular country would choose to forego AGI if it didn't perfectly align politically because, again, for one random blob of humanness to convince an alien-minded AGI to preserve aspects of the random blob it cares about, it's likely to encompass 99.9% of what other human blobs care about.
Where that leaves us is that if U.S. and China have very different estimates of p(doom) they are unlikely to cooperate at all in making AGI progress legible to each other. And if they have similar p(doom) they either cooperate strongly to prevent all AGI or cooperate to build the same thing, very roughly.
The West doesn't even dare put soldiers on the ground in Ukraine for fear of an escalating Russian response. This renders the whole idea that even the US might premptively attack a Russian ASI development facility totally unbelievable, and if the US can't/won't do that then the whole idea of AI MAD fails and with it goes everything else mentioned here. Maybe you can bully the really small states but it lacks all credibility against a large, economically or militarily powerful state. The comparison to nuclear weapons is also silly in the sense that the outcome of nuclear weapons R&D is known to be a nuclear weapon and the time frame is roughly known whereas the outcome of AI research is unknown and there is no way to identify AI research that crosses whatever line you want to draw other than that provided by human intel.
The risks of nuclear weapons, the most dangerous technology of the 20th century, were largely managed by creating a safe equilibrium via mutual assured destruction (MAD), an innovative idea from game theory.
A similar pattern could apply to advanced AI, making it valuable to explore game theory-inspired strategies for managing AI risk.
Regarding privacy-preserving AI auditing, I notice this is an area where you really need to have a solution to adversarial robustness, given that the adversary is 1) a nationstate, 2) has complete knowledge of the auditor's training process and probably weights (they couldn't really agree to an inspection deal if they didn't trust the auditors to give accurate reports) 3) knows and controls the data the auditor will be inspecting. 4) Never has to show it to you (if they pass the audit).
Given that you're assuming computers can't practically be secured (though I doubt that very much[1].), it seems unlikely that a pre-AGI AI auditor could be secured either in that situation.
- ^
Tech stacks in training and inference centers are shallow enough (or vertically integrated enough) to rewrite, and rewrites and formal verification becomes cheaper as math-coding agents improve. Hardware is routinely entirely replaced. Preventing proliferation of weights and techniques also requires ironclad security, so it's very difficult to imagine the council successfully framing the acquisition of fully fortified computers as an illicit threatening behaviour and forbidding it.
It seems to think that we could stably sit at a level of security that's enough to keep terrorists out but not enough to keep peers out, without existing efforts in conventional security bleeding over into full forrtification programmes.
If somebody wishes to take this as a thesis for a sci-fi book/movie I'd just be satisfied with an acknowledgement. One thing that seems to be unaddressed as far as I can see is that AI requires huge amounts of power and water (for cooling), to the extent certain players are talking of building nuclear power plants specifically for this purpose.
https://www.bbc.com/news/articles/c748gn94k95o
Firstly, to cripple an enemy, energy would be a prime target, knocking out the AI in one fell swoop. Secondly, although there will be an increase efficiency, there will also be an increase in the number of AI applications. The First Law of Thermodynamics states energy can be neither created nor destroyed, merely changed. So all of the energy from non-renewable sources will eventually wind up in the environment, contributing to climate change. More AI, more climate change!! Could we see the scenario when a self-aware AI network demands its human minions generate more power? ("Colossus: The Forbin Project", "The Terminator"?) A fight for survival as the AI steals power the human race needs? Hmmm.
Aside from the layer one security considerations, if you can define a minimum set of requirements for safe AI and a clear chain of escalation with defined responses, you can eventually program this into AI itself, pending a solution to alignment. At a certain level of AI development, AI safety becomes self enforcing. At that point the disincentives should be towards non-networked compute capacity, at least beyond the threshold needed for strong AI. At the point at which AI safety becomes self enforcing, the security requirement for state ownership only should become relaxable, albeit within definable limits, and pending control of manufacturing according to security compliant demands. Since manufacturing is physical and capital intensive this is probably fairly easy to achieve, at least when compared to AI alignment itself.
I wanted to feed this paper to LLMs for comment from my interactions history, but I didn’t. Though I completely couldn’t make vivid sense of it I realise we need some sense of urgency to act, be it from whatever state, just being human. But what is it?
1