Title is a question because I'm looking for an answer, not providing it. 

I wanted to stress-test some ideas and reason through some questions I have about concentration of power. These have been bubbling in the back of my head for a while, so I've taken two hours this morning to write some of them up and see how others feel.

The main project I'm currently working involves the mitigation of AI proliferation, particularly for offensive capabilities. The core issue is that a) advanced AI can plausibly allow you to make cheap, ultra-destructive weapons, and b) that the cost to train an advanced AI system declines quickly over time because of algorithmic efficiency improvements. As a result, the more you allow proliferation early on, the more invasive and ineffective enforcement becomes, until we hit a point where a few people are empowered to destroy the world at the expense of everyone else (maybe they built a fail-deadly, they're terrorists, irrational, try to build a superintelligence and end up misaligning it, etc).

In my model of things, these destructive technologies are offense dominant (impluasibly inefficient or impossible to defend against), which means that the only realistic way to control them is to limit their spread preemptively. While it's hard to predict the future, I'd expect that the first ASI labs to end up nationalized by the USG and PRC, meaning that offensive capabilities are initially monopolized. If they recognize the security interest of restricting AI proliferation (as the U.S and Soviet Union mutually agreed to do with nukes), they will use their strategic position to try and disempower other states/actors from getting access to offensive AI capabilities before they can become cheap to access. Perhaps they commandeer the entire AI supply chain, and put restrictions in place such that all future AI systems are aligned to the US-China coalition goals. 

As far as I can see, this would reasonably secure the world against the threat of black ball AI tech. But an important secondary issue it creates is concentration of power, wherein the rest of humanity is subjected to the whims of the USG and PRC, because they have a monopoly on violence through their control of ASI technology and the commandeering they just did, and have to continue to do to in order to make sure an unauthorized ASI is never built.

Here are some of my intuitions about this situation:

1. First, there doesn't seem to be a reasonable alternative to this sort of monopolization. Offense-dominant technology is only really curtailable by MAD style policy, which breaks down as the number of actors increases. If you want humanity to enjoy the benefits of powerful AI systems, then you need to make sure that the dual-use systems are preemptively controlled before they're distributed, which means that they're passed through some kind of central authority first. That central authority needs to have a strategic advantage over everyone else, in order to make sure that they're genuinely not building the dangerous technology (think IAEA, sanctions, bombing Iranian enrichment facilities, etc).
   1. As a simple example, consider allowing other countries/companies/individuals to train their own ASI. If this ASI might end up misaligned, and there are cheap weapons it can develop that can threaten everyone else, each of these actors is a danger to the whole of civilization. The only way to avoid this scenario is to disempower those future actors ahead of time, such that they can never build the misaligned ASI or comparable weapons in the first place.
   2. Basically, I think d/acc (at least the defensive and decentralized components) is an unviable strategy, because you won't be able to reliably defend against the widely distributed and cheap offensive technology ASI would allow. While I'm sure there are comprehensive, defense dominant solutions that can be widely distributed in some domains (cryptography being the classic example), I don't think this is true of all future offensive technologies, which it needs to be. If there's even a single cheap, existentially powerful technology that exists (and we already have promising candidates like mirror life or misaligned ASIs), then the plan of allowing AI to proliferate for defense will have the opposite effect of favoring bad actors.^[1]
2. This doesn't mean that ordinary people can't be allowed to use any AI technology. They can! It just needs to be centrally processed and controlled first to make sure the dual-use stuff is restricted in some way (certain robotics, AI r&d, biology, engineering, and material sciences applications most likely). This looks something like allowing the use of superhuman medical researchers to create wonderdrugs, although the program is backdoored/pre-aligned/monitored to make sure it cannot comply with requests to develop bioweapons.
3. Second, it seems to me that most people might end up ok despite the concentration of power required by having governments monopolize ASI, even in the most extreme dicatorial scenarios.
   1. The primary reason is that, given the abundance that ASI might enable, both in the efficient production of goods and explosive supply of labor, it would become very cheap (as a fraction of GDP) to have most people live extraordinarily good lives. Eliezer has made the point before that misaligned ASIs would have no reason to spare even a tiny fraction of their resources, because they won't have even a tiny preference to satisfy. While I think this is likely true of misaligned ASIs, it seems wrong to argue that the same is true of most humans, who do have some nonzero preference for being altruistic along with their other goals. If we end up in a situation in which almost all labor ends up controlled by a handful of company executives/politicians, it would only take a few of them committing to a permanent welfare payment for everyone to live great lives (at a small fraction of their actual resources equivalent to their preferences).
   2. A common criticism of this altruist/redistributionist perspective is that Moloch devours all, and that those who don't advocate for human welfare will eventually outcompete those that do. However, it seems unclear that there's a realistic mechanism for the non-welfare advocates to do this to the altruist decision makers, given the presence of offense-dominant weapons at the limit of intelligence. Even though the U.S has unbelievable conventional military superiority to North Korea, for instance, the fact that they have nuclear weapons means that we cannot arbitrarily impose our preferences about how North Korea should act onto them. Although the U.S might "win" a nuclear exchange with North Korea, the damage it would be able to cause to the things we value in the process is so large that we don't take the steps that would disempower them in the first place.
      
      Roughly speaking, you can swap out "U.S" and "North Korea" with "Optimizers" and "Altruists". Although there might be some (or even most) decision makers who would prefer to get rid of all the strategically/politically disempowered humans who are consuming resources to better optimize for more galaxies/inter elite status games/etc, there probably won't be a decisive action they can take against civilians or the altruists that advocate for them that doesn't end in catastrophic retaliation. If an altruist has their hands on offense-dominant ASI technology, then they can just commit to using it in retaliation if they confirm that the optimizers are trying to disempower them. As a somewhat modern parallel, it's reasonable for the U.S to discard plans to assassinate Kim Jong Un on the basis that it wouldn't be able to stop a retaliatory strike when the regime panics. And given the coordination/verification mechanisms of superintelligences for upholding these contracts, rather than relying on human decision making in the moment, you could probably get pretty ironclad guarantees about the behavior of other actors in spite of their preferences.^[2]
4. This situation would still lead to unimaginable imbalances in material wealth, probably many orders of magnitude more extreme than today. But I don't believe that inequality is intrinsically problematic from a welfare perspective: it's far more important that the people at the bottom meet the absolute threshold for comfort than it is for a society's Gini coeficient to be lower. The fact that Elon Musk can afford to send his kids to university for millions of years doesn't detract from the four years I spent attending on scholarship. The only direct threat the ultra-wealthy pose to my wellbeing is their capture of the state/important institutions that I use, which in the above scenario is compensated for by altruists who can cheaply afford for me to be happy and are able to threaten something their competitors care about over it. 

Given all of this, concentration of strategic power seems both necessary and... probably ok? Even in this worst-case scenario, where power is totally monopolized by a handful of people (say the Oversight Committee from AI 2027), all it takes is a small preference among some of them for civilians to live great lives for them to be able to do so. The state needs to clamp down on offense-dominant offensive capabilities to make sure that non-state actors never get access to them (or else someone will eventually use them), but afterwards, beneficial AI technology can be allowed to mostly flourish.  

Of course, I still think it would be better to have outcomes like value-lock-in of welfare/political rights for all humans at the inception of ASI, or for the U.S and China to somehow avoid intent alignment and instead aim straight at civilizational alignment where the ASI runs everything itself, rather than having to account for the preferences of very few. But I believe the above situation---nationalization of labs, intent aligned AI, realization of the strategic importance of non-proliferation, permanent monopolization of strategic power---represents the default path to dealing with the proliferation problem, so it seems important to reason about what it might look like and whether that's acceptable. 

1. ^{^}

   As an example of the kinds of defensive plans I'm talking about, I'm referring to Luke Drago and Rudolf L's proposal for defensive augmentation, or Helen Toner's adaptation buffers, where we prepare society ahead of time through the wide-scale distribution of technology.
   
   I have several thoughts on this, which I'll explore in a future, properly edited post on the strategic implications of proliferating powerful AI systems. My main critique is that I do not share their optimism that you can efficiently respond to all existential threats, given the cost-imbalance of defending against them. As the number of threats and actors increase, spending on defense has to grow exponentially. Offensive investments, on the other hand, are linear, since they can focus all of their effort on targeting the areas that are under-invested or even impossible to defend against. Ie, it's much cheaper to try and overwhelm defensive capacity by just launching two pandemics at the same time than it is to make sure you are prepared for multiple pandemics. 

   Analogously, we can think about how ridiculously inefficient trying to intercept nuclear ICMBs is, and how you could obsolete the hundreds of billions of dollars worth of interceptor research you'd need by a) building faster missiles, b) using more missles, or c), just putting your bomb in a shipping container and docking it next to Washington.

   And even if you secure one avenue (wastewater monitoring, DNA synthesis screening, and KYC ends up being enough defense-in-depth for bioweapons that it's basically solved), there's no guarantee you can solve them all. If you upgrade the problem from a pandemic to mirror life, for example, outbreak monitoring and antibiotic response plans start to break down completely. And so on for swarms of insect-sized LAWs, nanotechnology, misaligned ASIs, and the unknown-unknowns of weapons development that lurk at the top of the tech tree. The only reasonable solution to this problem is just to make sure other people don't have weapons in the first place, or preemptive non-proliferation.

   I do not have the confidence to distribute general ASI without a guarantee that people will not have access to these offense-dominant technologies, and the only way to make sure that their AI cannot do those things would be for a more powerful central authority to have achieved ASI first and used their first-mover advantage to strip all future offensive capabilities away from other actors. 

2. ^{^}

   Similarly, even if the Chinese government decided to try to be more competitive by ignoring the welfare of its own citizens, it wouldn't necessarily be able to impose its will on the U.S given the disproportionate leverage of offense-dominant weapons.