Very thought provoking question. To prevent what you described from happening, I think the best way is to have open source models that remain close to the frontier, so that ai is not concentrated in a few companies or governments.
I also wonder to what extent the end state you described is the most likely path. We will probably go through a period of "hybrid" workforce where ai needs to work with humans. There will be political parties, firms, governments, alliances, and rival states all competing for "power" (quoted because what counts as power can vary widely in this context). Each actor is incentivized to concentrate power themselves, but they also have incentives to prevent others from doing the same.
The mass still has valuable bargaining power during the hybrid state. First, they have economic leverage, like the assets that they own. Second, ai could significantly lower the cost of coordination and resistance. The self-preservation instinct should be very strong to avoid entering an equilibrium where there's no chance to fight back. Whether this is enough remains to be seen.
In the end, I suspect the world won't be a hegemony. Autocratic states are much more likely to end up in the scenario you described.
I'm glad my post got you thinking; thanks for the comment. As for open-source models, I’m still undecided on how I feel about them. On the one hand, they’re truly necessary to maintain independence from providers and avoid falling too far behind. On the other hand, AI is approaching an extremely dangerous level of capability, and when many people have such AI at their full disposal, the risks may start to become a reality.
Once the entire economy—or nearly the entire economy—is automated, people may lose their political power. Since people are no longer needed to carry out orders, those in power gain the opportunity to engage in repression with impunity and to consolidate their power even further. This possibility, for example, is discussed by Acemoglu et al.
Throughout history and to this day, the state remains physically dependent on the cooperation of its population. If a sufficient number of people refuse to follow orders, the system can break down. Therefore, it is in the state’s interest to cooperate with its population, at least to a minimal extent.
But if we imagine a world where most labor is performed without human involvement, the government may begin to disregard the interests of its citizens. Now that the government is far less dependent on the population, it can, in certain situations, disregard them. This relationship may be neutral most of the time, but if the interests of the population and the government diverge, what will protect people from repression?
In fact, this is a very broad issue. It touches on both philosophical and historical questions—such as what power and political will are, and what prevents rulers from being tyrants everywhere—as well as technical questions, such as who exactly will manage an automated economy and how. In this post, I’ll discuss just a couple of ideas, and only briefly. I hope others will continue the discussion.
A widely known and long-debated concept is UBI—universal basic income. But in its simplest form, and without alternative sources of income, UBI creates an additional condition for repression. In a world where people are unable to earn a living on their own and rely solely on government payments, they become extremely vulnerable. People can be manipulated by the threat of reducing or even taking away their UBI. They become puppets in the hands of those who control the distribution of money.
There are several variations on how this could be organized more safely. For example, UBO—universal basic ownership. Instead of simply paying out money, we make every inhabitant of the planet a co-owner of the economy. Absolutely everyone has a share of global capital by virtue of being born, and thus the payments are dividends (similar to what was described in the AI-2040 scenario). Something similar already exists: the Alaska Permanent Fund pays dividends to Alaskans from the sale of natural resources.
The right to be a co-owner of the economy cannot be transferred or sold to another person, which guarantees the individual’s financial security throughout their life. It may be possible to sell only a small portion of this share if a person needs additional money.
But if necessary, the government can still “cut off” dividends. To avoid such a scenario, ownership must be secured by something other than legislation. People must continue to have the ability to refuse to cooperate with the authorities, as has been the case throughout history.
I propose the idea not only of ownership but also of control over one’s share of the economy. For example, every automated asset contains a hardware security module (HSM), and to continue operating, it must regularly receive cryptographic authorization. Every person on Earth possesses the digital equivalent of their share, and by default, authorization is active, but they can revoke it at any time. The cryptographic key is sent to the HSM only as long as the majority of people have not revoked their authorizations. Verification is carried out by a special body that aggregates authorizations and checks whether they meet a baseline threshold—for example, 75% agreement. In this way, people still retain the ability to go on strike and halt part of production.
Like almost all other governance elements, payments must be organized by different bodies. This will help diversify the risk that a single bad actor could deprive a person of the resources necessary for survival. For example, although people receive dividends from the global economy, each country could independently make payments to every person on Earth. Alternatively, this could be done by various unrelated funds and trusts within a single country, each managed by different people or AI systems.
We should already be actively exploring possible ways to maintain political power in the wake of widespread automation, without limiting ourselves to UBI alone, since this may happen much sooner than many people expect.
That is, of course, if humanity survives at all.