This is a special post for quick takes by Mitchell_Porter. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Some musings on the shape of the world before complete AI takeover:
Some of the AI companies/leaders - the ones who talk about basic income - seem to have a daydream according to which AI (possibly in combination with robotics) will put everyone out of a job, except for the people who work at the AI companies; but all those unemployed people will get a payment doled out by their government.
If we assume, for the sake of discussion, a world in which there is still a US government, and still multiple AI companies, each of which now sits atop an AI/robot economy that uses its products... one may suppose an agreement in which a fraction of each company's income or resources is given to the government, which in turn gives it to the unemployed people so they can afford AI-economy goods and services. Someone might also become a customer/subscriber/dependent of a specific AI company and thereby get access to its subsidiary economy without going through the government. Finally, there may also be an "anti-AI" economy, among humans who agree to avoid AI products entirely.
It's not quite clear what happens to countries that don't have their own AI industries, but they might try to be part of such arrangements. One has to suppose that the AI powers also have a treaty or protocol that sharply regulates proliferation of core AI technology to new countries.
On top of the uncontrolled effects of AI on society, we could have a deliberate effort to create this kind of a world. However, one doesn't expect AI capabilities to stand still (unless AI treaties not only restrict AI development outside the trusted powers, but also restrict AI development within them), and so AI won't just produce entities fit to do white-collar work, it will produce entities fit to converse as equals with the new human aristocracy (by which I mean the remaining humans in charge of the AI companies and in charge of the government). To this picture of a managed AI future, one might therefore add a debate among humans "aristocrats" about the status of these most advanced AIs, to which they can speak as equals - should they be made full members of aristocratic society under certain conditions, do they always need a human caretaker otherwise, and so forth.
I am trying, here, to be as sympathetic as possible to the idea that AI can be developed, and take over all labor, without completely displacing or replacing the human race. However, I think it's more logical to suppose that AI swiftly surpasses even human CEOs and politicians, and becomes its own autonomous alien self, living in wild indifference to their pleas and commands. At that point a radically posthuman ecology of mind and machine is in motion, and these daydreams of peaceful human/AI coexistence are no longer relevant.
Isn't it the very outcome that mankind tries to avoid by ensuring that the AIs are aligned?
Yes, but it is also the default outcome, unless the first team to create superintelligence is successful at that complex and unprecedented design challenge.