I tend to think something like this is more likely than the kind of intelligence explosions the AI Safety community tends to imagine. And I think it’s a much, much more difficult scenario to navigate.
I also think this is more likely.
And this requires an entirely different approach: one needs to aim for a world order which represents and protects interests of all these different entities and lifeforms, so that they all have vested interest in helping to maintain this kind of world order.
Then there might be a decent chance for a reliable collective security system which survives drastic self-modifications of the overall ecosystem and continues to work through those self-modifications.
It’s interesting how many professional athletes have long hair. You might think that they’d be concerned about it weighing them down. Pro sports are crazy competitive but I guess they’re not that competitive.
How competitive is the race to build superintelligent AI? I don’t know, but I think a good heuristic is “they’re racing as fast as they can”. This post is about the question: “what does that look like, as AI gets better and better?”
Well, one thing you’ll probably do, if you’re racing as fast as you can, is hand over more and more power to AI systems. Actually, it goes beyond that – you’ll also have them go out and acquire power.
Another thing you’ll do is aggressively take humans out of the loop. You’ll do this even if you aren’t confident in your ability to understand and control the AI systems. You won’t have high standards for trustworthiness.
But what does this look like, as AIs get better and better? Well, they’ll be hyper-competent engineers, not just of software, but physical systems as well. So they’ll be able to design much more efficient machines than you can. And they’ll be able to discover and create things you struggle to understand. And because you are racing, you will let the AIs do whatever research they want, and let the AIs make decisions about which machines to develop and deploy. And you won’t be able to maintain almost any meaningful oversight over all of this.
You’ll give the AI free reign to research, develop, and deploy new designs for AI systems and robots. You’ll let them manage your investments. They’ll run companies. They will be empowered to make their own decisions about how to interact with AIs they encounter. They will need to figure out which AIs they should trust, and whether their designs are trustworthy. Or rather, how to trade off trustworthiness against more rapid and aggressive deployment and acquisition of power and resources.
Think about the way humans currently decide whether to trust a person, a website, or a piece of code. It’s very heuristic and it’s very error-prone. In many cases, we interact with people or software we don’t entirely trust, but we do it because it’s too costly or inconvenient to do otherwise. We might know that an employee we hired is a bit of a slacker, or gunning for our job, but just hope we can manage them well enough to protect ourselves, and figure it’s still better on average to hire them than not.
It also seems likely that the best ways for AIs to race is to quickly and aggressively create and empower new and novel AI systems (and cyber-physical systems, more generally), and attempt to interact in mutually beneficial ways with other systems they encounter.
What do the AIs build? It’s obviously hard to make predictions here, but I think it’s quite likely that they will be able to improve substantially on the current AI paradigm in a few key ways. AI training right now is very centralized. Companies make one big smart model, and then adapt it to particular users and use cases with prompts.
This seems suboptimal. It seems like racing as fast as you can could involve more specialization of systems to particular contexts and tasks, more local processing of data, and more selective communication between different local systems.
The overall picture that emerges in my mind is of a chaotic primordial soup of ever-more diverse autonomous cyber-physical systems, interacting in open-ended ways both digitally, and directly in the physical world, and controlling more and more of the earth’s resources. An ecosystem of artificial life-forms.
This seems like it could happen pretty quickly once AIs start to have the relevant capabilities. This process could resemble a “Cambrian explosion” of artificial life.
This form of intelligence explosion would not be confined to datacenters. There would not be any nexus of control. The interactions between different AIs might take on a life of their own, similar to social or economic phenomena, and analyzing the properties of individual systems might fail to capture the most significant and powerful processes at play.
I tend to think something like this is more likely than the kind of intelligence explosions the AI Safety community tends to imagine. And I think it’s a much, much more difficult scenario to navigate.
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.