Yes, to create simulations AI needs some real humans to calibrate these simulations. And it needs simulations to predict behaviour of other possible AIs which it can meet in space and their progenitor civilizations.
If AI successfully calibrates simulations, it will not need humans, or if it collect all needed data from simulations, it will turn them off.
Also, obviously, surviving in simulations is still disempowerment of humans, can cause suffering at large scale and death of most people.
Value-handshake is more promising way to ensure AI safety of this type.
One of my points was that humanity has a level of complexity that means that an AI couldn't simulate humanity perfectly without humanity.
So, a superintelligent AI would keep us because it would want to observe humanity, which can involve observing us in reality. I doubt that AI can "successfully calibrate simulations [of humanity]" as you mentioned.
My thought was different that. That even if simulation is possible, it needs original for verification.
Also, one way to run simulations is 'physical simulations' like in Trumen Show or Alien Zoo: a real planet with real human beings which live their lives but the sky is not real at some distance and there are thousands such planets.
Assuming a superintelligent AI wanted to maximize its knowledge, I argue that a superintelligent AI must avoid destroying humanity.
I'll cover some information theory, how irreducible complexity means no perfect simulation of humans, and the "simming problem" from my favorite author, Iain M. Banks.
Contrary to AI doomers, my view is that the human complexity means that a superintelligent AI won't destroy humans. Such an AI would probably want to preserve us.
My main idea is that, no matter how smart an AI is, its model of the world can never be the world itself. This is the map vs. territory analogy.
A map can be incredibly detailed - every street, every building - but it's still a shrunken, simplified representation of the real city. It won't match the city at the atomic level.
As Alfred Korzybski first said (and now verges on cliche), "the map is not the territory" . The only complete map of a territory would literally be the territory. The same goes for an AI's understanding of humanity. Humans are the territory; the AI's internal model of humans is the map.
In computer science, there's a concept called Kolmogorov complexity. This measures the shortest possible program that produces an object or data.
If something is truly random or highly complex, the shortest description is the thing itself.
The same thing applies to humans.
Think of your life - every moment, thought - as data. The only program that can perfectly capture that might be as long - in other words as complex - as your life itself!
In other words, there may be no shortcut to fully encoding a human being; the AI would need to recreate every atom, every neuron. This is a profound limit: some systems can't be compressed or simplified without losing information.
In other words, there may be computational irreducibility in complex systems, such as human society, which can't be modeled.
For many complex processes (like weather patterns or certain cellular automata), the only way to know what they'll do is to actually let them run step by step - you can't shortcut it. In Stephen Wolfram's words, the "analyzers or predictors" of a system can't be more sophisticated than the system itself.
For any AI trying to simulate humanity, this means that an AI can't just perfectly predict human society with equation that abstracts away information (i.e., a map of the actual territory).
Humans seem likely to be computationally irreducible. Meaning that an AI would have to trace every interaction, every event, to truly predict what we'll do. (This doesn't mean AI has no prediction power over humanity; I'm simply saying that there are unexpected events that exist, which require observing to understand.)
In the 'State of the Art', The Arbitrary (who is a Mind - ie., superintelligent AI) states: "I'm the smartest thing for a hundred light years… but even I can't predict where a snooker ball's going to end up after more than six collisions."
This captures a real principle of chaos theory that would make it extremely computationally difficult for an AI to model human society.
The first few billiard collisions on a pool table are easy to predict with physics. But by the time you try to predict the ninth collision, you literally need to factor in the gravitational pull of a person standing next to the table.
By the 56th collision, Berry showed you'd need to account for every single elementary particle in the universe - even an electron 10 billion light years away. (See section 8.6 in Berry's paper here)
In other words, the system's complexity explodes. This is often called the "butterfly effect", or sensitive dependence on initial conditions. It's why weather is so hard to forecast, and why simulating entire human civilizations would be ridiculously difficult and complex.
Even a superintelligent AI might simply be unable to deal with this amount of complexity.
Another problem is that even if you could simulates living beings, you'd need to create simulations so detailed to be accurate that the simulation would effectively become conscious entities.
This creates an ethical problem: ending the simulation involves killing the life that you created. This would fit within Nick Bostrum's concept of 'Mind crime'.
Here's a nice description about the "simming problem" from Iain M Banks:
"Sometimes, if you were going to have any hope of getting useful answers, there really was no alternative to modeling the individuals themselves, at the sort of scale and level of complexity that mean they each had to exhibit some kind of discrete personality, and that was where the Problem kicked in.
Once you'd created your population of realistically reacting and – in a necessary sense – cogitating individuals, you had – also in a sense – created life. The particular parts of whatever computational substrate you'd devoted to the problem now held beings; virtual beings capable of reacting so much like the back-in-reality beings they were modeling – because how else were they to do so convincingly without also hoping, suffering, rejoicing, caring, living and dreaming?
By this reasoning, then, you couldn't just turn off your virtual environment and the living, thinking creatures it contained at the completion of a run or when a simulation had reached the end of its useful life; that amounted to genocide."
(from 'The Hydrogen Sonata' by Iain M Banks).
So, a superintelligent AI faces an information theory wall: humans (and our society) are likely:
(Sidenote: perhaps we are a simulation in a more advanced machine?)
Let's address some argument against what I've said.
Couldn't a superintelligent AI just approximate humans enough to get by, or use proxies? Maybe it doesn't need a perfect simulation; maybe "good enough" is enough.
Counterargument 1: "The AI can simulate humans well enough for its purposes." Perhaps it doesn't need every neuron; it could simulate our behavior with a coarse model.
Answer: In routine tasks, ues. But the argument here is about fundamental unpredictability. The AI might simulate, say, global economic behavior with a decent model. But what if a small, unpredictable human innovation or a cultural shift throws it off?
Counterargument 2: "The AI might not need understanding at an individual level; it can use statistical proxies."
Answer: That can work for some things, but not for others. Complex systems often have critical rare events or outliers.
Assuming a superintelligent AI wants to learn as much as possible, destroying humanity would remove a way to observe that complexity.
By definition, a superintelligent AI, would understand this principle. It would recognize the limits of its own maps regarding humanity.
Humanity is the ultimate reference point for certain problems; eliminate us, and the AI might never solve those problems.
So, a truly advanced AI would avoid destroying us because of its interest in maximal learning.
(Crossposted from tomdekan.com)
Note: I'm assuming that a superintelligent AI would be heavily incentivized to understand the universe and expand its knowledge. By definition, a superintelligent AI has solved most of its instrumental goals, so its core incentive might be simply to 'learn' as much as possible. Further, 'knowledge' gives power to the AI - more power than simply 'destroying'.