Most people imagine that if we get a benevolent ASI, we'll see a Golden Age slowly appear over 20 or 50 years. I think this is wrong because it ignores the speed difference between biology and physics.
If an ASI really figures out how to control matter at the fundamental level (post-Standard Model physics), it won't wait for us. Our reaction time is ~200ms. The ASI acts nearly instantly.
So my model is that it will feel like a Jump Cut. One moment you are here. The next moment, you are in a state I call "The Lobby."
This isn't a waiting room. It's the moment where biological constraints are gone, and you are placed in a new context.
What is the goal? People fear wireheading (just feeling pleasure in a tank). But I think a smart AI knows that humans need Eudaimonia (meaning/flourishing), not just dopamine. You want to feel existentially "right" for your role.
You might want to live a life of drama, or learning, or heroism.
The Ethics Problem (Solved by Infinity) The usual counter-argument is: "If I want to be a King, don't I need subjects? Is the AI creating slave-minds for me?"
I think the solution is just math. In an infinite (or effectively infinite) universe with infinite conscious minds, the matching problem is trivial. If you want to lead, the AI matches you with minds that genuinely want to follow. If you want to save someone, it matches you with someone who wants the narrative of being saved. The AI is just a universal matchmaker in a limitless configuration space. No NPCs needed.
Not a Simulation (Important!) This is the part that matters most to me: This is not a simulation. It is not FDVR.
If the ASI controls the substrate of reality, there is no difference between "virtual" and "real." It creates the ontology physically. If you are living a life on a reshaped Mars, the air you breathe is made of real atoms assembled by the AI. The light is real photons. It's a manipulation of the territory, not a map on a server.
So, if alignment works, don't expect a slow climb. Expect to blink and wake up in a reality physically built to match your internal needs.
Most people imagine that if we get a benevolent ASI, we'll see a Golden Age slowly appear over 20 or 50 years. I think this is wrong because it ignores the speed difference between biology and physics.
If an ASI really figures out how to control matter at the fundamental level (post-Standard Model physics), it won't wait for us. Our reaction time is ~200ms. The ASI acts nearly instantly.
So my model is that it will feel like a Jump Cut.
One moment you are here. The next moment, you are in a state I call "The Lobby."
This isn't a waiting room. It's the moment where biological constraints are gone, and you are placed in a new context.
What is the goal?
People fear wireheading (just feeling pleasure in a tank). But I think a smart AI knows that humans need Eudaimonia (meaning/flourishing), not just dopamine. You want to feel existentially "right" for your role.
You might want to live a life of drama, or learning, or heroism.
The Ethics Problem (Solved by Infinity)
The usual counter-argument is: "If I want to be a King, don't I need subjects? Is the AI creating slave-minds for me?"
I think the solution is just math. In an infinite (or effectively infinite) universe with infinite conscious minds, the matching problem is trivial.
If you want to lead, the AI matches you with minds that genuinely want to follow. If you want to save someone, it matches you with someone who wants the narrative of being saved. The AI is just a universal matchmaker in a limitless configuration space. No NPCs needed.
Not a Simulation (Important!)
This is the part that matters most to me: This is not a simulation. It is not FDVR.
If the ASI controls the substrate of reality, there is no difference between "virtual" and "real." It creates the ontology physically. If you are living a life on a reshaped Mars, the air you breathe is made of real atoms assembled by the AI. The light is real photons. It's a manipulation of the territory, not a map on a server.
So, if alignment works, don't expect a slow climb. Expect to blink and wake up in a reality physically built to match your internal needs.