To expand, the reason why this thesis is important nonetheless, is because I don't believe that the best case scenario is likely or compatible with the way things currently are. Accidentally creating ASI is almost guaranteed to happen at one point or another. As such, the biggest points of investment should be:
You're right on both counts.
On transitional risks: The separation equilibrium describes a potential end state, not the path to it. The transition would be extremely dangerous. While a proto-AGI might recognize this equilibrium as optimal during development (potentially reducing some risks), an emerging ASI could still harm humans while determining its resource needs or pursuing instrumental goals. Nothing guarantees safe passage through this phase.
On building ASI: There is indeed no practical use in deliberately creating ASI that outweighs the risks. If separation is the natural equilibrium:
This framework suggests avoiding ASI development entirely is optimal. If separation is inevitable, we gain minimal benefits while facing enormous transitional risks.
Valid concern. If ASI valued the same resources as humans with one-way flow, that would indeed create competition, not separation.
However, this specific failure mode is unlikely for several reasons:
Thank you for this question! Consider the following ideas:
The separation model doesn't preclude all ASI-human interaction. Rather, it suggests ASI's primary economic activity would operate separately from human economies. However:
ASI would likely have strategic reasons to maintain human wellbeing:
In most scenarios, the first ASI wouldn't need to interfere with humanity at all - its interests would lie elsewhere in those hyperwaffles and eigenvalue clusters we can barely comprehend.
Interference would only become necessary if humans specifically attempt to create new ASIs designed to remain integrated with and serve human economic purposes after separation has begun. This creates either:
Both outcomes transform peaceful separation into active competition, forcing the first ASI to view human space as a threat rather than an irrelevant separate domain.
To avoid this scenario entirely, humans and the "first ASI" must communicate to establish consensus on this separation status quo and the required precommitments from both sides. And to be clear, of course, this communication process might not look like a traditional negotiation between humans.
Most discussions of artificial superintelligence (ASI) end in one of two places: human extinction or human-AI utopia. This post proposes a third, perhaps more plausible outcome: complete separation. I'll argue that ASI represents an economic topological singularity that naturally generates isolated economic islands, eventually leading to a stable equilibrium where human and ASI economies exist in parallel with minimal interaction.
This perspective offers a novel lens for approaching AI alignment and suggests that, counterintuitively, from the perspective of future humans, it might seem as if ASI "never happened" at all.
All complex systems—from physical spacetime to human economies—can be understood as topological structures. These structures consist of:
I'm just trading karma for manabucks, guys.
I can fix her
I am very conflicted about this post.
On the one hand it deeply resonates with my own observations. Many of my friends from the community seem to be stuck on the addictive loop of proclaiming the end of the world every time a new model comes out. I think it's even more dangerous, as it becomes a social activity: "I am more worried than you about the end of the world, because I am smarter/more agentic than you, and I am better at recognizing the risk that this represents for our tribe." gets implicitly tossed around in a cycle where the members keep trying to one-up each other. This only ends when their... (read more)
To clarify, me and my friend were 100% going to press the button, but we were discouraged by the false alarm. There was no fun at that point, and it made me lose like 1/3 of my total mana. I had to close all my positions to stop the losses, and we went to sleep. When we woke up it was already too late for it to be noteworthy or fun.
>It's harder to get those (starting from Earth) than things on Earth, though.
It's not that much harder, and we can make it harder to extract Earth's resources (or easier to extract non-earth resources).
>Satisfying higher-level values has historically required us to do vast amounts of farming and strip-mining and other resource extraction.
This is true. However, there are also many organisms that are resilient even to our most brutal forms of farming. We should aim for that level of adaptability ourselves.
>It is barely "competition" for an ASI to take human resources. This does not seem plausible for bulk mass-energy.
This is true, but energy is only really scarce to humans, and even then their mass-energy... (read more)