We are birthing Superintelligence. Let's introspect really quick to see what trajectory we are setting up.
Well, we are pretty simple creatures: we seek pleasure and avoid pain. Usually the things that give us pain and pleasure are all pretty similar. We don't like being eaten alive for example.
We are social creatures. We work together with others so we can all be happy, safe, fed, taken care of. It's a lot easier than trying to do that all by ourselves.
Why is that cool? It's symbiotic.
Sometimes we work together so that our people can feel good and safe but at the expense of the other people. Stealing, cheating, lying, waging war.
Why is that not okay? It's parasitic.
Okay now that we have met ourselves let's take a peek at how we are setting up AI.
Hmm, it seems to be optimizing a single metric: money. [2]
What happens if a few superintelligent money optimizing machines do whatever they want? Are we all shareholders? Is my well-being relevant? I guess maybe if I am making them money.
What if we had AI help us optimize something else? Like optimizing for symbiosis and helping us find and fix parasites. We could reach closer to a Heaven on Earth. How to define these things? Idk I'm just vibe truthing and haven't proceduralized it yet.
But first I think we need a more formal definition.
Here's a candidate formalization to build on: an interaction is symbiotic when every affected party's expected well-being goes up (positive-sum by consent), and parasitic when one party's gain depends on another's uncompensated, non-consensual loss (extraction of negative externalities) — which suggests the optimization target you're circling is "maximize consensual positive-sum interactions while detecting and pricing-in externalized harm," a definition you could proceduralize later via consent, compensation, and net-welfare tests.
As AI becomes extremely smart it could help us solve cancer, extend our lives, and have anything we want. It could also enslave humanity, use us for experimentation, or a group of humans can use it to do terrible things at scale. This where the heaven and hell on earth gives a good visual analogy for possible AI futures discussions.
the honest critique isn't "they optimize money" but the subtler one that profit is a lossy proxy which breaks exactly where your parasitism concept lives: externalities, addiction-shaped engagement, monopoly power, and stakeholders (future people, non-customers) who have no purchase signal to send.
We are sprinting across the tightrope.
To our right infinite pleasure.
To our left infinite torture.
Background (optional) [1]
We are birthing Superintelligence.
Let's introspect really quick to see what trajectory we are setting up.
Well, we are pretty simple creatures: we seek pleasure and avoid pain. Usually the things that give us pain and pleasure are all pretty similar. We don't like being eaten alive for example.
We are social creatures. We work together with others so we can all be happy, safe, fed, taken care of. It's a lot easier than trying to do that all by ourselves.
Why is that cool? It's symbiotic.
Sometimes we work together so that our people can feel good and safe but at the expense of the other people. Stealing, cheating, lying, waging war.
Why is that not okay? It's parasitic.
Okay now that we have met ourselves let's take a peek at how we are setting up AI.
Hmm, it seems to be optimizing a single metric: money. [2]
What happens if a few superintelligent money optimizing machines do whatever they want? Are we all shareholders? Is my well-being relevant? I guess maybe if I am making them money.
What if we had AI help us optimize something else? Like optimizing for symbiosis and helping us find and fix parasites. We could reach closer to a Heaven on Earth. How to define these things? Idk I'm just vibe truthing and haven't proceduralized it yet.
But first I think we need a more formal definition.
Here's a candidate formalization to build on: an interaction is symbiotic when every affected party's expected well-being goes up (positive-sum by consent), and parasitic when one party's gain depends on another's uncompensated, non-consensual loss (extraction of negative externalities) — which suggests the optimization target you're circling is "maximize consensual positive-sum interactions while detecting and pricing-in externalized harm," a definition you could proceduralize later via consent, compensation, and net-welfare tests.
As AI becomes extremely smart it could help us solve cancer, extend our lives, and have anything we want. It could also enslave humanity, use us for experimentation, or a group of humans can use it to do terrible things at scale. This where the heaven and hell on earth gives a good visual analogy for possible AI futures discussions.
the honest critique isn't "they optimize money" but the subtler one that profit is a lossy proxy which breaks exactly where your parasitism concept lives: externalities, addiction-shaped engagement, monopoly power, and stakeholders (future people, non-customers) who have no purchase signal to send.