If nothing else, I expect mildly-superhuman sales and advertising will be enough to ensure that the human share of the universe will decrease over time. And I expect the laws will continue being at least mildly influenced by deep pockets, to keep at least some such avenues possible. If you imagine a hard lock on these and other such things, well that seems unrealistic to me.

Like I said, people who have been outcompeted won't keep owning a lot of property for long. Even if that property is equity in AI companies, something or other will happen to make them lose it. (A very convincing AI-written offer of stock buyback, for example.)

Even if you have long term preferences, bold of you to assume that these preferences will stay stable in a world with AIs. I expect an AI, being smarter than a human, can just talk you into signing away the stuff you care about. It'll be like money-naive people vs loan sharks, times 1000.

Maybe an even better analogy is non-Euclidean geometry. Agent foundations is studying a strange alternate world where agents know the source code to themselves and the universe, where perfect predictors exist and so on. It's not an abstraction of our world, but something quite different. But surprisingly it turns out that many aspects of decision-making in our world have counterparts in the alternate world, and in doing so we shed a strange light on what decision-making in our world actually means.

I'm not even sure these investigations should be tied to AI risk (though that's very important too). To me the other world offers mathematical and philosophical interest on its own, and frankly I'm curious where these investigations will lead (and have contributed to them where I could).

They think (correctly) that AI will take away many jobs, and that AI companies care only about money and aren't doing anything to prevent or mitigate job loss.

You think AIs won't be able to offer humans some deals that are appealing in the short term but lead to AIs owning everything in the long term? Humans offer such deals to other humans all the time and libertarianism doesn't object much.

No, just regular people.

OpenAI currently creates a massive amount of value for humanity and by default should be defended tooth and nail.

Interesting how perspectives differ on this. From what I see around me, if tomorrow a lightning from God destroyed all AI technology, there'd be singing and dancing in the streets.

People who have been outcompeted won't keep owning a lot of property for long. Something or other will happen to make them lose it. Maybe some individuals will find ways to stay afloat, but as a class, no.

A happy path: merge with the AIs?

I think if you're serious about preserving human value into the future, you shouldn't start with something like "let's everyone install AI-brain connectors and then we'll start preserving human value, pinky promise". Instead you should start by looking at human lives and values as they exist now, and adopt a standard of "first do no harm".