People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy.
How? The basic mechanism is actually easy to understand on a human level. One of the things difficult about being a billionaire is staying true to your values and keeping good epistemics. Why? Because a growing number of people are optimizing against your mind. Not “against” in an adversarial sense, but in the sense that your specific mind believing or doing something is part of their optimization objective. People see you as a source of resources and try to befriend you so you’ll fund them, invest in their startups, donate to their causes, or have sex with you. Nearby egregores extend their tentacles to convert you.
Staying sane gets harder. You often see countermeasures, like billionaires setting up family offices or small organizations that also act as filters, shielding them from part of the pressure (“I do philanthropy only through my foundation and don’t make personal donations, please don’t reach out”).
In my view most billionaires lose in this game, the question is just to what degree.
Now imagine a hypothetical future with the ultra-rich (or, as people usually imagine it, with you as the billionaire), where the rest of the world consists of the permanent underclass and countries of geniuses in datacenters. In this world, the content of your mind is the key to an absurdly large share of resources. Unless some magical social realignment happens, the countries of geniuses will optimize against your mind, and millions of desperate people with no better game in town will be happy to help them (or the other way around).
I think the default expectation is that you will go crazy. Not relative to your new values (you may feel great), but from the perspective of who you are now: what you currently believe and care about. Unless, of course, the only thing you care about is that some future you, with your name, keeps the power.
The mechanism is similar to gradual disempowerment at the macro scale, just applied to individuals. What we believe and value is largely determined by culture, in a broad sense. Individually, we don’t have the capacity to observe most of the world, find abstractions, or get oriented on our own. We think in a collectively invented language, learn ethics socially, and even our interpretation of more biologically grounded signals, like emotions, is heavily culturally mediated.[1]
If humans somehow stay in control of all the wealth and political power, the content of our minds (what we believe, what we care about) becomes the key to the universe, the load-bearing part of the structure. With so much powerful optimization and abundant intelligence around, it seems likely that human-aligned culture will be replaced with something else. Political and economic power does not protect you against the content of your mind - if your mind goes crazy, power gets wielded by whatever beliefs and values took over.
As a post-AGI billionaire, can’t you protect your mind using AI? I think broadly yes, but roughly in proportion to how much your original human mind cedes control to your guardian AI. Again, the human analogues are easy to see: people wielding large amounts of nominal power, shielded by layers of guardians, protectors, and advisors. With AI, this means carefully filtering every bit of information, every pixel, sampling molecules from air you breathe and water you drink... You would likely also need to stop interacting with other humans, unless your protectors vet them extremely thoroughly and confirm they pose no risk of changing your mind. In the limit, what you believe and who you become are decided by AIs that are only nominally owned by you, the glorious meatpuppet.
Is there a better alternative? I think so, but not an individual one. Our survival and flourishing as individuals is tied to that of other humans, and largely depends on solving the civilizational alignment problem, including having a culture that is pro-human and pro-human-autonomy. Capital ownership alone won’t buy you a way to escape gradual disempowerment
I think the post repeats the old idea of modesty: "The sleep of a laborer is sweet, whether they eat little or much, but as for the rich, their abundance permits them no sleep" (Ecclesiastes 5:12), which is likely still repeated in poor African villages today. Despite this, I still think having more capital is generally better for well-being.
Are you saying that AGI increases the pressure so much that it actually becomes a real problem? I don't see what changes. Do you believe having a net worth of $10k is better than $1k? What about $100k? At what point does this trend break?
This is more of a nitpick, but you start the post with "owninglarge amounts of capital will get you disempowered sooner than average". However, in your scenario with unemployed masses and agents wanting something from you, it seems that you actually get disempowered later. What do you mean by disempowerment?
I think it related but does not directly repeat Ecclesiastes 5:12. As I understand that, the quote points to something like "holding capital is taxing/has the downside of worrying about loosing that capital, so the effect on wellbeing is complicated".
What I'm saying is not really about wellbeing.
With humans and US income distribution, I guess it is mostly not a big problem until the range of hundreds (?) of millions of dollars, roughly because the numbers don't add up. People's semi-arbitrary discretionary spending is not that large part of their capital * influenceability by individual humans is not that large; opportunity costs.
This is an interesting observation/framing, but what you frame as "going crazy" seems kind of fine to me. Fundamentally shouldn't I want to update my own values towards future-me's? And/or what the general future society recognizes as morally correct?
As an analogy, today we tend to think of moral heroes as eg the people who ran the underground railroad in defiance of slavery, or Schindler saving lives in defiance of his local environment. Insofar as future me and/or future society has a different set of values, I don't think I should be too scared about recognizing, learning, and updating in that direction.
(of course, me right now also kind of likes my current values and wants future me/future society to trade in that direction too)
People worried about a “permanent underclass” often assume that owning capital will protect them from disempowerment, or at least that they will be disempowered last, after the jobless masses dependent on UBI. I think something else is more likely: owning large amounts of capital will get you disempowered sooner than average, just by a different path, namely going crazy.
How? The basic mechanism is actually easy to understand on a human level. One of the things difficult about being a billionaire is staying true to your values and keeping good epistemics. Why? Because a growing number of people are optimizing against your mind. Not “against” in an adversarial sense, but in the sense that your specific mind believing or doing something is part of their optimization objective. People see you as a source of resources and try to befriend you so you’ll fund them, invest in their startups, donate to their causes, or have sex with you. Nearby egregores extend their tentacles to convert you.
Staying sane gets harder. You often see countermeasures, like billionaires setting up family offices or small organizations that also act as filters, shielding them from part of the pressure (“I do philanthropy only through my foundation and don’t make personal donations, please don’t reach out”).
In my view most billionaires lose in this game, the question is just to what degree.
Now imagine a hypothetical future with the ultra-rich (or, as people usually imagine it, with you as the billionaire), where the rest of the world consists of the permanent underclass and countries of geniuses in datacenters. In this world, the content of your mind is the key to an absurdly large share of resources. Unless some magical social realignment happens, the countries of geniuses will optimize against your mind, and millions of desperate people with no better game in town will be happy to help them (or the other way around).
I think the default expectation is that you will go crazy. Not relative to your new values (you may feel great), but from the perspective of who you are now: what you currently believe and care about. Unless, of course, the only thing you care about is that some future you, with your name, keeps the power.
The mechanism is similar to gradual disempowerment at the macro scale, just applied to individuals. What we believe and value is largely determined by culture, in a broad sense. Individually, we don’t have the capacity to observe most of the world, find abstractions, or get oriented on our own. We think in a collectively invented language, learn ethics socially, and even our interpretation of more biologically grounded signals, like emotions, is heavily culturally mediated.[1]
If humans somehow stay in control of all the wealth and political power, the content of our minds (what we believe, what we care about) becomes the key to the universe, the load-bearing part of the structure. With so much powerful optimization and abundant intelligence around, it seems likely that human-aligned culture will be replaced with something else. Political and economic power does not protect you against the content of your mind - if your mind goes crazy, power gets wielded by whatever beliefs and values took over.
As a post-AGI billionaire, can’t you protect your mind using AI? I think broadly yes, but roughly in proportion to how much your original human mind cedes control to your guardian AI. Again, the human analogues are easy to see: people wielding large amounts of nominal power, shielded by layers of guardians, protectors, and advisors. With AI, this means carefully filtering every bit of information, every pixel, sampling molecules from air you breathe and water you drink... You would likely also need to stop interacting with other humans, unless your protectors vet them extremely thoroughly and confirm they pose no risk of changing your mind. In the limit, what you believe and who you become are decided by AIs that are only nominally owned by you, the glorious meatpuppet.
Is there a better alternative? I think so, but not an individual one. Our survival and flourishing as individuals is tied to that of other humans, and largely depends on solving the civilizational alignment problem, including having a culture that is pro-human and pro-human-autonomy. Capital ownership alone won’t buy you a way to escape gradual disempowerment
In fact, even our biology was partly shaped by culture, through selection similar to the domestication of animals.