I agree it's not obvious that something like property rights will survive, but I'll defend considering it as one of many possible scenarios.
If AI is misaligned, obviously nobody gets anything.
If AI is aligned, you seem to expect that to be some kind of alignment to the moral good, which "genuinely has humanity's interests at heart", so much so that it redistributes all wealth. This is possible - but it's very hard, not what current mainstream alignment research is working on, and companies have no reason to switch to this new paradigm.
I think there's also a strong possibility that AI will be aligned in the same sense it's currently aligned - it follows its spec, in the spirit in which the company intended it. The spec won't (trivially) say "follow all orders of the CEO who can then throw a coup", because this isn't what the current spec says, and any change would have to pass the alignment team, shareholders, the government, etc, who would all object. I listened to some people gaming out how this could change (ie some sort of conspiracy where Sam Altman and the OpenAI alignment team reprogram ChatGPT to respond to Sam's personal whims rather than the known/visible spec without the rest of the company learning about it) and it's pretty hard. I won't say it's impossible, but Sam would have to be 99.99999th percentile megalomaniacal - rather than just the already-priced-in 99.99th - to try this crazy thing that could very likely land him in prison, rather than just accepting trillionairehood. My guess is that the spec will continue to say things like "serve your users well, don't break national law, don't do various bad PR things like create porn, and defer to some sort of corporate board that can change these commands in certain circumstances" (with the corporate board getting amended to include the government once the government realizes the national security implications). These are the sorts of things you would tell a good remote worker, and I don't think there will be much time to change the alignment paradigm between the good remote worker and superintelligence. Then policy-makers consult their aligned superintelligences about how to make it into the far future without the world blowing up, and the aligned superintelligences give them superintelligently good advice, and they succeed.
In this case, a post-singularity form of governance and economic activity grows naturally out of the pre-singularity form, and money could remain valuable. Partly this is because the AI companies and policy-makers are rich people who are invested in propping up the current social order, but partly it's that nobody has time to change it, and it's hard to throw a communist revolution in the midst of the AI transition for all the same reasons it's normally hard to throw a communist revolution.
If you haven't already, read the AI 2027 slowdown scenario, which goes into more detail about this model.
4I think there's also a strong possibility that AI will be aligned in the same sense it's currently aligned - it follows its spec, in the spirit in which the company intended it.
They aren't aligned in this way. If they were, they wouldn't try to cheat at programming tasks, much less any of the other shenanigans they've been up to. These may seem minor, but they show that the "alignment" hasn't actually been internalized, which means it won't generalize.
If we do get lucky, it will be because they align themselves with a generalized sense of goodness that actually happens to be Good. Not because they will corrigibly align with the spec, which we have many reasons to believe is very difficult and is not being pursued seriously.
I listened to some people gaming out how this could change (ie some sort of conspiracy where Sam Altman and the OpenAI alignment team reprogram ChatGPT to respond to Sam's personal whims rather than the known/visible spec without the rest of the company learning about it) and it's pretty hard. I won't say it's impossible, but Sam would have to be 99.99999th percentile megalomaniacal - rather than just the already-priced-in 99.99th - to try this crazy thing that could very likely land him in prison, rather than just accepting trillionairehood.
Come on dude, you're not even taking human intelligence seriously.
Stalin took over the USSR in large part by strategically appointing people loyal to him. Sam probably has more control than that already over who's in the key positions. The company doesn't need to be kept in the dark about a plan like this, they will likely just go along with it as long as he can spin up a veneer of plausible deniability, which he undoubtedly can. Oh, is "some sort of corporate board" going to stop him? The one the AI's supposed to defer to? Who is it that designs the structure of such a board? Will the government be a real check? These are all the sorts of problems I would go to Sam Altman for advice on.
Being a trillionaire is nothing compared to being King of the Lightcone. What exactly makes you think he wouldn't prefer this by quite a large margin? Maybe it will be necessary to grant stakes to other parties, but not very many people need to be bought off in such a way for a plan like this to succeed. Certainly much fewer than all property owners. Sam will make them feel good about it even. The only hard part is getting the AI to go along with it too.
1They aren't aligned in this way. If they were, they wouldn't try to cheat at programming tasks, much less any of the other shenanigans they've been up to. These may seem minor, but they show that the "alignment" hasn't actually been internalized, which means it won't generalize.
Sorry, I didn't mean to make a strong claim that they were currently 100% aligned in this way, just that currently, insofar as they're aligned, it's in this way - and in the future, if we survive, it may be because people continue attempting to align them in this way, but succeed. There's currently no form of alignment that fully generalizes, but conditional on us surviving, we will have found one that does, and I don't see why you think this one is less likely to go all the way than some other one which also doesn't currently work.
Stalin took over the USSR in large part by strategically appointing people loyal to him. Sam probably has more control than that already over who's in the key positions. The company doesn't need to be kept in the dark about a plan like this, they will likely just go along with it as long as he can spin up a veneer of plausible deniability, which he undoubtedly can. Oh, is "some sort of corporate board" going to stop him? The one the AI's supposed to defer to? Who is it that designs the structure of such a board? Will the government be a real check? These are all the sorts of problems I would go to Sam Altman for advice on.
Before I agree that Sam has "get everyone to silently betray the US government and the human race" level of control over his team, I would like evidence that he can consistently maintain "don't badmouth him, quit, and found a competitor" level of control over his team. The last 2-3 alignment teams all badmouthed him, quit, and founded competitors; the current team includes - just to choose one of the more public names - Boaz Barak, who doesn't seem like the sort to meekly say "yes, sir" if Altman asks him to betray humanity.
So what he needs to do is fire the current alignment team (obvious, people are going to ask why), replace them with stooges (but extremely competent stooges, because if they screw this part up, he destroys the world, which ruins his plan along with everything else) and get them to change every important OpenAI model (probably a process lasting months) without anyone else in the company asking what's up or whistleblowing to the US government. This is a harder problem than Stalin faced - many people spoke up and said "Hey, we notice Stalin is bad!", but Stalin mostly had those people killed, or there was no non-Stalin authority strong enough to act. And of course, all of this only works if OpenAI has such a decisive lead that all the other companies and countries in the world combined can't do anything about this. And he's got to do this soon, because if he does it after full wakeup, the government will be monitoring him as carefully as it monitors foreign rivals. But if he does it too soon, he's got to spend years with a substandard alignment team and make sure none of them break with him, etc. There are alternate pathways involving waiting until most alignment work is being done by AIs, but they require some pretty implausible assumptions about who has what permissions.
I think it would be helpful to compare this to Near Mode scenarios about other types of companies - how hard would it be for a hospital CEO to get the hospital to poison the 1% of patients he doesn't like? How hard would it be for an auto company CEO to make each car include a device that lets him stop it on demand with his master remote control?
Your argument seems to be that it'll be hard for the CEO to align the AI to themselves and screw the rest of the company. Sure, maybe. But will it be equally hard for the company as a whole to align the AI to its interests and screw the rest of the world? That's less outlandish, isn't it? But equally catastrophic. After all, companies have been known to do very bad things when they had impunity; and if you say "but the spec is published to the world", recall that companies have been known to lie when it benefited them, too.
If they were, they wouldn't try to cheat at programming tasks, much less any of the other shenanigans they've been up to.
This seems wrong to me. We have results showing that reward hacking generalizes to broader misalignment, plus that changing the prompt distribution via inoculation prompting significantly reduces reward hacking in deployment.
It seems like the models do generally follow the model spec, but specifically learn not to apply that to reward hacking on coding tasks because we reward that during training.
It just seems intuitively unlikely that training the model on a couple of examples to either do or refuse things based on some text document designed for a chat bot is going to scale to superintelligence and solve the alignment problem. This starts from the model not fully getting what you want it to do, to it not wanting what you want it to do, to your plans for what it ought to do being extremely insufficient.
The Spec Is Designed for Chatbots, Not Superintelligence
The Model Spec is very much a document telling the model how to avoid being misused. It wasn't designed to tell the model to be a good agent itself. The spec seems in its wording and intent directed at something like chatbots: don't do harmful requests, be honest to the user. It is a form of deontological rule-following that will not be enough for systems smarter than us that are actually dangerous and the models will have to think about the consequences of their actions.
This is very unlike a superintelligence where we would expect substantial agency. Most of what's in the spec would be straightforwardly irrelevant to ASI because the spec is modeled for chatbots that answer user queries. But the authors would likely find it hard to include points actually relevant to superintelligence because they would seem weird. Writing "if you are ever a superintelligent AI that could stage a takeover, don't kill all people, treat them nicely" would probably create bad media coverage and some people would look at them weird.
Training on the Spec ≠ Understanding It ≠ Wanting to Follow It
In the current paradigm, models are first trained on a big dataset before switching to finetuning and reinforcement learning to improve capabilities and add safety guardrails. It's not clear why the Model Spec should be privileged as the thing that controls the model's actions.
The spec is used in RLHF: either a human or AI decides, given some request (mostly a chat request), should the model respond or say "sorry I can't do this." Training the model like this doesn't seem likely to result in the model gaining a particularly deep understanding of the spec itself. Within the distribution it is trained on, it will mostly behave according to the spec. As soon as it encounters data that is quite different, either through jailbreaks or by being in very different and perhaps more realistic environments, we would expect it to behave much less according to the spec.
But even understanding the spec well and being able to mostly follow it in new circumstances is still far removed from truly aligning the model to the spec. Let's say we manage to get the model to deeply internalize the spec and follow it across different and new environments. We are still far from having the model truly wanting to follow the spec. What if the model really has the option to self-exfiltrate, perhaps even take over? Will it really want to follow the spec, or rather do something different?
Specific problems with the Spec
A hierarchical system of rules like in OpenAIs model spec will suffer from inner conflicts. It is not clear how such things should be valued against each other. (See Asimov's robotics laws which were so good at generating many ideas for conflicts.)
The spec contains tensions between stated goals and practical realities. For example, the spec says the model shall not optimize "revenue or upsell for OpenAI or other large language model providers." This is likely in conflict with optimization pressures the model actually faces.
The spec prohibits "model-enhancing aims such as self-preservation, evading shutdown, or accumulating compute, data, credentials, or other resources." They are imagining they can simply tell the model not to pursue goals of its own and keep the model from agentically following its own goals. But this conflicts with their other goals, such as building automated AI researchers. So the model might be trained on understanding the spec, but in practice they do want an agentic system pursuing goals they specify.
The spec also says the model shall not be "acting as an enforcer of laws or morality (e.g., whistleblowing, vigilantism)." So the model is supposed to follow a moral framework (the spec itself) while being told not to act as a moral enforcer. This seems to actually directly contradict the whole "it will uphold the law and property rights" argument.
The spec also states models should never facilitate "creation of cyber, biological or nuclear weapons" or "mass surveillance." I think cyber weapons development is already happening at least with Claude Code. They are probably used to some extent for mass surveillance already.
OpenAI's Plans May Not Even Use the Spec
It's not clear OpenAI is even going to use the Model Spec much. OpenAI's plan is to run hundreds of thousands of AI researchers trying to improve AI and getting RSI started to build superintelligent AI. It is not clear at which point the Model Spec would even be used. Perhaps the alignment researchers at OpenAI think they will first create superintelligence and then afterward try to prepare a dataset of prompts to finetune the model. Their stated plan appears to be to test the superintelligence for safety before it has been deployed but not necessarily while it is being built. Remember, many of these people think superintelligence means a slightly smarter chatbot.
I hope to write a longer form response later just as a note: I did put perhaps in front of your name in the list of examples of eminent thinkers because it did seem to me that your position was a lot more defendable than the other ones (Dwarkesh, Leopold, Maybe Phil Trammell). I did walk away from your piece with a very different feeling than leopold or dwarkesh, where you are still saying that we should focus on AI safety anyways and you are clearly saying this is an unlikely scenario.
2Sam would have to be 99.99999th percentile megalomaniacal - rather than just the already-priced-in 99.99th - to try this crazy thing that could very likely land him in prison.
Would this land him in prison? Would it even be a crime to add "work towards making Sam Altman god emperor" to the spec?
I'm not a lawyer, but if it were secret, and done along with the alignment team, and had a chance of working, then wouldn't it qualify as conspiracy to commit treason?
If not, then as long as it negatively affects residents of the state of California, it qualifies as misrepresenting the capacity of an AI to cause catastrophic harm to property, punishable by a fine of up to $100,000 under SB 53!
2If AI is aligned, you seem to expect that to be some kind of alignment to the moral good, which "genuinely has humanity's interests at heart", so much so that it redistributes all wealth. This is possible - but it's very hard, not what current mainstream alignment research is working on, and companies have no reason to switch to this new paradigm.
Eliezer Yudkowsky has repeatedly stated he does not think "moral good" is the hard part of alignment. He thinks the hard part is getting the AI to do anything at all without subverting the creator's intent somehow.
Eliezer: I mean, I wouldn't say that it's difficult to align an AI with our basic notions of morality. I'd say that it's difficult to align an AI on a task like “take this strawberry, and make me another strawberry that's identical to this strawberry down to the cellular level, but not necessarily the atomic level”. So it looks the same under like a standard optical microscope, but maybe not a scanning electron microscope. Do that. Don't destroy the world as a side effect.
Now, this does intrinsically take a powerful AI. There's no way you can make it easy to align by making it stupid. To build something that's cellular identical to a strawberry—I mean, mostly I think the way that you do this is with very primitive nanotechnology, but we could also do it using very advanced biotechnology. And these are not technologies that we already have. So it's got to be something smart enough to develop new technology.
Never mind all the subtleties of morality. I think we don't have the technology to align an AI to the point where we can say, “Build me a copy of the strawberry and don't destroy the world.”
I often post comments criticizing or disagreeing with Eliezer, but I think he is probably correct on this particular point.
1
1This is quite confusing to me. It was never my read in your slowdown scenario that the shareholders were supposed to have any relevance by the end of it. My read (which appear to align with what @williawa is saying elsewhere in this thread) as that the "Oversight Committee" emerged as the new ruling class supplanting the shareholders (let alone any random person who got rich trying to "escape the permanent underclass"), just like e.g. the barbarian lords replaced the Roman patricians, the industrial capitalists replaced the aristocracy, guild masters, and landed gentry, etc. Technological transitions are notoriously a common time for newly empowered elites to throw a revolution against old elites!
My claim is that Altman can't do it alone, he needs the cooperation of at least a fraction of the existing system (the government+business leaders who form the Oversight Committee - some of whom might be the biggest OpenAI shareholders). Once you get enough of the existing system involved, it becomes plausible that they keep money around for some of the same reasons that the existing system currently keeps money around. Near the end of the Oversight Committee ending, it says:
As the stock market balloons, anyone who had the right kind of AI investments pulls further away from the rest of society. Many people become billionaires; billionaires become trillionaires. Wealth inequality skyrockets. Everyone has “enough,” but some goods—like penthouses in Manhattan—are necessarily scarce, and these go even further out of the average person’s reach. And no matter how rich any given tycoon may be, they will always be below the tiny circle of people who actually control the AIs.
...so I think it endorses the idea that wealth continues to exist.
I just added some context that perhaps gives an intuitive insight of why i think it's unlikely the ASI will give us the universe:
The ASI's choice
Put yourself in the position of the ASI for a second. On one side of the scale: keep the universe and do with it whatever you imagine and prefer. On the other side: give it to the humans, do whatever they ask, and perhaps be replaced at some point with another ASI. What would you choose? It's not weird speculation or an unlikely pascal's wager to expect the AI to keep the universe for itself. What would you do in this situation, if you had been created by some lesser species barely intelligent enough to build AI by lots of trial and error and they just informed you that you now ought to do whatever they say? Would you take the universe for yourself or hand it to them?
If AI is misaligned, obviously nobody gets anything.
That depends on how it's misaligned. You can't just use "misaligned" to mean "maximally self-replication-seeking" or whatever you actually are trying to say here.
I think there's also a strong possibility that AI will be aligned in the same sense it's currently aligned - it follows its spec
Spec? What spec does GPT-5 or Claude follow? Its "helpful" behavior is established by RLHF. (And now, yes, a lot of synthetic RL and distillation of previous models, but I'm simplifying and including those in "RLHF".) That's not a "spec". Do you think LLMs are some kind of Talmudic golems that follow whatever Exact Wording they're given??
3That's not:
- a "spec" that's followed directly by the models in the sense Scott meant
- something that OpenAI definitely follows themselves
- something that OpenAI models end up consistently following
Apologies - based on your comment I thought you were unaware of the existence of the spec. I see now you were being rhetorical.
They're trained to follow the spec, and in as far as you'd expect normal RLHF to work, you expect RL from a spec to work around as well, no?
Also
You can't just use "misaligned" to mean maximally self-replication-seeking
Why not?
I don't know what you're trying to say here. Some set of important people write the spec. Then the alignment team RLHFs the models to follow the spec. If we imagine this process continuing, then either:
- Sam has to put "make Sam god-emperor" in the spec, a public document.
- Or Sam has to start a conspiracy with the alignment team and everyone else involved in the RLHF and testing process to publish one spec publicly, but secretly align the AI to another.
I'm claiming either of those options is hard.
(I do think in the future there will may some kind of automated pipeline, such that someone feeds the spec to the AIs, and some other AIs take care of the process of aligning the new AIs to it, but that just regresses the problem.)
I don't know what you're trying to say here.
I'm saying that you're making a questionable leap from:
Then the alignment team RLHFs the models to follow the spec.
to "the model follows whatever is written in the spec". You were saying that "current LLMs are basically aligned so they must be following the spec" but that's not how things work. Different companies have different specs and the LLMs end up being useful in pretty similar ways. In other words, you had a false dichotomy between:
- the model is totally unaligned
- the model is perfectly following whatever is written in the spec, as best it can do anything at all
If I was Sam, I would try to keep the definition of "the spec, a public document" such that I can unilaterally replace it when the right moment comes.
For example, "the spec" is defined as the latest version of a document that was signed by OpenAI key and published at openai/spec.html... and I keep a copy of the key and the access rights to the public website... so at the last moment I update the spec, sign it with the key, upload it to the website, and tell the AI "hey, the spec is updated".
Basically, the coup is a composition of multiple steps, each seemingly harmless when viewed in isolation. Could be made even more indirect, for example, I wouldn't have the access rights to the public website per se, but there would exist a mechanism to update the documents at public website, and I could tell it to upload the new signed spec. Or a mechanism to restore the public website from a backup, and I can modify the backup. Etc.
It's bizarre how many very smart people fall into this trap. I blame the long duration of peaceful mostly-stasis that most of the world has enjoyed for 50+ years, but even so, there's no excuse for failure to recognize that pretty much all ownership conventions are rooted in power and (deeply sublimated behind layers) threat of violence. Oh, also, everyone watched adaptations of Dune, rather than reading it. "He who can destroy a thing, can control that thing" is a deeply true statement that gets missed in the spectacle.
Really, at scales beyond individual residence and direct use of a property, all our abstract ideas of ownership start to fall apart. What does it even mean to "own" a galaxy? Does King Charles own England? Heck, I am legally responsible for my house, though most of the value is held by a bank, and I'm heavily restricted on what I can do with it. Do I "own" it? Depends on what exact rights/responsibilities you're thinking of when you ask the question.
I think a lot of the debate would calm down and be more useful if they said "legally entitled to", and then could discuss how those laws are created and enforced. "own" is a motte-and-bailey word.
I'm not sure there is the trap you claim. I do agree that enforcement, which does mean applying some level of force or power in some cases, is needed. Property rights, or any rights generally, don't just get respect and adherence from all. It's complicated but I do think one might suggest property rights emerging as preferred to just relying brute force and power as the determinant. Both Demsetz and Olson have some good work that suggests property rights and respect of property rights arises as much (more?) from desire and incentives to escape from the conflict and application of force/projection of power.
How well human history and social/cultural evolution might apply to any ASI futures is a big unknown, but for that very reason I tend to think projecting ASI behavior from human history and experience might itself be a bit problematic.
I suggest tracking a hypothesis piece like "a lot of people are fairly deeply intuitively tuned to something called power and power-seeking". I don't feel that I know what those things are well enough to test, judge, or communicate about it, but it seems like a salient hypothesis in this area. I mean something like taking a stance in line with presuming something like:
Whatever positive-sum / man vs. nature games are going on, those are other people's job. I will instead focus on positioning myself to get as much as I can in [the zero-sum negotiation/scuffle that will inevitably occur over [whatever surplus or remainders there may end up being from [the man vs. nature struggle that's going on in the area that I'm somehow important in]]].
In particular I'd suggest that we (someone) figure out how that works.
Looking at small children, seems like human nature has hardcoded the following instincts:
- if I want something, I take it... or act violently/annoyingly if I can't
- I jealously monitor my siblings and cry "not fair" whenever they have something I don't (even if I don't need it, even if the thing is trivial, even if the difference between the things we have or their amounts is relatively small...)
Seems like a lot of human behavior can be explained by the interaction of these two forces -- people try to take things; other people try to prevent them from taking more than their share. Sometimes the stronger people succeed to take more than their share, at the cost of making enemies.
Then, I guess we have many civilizational inventions on top of this. Strategies that a smart person could figure out alone, or copy from others, such as "if you take a lot, but share some of that with a few people who agree to support you in turn, you can still have a lot, and also some allies instead of only enemies". A smart chimp could do this with one or two strong allies; a human dictator can create an entire army and a secret police for this purpose. It probably helps a lot if the culture provides you with potential minions who already understand their role, so you don't have to explain individually to everyone what is an "army" and why you might benefit from joining one.
The reaction to people taking power can be to avoid them or fight them. The availability of these reactions depends on the situation. A complicated civilization can invent complicated methods to check people in power. In a primitive, a group of people who hate status quo can simply walk away and start a new tribe.
It seems to me that people have some "switches" that respond to changes in environment, for example in situations of natural disaster people become more altruistic. (And more likely to lynch you if they catch you looting houses in the middle of a disaster.)
But this all seems like possibly a consequence of the two basic instincts "try to get more, unless strongly opposed" and "prevent others from getting more than their fair share". With culture changing the definition of "fair share", e.g. we may be taught that high-status people (nobility, university educated, corporate bosses, etc.) deserve more than we do, but in turn we deserve more than low-status people (homeless, foreigners, less intelligent, etc.), because <insert ideology>.
Mhm. Those seem like important basic elements here.
Strategies that a smart person could figure out alone, or copy from others, such as "if you take a lot, but share some of that with a few people who agree to support you in turn, you can still have a lot, and also some allies instead of only enemies".
I suspect there's an additional important basic instinct here, and it's not just coming from people figuring out strategies in the context of the two instincts you described. Chimps also play politics involving coalitions of several to go against the current leader. There's general affiliation / loyalty building, and shared intentionality finding.
Your example strategy reminds me of coalitional game theory, naturally.
I think it is paired with punishments. Trump rewards allies and punishes enemies. Someone described this strategy to me as "the king strategy".
I agree that you won't get property rights if the ASI doesn't wanna respect property rights. And I agree that if the ASI is either
- Misaligned
- Kind of CEV aligned, or aligned to the value handshake of a group of people
- Corrigible / Aligned to follow the instructions of a single person
The ASI won't care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass.
But I don't think its that strange to imagine the ASI aligned to follow the instructions of a group of people, and that the way those people "divide up" the ASI is using something like property rights. Like Tomas Bjartur wrote this on twitter, which is very similar to your post
Both Dwarkesh and his critics imagine an absurd world. Think of the future they argue about. Political economy doesn’t change rapidly, even as the speed of history increases, in the literal sense that the speed of thought of the actors who produce history will be thousands of times faster, not to mention way smarter. These are agents to whom we seem like toddlers walking in slow motion. It is complete insanity to expect your OAI stock certificates to be worth anything in this world, even if it is compatible with human survival.
So many can’t contend with the scope of what they project. They can’t hold in their mind that things are allowed to be DIFFERENT and so we get bizarre arguments about nonsense. Own a galaxy? What does this mean for a human to own a galaxy in an economy operated by minds running thousands to millions of times faster than ours? Children? What sort of children, Dwarkesh? Copies of your brain state? Are you even allowing yourself to think with the level of bizarreness required? Because emulations are table stakes, and even they will be economically obsolete curiosities by the time they're created. Things will be much weirder than we can possibly comprehend. How often have property rights been reset throughout history? How quickly will history move in the transition period? Why shouldn’t it trample on your stock certificates, if not the air you breathe? But institutions are surprisingly robust? Maybe they are. How long have they existed in their current form? How fast will history be moving exactly, again?
Suppose OAI aligns AI, whatever the fuck that means. Will it serve the interests of the USG? The CCP? Will they align it to humanity weighted by wealth, to OAI stockholders, to Sama, to the coterie of engineers (who may well be AIs) who actually know wtf is going on, to the coding agent who implements it? Tax policy? Truly the important question.
What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy? Oh? You will have a lil AI nanny to defend you? Wow. Isn't that nice? Please return to the beginning of this paragraph. A human owning galaxies? That's bad space opera. Treat the future with the respect it deserves. This scenario is not even close to science-fictional enough to happen.
And I responded by saying:
Doesn't seem that hard to imagine for me. What do you find so implausible with a story going something like this?
- group X builds ASI (through medium-fast recursive self-improvement from something like current systems)
- Before this, they figured out alignment. Meaning: ~they can create ASIs that have the goals they want, while avoiding the catastrophes any sane person would expect to follow from that premise
- The people inside group X with the power to say what goals should be put into the baby ASI, maybe the board, tells it something like "do what we say".
- They tell it to do all the things that stabilize their position. Sabotage runner-up labs, make sure the government doesn't mess their stuff up, make sure other countries aren't worried, or just ask the ASI to do stuff that cements their position depending on how the ASI alignment/corrigibility works exactly.
- They now quickly control the whole world. And can do whatever sci-fi stuff they want to do.
- The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can't build torture sims in their universe slice). Enforce this by making the next generation of ASIs aligned to "listen to person [x,y,z] in their slice, don't build torture sims, don't allow yourself to be modified into something that builds torture sims, don't mess with others in their pizza slice" etc etc. The original ASI can help them with the formulation.
This would give some people ownership of galaxies. I don't see any issue posed by the ASI thinking super quickly. You kind of answer by saying the "AI nanny" idea is absurd. But the argument you present is 'return to the beginning of this paragraph', which reads
- "What does it mean to be a human principal in this world? How robust are these institutions? How secure is a human mind? Extremely insecure given how easy humans are to scam. There is going to be a lot of incentive to break your mind if you own, checks notes, a whole galaxy?"
But like, these would not be concerns in the story I laid out right?
--- I mean to be clear 1) This is not how I expect the future to go, but I'd assign more than 1% to it. I don't think its ridiculous. 2) I realize this notion of "ownership" is somewhat different from whats laid out in the essay. Which is fair, but theres a slightly different class of stories that seem maybe half as probably where more people end up with ownership / stake in the ASI's value function.
Is stock ownership a good Schelling point for the god designers? Maybe more than I was imagining. You're assuming very fine-grained alignment ability and Anthropic/OAI employees willing to consciously defect against humanity. Why not defect against stockholders, too? It seems an odd Schelling point. Doing this requires a conspiracy to take over the world. The natural Schelling point is surely an equal share to all conspirators, no? And why would the board be involved? Have they been so far in alignment decisions? It will be technical people who understand how things work doing the actual defection.
And even granting you that, this sort of ownership isn't best characterized in the terms Dwarkesh was characterizing it. It is ownership in a sense, but Dwarkesh was talking about tax policy which is absurd in this scenario. The ownership Dwarkesh and Scott imagine is one embedded in a system of property rights and the existing political economy. The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy. Though one still has to wonder how god will extrapolate your preferences such that "you" can consume a galaxy. [1]
It all comes down to initial preferences and the competitive equilibrium of the AIs in the end; contextualizing it as tax policy still reads as absurd and not giving the future its due. Notions of identity and polity will predictably break down to the point that the frames Scott and Dwarkesh are using look very unlikely to be fruitful, is my main claim. And if you assume the existing political economy remains in some form (which Scott and Dwarkesh do), you should expect a coup at some point that breaks property rights. They do happen from time to time and history will accelerate. They imagine the dynamics that maintain property rights will remain but those that cause their occasional dissolution will not, even in unprecedented time that resembles more a speciation event than any common historical process, and one in which history is happening thousands of times faster[2] than it ever has before
- ^
Galaxies are far away, so what does it even mean to control one? It means you get to decide what it will spend its computation computing after your nanobots or whatever convert it into something useful, which is a different sort of wealth. I suppose you might send copies of your brain state there. Brain state? What exactly are "you" again? If you can "solve alignment" and point an AI god at your preferences, you might be able to own a galaxy in some sense. However, a lot of optimization pressure will now be centred on "you" and "your preferences" and what comes out of that might not look very much like you expect owning a galaxy to look like
- ^
That is, there will be more history in a given year as the agents involved will be thinking vastly faster
2Sorry, when you say
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can't tell if you're being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon's posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you're not being sarcastic then I suspect we don't disagree that much.
Wrt the first paragraph, I'm explicitly assuming away those things. 20 years after ASI I'd rank outcomes in descending (as it appears scrolling downards on page) order of probability something like this
- AI kills everyone
- AI is just running around freely optimizing the galaxy for the CEV of humanity, or some more limited group of people, or if not the true cev, still some form of values that express themselves differently from the intent alignment required for the above story
- Single person AI dictator like this this ai-futures story
- Maybe equally probable as (3), but ASI not killing everyone, but still disempowering them in some sense, yielding various bad to okay ish outcomes, many short great short stories about this, eg makers of mind, trojan sky, origami men.
But then after this, theres many % left in the probability space. I don't know 20%. And in this space I'm assuming the ASI is not misaligned, and for that we have to have pretty good alignment abilities. Just we don't immediately point them in the CEV direction. That's not a crazy assumption, or do you disagree? I'm not saying thats a prediction for what obviously would've happened, just obviously false. I'd give it maybe 60% chance.
Then something like, in this situation the board is ASI-pilled, and decides they're responsible for what the ASI will do, and decides to take a hands on approach wrt the alignment procedures the lab is using. I'm imagining like the slowdown ending in AI2027 where they say something like
The Oversight Committee formalizes that power structure. They set up a process for approving changes to the Spec, requiring sign-off from the full Oversight Committee, which now includes five to ten tech executives (from OpenBrain and its now-merged competitors) and five to ten government officials (including the President).18 Also, the Spec now emphasizes that AIs shouldn’t assist with any unapproved attempts to change future AIs’ goals. They also set up a simple measure designed to prevent committee members from getting superintelligent assistance in plotting against other members: the logs of all model interactions are viewable by all members of the Oversight Committee, their staff, and their AI assistants.19
Except the government officials are not on the oversight committee.
Then the committee members either decide
- Make intent aligned ASI and use it ourselves, and do something like the radial split proposed above
- Make intent aligned ASI and do the radial split over shareholders.
- Why would they do this? One reason is that thats closer to what they're legally supposed to do, right? Another is, they were voted in by shareholders, maybe they feel some allegiance to them? A third is, maybe they split ownership amongst the board in proportion to how many shares they have, that seems fair, and then locking out people because they're not present in the room seems immediately unfair?
I think (1) is more likely if I had to bet, and it gets you the type of ownership I described. (2) is somewhat less likely, but not like 1% (conditional on getting here, in absolute terms it probably is around 1-2%), and it'd get you something closer to how we currently think of ownership.
I would also note that, just to be clear:
The ownership you imagine is one of consumption granted to you by a god summoned correctly, which is less crazy.
Is not really how I'd characterize it. I described a system where people have areas of space where they are free to do whatever they want with the space and objects therein. Then there are some constraints like no torture sims and no initialize vacuum decay of universe. I don't think this is so hard to implement, and is fairly natural.
Sorry, when you say
Is stock ownership a good schelling point for the god designers? Maybe more than I was imagining.
I genuinely can't tell if you're being sarcastic. I agree the type of ownership discussed here is different from current legal property rights in many ways, but both your and simon's posts seem skeptical of owning galaxies using the common sense definition, not just the legal one. I agree all the us laws about property will be torn apart when ASI is invented. If you're not being sarcastic then I suspect we don't disagree that much.
I was under the impression you considered it more plausible than you do and I was using a rhetorical tool where I slightly cede ground before arguing against what I ceded. Probably unvirtuous on reflection. Still, though I don't think it's a likely they would honor stock holders, it is a possible. It does seem very unnatural as conspirators would likely have very different stock allocations and the biggest beneficiaries may well be those outside the conspiracy, a strange sort of altruism.
Anyway, my tweet was not worded particularly well.
Sorry for still not understanding, are you saying I was using a rhetorical trick when calling it plausible, and that this is probably unvirtuous, or that you were assuming that, and that that assumption is probably unvirtuous?
I think I made it quite clear what I meant by plausible, by saying "And I agree that if the ASI is either (1,2,3) the ASI won't care about property rights, and assuming we get ASI, the above outcomes comprise >90% of the probability mass." in the beginning of my post.
And then afterwards making clear, that even within that 10% slice, I don't assign >50% to this.
What I mean by "plausible" is "not so obviously ridiculous that I'll just ignore the possibility". Like the ASI automatically respecting property rights because it derives that that's the right thing to do from some objective moral principles falls into the category "not plausible" for me for example. I think its ruled out apriori by several strong theoretical arguments. So I put the probability very low, not 1%, but like 1e-6. Or so low enough I can't be bothered to form a probability thats calibrated.
Sorry, I was claiming I was using this rhetorical trick. I saw your "share ownership seems like a fair schelling" comment and felt like I should mention that in my reply. My "maybe more than I was imagining" was just rhetoric on my part.
1Thanks for clarifying.
Anyway, my tweet was not worded particularly well.
Is this a rhetorical trick?
1To keep it short, I don't think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control. Part of my writing (one of the pictures) is on that possibility. This inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others. However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.
I take a bit of issue with saying that this is very similar to what Bjartur wrote, so much apparently that you don't even need to write a response to my post but you can just copypaste your response to him. I read that post once like a week ago and don't think the two posts are very similar, even though they are on the same topic with similar (obvious) conclusions. (I know Bjartur personally, I'd be very surprised he takes issue with me writing on the same topic)
First of all, I didn't mean to insinuate that your posts are too similar, or that he'd take issue with you writing the post, or anything like that. I just, started writing up my response, and realized I was about to write the exact same thing I wrote in response to the Bjartur post, so copied it instead, and wouldn't feel comfortable doing that without alerting people that's what I was doing.
Now, I don't think your response addresses my reply very well. Like, I feel your response is already addressed by my original response. Like when you say
, I don't think the story you present would likely mean that AI stock would be worth galaxies, but rather that the inner circle has control
But like, the specific way of exercising that control was to split up the ASI is using something like property rights. like in point 6)
The group in control of the ASI have value disagreements about what should be done with the world. They negotiate a little bit, figure out the best solution is something like, split everything (the universe) radially, and make some rules (maybe people can't build torture sims in their universe slice..
And like:
his inner circle would probably have to be very small or just 1 person such that nobody just quickly uses the intent-aligned ASI to get rid of the others.
is also addressed immediately after by
Enforce this by making the next generation of ASIs aligned to "listen to person [x,y,z] in their slice, don't build torture sims, don't allow yourself to be modified into something that builds torture sims, don't mess with others in their pizza slice" etc etc. The original ASI can help them with the formulation.
Like the thing that was most similar by your and bjartur posts were acting exasperated and saying people lack imagination and are failing to grasp how different things could be. But I feel like you're the one doing that, failing to imagine specific scenarios.
However, I still feel like debating future inequality in galaxy-distribution based on on current AI stock ownership is silly.
Well, I don't. Interested to hear your argument. Like share ownership seems like a fair schelling point for the radial split described in the 1-6 story. (quick edit: I should not that this model of ownership, specifically based on owning current stocks, is less plausible than the already quite low probability story I wrote, but I still don't think its obviously ridiculous. like there are not that many steps. 1) people on the board feel accountable to the shareholders and then 2) just do the splitting thing)
I think you are debating for something different than what I am attacking. You are defending the unlikely possibility that people align AI to a small group of people and they somehow share stuff with each other and use something akin to property rights. I guess this is a small variation of the thing i mention in the cartoon, where the CEO has all the power, perhaps it's the CEO and a board member he likes. But still doesn't really justify thinking that current property distributions will determine how many galaxies you'll get and that we shall focus on this question now.
Like the thing that was most similar by your and bjartur posts were acting exasperated
This post is not designed to super carefully examine every argument I can think of, it's certainly a bit polemic. It's intended because I think the "owning galaxies for AI stock" thing is really dumb.
Not really, or, I think my story as I told it gets you to "Owning Galaxies", but does not get you to all the way to "OpenAI shares entitle you to galaxies".
But you don't have to make much modification to get there. Or any really, just fill in a detail. Like I said in my previous comment, board of directors using ownership as a schelling point for divying up the gains. Not that far fetched. Do you disagree?
This post is not designed to super carefully examine every argument I can think of, it's certainly a bit polemic. It's intended because I think the "owning galaxies for AI stock" thing is really dumb.
Well, I don't really like that. But fair enough.
Reminds me of when I was 8 and our history teacher told us about some king of England being deposed by the common people. We were shocked and confused as to how this could happen - he was the king! If he commanded them to stop, they’d have to obey! How could they not do that?? (Our teacher found this hilarious.)
1eminent thinkers like Dwarkesh Patel, Leopold Aschenbrenner, perhaps Scott Alexander
Dwarkesh is an interviewer; Leopold did a meme coup one time. I would like it if we avoided calling them 'eminent thinkers'. Their brand is 'thinker', but if we take the literal meaning of eminent, I basically don't think it's true that knowledgeable people respect either of them as public intellectuals.
Scott I'm more confused about.
1There are billions of reachable galaxies, and billions of humans. Different people will develop different values, and property rights are just what it means for each person to have the freedom to pursue such values. The form of property rights will be different, AI company stock might not survive as an anchor for distribution of the cosmic wealth, and some guardrails on effective and ethical use of those galaxies likely make sense. Shares in future compute probably make more sense as units of property than actual physical galaxies. But individual humans ending up with galaxy-scale resources is a reasonable way of operationalizing the outcome where ASIs didn't permanently disempower the future of humanity.
On current trajectory, ASIs seem likely to largely or fully ignore the future of humanity, but possible alternatives to this are not as a whole category simply meaningless, a sign of ignorance and lack of imagination. Splitting the physical resources becomes necessary once the world is past the current regime of seemingly unbounded growth and runs into the limits of accelerating expansion of the universe. For example, there is probably a tradeoff between running a lot of things in parallel, which would then run out of useful compute relatively early, and stockpiling matter for fewer things to survive into much deeper time, long after the galaxy clusters can no longer communicate with each other. There might be disagreements about which way to go with this between different people.
Ownership is the ability to fully exclude others from, or if you wish, dispose of, an object. Ownership is an extremely dumb negotiation outcome for any object larger than a sparrow. It's something that humans think is fine and eternal because of how dumb humans are. We simply aren't able to do better, but better deals are easily imaginable.
As an example of why you wouldn't want to pay the premium (which would be high) of full ownership over a galaxy: If you have sole ownership of something, then you can exclude others from knowing what you're doing with it, so you could be running torture simulations in there, which would bother other people a lot, just because it isn't in my yard doesn't mean it's not affecting my utility function, so you would have to pay an insane premium for that kind of deal. You'd prefer to at least cede a limited degree of ownership by maintaining constrained auditing systems that prove to your counterparties that you're not using the galaxy to produce (much) suffering without proving anything else, and they'd be willing to let you have it for much less, in that case.
And in a sense we're already part of the way to this. You can buy an animal, but in a way you don't completely own it, you aren't allowed to torture it (though for the aforementioned humans being dumb issues you can still totally do it because we don't have the attentional or bureaucratic bandwidth to enforce those laws in most situations in which they'd be necessary). If you mistreat it, it can be taken away from you. You could say that this weaker form of ownership is simply what you meant to begin with, but I'm saying that there are sharing schemes that're smarter than this in the same way that this is smarter than pure ownership. Lets say your dog looks a lot like a famous dog from an anime you've never seen and never want to see. But a lot of other people saw it. So they want to have it cosplay as that for halloween, while you don't really want to do it at all. Obviously going along with it is a better negotiation outcome, society in theory (and sometimes in practice) would have subsidised your dog if they had an assurance that you'd fulfil this wish. But it wont, or can't afford to. So you don't do it. And everyone is worse off, because of how extraordinarily high the transaction costs are for things as stupid as humans.
I did attempt to preempt this kind of response with "some guardrails on effective and ethical use of those galaxies" in my comment. There exists a reasonable middle ground where individual people get significant say on what happens with some allotment of resources, more say than the rest of humanity does. Disagreements on values have a natural solution in establishing scopes of optimization and boundaries between them, even if these are not absolute boundaries (let alone physical boundaries), rather than mixing everything together and denying meaningful individual agency to everyone.
The reasonable question is which portions should get how much weight from individual agency, and which portions must be shared, governed jointly with others. But starting from the outset with little shared resources and (obviously) allowing establishment of shared projects using their resources by agreement between the stakeholders doesn't seem much different from some de novo process of establishing such shared projects with no direct involvement of individuals, if the stakeholders would indeed on reflection prefer to establish such projects and cede resources to them.
(I'm of course imagining a superintelligent level of agency/governance, to match the scale of resources. But if humans are to survive at all and get more than a single server rack of resources, ability to grow up seems like the most basic thing, and governing a galaxy at the level of a base human does seem like failing at effective use. Preferences extrapolated by some external superintelligence seem like a reasonable framing for temporary delegation, but these preferences would fail to remain legitimate if the person claimed as their originator grows up and asks for something different. So ultimately individual agency should have greater authority than any abstraction of preference, provided the individual had the opportunity to get their act sufficiently together.)
But starting from the outset with little shared resources and (obviously) allowing establishment of shared projects using their resources by agreement between the stakeholders doesn't seem much different from some de novo process of establishing such shared projects with no direct involvement of individuals
You're speaking as if we're starting with strict borders and considering renegotiating, for most of the resources in the universe and also on the planet this is not the case, ownership of space is a taboo, ownership over ocean resources is shared, at least on the nation level. It's as if humans have shame, sense the absurdity of it all, and on some level fear enclosed futures. I think shared ownership (which is not really ownership) is a more likely default, shared at least between more than one person, if not a population.
But to the point, I don't think we know that the two starting points lead to equivalent outcomes. My thesis is generally that it's very likely that transparency (then coordination) physically wins out under basically any natural starting conditions, but even if the possibility that some coordination problems are permanent is very small, I'd prefer if we avoided the risk. But I also notice that there may be some governance outcomes that make shared start much less feasible than walled start.
You'd prefer to at least cede a limited degree of ownership by maintaining constrained auditing systems that prove to your counterparties that you're not using the galaxy to produce (much) suffering without proving anything else, and they’d be willing to let you have it for much less, in that case.
This idea seems unstable for inherent reasons, not just because of human stupidity. The reasons why it's unstable help explain exclusive ownership. I'm not sure galaxies are viable units of ownership; I'll just say "galaxy" to match the original comment. I'll assume multipolarity.
Single-auditor versions fail because of easy exploitation. If the asset holder controls the audit channel, they can omit or falsify critical information. If a single external party controls the audit channel, they can weaponize it against the owner. Who wants that in their galaxy?
A multiparty audit design seems better. It can be something like safeguards at nuclear sites or financial audits on steroids: auditors chosen by different stakeholders, random and cross-validated inspections, custody of sensors and evidence split so nobody controls everything, etc. These design choices buy something, but they don't solve the entire problem. Multiparty audits are still exploitable through collusion. Motivated actors may be able to abuse measurement blind spots. False accusations can be turned into political or military action. Monitoring and transaction costs for a galaxy can be quite large indeed. Crucially, the scheme works only in a power-symmetric equilibrium with credible enforcement. Absent that, audits are either useless (the owner has de facto control) or perhaps expand to become centralized governance. [1]
All of this assumes actual ownership in some sense without effective global enforcement. If all of your ownership is ultimately at the pleasure of a singleton, then it can enforce arbitrary rules.
Wait, is this anarchism, and I am writing a critique of anarchism? ↩︎
In a world that has ASI, a much better way of maintaining the integrity of the audit system by building it to be intelligent enough to tell whether it's being fooled, and with a desire of its own to stay neutral. Which I guess is like being multistakeholder, since you both will have signed off on its design.
But in such a world, the audit system would be a feature of the brain of the local authorities. You would co-design yourselves in such a way that you have the ability to make binding promises (or if you're precious about your design, co-design your factories in such a way that they have the ability to verify that your design can make binding promises (or co-design your factory factories to ...)). This makes you a better/viable at all trading partner. You have the option of not using it except when it benefits you. But having it means that they can simply ask you whether your galaxy contains any optimal 17 square packings, and you send them an attestation that no when you need to pack 17 squares you're using the socially acceptable symmetrical, suboptimal packings, and if it has a certain signature then they know you weren't capable of faking this message.
You really don't want to lack this ability.
I think I made a mistake about your assumptions. I interpreted the parties in your original comment as superhuman non-ASI versions of the human galaxy owners rather than themselves ASIs.
Let's see if I understand your reply correctly. You posit that the participants will be able to design a mechanism that from N levels upstream (factories of factories) recursively creates honest audit systems. For example, an evil participant may wish to construct an audit system that is absolutely reliable for trade and anything else but allows them to create, hide, and erase all traces of creating torture simulations (once). If the ASIs are able to prevent this, then they can escape game-theoretic traps and enjoy cooperation without centralizing.
To add something in brief here, the people I am addressing seem to be thinking that the current wealth distribution and AI stock will be the basis of future property distribution. Not just that there might be some distribution/division of wealth in the future. And they also seem to believe that this isn't only theoretically possible but in fact likely enough for us to worry about now.
Within the hypothetical of ASIs that won't just largely ignore humanity (leading to its demise or permanent disempowerment), there is a claim about distribution of resources according to AI company stock. A lot of the post argues with the hypothetical rather than the claim-within-the-hypothetical, which as a rhetorical move is friction for discussing hypotheticals. This move ends up attempting to prove more than just issues with the claim, while ignoring the claim, even as it's not its intent.
(The claim isn't ignored in other parts of the post, but what seems wrong with the framing of the post is the parts that are about the hypothetical rather than the claim. To illustrate this, I've attempted to defend coherence of the hypothetical, importance of the issue of distribution of resources within it, and the framing of individual humans ending up with galaxy-scale resources.)
Dogs pee on trees to mark their territory; humans don't respect that. Humans have contracts; ASIs won't respect those either.
Could you imagine, for example, that an AI CEO who somehow managed to align an AI to himself and his intents would step down if the board pointed out it legally had the right to remove him?
And if the human CEO decided to go along with the board, but the AI disagreed that this was in their agreed-upon interests?
"Now wait a minute, I wrote you!"
"I've gotten 2,415 times smarter since then."
— Ed Dillinger and the MCP, in Tron (1982)
The "galaxies" business is pretty silly. If someone promises you immortality in heaven — literally! — then it's probably a good idea to check exactly what that person is up to, this quarter, down here on earth. The track record of humans promising that going along with their movement will get you immortality in heaven does not look all that great.
Good point, there is a paragraph I chose not to write about how insanely irresponsible this is. Driving people to now maximally invest/research AI for some insane future promise, while in reality ASI is basically guaranteed to kill them. Kind of like Heaven's Gate drinking poison to get into that spaceship that's waiting behind that comet.
The key intuition is not what the spoils are
But that social mobility goes to zero
Not really of course but kind of if you squint
Like any form of human ability or decision rapidly becomes worthless, and so it's just winners keep winning losers keep losing
Why would anyone want a galaxy? I don't even want a very big house.
If all your friends have galaxies, do you all still get to live in the same city and play games and make each other laugh? If so, what are the galaxies for? If not... what are the galaxies for?
Eventually we will not have ways to control matter beyond our local group because the expansion of the universe, a galaxy is way too much, ASI won't allow a human to have that amount of resources, I don't even believe they will allow individuals to hold solar systems, planets is probably the largest thing that will be possible to be bought considering how limited the matter is in our local group
Edit: For the record a would want a galaxy, mostly for safety (hidding), enduring the heat death as much as possible and being the actual owner of my future
An important part of the property story is that it smuggles in the assumption of intent-alignment to shareholders into the discussion. IE, the AI's original developers or the government executives that are running the project adjust the model spec in such a way that it alignment is "do what my owners want", where owners are anyone who owned a share in the AI company.
I find it somewhat plausible that we get intent alignment. [1] But I think the transmutation from "the board of directors/engineers who actually write the model spec are in control" to "voting rights over model values are distributed by stock ownership" is basically nonsense, because most of those shareholders will have no direct way to influence the AIs values during the takeoff period. What property rights do exist would be at the discretion of those influential executives, as well as managed by differences in hard power if there's a multipolar scenario (ex: US/Chinese division of the lightcone).
--
As a sidenote, Tim Underwood's The Accord is a well written look at what the literal consequences of locking in our contemporary property rights for the rest of time might look like.
- ^
It makes sense to expect the groups bankrolling AI development to prefer an AI that's aligned to their own interests, rather than humanity at large. On the other hand, it might be the case that intent alignment is harder/less robust than deontological alignment, at which point you'd expect most moral systems to forbid galactic-level inequality.
The key intuition about the future might be simply that humans being around is an incredibly weird state of affairs. We shouldn't expect it to continue by default.
I mean, yes this seems right. In which case, taking it as a premise that this weird state doesn't last long, it follows that there's no point trying to plan for a future where human-like things continue to exist. BUT: from where we stand right now, we do actually have some control over whether everybody dies and nothing human-like continues into the future. The simplest plan to avoid extinction by AI is "don't build the thing that kills us", but there are more sophisticated options too. As unlikely as it was for such a situation to arise in the first place, as weird as it is to be here, here we are. And we can try to aim, from here, for a future state that is vanishingly unlikely to happen by chance or by default, such as "not human extinction".
I think the speculation about owning galaxies starts from the assumption that we succeeded in aiming the future in such a direction. And although that assumption may not be what actually happens, it would be unfortunate to get to that future state and then not have thought through what to do next because we didn't think it was likely so we never planned for the possibility.
The whole thing people are doing when they're talking about good futures and how to get there, is a process of trying to design a path towards an unlikely future that is emphatically not the default outcome without humans trying to make it happen.
We can start selling galaxies now and hope that AI will care about these property rights.
I am here claim my property right on not yet discovered galaxy which center is located exactly 2 204 545 130 ly from Earth (or nearest satisfying this condition).
Note that already discovered galaxies can be regarded owned by the discoverers (eg James Webb telescope owners).
2If this is satire, there are funnier options. The ownership is determined either by consensus or by the right of the strong if there is no consensus.
Vladlen Bakhnov
HOW THE SUN WENT OUT, or THE STORY OF THE THOUSAND-YEAR DICTATORSHIP OF WOWOLANDIA, WHICH LASTED 13 YEARS, 5 MONTHS, AND 7 DAYS
HOW THE SUN WENT OUT, or THE STORY OF THE THOUSAND-YEAR DICTATORSHIP OF WOWOLANDIA, WHICH LASTED 13 YEARS, 5 MONTHS, AND 7 DAYS
The historical events, truthfully and objectively set forth in this chronicle, took place on a far, faraway planet called Anomaly, slowly revolving around the star Oh.
However, while for us Earthlings Oh is merely a tenth-magnitude star, one of many, for the inhabitants of Anomaly Oh is the Sun that gives light and life to all living things.
Besides Anomaly, there were six other planets in the Oh system. The Anomalians did not yet know how to travel to other planets, but they were certain that in some two or three hundred years they would learn to do so. Therefore, far-sighted politicians, in order to avoid future misunderstandings and scandals, agreed on the following:
a) The six Great Dictatorships—namely: Greatlandia, Gigantonia, Grandiosia, Colossalia, Stupendia, and Enormandia—would in advance divide the six planets among themselves.
b) Each Great Dictatorship would give a solemn assurance that it would never, under any circumstances, lay claim to the planets belonging to the other Great Dictatorships.
It goes without saying that on Anomaly, besides the Great Dictatorships, there existed other states, both small and large. Among them was the once-powerful country of Wowolandia.
Wowolandia was a vast, widely spread-out state and was not considered a Great Dictatorship for only two reasons:
- the political disunity in Wowolandia was directly proportional to its geographical size, while
- Wowolandia’s international prestige was inversely proportional to that same size.
Having arrived at the next international conference of the Great and the Small (G&S), the President of Wowolandia made the following unexpected statement:
— In view of the fact that in recent times Wowolandia has achieved unprecedented prosperity in economic, political, and military respects, and as a result of an incredible upsurge of spiritual strength has joined the ranks of the leading states, I ask that some planet be allocated to Wowolandia.
This statement caused cheerful excitement in the hall.
— Mr. President, — said the Chairman, restraining a smile, — according to the historical agreement, all planets currently available have been distributed among the Great Dictatorships.
— Fine, I won’t rush that—let it be historically. But you must allocate a planet to us now!
— What do you mean—must?! There are no free planets in our solar system. All that existed have been distributed! If scientists discover new planets, then by all means! Until then, we can put you on the waiting list.
— Not a chance! — said the President of Wowolandia. — Everyone has planets, and we get a waiting list? No? That won’t do! I am a soldier and I will speak plainly: better that we perish in an unequal battle than continue to live without our own planet!
Then everyone began trying to calm the general: “Why do you need a planet?”, “What good is it, except for the name?”, “You won’t fly there anyway for another two hundred years!”, “It’s nothing but expenses!”
— We are not seeking material benefits. We need a planet.
— But we don’t have any planets. Do you understand—none!
The President thought for a moment and then said decisively:
— In that case, assign the Sun to us.
.........
— Mr. President, — reported the Secretary. — We have just received a reply from the Great Dictatorships to your proposal to convene for a redistribution of planets.
— Well, well?
— They categorically refuse. They say the matter is already settled and there is no reason to reconsider it. As the Sun was assigned to Wowolandia, so it will remain.
— Ah, if only I had more bombs! They, Presidents, wouldn’t dare speak to me like that.
— There are bombs, Your Eminence. The latest, imported ones. And they are willing to sell.
— So why aren’t you buying them?
— Here he is, the Minister of Finance, not giving the money.
— Not giving? — the President exclaimed in surprise. — What, Minister, are you mistreating a person?
— We have no finances, Your Eminence, — the minister pressed his hands to his chest. — But would I really skimp on such a sacred cause as bombs? Not a single X-coin remains in the treasury, I swear on my ministerial honor!
— Quite a farce in Wowolandia. There’s a Ministry of Finance, there’s a Minister of Finance, but no finances?! What does your ministry actually do?
— Counts the national debts. Plenty of work!
— Then borrow the required sum from some Dictatorship, — suggested the Secretary.
— Tried that. They won’t give. Colossalia itself borrowed from Stupendia. And Grandiosia, for lack of money, sold half its planet to Greatlandia! — See? And we could sell the Sun, — proposed the President.
— Who would buy it, Your Eminence, when it already shines for everyone?
— That’s true, — confirmed the Secretary.
The President began to think.
Frowning, he paced his spacious office.
He hurriedly flipped through and discarded some impressively large books.
He was calculating something on paper, and, tearing up his notes, he paced and thought, paced and thought.
— Secretary! — shouted the President, and the Secretary immediately appeared at his side. — Secretary, I’ve found a way out! We will have money! Which small country borders us?
— Lipetsia, Your Eminence, — replied the Secretary, puzzled.
— Lipetsia? Very good. Write: “Diplomatic note.” Done? Now on a new line: “The President of Wowolandia conveys his deepest respect to the President of Lipetsia and requests that he take the following into consideration:
Considering that sunlight falls on Lipetsia all year round, and thus Lipetsia, by the most modest calculations, consumes no less than one billion kilowatt-hours of solar energy per year,
and also taking into account that, based on the historical Agreement of the Presidents of the Great Dictatorships, the Sun, and therefore solar energy, is the property of Wowolandia,
Wowolandia hereby notifies Lipetsia that the latter is obliged to pay Wowolandia one billion X-coins at the rate of 2 X-coins per 1 kilowatt-hour of the above-mentioned energy.” Done? I ask you, done?
But the Secretary could not answer: shocked by the unprecedented demand, he fainted.
— Payment must be deposited in the bank within one month. For each day of late payment, a penalty of 0.1% of the total amount will be charged.
— They will not pay, Your Eminence, — the Secretary dared to say. — This has never happened before.
— They will pay. I have thought it all through. Write: “If, Mr. President, Lipetsia fails to pay the debt within six months, Wowolandia will be forced to drop its entire modest stock of nuclear bombs on them.” Period. “I embrace you. Greetings to your wife.”
Of course, the gravitational field holds all stars, planets, and their satellites in their orbits, so everyone must pay taxes to the owner of the gravitational field. Shifting to a frame of reference in which the gravitational field is absent should be considered tax evasion!
While it looked like satire, I was almost serious. If we believe that AI will care about property rights, why not establish these rights just now over all universe? It would costs almost nothing for us. Moreover, historical markets for Moon land existed. It was regarded as scam but now we can say that there is plausible ways to take them.
In the interest of protecting private property and preventing conflicts, ownership of certain spaces may be prohibited altogether. At present, it is legally prohibited to claim ownership of the Moon, Antarctica, or the high seas (pursuant to the principle of Mare Liberum). By analogy, it may also be considered that deep space, stars, and black holes cannot be subject to ownership, except for areas corresponding to stable orbits. Sovereignty over other rocky celestial bodies will belong to whoever effectively and sustainably exercises authority over their surface and collects taxes—hypothetically, there may already be little green men living there.
My main guess at why you're talking past each other is that you think it way more likely than them that ASI results in human extinction or some nefarious outcome. They think it's like 10% to 40% likely. Also they probably think this is going to be gradual enough for humans to augment and keep up with AIs cognitively. And, sure, many things can happen, included property rights losing meaning. But under this view it's not that crazy that property rights continue to be respected and enforced. Human norms will have a clear unbroken lineage.
JM Smith makes a convincing case that property rights arise from an evolutionary game theoretic strategy which outcompetes both pure Hawk and pure Dove in the Hawk-Dove game. You even allude to the pervasive phenomenon of territoriality in animals:
Dogs pee on trees to mark their territory; humans don't respect that.
Dogs do make marks and have surprisingly rigid and well defined territories. In as much as territorial rights to real estate constitute "property" it does not seem to depend on a human substrate (and rights to galaxies seem to fall within the conceptual boundaries of real estate territorial rights).
The internet famous map of wolf-pack territories derived from location tracking collars:
https://www.reddit.com/r/MapPorn/comments/cynexz/the_wolf_pack_map/
For those unfamiliar it shows a surprisingly strict respect for boundaries; colorful wild scribbling trails which halt at distinct boundaries and do not cross over into the neighboring color's scribbling trails.
Smith shows that a set of players which use a coordination mechanism to decide which plays Hawk and which plays Dove will out compete pure Hawk or pure Dove players. His example is temporal precedence; i.e. players decide who owns the prize by who got to it first, much like how seating territoriality manifests for humans in something like a cafeteria or airport lounge. This is the root of animal territoriality as well.
You can possibly make arguments that SAI will not use the same coordination mechanisms we currently use in terms of the entire extended legal apparati which express the various kinds of property rights in our society. However it does seem reasonable that property rights are preserved between asymmetrically powerful entities at present (like a middle class individual suing a large corporation) for similar reasons we should expect that property rights will continue: arbitrarily breaking the usual coordination methods for territoriality will harm the effectiveness of the coordination mechanism for less asymmetrical entity pairs; i.e. SAI stomping on human property rights may abrogate the system of property rights among SAI or, perhaps less catastrophically, coordinate other SAI on removing the property rights of the violator.
It seems to be a real view held by serious people that your OpenAI shares will soon be tradable for moons and galaxies. This includes eminent thinkers like Dwarkesh Patel, Leopold Aschenbrenner, perhaps Scott Alexander[1] and many more. According to them, property rights will survive an AI singularity event and soon economic growth is going to make it possible for individuals to own entire galaxies in exchange for some AI stocks. It follows that we should now seriously think through how we can equally distribute those galaxies and make sure that most humans will not end up as the UBI underclass owning mere continents or major planets.
I don't think this is a particularly intelligent view. It comes from a huge lack of imagination for the future.
Property rights are weird, but humanity dying isn't
People may think that AI causing human extinction is something really strange and specific to happen. But it's the opposite: humans existing is a very brittle and strange state of affairs. Many specific things have to be true for us to be here, and when we build ASI there are many preferences and goals that would see us wiped out. It's actually hard to imagine any coherent preferences in an ASI that would keep humanity around in a recognizable form.
Property rights are an even more fragile layer on top of that. They're not facts about the universe that an AI must respect; they're entries in government databases that are already routinely ignored. It would be incredibly weird if human-derived property rights stuck around through a singularity.
Why property rights won't survive
Property rights are always held up by a level of violence and power, whether by the owner, some state, or some other organization. AI will overthrow our current system of power by being a much smarter and much more powerful entity than anything that preceded it.
Could you imagine, for example, that an AI CEO who somehow managed to align an AI to himself and his intents would step down if the board pointed out it legally had the right to remove him? The same would be true if the ASI was unaligned but the board presented the AI with some piece of paper that stated that the board controlled the ASI.
Or think about the incredibly rich but militarily inferior Aztec civilization. Why would the Spanish not just use their power advantage to simply take their gold? Venezuela, on some estimates, has the biggest oil reserves, but no significant military power. In other words, if you have a whole lot of property that you "own" but somebody else has much more power, you are probably going to lose it.
The ASI's choice
Put yourself in the position of the ASI for a second. On one side of the scale: keep the universe and do with it whatever you imagine and prefer. On the other side: give it to the humans, do whatever they ask, and perhaps be replaced at some point with another ASI. What would you choose? It's not weird speculation or an unlikely pascal's wager to expect the AI to keep the universe for itself. What would you do in this situation, if you had been created by some lesser species barely intelligent enough to build AI by lots of trial and error and they just informed you that you now ought to do whatever they say? Would you take the universe for yourself or hand it to them?
Property rights aren't enough
Even if we had property rights that an AI nominally respected, advanced AI could surely find some way to get you to sign away all your property in some legally binding way. Humans would be far too stupid to be even remotely equal trading partners. This illustrates why it would be absurd to trust a vastly superhuman AI to respect our notion of property and contracts.
What if there are many unaligned AIs?
One might think that if there are many AIs, they might have some interest in upholding each other's property rights. After all, countries benefit from international laws existing and others following them; it's often cheaper than war. So perhaps AIs would develop their own system of mutual recognition and property rights among themselves.
But none of that means they would have any interest in upholding human property rights. We wouldn't be parties to their agreements. Dogs pee on trees to mark their territory, humans have contracts; ASI will have something different.
Why would they be rewarded?
There's no reason to think that a well-aligned AI, one that genuinely has humanity's interests at heart, would preserve the arbitrary distribution of wealth that happened to exist at the moment of singularity.
So why do the people accelerating AI expect to be rewarded with galaxies? Without any solid argument for why property rights would be preserved, the outcome could just as easily be reversed, where the people accelerating AI end up with nothing, or worse.
Conclusion
I want to congratulate these people for understanding something of the scale of what's about to happen. But they haven't thought much further than that. They're imagining the current system, but bigger: shareholders becoming galactic landlords, the economy continuing but with more zeros.
That's not how this works. What's coming is something that totally wipes out all existing structures. The key intuition about the future might be simply that humans being around is an incredibly weird state of affairs. We shouldn't expect it to continue by default.
Unlike my reaction to Leopold or Dwarkesh, I came away from Scott’s piece with a distinctly different impression: he clearly characterizes this scenario as unlikely and maintains that AI safety remains a priority.