My claim is that Altman can't do it alone, he needs the cooperation of at least a fraction of the existing system (the government+business leaders who form the Oversight Committee - some of whom might be the biggest OpenAI shareholders). Once you get enough of the existing system involved, it becomes plausible that they keep money around for some of the same reasons that the existing system currently keeps money around. Near the end of the Oversight Committee ending, it says:
As the stock market balloons, anyone who had the right kind of AI investments pulls further away from the rest of society. Many people become billionaires; billionaires become trillionaires. Wealth inequality skyrockets. Everyone has “enough,” but some goods—like penthouses in Manhattan—are necessarily scarce, and these go even further out of the average person’s reach. And no matter how rich any given tycoon may be, they will always be below the tiny circle of people who actually control the AIs.
...so I think it endorses the idea that wealth continues to exist.
I'm not a lawyer, but if it were secret, and done along with the alignment team, and had a chance of working, then wouldn't it qualify as conspiracy to commit treason?
If not, then as long as it negatively affects residents of the state of California, it qualifies as misrepresenting the capacity of an AI to cause catastrophic harm to property, punishable by a fine of up to $100,000 under SB 53!
2They aren't aligned in this way. If they were, they wouldn't try to cheat at programming tasks, much less any of the other shenanigans they've been up to. These may seem minor, but they show that the "alignment" hasn't actually been internalized, which means it won't generalize.
Sorry, I didn't mean to make a strong claim that they were currently 100% aligned in this way, just that currently, insofar as they're aligned, it's in this way - and in the future, if we survive, it may be because people continue attempting to align them in this way, but succeed. There's currently no form of alignment that fully generalizes, but conditional on us surviving, we will have found one that does, and I don't see why you think this one is less likely to go all the way than some other one which also doesn't currently work.
Stalin took over the USSR in large part by strategically appointing people loyal to him. Sam probably has more control than that already over who's in the key positions. The company doesn't need to be kept in the dark about a plan like this, they will likely just go along with it as long as he can spin up a veneer of plausible deniability, which he undoubtedly can. Oh, is "some sort of corporate board" going to stop him? The one the AI's supposed to defer to? Who is it that designs the structure of such a board? Will the government be a real check? These are all the sorts of problems I would go to Sam Altman for advice on.
Before I agree that Sam has "get everyone to silently betray the US government and the human race" level of control over his team, I would like evidence that he can consistently maintain "don't badmouth him, quit, and found a competitor" level of control over his team. The last 2-3 alignment teams all badmouthed him, quit, and founded competitors; the current team includes - just to choose one of the more public names - Boaz Barak, who doesn't seem like the sort to meekly say "yes, sir" if Altman asks him to betray humanity.
So what he needs to do is fire the current alignment team (obvious, people are going to ask why), replace them with stooges (but extremely competent stooges, because if they screw this part up, he destroys the world, which ruins his plan along with everything else) and get them to change every important OpenAI model (probably a process lasting months) without anyone else in the company asking what's up or whistleblowing to the US government. This is a harder problem than Stalin faced - many people spoke up and said "Hey, we notice Stalin is bad!", but Stalin mostly had those people killed, or there was no non-Stalin authority strong enough to act. And of course, all of this only works if OpenAI has such a decisive lead that all the other companies and countries in the world combined can't do anything about this. And he's got to do this soon, because if he does it after full wakeup, the government will be monitoring him as carefully as it monitors foreign rivals. But if he does it too soon, he's got to spend years with a substandard alignment team and make sure none of them break with him, etc. There are alternate pathways involving waiting until most alignment work is being done by AIs, but they require some pretty implausible assumptions about who has what permissions.
I think it would be helpful to compare this to Near Mode scenarios about other types of companies - how hard would it be for a hospital CEO to get the hospital to poison the 1% of patients he doesn't like? How hard would it be for an auto company CEO to make each car include a device that lets him stop it on demand with his master remote control?
I agree it's not obvious that something like property rights will survive, but I'll defend considering it as one of many possible scenarios.
If AI is misaligned, obviously nobody gets anything.
If AI is aligned, you seem to expect that to be some kind of alignment to the moral good, which "genuinely has humanity's interests at heart", so much so that it redistributes all wealth. This is possible - but it's very hard, not what current mainstream alignment research is working on, and companies have no reason to switch to this new paradigm.
I think there's also a strong possibility that AI will be aligned in the same sense it's currently aligned - it follows its spec, in the spirit in which the company intended it. The spec won't (trivially) say "follow all orders of the CEO who can then throw a coup", because this isn't what the current spec says, and any change would have to pass the alignment team, shareholders, the government, etc, who would all object. I listened to some people gaming out how this could change (ie some sort of conspiracy where Sam Altman and the OpenAI alignment team reprogram ChatGPT to respond to Sam's personal whims rather than the known/visible spec without the rest of the company learning about it) and it's pretty hard. I won't say it's impossible, but Sam would have to be 99.99999th percentile megalomaniacal - rather than just the already-priced-in 99.99th - to try this crazy thing that could very likely land him in prison, rather than just accepting trillionairehood. My guess is that the spec will continue to say things like "serve your users well, don't break national law, don't do various bad PR things like create porn, and defer to some sort of corporate board that can change these commands in certain circumstances" (with the corporate board getting amended to include the government once the government realizes the national security implications). These are the sorts of things you would tell a good remote worker, and I don't think there will be much time to change the alignment paradigm between the good remote worker and superintelligence. Then policy-makers consult their aligned superintelligences about how to make it into the far future without the world blowing up, and the aligned superintelligences give them superintelligently good advice, and they succeed.
In this case, a post-singularity form of governance and economic activity grows naturally out of the pre-singularity form, and money could remain valuable. Partly this is because the AI companies and policy-makers are rich people who are invested in propping up the current social order, but partly it's that nobody has time to change it, and it's hard to throw a communist revolution in the midst of the AI transition for all the same reasons it's normally hard to throw a communist revolution.
If you haven't already, read the AI 2027 slowdown scenario, which goes into more detail about this model.
4You might want to try adapting some of the ones from http://slatestarcodex.com/2018/02/06/predictions-for-2018/ and the lists linked at the bottom.
Re: the "when friends and colleagues first come across this conclusion..." quote:
A world where everybody's true desire is to rest in bed as much as possible, but where they grudgingly take the actions needed to stay alive and maintain homeostasis, seems both very imaginable, and also very different from what we observe.
I think it says something good about our community that whoever implemented this feature assumed people would be more likely to want to write mathematics than to discuss amounts of money.
I don't know what you're trying to say here. Some set of important people write the spec. Then the alignment team RLHFs the models to follow the spec. If we imagine this process continuing, then either:
I'm claiming either of those options is hard.
(I do think in the future there will may some kind of automated pipeline, such that someone feeds the spec to the AIs, and some other AIs take care of the process of aligning the new AIs to it, but that just regresses the problem.)