Review
Seeing OpenAI's impressive technology should make media commentators think it more likely that AGI is possible and existentially dangerous. Some do, but overall this shift in views does not seem consistent enough; the default reaction still seems to be to view the whole AI thing as just another profitable technology development.
If you did not pay much attention to AI before ChatGPT and then started, you were very likely surprised and impressed. This may even be true if you closely followed AI developments, and even if GPT-3.5 did not impress you, GPT-4 likely did. This also seems to be true of many journalists; AI reporting has become a topic for mainstream media. LLMs (a word that only AI experts knew 2 years ago) are very powerful, and even if you don't immediately see how they will transform the economy, you feel their power if you use them. The same is true of image generation or voice generation AI. The power of translation and other applications like protein folding may be less immediately obvious, but they add more facets of AI power.
Yet there is a strange missing reaction of society's mainstream to these developments, which became particularly visible in the reporting and commentary on the firing of Sam Altman as CEO of OpenAI.
Imagine a person came to you and claimed to be a magician. She had created a magic workshop in order to summon a super-powerful demon, and on the way there she will create smaller versions of the demon. You laugh because demons don't exist. But she creates the small demons. These demons are not super-powerful, but you would not have expected them to be possible. How should the small demons change your mind about the possibility of a super-powerful one?
OpenAI claims that it can create and also tame the demon called AGI. But media coverage usually ignores that it has demonstrated the ability to create small ones explicitly as steps toward creating the super-powerful one.
Consider the following claims that you could have had opinions on in January 2019:
Suppose that in 2019, you think that all of this is science fiction. Now you fall asleep in 2019 and wake up five years later, and you see all the AI tools available. It seems that the first claim has become true. It seems natural to also increase the likelihood you assign to claims 2-3 being true. After all, creating a capable AI seems like a necessary step that you need to complete before creating an extremely capable A(G)I, and being dangerous is more likely if A(G)I is extremely capable. However, maybe you believe the latter two claims are still quite unlikely.
But then you realise that the organization that has made the AI that changed your claim-1 worldview, OpenAI, had a charter that said that its
"mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work—benefits all of humanity. We will attempt to directly build safe and beneficial AGI [...]".
So you now know that an organization that can build powerful AI, and that knows how to create frontier technology, also claims that it can build AGI. This could be a marketing claim, but it seems that you should take the organization's claims seriously. (At closer inspection, you see that the firm has very smart software engineers, researchers, and managers, which may be seen as additional signals that the claims in the charter are to be taken seriously; these people will probably not coordinate around nonsense claims.) Okay, fine, you update towards a higher likelihood of claim 2 being possible.
But OpenAI's charter also states:
We are committed to doing the research required to make AGI safe, and to driving the broad adoption of such research across the AI community.
We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.”
This should give you pause: The people developing AGI think that it is necessary to put serious effort into making sure that AGI is made safe, and they make unusual commitments to do so.
So by now you have started believing that the three claims above are either true or have some likelihood that you cannot ignore. However, this comes with the belief that OpenAI is very capable, and in particular that it is capable of doing what it claims, so you should also believe that OpenAI will likely be also able to make AGI safe, right?
Maybe. But if so, this is a probability conditional on creating AGI, and there is no reason to believe that the conditional probability is 100%. So you should now put some non-zero probability on the possibility that AGI can be created, that it is important to make it safe, and on the possibility that making it safe won't succeed.
Yet for some reason, media commentators mostly stop at claim 1 of the list, and the economic and business implications of what can be expected in the next few years from a company that creates such things - as though this were a company building a new car, a new spaceship, or some other mundane technology.
Some day, you read the news that the board has fired CEO Sam Altman. Wait, what? Sam Altman, the admired CEO of OpenAI? The board does not explain this very well, and you see the protests of smart people at OpenAI against this. In the end, the board seems disempowered, and Altman is back in charge. What will you conclude, and what should you conclude?
The answer depends on whether you believe that the fact that many people at OpenAI defended Sam Altman should tell you that he indeed should not have been fired. That is, maybe you believe that so many smart people defending Altman shows that the board was wrong, or you may believe that these people are probably biased for some reason.
In any case, for people wanting to understand the situation, the firing should have been a cause to take a look at the institutional structure of OpenAI (see, for example, here), understand why this structure was chosen in the first place, and consider whether there was more to this conflict than just ordinary power play. You could then still have concluded that this whole AGI thing and the potential danger is overblown; but it does not seem possible to simultaneously treat OpenAI as a technology gamechanger and ignore the mode of thinking that inspired both OpenAI's technology and its institutional structure.
The Economist, for example, did make that mistake. In a column about Elon Musk ("Elon Musk’s messiah complex may bring him down", December 2023), The Economist writes:
"Most troubling is the messiah complex. ... At times he sounds like a capricious Greek god who believes he holds the fate of the world in his hands. ...
Saving humanity is in vogue right now. It is a dangerous fetish. Last month a charter to protect the world from the dangers of rogue AI almost destroyed OpenAI, maker of ChatGPT. A year ago Sam Bankman-Fried, now a convicted fraudster, claimed that the disastrous risks he took with his FTX crypto-exchange were in service of humanity. Such missionary zeal is not new in business. It pushed Henry Ford, inventor of the Model T, to raise workers’ living standards. But his saviour complex got the better of him and he ended up spewing antisemitic bile.
Mr Musk’s hubris, too, may end badly. ... Future generations will probably judge him the way today’s judges Ford: a handful will decry his flawed character; most will remember the majesty of his creations."
The disappointing part is not just the superficial pattern-matching ("like a capricious Greek god"), or the one-sidedness of only mentioning some examples of powerful people as evidence against trying to "save the world", while not mentioning Bill Gates's activities aimed at preparing the world against pandemics, for example. Rather, the disappointment comes from seeing that one of the most influential magazines in the (Western) world does not realize the inconsistency of mocking the board's actions for "saving humanity" while ignoring that all actions of OpenAI, its whole business model, is justified by "saving humanity".
The columnist's only argument that the actions of the board of OpenAI were obviously bad is that it almost destroyed OpenAI, as though it were obvious that the world is a better place with OpenAI. The columnist does not seem to think that this claim has to be proven, but just presupposes that destroying profitable companies is bad. But if the profitable company was founded to summon super-powerful demons, and it is profitable because the small demons it is already able to summon can write your homework and your diet plan, it seems you should at least try to understand what is going on.
Thanks to JustisMills for very valuable feedback.