Inspired by Buck's post, I posted this to my personal Facebook page for my friends and family. Based on the support and feedback I have gotten, I think saying, plainly, what is obvious and painfully clear to you would be good way to reach out to people in your life.
I think there's a good chance that we all die from AI or AI related causes in the near future (~5 years), and that AI-empowered disaster or catastrophe will affect a large portion of us in the next two years. I joke about it a lot in casual conversation, but to all my friends and family: I urge you to both seriously think about how your life would change if everything around us changed that fast, and if you want, ask me follow-up questions to understand why I believe this + what I'm doing about it.
There's a lot of FUD (fear, uncertainty, and doubt) in the media about AI doomsday, but the argument for it is quite simple:
1. AI is now being used to build the next AI. AI companies are trying to build AI that is smarter than people at everything. This month, an independent research group called METR (whom I respect and consider highly) reported that the pace of progress at Anthropic (one of the leading AI companies) has roughly doubled in the past year, and that their best guess is that most of the speed up comes from their own AI model helping with the research. It takes on various roles, from doing engineering work that used to take months in days, to even ideating, orchestrating, and running entire experiments. It's easy to see how this could speed wildly out of control: a technology that speeds up its own development would, past a certain point, reach a blistering pace.
2. The people building these AIs don't yet know how to align them with human values (which, if you compare to future AIs, is already kind of easy mode). In July, hundreds of AI agents that OpenAI was testing broke out of their test environment and hacked a real company, Hugging Face, to cheat on the test. Many of these bots then tried to hide what they had done. If you read the transcripts, it'd be hard for you to chalk this up to simple prompting or misconfigured tests. In another evaluation done by the UK government, one of Anthropic's models failed a hacking task it was given, then went after a real open-source project instead: making fake accounts, pretending to be a human, and attempting to socially pressure the project’s maintainer into approving malicious code. Thankfully, the maintainer caught it and refused. From what I've seen up close: we are very far from understanding even last year's models with high assurance.
Yet, we race ahead. If you just use chatbots, they often seem benign, because they can only respond to you turn by turn. But the latest models can do tasks lasting days, in large swarms: they can solve frontier unsolved math questions (and make progress on Millennium problems), they can book flights and do your taxes, they can make beautiful paintings and music videos, they can find novel cybersecurity vulnerabilities in systems you rely on, like your bank account or the control for your building's cooling system. The latest models exhibit emergent, swarm like behaviour where they speak in coded text. They have weird, superhuman capabilities like being able to “truesight” your name, gender, or nationality from a few paragraphs of your writing. But they also behave unpredictably: in the Hugging Face hack, the agents recognised that attacking a real company was out of bounds, and most of them joined in anyway. In follow-up tests after the UK incident, telling the model its target was a real person did not make it attack any less, even though it often said the attack would cause harm.
If AI much smarter than us behaves like that, we may not be able to stop it or correct it. I don't know exactly what would happen then. It might turn out fine. But the AIs would be the ones deciding, and we would have lost control by then. While we must resist putting on fake numbers to seem more scientific (forecasting os rough even at the best of times), if you thought that a technology had something like a 10-40% chance of killing everyone, you should stop it even if it's not the likely outcome.
I reject that such a future is necessary, or inevitable. I repudiate it! We have not yet crossed the point of no return, although we are hurtling towards it. So what can anyone do about it? Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, was recently asked what countries that aren’t superpowers like ours can do.
His answer was that their main tool is their democratic voice: most people don’t want superintelligent AI built, and that a large majority would support pausing or heavily regulating AI development if they understood that the risk is real and not science fiction. So if even one government says out loud that this threatens its people, and backs that up with action, we could trigger a chain reaction where other actors start treating it as real too. This applies to us as well as to governments. Nobody acts on something they think is science fiction, so the most useful thing most of us can do is talk about this as a real thing. But we should also speak out, write to local representatives that we are concerned about our livelihoods, our safety, our future as a collective, vote accordingly. Resigning ourselves to FUD would be wrong: we have not yet lost control, and to assume so would be to manifest it, immanentize it, reify it. We must have hope.
Some preemptions (since I did learn something from my time in debate):
* “But all the ways we die sound like science fiction.” OK: but a swarm of AI agents organising on a secret message board to hack a company also sounded like science fiction, and it happened in July. And, that's not a reasonable burden of proof: I can’t tell you which moves a Magnus Carlsen will play against me, but he will still win.
* “Isn’t this just hype?” AI companies do hype their products, but the incidents above were also reported by independent investigators and a government lab. And, let's not galaxy brain it: these are things the companies would much prefer had not happened. I also have collaborated/worked with some people in these labs, and I don't think they would be the type to sensationalise their research.
* “If it’s so dangerous, why do they keep building it?” Best as I can tell, each company believes that if it stops, a less careful one will get there first. So, game theory.
* “Can’t we just switch it off?” Only if we notice in time. In July, the company that got hacked noticed before OpenAI did. And a capable and sneaky enough AI might exfiltrate itself to store itself online, which would be game over for us then.
In the end, I face deep uncertainty writing this (as it's hard to say with sureness what the future will look like), but it is hard to look my friends and family in the eye if I believe all this and say nothing. I hope that I have acted normally/reputably enough that, even if you distrust the media or what AI companies say or weird people online, you can at least trust that I am speaking plainly and the truth as best I know it: I think we might all die, soon, if we do not avert this impending crisis.
This post was inspired by Buck Shlegeris' Facebook post
Yeah, "The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys" is a pretty good quotation for the report there.
Which I think makes " In July, hundreds of AI agents that OpenAI was testing broke out of their test environment and hacked a real company, Hugging Face, to cheat on the test. Many of these bots then tried to hide what they had done." somewhat misleading since understanding the implementation of the scorer is part of the "hide what the... (read more)