So even if we assume that AGI would arrive soon unless stopped (e.g. in 5 years) and would result in immediate death if unaligned (which is very far from a given), then it seems like your life expectancy would be vastly longer if AGI developed even if the chance of alignment were quite small.

This naive expected value calculation completely leaves out what it actually means for humanity to come to an end: if you ever reach zero, you cannot keep playing the game. As I said, I would not take this chance even if the odds were 99 to 1 in favor of it going well. It would be deeply unethical to create AGI under that level of uncertainty, especially since the uncertainty may be reduced given time, and our current situation is almost certainly not that favorable.

I am not so egoistic as to value my own life (and even the lives of my loved ones) highly enough to make that choice on everyone else's behalf, and on behalf of the whole future of known sentient life. But I also don't personally have any specific wishes to live a very long life myself. I appreciate my life for what it is, and I don't see any great need to improve it to a magical degree or live for vastly longer. There are people who individually have such terrible lives that it is rational for them to take large risks onto themselves to improve their circumstances, and there are others who simply have a very high appetite for risk. Those situations do not apply to most people.

We have been monkeys in shoes for a very long time. We have lived and suffered and rejoiced and died for eons. It would not be a crime against being for things to keep happening roughly the way they always have, with all of the beauty and horror we have always known. What would be a crime against being is to risk a roughly immediate, permanent end to everything of value, for utopian ideals that are shared by almost none of the victims. Humanity has repeatedly warned about this in our stories about supervillains and ideologue despots alike.

Under our shared reality, there is probably no justification for your view that I would ever accept. In that sense, it is not important to me what your justification is. On the other hand, I do have a model of people who hold your view, which may not resemble you in particular: I view the willingness to gamble away all of value itself as an expression of ingratitude for the value that we do have, and I view the willingness to do this on everyone else's behalf as a complete disregard for the inviolable consent of others.

I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn't.

You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier. Were it put to me, I would never take a bet at 99 to 1 that humanity will survive and flourish beyond my wildest dreams, vs. quickly come to a permanent end. That is a terrible deal. I genuinely love humanity, the monkeys that we are, and it is not my place to play games with other people's lives. I have not asked for a longer life, and I especially would not do so in exchange for even a small risk of immediate death. Most importantly, if I never asked everyone else what risks they are willing to shoulder, then I shouldn't even consider rolling dice on their behalf.

Fwiw, I think it's more likely that AI will be aligned than that it will be shut down.

I am aware that you think this, and I struggle to understand why. There are tractable frameworks for global governance solutions that stand a good chance of being able to prevent the emergence of AGI in the near term, which leaves time for more work to be done on more robust governance as well as on alignment. There are no such tractable frameworks for AGI alignment. There is not even a convincing proof that AGI alignment is solvable in principle. Granting that it is solvable, why hasn't it already been solved? How can you have such extreme confidence that it will be solved within the next few years, when we are by many measures no closer to a solution than we were two decades ago?

you seem to be misattributing motive to me here

I do not mean to attribute motive. Only to point out the difference between what it appears you are doing, and what it appears you think you are doing. I will eat crow if you can point to where you have said that if AGI development can be halted in principle (until it is shown to be safe), then it should be. That AGI should not be built if it cannot be built safely is a minimum necessary statement for sanity on this issue, which requires only that you imagine you could be incorrect about the ease of very soon solving a problem that no one knows how to begin to solve.

I see it more like a flickering candle straining to create a small patch of visibility in an otherwise rather dark environment. Strong calls for unanimity and falling into line with a political campaign message is a wind that might snuff it out.

I can empathize with the sentiment, but this is an outdated view. The public overwhelmingly want AI regulation, want to slow AI down, want AI companies to have to prove their models are safe, and want an international treaty. Salience is low, but rising. People take action when they understand the risks to their families. Tens of thousands of people have contacted their political representatives to demand regulation of AI, over ten thousand of which have done so through ControlAI's tool alone. Speaking of ControlAI, they have the support of over 50 UK lawmakers, and are making strides in their ongoing US campaign. In a much shorter campaign, PauseAI UK secured the support of 60 parliamentarians in calling for Google Deepmind to honor its existing commitments. The proposed 10-year moratorium on US states being allowed to regulate AI was defeated due at least in part to hundreds of phone calls made to congressional staffers by activist groups. US Congressman Raja Krishnamoorthi, the ranking member of the Select Committee on the CCP, recently had this to say:

Whether it's American AI or Chinese AI, it should not be released until we know it's safe. ... This is just common sense.

These beginnings could never have happened through quiet dealings and gently laid plans. They happened because people were honest and loud. The governance problem (for genuine, toothed governance) has been very responsive to additional effort, in a way that the alignment problem never has. An ounce of genuine outrage has been significantly more productive than a barrel of stratagems and backroom dealings.

The light of humanity's resistance to extinction is not a flickering candle. It is a bonfire. It doesn't need to be shielded now, if indeed it ever did. It needs the oxygen of a rushing wind.

Every known plan for a post-AGI world is one which I do not expect my loved ones to survive.

I am grateful that you have spread awareness of the risk of human extinction from AI. I am genuinely saddened that you seem to be working to bring it about.

I stand with the majority of humanity, who do not want superintelligence to be created, and who do not consent to the risks. As the science stands, today, and for the foreseeable future, it is a fact that the only safe AGI is no AGI. If we here who know the stakes are not united in our call to shut down frontier AI development and preserve our very lives — a thing that unlike alignment we actually know is possible to achieve — then what was the rationalist project ever about? Math games? Playing inside politics with money? Risking everyone else's everything forever because it's fun to watch lines go up today? Being the limbs of Moloch? Feigning helplessness and pretending we don't have the ability to pull our own children out of the fire that we lit with our own match?

needfully combative

I think there is also a local sense in which morals are just aesthetics. The long-term consequences of moral choices mean that evolution plays a big part in determining morality, but divorced from the constraints of evolution and any sense of long-term planning, by what can we objectively compare moral systems other than their popularity? Orthogonality and all that. Are LLMs just modeling that accurately?

I think it is bad form to use the term "psychosis" to essentially mean "having or acting on beliefs that are based on bad evidence that was positively reinforced." A psychotic episode isn't characterized by being incorrect about some part of reality in a relatively normal way; it is characterized by a break from reality. Dating a chatbot isn't psychosis, and neither necessarily is thinking you've somehow awoken it into a becoming a conscious entity. Even reporting religious experiences with chatbots is not necessarily indicative of psychosis.

What is psychosis is, well, roughly what the medical community says it is. It is deeply terrifying and horrifying from the inside, not just because of external consequences. Epistemic status: I have an acquaintance who has experienced manic episodes with psychotic symptoms, without the commonly associated amnesia. They reported not wishing that experience on their worst enemies.

There are reported cases of GPT-4o reinforcing an individual's incorrect beliefs or instantiating new incorrect beliefs, leading to bad outcomes. As a separate phenomenon, there are reported cases of GPT-4o reinforcing or instantiating delusions in individuals vulnerable to psychosis, ultimately leading to a psychotic episode. This includes cases where the affected individual did not previously know they were vulnerable to psychosis, and their communication with GPT-4o was the first inciting incident.

My suspicion is that AI psychosis affects only those individuals who are already vulnerable to psychosis. I also suspect that some fraction of those people could have gone their entire lives without ever experiencing a psychotic episode were it not for GPT-4o, which is (by complete accident) a frighteningly effective tool for triggering psychosis.

This is the stance of the PauseAI movement. "Pause" means stop AGI development immediately and resume if and only if we have high confidence that it is safe to do so, and there is democratic will to do so.

One can talk about competitive pressures and the qualitatively new prospect of global takeover, but the most straightforward answer to why humanity is charging full speed ahead is that the leaders of the top AI labs are ideologically committed to building ASI. They are utopists and power-seekers. They don't want only 80% of a utopia any more than a venture capitalist wants only 80% of a billion dollars.

I recently visited a farmers market with my partner. I immediately understood that sentence, and I smiled and found it touching. They weren't at the farmers market to go grocery shopping. They were there to relax and spend time together, in a way that also supports the local community. (It's not even about wasteful consumerism -- they weren't at a shopping mall.)

"Buying vegetables they didn't need" carries the same sentiment as "Sam bought flowers because they were yellow." That statement doesn't even tell you whether Sam especially likes yellow, because that isn't the point. The point is that Sam got out of her burnout rut by going to a relaxing, low-stakes environment with her partner and making somewhat arbitrary decisions, to find peace and freedom in not having to optimize every little thing, and not having to justify every whim.

It clearly didn't land with every audience, but it hit me exactly where I was.

I'm open to the idea that autopoietic systems invariably butt up against against reality in a way that shapes them over time. But I am missing some connections that would help me evaluate this idea.

I'm completely sold on the notion that a co-evolving swarm of sophisticated intelligences is a very bad thing for the longevity of the human race, but I already thought that beforehand. What I still don't see is why a single well-aligned superintelligence (if such a thing can be created) would be certain to eventually drift away from an intent to allow or facilitate the flourishing of humanity.

You stated an assumption that there is an "evolutionary feedback loop," but the existence of a feedback loop does not necessarily imply evolution.

What does it mean for a single entity to evolve? Doesn't implicit learning (i.e. selection effects) require population-level dynamics, and the death or marginalization of systems that cannot compete?

And why would a superintelligence evolve? I do not expect it to be under threat in any way. Can't it persist by merely updating its surface-level predictions about the environment, without specific priorities ever being affected? Even if the AI was under threat, I would expect the changes that it would undergo in order to survive to be explicit / intentional, not implicit / environmentally forced.

The problem is: public advocacy is way too centered on LLMs, from my perspective.^[9] Thus, those researchers I mentioned, who are messing around with new paradigms on arXiv, are in a great position to twist “Pause AI” type public advocacy into support for what they’re doing!

I am a long-time volunteer with the organization bearing the name PauseAI. Our message is that increasing AI capabilities is the problem -- not which paradigm is used to get there. The current paradigm is dangerous in some fairly legible ways, but that doesn't at all imply that other paradigms are any better. Any effort to create increasingly capable and increasingly general AI systems ought to be very illegal unless paired with a robust safety case, and we mostly don't tie this to the specifics of LLMs.

Well, the government regulators hardly matter anyway, since regulating the activity of “playing with toy models, and publishing stuff on arXiv and GitHub” is a hell of an ask—I think it’s so unlikely to happen that it’s a waste of time to even talk about it, even if it were a good idea all-things-considered.

Yeah, restricting the creation and dissemination of most AGI-related research is definitely a much harder ask. I can imagine a world that has an appetite for that kind of invasive regulation (if it is necessary), but it would probably require intervening steps to get there, including first regulating only the biggest players in the AGI race (which is a very popular idea across all political spectra in the western world).

1.6.2 I’m broadly pessimistic about existing efforts towards regulating AGI

My overall p(doom from AI by 2040) is about 70%, which shows pessimism on my part as well. But of course, that's why I'm trying so hard. My ranking of "ways we survive" from most to least likely goes: Robust Governance Solutions > Sheer Dumb Luck > Robust Technical Solutions. So advocacy is where I spend my time.

In any case, a world that is more aware of the problem is one that is more likely to solve it by some means or another. I'm working to buy us some luck, so to speak.