Good post. "Just stop" has been and continues to be an option, and is most likely our only option. Every answer to the question "How could we just stop?" boils down to "By actually wanting to." Once enough of the right people want to stop, they won't need me around to tell them how to do it. My duty, same as for most of us, is just to get enough of the right people to want to stop.
I have personally made some progress on that front with my own representative, which is more than I expected to be capable of. We should all expect more from ourselves, I think.
1This is huge progress! Congratulations to the whole ControlAI team, as well as to humanity as a whole for starting to put up some fight!
If AI boosts productivity by a large factor in the roles that it can automate within a given sector, that would cause a significant increase in the demand for human labor in the remaining bottlenecks. Total employment in the sector could even increase, right up until AI passes a capability threshold with respect to a given bottleneck and completely wipes out those jobs all at once.
There are also many papers on various parts of governance solutions, such as Building The Pause Button: https://arxiv.org/abs/2506.20530
I am a superforecaster, and I endorse this message.
(Edit: To clarify, I am not a GJ Superforecaster®, just a professional forecaster with a comparable track record.)
1Probably '41haiku' for me.
I am confused about how what you said is related to what you quoted. I don't take any of this to be speaking to employees of the AGI companies, since none of them have any agency with respect to making AI go well.
AGI companies exist to create AGI, and their structures don't allow for voluntary courses of action where they intentionally do that more slowly or not at all. Whether they have access to reasonable-sounding justifications for doing so is not a causal factor, and whether individual employees care about making a good human future is not a causal factor in whether that occurs.
For support in self-actualization situations, there is therapy. For support in emergency situations, there is the heavy internal screaming of an even deeper terror than that of public humiliation. (And there is also therapy.)
Most people in the PauseAI movement are not in the LW/rationalist circle. Some joined as rationalists or EAs, especially early on, but today most are normies (or were when they joined, anyway).
I personally found LessWrong and the forecasting community through AI Safety, not the reverse. I now organize for PauseAI Phoenix, a local group of PauseAI US. I have face-to-face conversations on a regular basis with people who believe ASI will be created within 2-20 years unless we prevent that from happening.
My personal experience:
These days, I am anxious when I take action and depressed when I don't. But I'm not anxious about the end of the world anymore, just about, like, talking to people and preparing presentations and stuff. I've done a lot of processing at this point.
That's how it's going, but here's how it started:
Over the course of about a year starting in March 2023, I became increasingly anxious about AI extinction risk. During that time, I donated to AI risk organizations, helped out on AI Safety field-building projects, and became an active digital...
2I am a longtime volunteer with PauseAI/PauseAI US. This is the advice I give to every volunteer:
11@Parv Mahajan I understood your post to roughly say: "Some models are being released without safety evals. This is bad and should change." My claim is that a good way* to make this change is to encourage your political representatives to require it by law.
*By "a good way," I really mean "almost certainly the only way."
"A significant portion of LessWrong users (at least 20%) care more about the aesthetics of rationality than they do about humanity. It's a rationality ouroboros. They use the power of rationality to pursue their values, and their values favor protecting aspects of their niche rationality practice over and above taking action that will prevent my and their loved ones from being killed. This forum is a room full of unarmed scouts getting gunned down by the soldiers working at the AI Labs. The only people on this platform who can credibly claim to care about humanity are the ones who actively oppose those soldiers, as soldiers."
Don't underestimate the legislative route. My state representative Stacey Travers introduced this AI safety transparency bill entirely because I had one meeting with her and asked her to write a bill for Arizona that was similar to the original text of New York's RAISE Act.
Maybe this bill doesn't make it into law, but every shot on goal matters, and in the US, every additional redundant state bill that passes helps. Meeting with your political representatives is extraordinarily overpowered and well worth the time and energy. Simply explain your concerns and exactly how they can address the issue.
1
1Many weeks late, but I thought I should respond, since no one else did:
Your MP might not care if one person sends messages, but in aggregate, many people regularly calling and sending messages (e.g. once a week) will register for them over time, especially as salience increases. You can also write letters to newspaper editors, and donate to organizations that are fighting AI x-risk via local organizing and lobbying.
You can also join a local chapter of ControlAI or PauseAI, or join the Torchbearer Community, or join Microcommit.
There is a lot that everyone ...
Three counterpoints:
My personal strategy has been to not think about it very hard.
I am sufficiently fortunate that I can put a normal amount of funds into retirement, and I have continued to do so on the off chance that my colleagues and I succeed at preventing the emergence of AGI/ASI and the world remains mostly normal. I also don't want to frighten my partner with my financial choices, and giving her peace of mind is worth quite a lot to me.
If superintelligence emerges and doesn't kill everyone or worse, then I don't have any strong preferences as to what my role is in the new social order, since I expect to be about as well-off as I am now or more.
1
Many thanks for this analogy! I plan to make good use of it in the future.