Note: I think this is partially me working through my plans and partially my emotions, so imo it's less epistemically rigorous than ideal for LessWrong.
I might have less than a year left to affect the insane, reckless, Faustian race to ASI, and I plan to spend that time doing everything I can to push for a pause.
I currently think there’s a decently high chance that by default, there’s 6-12 months left before my personal point of no return: the point after which my own actions no longer plausibly affect the future. This is different from other people’s points of no return: for example, people like Dario Amodei (CEO of Anthropic) or Donald Trump may have slightly longer times to wake up and change course. This is also different from other timelines that people sometimes throw around, like timelines to AGI or ASI or fully automated remote-worker or whatever. However, it is the most important for my own decisions.
I do not defend this position here. Other people have written partial explanations that I like (Karthik, AI Futures Project, Sentinel, Rutger Bregman). I may also write more accessible explainers in the near future.
While I don’t have strong confidence that I actually only have 6-12 months left, I think very few people are working on this by default, and AI Safety is growing very quickly (though still far smaller than needed), and newcomers to the field can cover the scenarios where we have more time.
Given this, I think my best strategy is to push hard for a pause on training AI: specifically a complete moratorium on training stronger AI models than the ones we currently have and severe limitations on AI R&D, until scientists, policymakers, and the general public have more time to understand what’s going on and catch up.
I think pushing for a pause would be the best use of my time. While it’d be very difficult to execute, it is less robustly doomed than trying to make things better on the margin and slightly increasing the chances that the crazy race to superhuman AI is conducted safely. I’m not very confident in this position and do not defend it in detail here, though I wrote some other strategies I’ve considered below in the next section.
I’m less opinionated about what happens after a pause. Reasonable options range from resuming carefully under real engineering standards to pausing indefinitely, if humanity collectively decides that the risks posed by superhuman AI is too high.1
Regardless, we should not be sleepwalking into disaster. A pause gives humanity time to regroup and rethink things.
This is the strategy. What are tactics? What should I concretely do?
Here I am less certain. Right now, in the next (uncertain but limited) number of days, I’m dropping all my research and writing projects to re-orient myself to this new reality and think through what I can do to increase the odds of a pause.
Current candidate options for what seems live to me: a) moving to Sacramento and providing evidence and arguments for the California government to mandate a pause, b) doing something similar in DC, c) serving as a research/writing assistant for more prominent public intellectuals pushing for a pause, d) helping advocacy and protest movements working towards a pause, or e) spearheading long-shot plans to increase the odds of a pause.
And of course, just writing a lot online.
(Extremely open to other ideas!)
What are other strategies I could be pursuing?
Is pausing the best strategy? I think so, but I’m not confident. Here are two options that I’ve considered and tentatively rejected (I’ve also considered other options, but they seem even more doomed, at least for me).
Help key decisionmakers “manage the transition better”
If a pause won’t happen, I’d rather the transition to superhuman AI be done well (or okay, or at least poorly) rather than disastrously.
I considered trying to join Anthropic or OpenAI. I currently think I wouldn’t be a good fit2. I dislike the organizations too much, and working on a doomsday machine to make it slightly less doomed just doesn’t sit well with me. I might still re-consider it if nonconflicted people argued me into believing that the impact is high enough, and I should swallow my disgust.
I also considered working to help government people re-orient and make better non-pause decisions. This still seems like a live option to me, and I will continue looking for potentially good options.
Macrostrategy automation and trying to solve non-alignment AI issues with AI assistance
Under some models of AI progress, the most urgent time isn’t now but several months in the future when we’ll have 3-6 months of rapidly accelerated non-technical research due to AI assistance, before my own irrelevance. Maybe I’d have 5-100x speedup on non-alignment problems that matter a lot, like how to counteract the effects of AI propaganda or making sure humanity doesn’t go insane through the transition.
Preparing for that time seems somewhat competitive, and there are a relatively small number of people who can do this well, myself marginally included.
I currently feel less excited about this than working towards a pause, but I will continue thinking about it and talking to my colleagues at Forethought about this possibility.
Focus on counteracting AI persuasion in short-timelines worlds
An angle that I feel somewhat excited about/drawn to is trying to help harden specific decision-makers and decision-making against future AI superhuman persuasion (my research area this year). While somewhat less likely with faster timelines than slower ones, I still think there are significant worlds where superhuman AI persuasion will be a big part of AI takeover attempts, and hardening within-AI-company and USG decision processes against AI persuasion seems fairly competitive with other options, and plays well into my research interests and backgrounds.
I can imagine doing this either within or outside of the existing institutions.
Other personal updates:
Emotionally it’s… not been great. But I’ve been handling this re-orientation better than I personally expected (so far). My dominant feeling is more like determination plus light anxiety than deep fear, dread, or resigned acceptance.
I’m consciously aware that there’s probably only enough time left to make a single medium-sized career pivot (likely still in the researcher/writer space). Not many more pivots left from here on out.
I plan to not do much in field-building or mentorship, other than continuing existing relationships or maybe upskilling people who are highly experienced in other fields into AI safety/AI macrostrategy. Time is too short.
I’m slightly worried about going crazy in the next year, partially due to the situation seeming increasingly insane, and partially due to actual Claude psychosis as the models get stronger. Not sure what to do about it. Just flagging it as a risk.
I’m comparatively more worried about just “hiding away”/”shutting down”, and living an unendorsed life of being depressed and internet/media distractions. I’ve set up some safeguards with friends to reduce this risk, and plan to do more.
I plan to stop doing difficult personal/professional growth stuff unless it’s in-the-moment enjoyable, extremely critical, or very AI related. The basic model is that I already had over 30 years to grow (more than a decade as an adult), so the likely gains of further growth are unlikely to be worth the time and attention costs. I can do more growth after the singularity, or after a pause. AI-related stuff is an exception because squeezing in crazy uplift seems plausible. I honestly feel kind of sad about this; I’m someone for whom personal growth is important, or at least someone who likes to imagine themselves as someone for whom personal growth is important3.
I plan to be more open with my information and be less discreet with semi-private information (unless I think they’re other people’s secrets or I explicitly promised confidentiality). Openness is critical to the world orienting better, and there’s just less time left for information hazards and attention hazards to backfire.4
Will be less critical of other safety people’s plans going forwards. Time is too short. Generally want to be less of a roadblock to good things happening.
Planning to postpone elective surgeries I was vaguely considering.
I plan to get in less internet fights, except occasionally with popular/important people. Tbh this is going to be really hard for me, but I’ll try my best!
I no longer look both ways before crossing a one-way street.
What should other people do?
For people currently working in AI safety:
If you currently work in technical AI safety, AI macrostrategy, AI policy, etc, presumably you’ve already heard of some of these arguments and updates and already think they’re somewhat important.
If you haven’t already done so, you should definitely think about this and talk to your colleagues (both internally and externally) about the updates towards shorter timelines and what it means for you and your organization! My guess is that there are a lot of things most people in the field should be changing about their work.
I don’t have strong opinions about what specifically you should be changing to, without knowing more of your specific situation and context.
Technical people should be split between some combination of a) providing evidence of risk and why a pause is critical, b) working out the technical details of a pause, c) emergency alignment and interpretability techniques that just might work if we can’t pause, d) more long-run alignment, interpretability, control etc, strategies that work only conditional upon us succeeding on getting a pause (or if I am wrong about how much time we have left).
If you’re non-technical, obvious options include a) persuading and pressuring AI companies to unilaterally pause, b) lobbying Congress (if you’re Democrat) or c) the Executive Branch (if you’re Republican) for a pause, d) lobbying California to mandate a pause, or e) raising awareness and increasing public demand and political will for a pause.
Other options include pushing other states to demand the AI companies pause (most saliently Delaware but I think other states have leverage too), building momentum in China to pause if the US does so (this is delicate; if you’re in a situation to do so presumably you know more than me on how to do so), and increasing international pressure on a global AI moratorium.
For people not currently working on AI safety
A very important strategic consideration here is that because time to the point of no return might be very short but we can’t know for sure, and we likely may never know for sure until it’s too late, the safety field should likely bifurcate strategies where some people work on the assumption of very short (<1 year) time to have an impact, and other people work on the assumption of more time to have an impact.
Since the field is growing very quickly but it takes people time to get up to speed, a natural bifurcation is for most people currently in the field to work under the ~1 year clock, and most people currently outside of it to get up to speed and do things that are more promising given more time.
All things considered, I think those worlds are still very salient, either because we’re wrong about the future speed of automation of AI research, or because the pause actually worked. However, I don’t have advice for how to join the AI safety effort later on, over and beyond the standardadvice.
With that in mind, here are things I think people not currently in AI safety should do:
(For some people) Think of highly context-specific and long-shot plans to increase the odds of a pause. My guess is that for the right people in the right places, that’s your individual highest bet of making sure we don’t die from AI. I don’t know what your context is so it’s hard to say what plans can and can’t have a chance of working. But for example I don’t have special connections with the Vatican (and they might actively dislike me), but other people more connected with the Vatican or other faith leaders might have ideas on how faith leaders can sound the alarm and call for a moratorium. Similarly for people with connections to ASML/Netherlands government, TSMC/Taiwanese gov’t, mass media, compute providers like Amazon, other tech leaders, other members of civil society, and so on.
If you do have a plan that you think has some chance of working, please don’t hesitate to reach out! I’m available over substack DMs, and can either give some feedback or connect you with other people better suited to evaluating those strategies.
To be clear, please don’t do (or plan, or threaten) anything violent! Violence predictably backfires in almost all circumstances, will not stop the suicidal race to ASI, and makes the jobs of all the rest of us harder.
Get oriented. Learn more about AI and AI safety and understand or consider more durable ways you can contribute, whether on technical AI safety, on policy, or something else.
Join protests and other forms of direct action against the reckless, madcap, Faustian race to AGI. Do not just accept the race as a fact of nature. Get involved with organizations like DONOTACCEPT ai, Humans In Control, and Who Decides AI.
At some point (if things go okay), AI protests will be so popular that each individual in a protest will be less important, and there are more impactful ways to get involved in AI safety. But we’re (sadly) still in the early part of the exponential for AI saliency and worries about AI risk.
I will go to an AI Treaty March myself this Saturday. I’m highly uncertain whether this march will actually increase the odds of a US-China treaty, but I want to go partially so there’s momentum behind future marches, and partially to build the muscle for protests myself.
Call or email your political leaders and demand action. Especially for Americans. This mad race must end, and the American Congress and the Executive Branch can make it happen.
Slacktivism. If nothing else, it’s pretty good to understand the issues well enough, and be correct, polite, and kind in online discussions of AI and AI risk. Make the stakes as clear as you can and link to ways people can contribute.
A final, personal, message: This is deep and heavy stuff. I wish I could offer more emotional solace or reassurance. I don’t think I’m the right person, or this is the right situation, for that. What I could say is that I truly believe the fight is well worth it. We must not accept the industry’s framing of this mad, suicidal, race as inevitable. We can and must do our best to end it.
The Linchpin is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Options that seem salient to me right now: a) unpause and proceed ahead on an “AI as normal engineering” assumption, except we take normal engineering safety standards seriously, rather than the slapdash “safety” and “security” measures that currently plague those companies*, b) unpause and proceed according to an international verified slowdown between the US and China, with total research transparency and extensive checks on safety between each development cycle, c) don’t unpause until companies have safety cases that they can constrain the risk of further AI development below a manageable threshold, those safety cases are verified by external evaluators and openly communicated and debated, and humanity agrees that unpausing is worth the risk, d) pause for 6 months, decide whether the risks of unpausing outweigh the benefits, if not, pause for 1 year, then do this again, pause for 2 years, then 4, then 10(?), and so forth, e) do limited AI research into safety, and don’t unpause until we discover AI architectures that are provably or demonstrably much safer than the status quo, f) unpause with a focus on specialized AI architectures for specific tasks (like self-driving cars and medicine), don’t do AI research into RSI or other dangerous scary general AI, or e) don’t unpause period until humanity changes our minds; the technology for supplanting all of human labor and cognition is too dangerous for Man.
Or in my darker moments, someone who likes to imagine themselves as someone who likes to imagine themselves as someone for whom personal growth is important.
Subtitle: This mad race must end.
Note: I think this is partially me working through my plans and partially my emotions, so imo it's less epistemically rigorous than ideal for LessWrong.
I might have less than a year left to affect the insane, reckless, Faustian race to ASI, and I plan to spend that time doing everything I can to push for a pause.
I currently think there’s a decently high chance that by default, there’s 6-12 months left before my personal point of no return: the point after which my own actions no longer plausibly affect the future. This is different from other people’s points of no return: for example, people like Dario Amodei (CEO of Anthropic) or Donald Trump may have slightly longer times to wake up and change course. This is also different from other timelines that people sometimes throw around, like timelines to AGI or ASI or fully automated remote-worker or whatever. However, it is the most important for my own decisions.
I do not defend this position here. Other people have written partial explanations that I like (Karthik, AI Futures Project, Sentinel, Rutger Bregman). I may also write more accessible explainers in the near future.
Regardless, humanity is absolutely not ready for what comes next, whether that’s human extinction, complete loss of control to rogue AIs or a few oligarchs, or something far weirder.
While I don’t have strong confidence that I actually only have 6-12 months left, I think very few people are working on this by default, and AI Safety is growing very quickly (though still far smaller than needed), and newcomers to the field can cover the scenarios where we have more time.
Given this, I think my best strategy is to push hard for a pause on training AI: specifically a complete moratorium on training stronger AI models than the ones we currently have and severe limitations on AI R&D, until scientists, policymakers, and the general public have more time to understand what’s going on and catch up.
I think pushing for a pause would be the best use of my time. While it’d be very difficult to execute, it is less robustly doomed than trying to make things better on the margin and slightly increasing the chances that the crazy race to superhuman AI is conducted safely. I’m not very confident in this position and do not defend it in detail here, though I wrote some other strategies I’ve considered below in the next section.
I’m less opinionated about what happens after a pause. Reasonable options range from resuming carefully under real engineering standards to pausing indefinitely, if humanity collectively decides that the risks posed by superhuman AI is too high.1
Regardless, we should not be sleepwalking into disaster. A pause gives humanity time to regroup and rethink things.
This is the strategy. What are tactics? What should I concretely do?
Here I am less certain. Right now, in the next (uncertain but limited) number of days, I’m dropping all my research and writing projects to re-orient myself to this new reality and think through what I can do to increase the odds of a pause.
Current candidate options for what seems live to me: a) moving to Sacramento and providing evidence and arguments for the California government to mandate a pause, b) doing something similar in DC, c) serving as a research/writing assistant for more prominent public intellectuals pushing for a pause, d) helping advocacy and protest movements working towards a pause, or e) spearheading long-shot plans to increase the odds of a pause.
And of course, just writing a lot online.
(Extremely open to other ideas!)
What are other strategies I could be pursuing?
Is pausing the best strategy? I think so, but I’m not confident. Here are two options that I’ve considered and tentatively rejected (I’ve also considered other options, but they seem even more doomed, at least for me).
Help key decisionmakers “manage the transition better”
If a pause won’t happen, I’d rather the transition to superhuman AI be done well (or okay, or at least poorly) rather than disastrously.
I considered trying to join Anthropic or OpenAI. I currently think I wouldn’t be a good fit2. I dislike the organizations too much, and working on a doomsday machine to make it slightly less doomed just doesn’t sit well with me. I might still re-consider it if nonconflicted people argued me into believing that the impact is high enough, and I should swallow my disgust.
I also considered working to help government people re-orient and make better non-pause decisions. This still seems like a live option to me, and I will continue looking for potentially good options.
Macrostrategy automation and trying to solve non-alignment AI issues with AI assistance
Under some models of AI progress, the most urgent time isn’t now but several months in the future when we’ll have 3-6 months of rapidly accelerated non-technical research due to AI assistance, before my own irrelevance. Maybe I’d have 5-100x speedup on non-alignment problems that matter a lot, like how to counteract the effects of AI propaganda or making sure humanity doesn’t go insane through the transition.
Preparing for that time seems somewhat competitive, and there are a relatively small number of people who can do this well, myself marginally included.
I currently feel less excited about this than working towards a pause, but I will continue thinking about it and talking to my colleagues at Forethought about this possibility.
Focus on counteracting AI persuasion in short-timelines worlds
An angle that I feel somewhat excited about/drawn to is trying to help harden specific decision-makers and decision-making against future AI superhuman persuasion (my research area this year). While somewhat less likely with faster timelines than slower ones, I still think there are significant worlds where superhuman AI persuasion will be a big part of AI takeover attempts, and hardening within-AI-company and USG decision processes against AI persuasion seems fairly competitive with other options, and plays well into my research interests and backgrounds.
I can imagine doing this either within or outside of the existing institutions.
Other personal updates:
What should other people do?
For people currently working in AI safety:
If you currently work in technical AI safety, AI macrostrategy, AI policy, etc, presumably you’ve already heard of some of these arguments and updates and already think they’re somewhat important.
If you haven’t already done so, you should definitely think about this and talk to your colleagues (both internally and externally) about the updates towards shorter timelines and what it means for you and your organization! My guess is that there are a lot of things most people in the field should be changing about their work.
I don’t have strong opinions about what specifically you should be changing to, without knowing more of your specific situation and context.
Technical people should be split between some combination of a) providing evidence of risk and why a pause is critical, b) working out the technical details of a pause, c) emergency alignment and interpretability techniques that just might work if we can’t pause, d) more long-run alignment, interpretability, control etc, strategies that work only conditional upon us succeeding on getting a pause (or if I am wrong about how much time we have left).
If you’re non-technical, obvious options include a) persuading and pressuring AI companies to unilaterally pause, b) lobbying Congress (if you’re Democrat) or c) the Executive Branch (if you’re Republican) for a pause, d) lobbying California to mandate a pause, or e) raising awareness and increasing public demand and political will for a pause.
Other options include pushing other states to demand the AI companies pause (most saliently Delaware but I think other states have leverage too), building momentum in China to pause if the US does so (this is delicate; if you’re in a situation to do so presumably you know more than me on how to do so), and increasing international pressure on a global AI moratorium.
For people not currently working on AI safety
A very important strategic consideration here is that because time to the point of no return might be very short but we can’t know for sure, and we likely may never know for sure until it’s too late, the safety field should likely bifurcate strategies where some people work on the assumption of very short (<1 year) time to have an impact, and other people work on the assumption of more time to have an impact.
Since the field is growing very quickly but it takes people time to get up to speed, a natural bifurcation is for most people currently in the field to work under the ~1 year clock, and most people currently outside of it to get up to speed and do things that are more promising given more time.
All things considered, I think those worlds are still very salient, either because we’re wrong about the future speed of automation of AI research, or because the pause actually worked. However, I don’t have advice for how to join the AI safety effort later on, over and beyond the standard advice.
With that in mind, here are things I think people not currently in AI safety should do:
AI Treaty March here
A final, personal, message: This is deep and heavy stuff. I wish I could offer more emotional solace or reassurance. I don’t think I’m the right person, or this is the right situation, for that. What I could say is that I truly believe the fight is well worth it. We must not accept the industry’s framing of this mad, suicidal, race as inevitable. We can and must do our best to end it.
The Linchpin is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
1
Options that seem salient to me right now: a) unpause and proceed ahead on an “AI as normal engineering” assumption, except we take normal engineering safety standards seriously, rather than the slapdash “safety” and “security” measures that currently plague those companies*, b) unpause and proceed according to an international verified slowdown between the US and China, with total research transparency and extensive checks on safety between each development cycle, c) don’t unpause until companies have safety cases that they can constrain the risk of further AI development below a manageable threshold, those safety cases are verified by external evaluators and openly communicated and debated, and humanity agrees that unpausing is worth the risk, d) pause for 6 months, decide whether the risks of unpausing outweigh the benefits, if not, pause for 1 year, then do this again, pause for 2 years, then 4, then 10(?), and so forth, e) do limited AI research into safety, and don’t unpause until we discover AI architectures that are provably or demonstrably much safer than the status quo, f) unpause with a focus on specialized AI architectures for specific tasks (like self-driving cars and medicine), don’t do AI research into RSI or other dangerous scary general AI, or e) don’t unpause period until humanity changes our minds; the technology for supplanting all of human labor and cognition is too dangerous for Man.
*Treating AI risks as solvable normal engineering feels doomed to me, but not as completely doomed as treating them as solvable by normal engineering and then not applying normal engineering safety standards, which is the current default.
2
Not saying that they’d necessarily hire me even if I applied, to be clear.
3
Or in my darker moments, someone who likes to imagine themselves as someone who likes to imagine themselves as someone for whom personal growth is important.
4
Among the information I actually have, to be clear. It’s not like I have the plans for a working bioweapon stored in my head. Your mileage may vary.