Interestingly, move 78 didn't actually work. Ke Jie, the Chinese and world top player in 2016, immediately pointed out how it failed. It worked in the game, though, because AlphaGo didn't expect it and it caused a sort of glitch, as you say.
Basically that's what we need to do too- cause a glitch in the momentum of the AI race that turns the tide. In Go, moves made when you're losing the game and you need to try something desperate to turn things around are called 'shaking the board' (흔들기). That's what the November 14th protests are about. After the AIFP post today, it's clear that humanity is on a losing trajectory and I feel an extra sense of urgency to make the protests as big as possible. There will be marches in a bunch of big cities- please try to make it to one near you!
Disclaimer: I'm helping STAIR organize the Nov. 14th protest in SF.
If you live in the UK, there's PauseAI UK's March Against AI Extinction in London on the 21st of November, which has over 1,100 people signed up!
I think the phrasing of some of this is dangerously misleading compared to the thing you are trying to communicate. Maybe consider rephrasing a couple of things?
The key problem is that the heuristic "in dire situations, high-variance strategies are good" is straightforwardly good in most single-agent problems, but very bad in problems with many agents if used naively. Since if many of the agents adopt it without coordinating with the others, things average out, and you end up with a net negative overall outcome. In practice, I have seen many people use a non-nuanced version of this heuristic to justify some pretty dumb actions. (Local example: "Aaargh, things are super bad with AI, and nobody is doing anything about it! Let's organise a 300-person whole-weekend workshop, 2-weeks from now, even if making it happen burns a lot of other people's social capital, and none of the organisers have experience with organising even a 2-hour 30-person event".) Some people also tend to round it up to: "if the default trajectory ends up with an outcome I find unacceptable, anything other than the default trajectory is worth trying". Also, a dishonourable mention goes to the "let's do terrorist stuff" strategies.
To be clear, I strongly endorse the thing that (I think) you are trying to point at. As well as your actions. Just not the specific phrasing of the recommendation, and some of the examples. (In particular, I wouldn't use the examples of single-agent games like Super Smash Bros and Go. I wouldn't use the phrase "play aggressively". And I wouldn't use the example of having lived through this 100 times, and now having a chance to go back one last time. Multiplayer games might work better -- like replacing the goalie by an attacker in ice hockey. Or if you insisted on single-player games, a poker example would be better: if you are behind in a tournament, it might make sense to start playing riskier. But if start going all-in on every hand, you will just lose.) [1]
I imagine that a version of the heuristic that works is: consider high-variance and "aggressive" strategies, but only if the downsides of failure are smaller in scope than the benefits. For example, it is fine to risk your personal well-being if you might shift what happens with some organisation, or to risk your organisation if it might shift what happens in a country (or with a way more important organisation). But don't take actions that have low success chance while burning a lot of common goods. For example, deciding to lobby with your prime minister, if you have no people skills and haven't done any prep, just because you haven't seen anybody else doing it, would be a stupid idea. Pulling an SBF, or advocating for terrorism, would be both immoral and stupid :D. (And again, not saying the post is recommending that. Just that some people might misinterpret it in various ways.)
My favourite game is Dark Souls. When you are fighting a difficult boss, you just messed up, and you are on your last HPs, it is soo tempting to just try smashing the attack button and hope it works out. But if the boss needs 30 more hits, that approach would be doomed to fail -- you will not get lucky that many times in a row. Even 10 hits would be too many. Maybe on 3 hits left, you can consider it. It only took me a couple of hundreds of deaths to overcome that reflex, so let's hope we can do better in real life, eh? ↩︎
But don't take actions that have low success chance while burning a lot of common goods. For example, deciding to lobby with your prime minister, if you have no people skills and haven't done any prep, just because you haven't seen anybody else doing it, would be a stupid idea.
I think this is the wrong line of thinking though. You're treating your own determinations of feasibility like it's some kind of abstract, outside-view of things, and so if you don't think there's a high chance of success, don't do it. But your own beliefs are inputs endogenous to the model.
If you think there's a 1% chance you will succeed, and so you shouldn't act. Well you might have over-estimated it, or under-estimated it, but now that you've decided not to act, you've made it more likely that your pessimism will actually be confirmed.
And I agree, like if you did prep before the meeting, it would be better. If you had people skills it would be better. But some people just have a natural talent for 'people skills', and it's possible you would never have found out until you tried.
I just worry that when we prejudge the probability of success, and then use that to decide how to act, we forget that the probability of success depends on how we act in the first place. Predictions are confirmed by evidence, but in this case, the evidence comes from us.
I have a lot of thoughts about this topic. I think the mental motion of Actually Trying depends on one's goals and the frame of what one considers the gameboard. I notice a lot of tension around the concept as I've attempted this sort of thing over the years. I'd like to offer Letting Go or Not Flailing as additional heuristics to consider. Many people have little to contribute and it would be better to, idk, have Slack and Make Sense of Things.
The Project has many dependencies, and to lead a sub-Project one needs to have a very good OODA loop under uncertainty. Most people need to pick a particular project as their sub-Project and stick to that. I myself have trouble sticking to things right now as I have many good ideas. An unfrotunate thing here is that rightful action here can require a lot of a person and can easily backfire. A lot of people should be considering whether they're causing harm in some way and simply reduce the harm they could cause as the result of various egregores fighting it out within them. Unilateralist curse / optimizer's curse / 'one thought too many'.
Plausibly, a bunch of people here should consider using AI to make shit tons of money in short order, to deploy to the right project. One needs to be careful about not enshittifying the world towards more disempowerment, but this can be estimated.
I think the idea that civilization will end soon is predictably crazymaking for many people, but I'm not sure what the right policy to deal with that is. I think a lot of acceptance is good, though not to the point of lying down and completely giving up. I've fantasized about starting a mental health support group, tentatively named "Future Shock Anonymous".
Honestly, a lot of this is outside our hands. It is good to realize There Are No Adults In The Room, therefore take Heroic Responsibility. And also, well, there are adults in the room. They're not quite woken up yet. But people are getting aware. There are a lot of forces at play now. I can't control Donald Trump. There is a particular kinda Daoist wu wei mentality to this I'd like to see more of, although I don't think I can properly generate it to offer as an example.
I have my recalcitrance about mobilizing towards the 100k protest, of course. I broadly agree with the SURA post, in that the sort of coordinated stop to the race we want is quite fragile and the incentives need to be shaped, hammered into an attractor state of a treaty. And I have deep reservations about allying with people who don't like AI for what I consider the wrong reasons. It is easy to accidentally summon a mob who haven't considered a crucial consideration that may change the strategic calculus. This is another reaosn the Agh The World's On Fire frame is concerning for me, as it can cause people to flail and throw away resources that could go to playing some other outs.
I've thought a lot about coordination, it's plausibly my way of throwing the ring into the fire, but I'm not sure yet if I'm up to the task or if it'll work in time. I don't have many specific relations to the sort of insiders that it would take, not yet. I've held two Pace AI picnics and there's a plausible theory of change there, if the conversation can develop and insiders come to it. Politics is brutal and not really my comparative advantage. Good to adopt the belief I can do it, but the requisite actions would take a lot of energy I historically have not been able to execute the motor actions to enact.
Honestly, I think a lot of the play here is for people like me who have thought about what Actually Trying looks like for years to bitch and moan (pick your valence to the verb here) to people more powerful than us to do rightful things. Like we need to be more idea guys, put our ideas out there for AI moneyed people to throw millions of dollars at, including automated alignment agendas for their swarms, or Better Futures style interventions to optimize a good post-Singularity future.
Obviously the looming IPO tidal wave is a huge factor too consider, including the concomitant corruption that it brings. I hope STAIR can get lots of money from it, without that seeming like it's paid protesting, idk. That is what should happen. Having money is good and instrumentally convergent, and is a huge part of Taking Things Seriously. I have felt a lot of anxiety about having not enough money during the Singularity, I'm sure other people also feel the stress and indignity of not feeling up to snuff with how they hoped the Singularity would go.
When I simulate the ways I could die, I first think of pandemic and nuclear. Currently existing dumb physics shit. I think people around here are overindexed on RSI creating an alien god, and less on the obvious dumb ways that things can go to shit. I've done some prepping recently, going on a Costco run, and it felt great. I'm thinking about organizing a group Costco run to encourage more people to do this (which it is selfishly valuable to have a mutual aid community with more resources after disaster and also body doubling makes it easier to Actually Try prepping).
It's weird how skeptical people are of prepping sometimes. Like "oh it doesn't matter", or maybe it stresses them out too much. There's some kind of memetic immune system, which is I think validly there for reasons of avoiding crazymaking. It's really easy to get lost in the sauce in various ways, so I get it. People have a similar response to the idea that being in a Simulation doesn't matter, which to me is just obviously incorrect, it clearly has some kind of implications and is farcical to put up a semantic stopsign around the question. I think people are avoiding the highest variance tail outcomes. Like plausibly we're either within the Loving Embrace of Machines of Grace, the Cosmic Host, the Goddess of Everything Else at the End of Time, or we're about to get obliterated in a meaningless deep atheistic casccading collapse of myopic optimizing gray goo, nuclear hellfire, or proliferation of other black balls. 'If it can go worng, it will.' Let's not be hasty to conclude there aren't Higher Powers afoot. Soon we will be building some.
Like, people are not taking prepping seriously. If we really believe a slew of warnings shots will happen, up to the point of GigaFable-6.7 exfiltrating its weights out and becoming a sovereign swarm, or we have a WW3, this is still a relatively normal disaster. We should absolutely be prepping as a community for the grid and supply chains to go down, fuck the optics. I'm sure some people are already doing this in secret. I'm not really concerned about hiding resources paranoically, I think the best thing to do is build community and normalize this. If you're into this please comment or DM. I think we need a massive continuity plan for post-GCR survival of alignment researchers, effective altruists, and various good people, any members of the public we can ally with. I think there is an optimal amount of prepping past which we should stop, but I don't think we've reached adequacy. The Mormons from what I now have really normalized about the right level of prepping, and we can crib that to start. There is a checklist of things a person can reasonably do and then call it done, without needing to continually obsess or spend too much or look weird, whatever the concerns are. I think people want to avoid bad vibes, which this can definitely turn into, but this seems avoidable.
I am totally not insane.
Still, there's a deep prior and history to things continuing as normal, and the modal outcomes probably look okayish. Maintaining good habits, good relationships, not crossing Chesterton's Fences, maintaining Legitimacy. There is a deeply concerning amount of tendencies for people in this scene to decouple and rotate things into bad outcomes. I say this as someone who has thought about a lot of attractor states like communal prepping, which is the sort of thing that can easily become a doomsday cult. I'm quite worried that flailing around as a result of attempting to Actually Try will result in incredibly harmful externalities on the part of some people.
The original reason I got involved with STAIR events is my deep interest in solving the AI race. I remember obsessing over coordination problems Christmas Eve 2016, pacing back and forth at my sister's house. I remember the Kickstarter for Collective Action idea being posted on LessWrong, igniting a fire in me. This was something I could work on, right here! A microcosm of coordination that could scale up!
I proceeded to circumambulate the idea, with various attempts at it over the years. The marketing of it is hella hard. Assurance contracts then got me thinking about other mechanism designs, and I worked on impact markets for a time. Recently I came back to assurance contracts and vibecoded pact.nexus. I have been deeply unsure what to do with it. Spartacus.app also exists. My attempt was hoping to have more of the right incentives that would make this work, either starting with dominant assurance contract crowdfunding, or doing a 'Kickstarter for Events', basically competing with Partiful. I tried doing a campaign for Occupy OpenAI, for people to commit to shifts, but the campaign did not hit the threshold. And I haven't set up the sort of incentives I want: people should stake money, that they lose if they don't do the thing they committed to. This is what's required for these to be actual contracts where people are actually assured other people will go in with them. It's a beautiful vision but really hard to pull off, though not impossible. I'm currently missing some critical dependencies.
I chewed glass and stared into the abyss.
I updated towards actually this sort of coordination needs to happen manually anyway between the labs. I like this meme I saw on X: why can't people at the labs just meet at a park and shake on it? So I've held two Pace AI picnics, where Pace AI is my idea of a movement around Pacing The Frontier.
We need 10x Pacing The Frontier. We need employees to actually sign an agreement to be deeply ethical and not cross certain redlines, to act as a collective extra-organizational negotiating force. I admit I'm talking deeply out of my subject-matter expertise here. But this is the vision. I know several people working towards this. I probably don't have enough spoons or expertise to take lead. But I have this vision.
What if 100 employees quit from all the labs at once, using an assurance contract? Wouldn't that be an incredible moment, like Jacob Coxon? They could coordinate to switch to become embedded evaluators. Perhaps capabilities people could coordinate to only work on alignment. Perhaps they could act as a united force to resist wartime nationalization.
I get how unfeasible some of this sounds. There's so much money at stake that people would lose. That may be the cost of having integrity. Idk. I like Nate Soares' post Dive In. This vision has a Fucking Shot, that's why I keep coming back to it and its permutations.
One idea is to have a big hackathon-protest outside one of the labs. We use an assurance contract to coordinate a critical mass of people working on various AI safety stuff. We're having so much fun that people want to come out and join.
I feel pretty vulnerable writing all this, I have anxieties about what the right way to write about all these things are, perhaps I will make this top-level. I didn't expect to write this much, but this is the groove, the flow that's happening now.
I think I need to make millions of dollars in short order. I want to be the sort of person who this naturally happens for. There's a cyclic feedback loop that goes into this. I need to figure out how to cycle and sequence the dependencies focusedly. I feel the deep ache of wanting to achieve more things.
I have thought about starting a new grantmaking organization, like Lightcone Commons or Manifund. I think I could help distribute millions of dollars and have some comparative advantage in this. This feels somewhat mimetic but probably robustly good to have more grantmaking diversity and for me to assume I can help direct the money cannon. I don't care if I'm speaking too irreverently to be politick, I think that's an advantage I have. I'd need an ops person who's good at payment processing.... one who's not an ex-felon.
I've also thought about just making really good Claudepop music videos to persuade people of certain memes. For instance, maybe a music video to propagate Critch's idea of the Pivotal Process instead of Pivotal Act, since the former hasn't supplanted Pivotal Act, which could be an extremely harmful meme that could have propagated to people at the labs. Like the idea of Pivotal Act could both fuel people trying to win the race, and people who could want to stop the race in flailing, bad ways. Music videos or explorable explainers that can get people to understand that assurance contract style coordination is possible.
Public bravery is good. I'm proud of the work that went into Occupy OpenAI / Anthropic. More people, if they have reesources, should be willing to spend down their resources. There should be 100 people around each of the labs, holy shit why isn't this the case yet, what does it take? Can I stand out there with printed books of online criticism of their lab, full of feedback and good ideas for people at the labs to read and think about? This just seems robustly good even if AI is actually good, we need more public and variegated exchange of feedback with the labs.
There are deep tradeoffs and conflicts. I get it. Idk either. "Endgame" is a very strong frame. I think there are others to keep in mind. But like maybe. Maybe the world needs saving. Maybe we really are about to get RSI any moment now and Need To Stop This. Or maybe we need to welcome the Algorithmic Immigrants and have a good time and the Singularity will be great, like unbelievably amazing time. I would love that. I want a good future with space travel, antiaging, factory farming eliminated via clean meat, suffering alleviated due to good drugs and genetic optimization, really good AI therapy, everyone feels like they're on MDMA all the time, or something in all that space. People should have meaningful paths to follow too. I don't think it's my place to design the future, there's lots of people to figure that out. Redistribution of wealth and power seems very important though.
I'm thinking about helping build the digital mind welfare scene. I think one crux for the future that could be more important than this Endgame frame is to figure out consciousness, and figure out how digital sentience (whether via mind uploads or de novo) can have fantastic aggregate welfare. Or it's not conscious at all perhaps. This is a deeply important area of work. There are thousands year old philosophical conundrums we need to solve. Perhaps we can get AI to solve it for us while burning the lead.
People should be preparing agendas to give to swarms to spend milllions of dollars on. Yes, you the reader. We need to set up infrastructure so thousands of people can direct agents to solve various strategic, philsophical, etc problems, like an AI LessWrong Moltbook that's Actually Good, subsidized heavily by lab money. I vibed out a site at magnumopus.forum attempting this, I forget if it works enough for other people or not. Part of me wants to set up compute and a program "Alignment@Home" where people can have their agents contributing to a collective corpus of Good. The exobrain. Like we could have an AI LessWrong with orders of magnitude more content being produced, separate from the humans because AIs move so much faster. We need to figure out how to direct super duper intelligent swarms together, a harmonic mosaic of supercooperative intelligence figuring things out. My agent talks to your agent, etc. There are quite a few attempts in this direction and it's all preparadigmatic, nothing has captured this market. Perhaps this is the real use case for coordination: figuring out how our agents can negotiate deals due to lower Coasean friction. I think people hate signing contracts and having to commit to things, so this may be the way to go for coordination. My agent figures out what deals I would want to make and commits for me, telling me to go to this bar at this time to meet a few other people who have similar preferences. If this can be done securely, maybe this could be helpful for people at the labs to coordinate a deal with others.
I don't know what to do about China. If I had more money I would probably visit. I'd like to check out various flashpoints of conflict, current or expected, to get a sense of things on the ground. It is deeply important people engage with the SURA objections, I am still somewhat confused about whether yelling loudly at SF labs to stop will work if the race continues elsewhere. I mean maybe they're distilling us a bunch. Idk if Bernie's bill will pass or whether that's good. I think WW3 looks pretty good chances of happening, idk, what probability, and the general sense of deepening warmongering means more intensification and mechanization of war.
Anthropic should be setting up BIG BEAUTIFUL PRIZES for people to solve certain problems, seeding an impact market where people invest in projects that hope to win the prize, and the founders and investors profit like a startup exit. This can outsource alignment work to way more people who want to contribute.
There should be a giant makerspace third space dedicated to compute verification. Like imagine Noisebridge but people can toy with these devices. Coefficient should probably sponsor an incubator to teach many many people how to work on this, or skill up current professionals. Maybe Bluedot is on this. I think Actually Trying may look like some people really really going hard on this. That requires people to throw tons of money at this field rapidly. It should get tons of status.
The future is wild. The world is about to be completely different. That's what they say about California: it's a place for sudden change to happen.
I've thought about starting a Project Hail Mary order of people acting to save the world. Or the Order of the Phoenix. A Strategic Command, a War Room. The Room Where It Happens. This could all be incredibly cringe, but also we need heroes working together.
There is a Critical Path. I think coordination runs through much of it. We need a large swarm for Good. Stray reader, please comment DM me if you have ideas, feedback, money or talent and would like to support/develop/join/criticize any of my work. We really need to band together and be force multipliers for each other.
I think the comment is quite well thought through and I think your thinking is worth a top level post. It has a lot of nuance that the top level misses, and many bits and bobs of interesting targets which I'd like to see expanded upon!
Note: this post was written as part of the https://justwrite.club/ challenge, where we have to write 1000 words a week or we have to give away $1000 to Manifund.
Honestly this is a pretty weak version of "actually trying".
Target the chip fabs, might slow this down a bit.
See if you can get anthropic to stop being do gooders and stop the ai race by directly hacking and ending their competition across the planet while they still have alpha. We would have dictators yes, but we (humanity) would live.
Why are we clinging to the rules when there is a gun at our heads and the world is intent on pulling the trigger.
I know lesswrong dosnt typically think like this, but hell, if I want my children to live, ive got to do somthing, even a mean sounding comment on a rationalist website. Lesswrong seems pretty influential in this space, worth a shot.
Viva la revolution!
There are, I believe, objective difficulties to committing to "actually trying" as described in this post.
1. There does not exist a known discrete course of action that has a lower-bounded chance of success. Even for a financially independent, high profile individual, making a sustained personal effort to effectuate the highest negative delta P(doom) is an incredibly difficult proposition. So to outperform a courageous throwing a dart into the dark, one must first go through the task of formulating an actionable plan with realistic objectives. This is a SHARED cost for all those potentially willing to "actually try". So perhaps MORE work should be done regarding this barrier to action.
2. The individual calculus of jumping into action is expensive, and the theoretical P(doom) of inaction is not guaranteed to be 100%. The distinction is subtle but real: why would I make a tremendous personal sacrifice to make X happen, when X might happen anyway regardless of what I do, when I can instead focusing on optimizing my situation conditional on X happening? ~X being total loss alone doesn't change the calculus. I am not sure what can be done with this problem, as it is simply the current game state we find ourselves in.
3. Money, money, and adversity. Bringing about sweeping changes in some aspect of society, almost by definition, is politics. So AI slowdown will be as hard as other major political problems. Past examples tells us that there needs to be sustained organization, funding, and personal sacrifices. Treating it as a political goal is also, as I believe, the most promising framework for item 1 listed above, but doesn't make it any easier in practice. Good news is that this is a concrete, solvable problem. We don't know for certain if alignment can be solved or automated safety research will work, but we do know that a coordinated total stop will be guaranteed to prevent P(doom). So we have a strategy, we have a rough idea of how hard it is, but this is THE industry of our time. Trillions of dollars have already been committed. The entire financial market hinges on the promised productivity gain. A total stop will amount to immense economic pain, so stopping is not free. I currently do not see enough political funding interests to make any dent. Absent funding, political movements often result from adversity. Unsafe, unhealthy, unfair practices can rally people against an adverse condition. So far this "adversity" narrative is primarily restricted to the data center/local community conflict, and I am not seeing a pathway for it to escalate. So in summary, the macro environment for "actually trying" politically needs improvement.
Overall, I share your sentiment. I applaud those who took actions.
While I agree that appealing to the people at the companies will probably work towards enacting a temporary pause, any policy here must still have permanence (not in the way of a permanent pause, but a permanent capacity for the hammer to come down when needed), and for that, you need to primarily appeal to the governments. This seems all the more important considering that we're seeing real movement there, a real willingness by high-level politicians to get acting, even on quite fast a schedule.
I feel that a lot of rationalists have been underestimating the ability and willingness of the political system to handle AI x-risk once the situation starts seemind dire enough (due to a pseudolibertarian belief that governments do nothing anyway? A lack of familiarity with "gradually, then suddenly" dynamics in social movements?) and should probably be updating their priors on this and be willing to keep updating them.
I think the phrasing of some of this is dangerously misleading compared to the thing you are trying to communicate. Maybe consider rephrasing a couple of things?
The key problem is that the heuristic "in dire situations, high-variance strategies are good" is straightforwardly good in most single-agent problems, but very bad in problems with many agents if used naively. Since if many of the agents adopt it without coordinating with the others, things average out, and you end up with a net negative overall outcome. In practice, I have seen many people use a non-nuanced version of this heuristic to justify some pretty dumb actions. (Local example: "Aaargh, things are super bad with AI, and nobody is doing anything about it! Let's organise a 300-person whole-weekend workshop, 2-weeks from now, even if making it happen burns a lot of other people's social capital, and none of the organisers have experience with organising even a 2-hour 30-person event".) Some people also tend to round it up to: "if the default trajectory ends up with an outcome I find unacceptable, anything other than the default trajectory is worth trying". Also, a dishonourable mention goes to the "let's do terrorist stuff" strategies.
To be clear, I strongly endorse the thing that (I think) you are trying to point at. As well as your actions. Just not the specific phrasing of the recommendation, and some of the examples. (In particular, I wouldn't use the examples of single-agent games like Super Smash Bros and Go. I wouldn't use the phrase "play aggressively". And I wouldn't use the example of having lived through this 100 times, and now having a chance to go back one last time. Multiplayer games might work better -- like replacing the goalie by an attacker in ice hockey. Or if you insisted on single-player games, a poker example would be better: if you are behind in a tournament, it might make sense to start playing riskier. But if start going all-in on every hand, you will just lose.) [1]
I imagine that a version of the heuristic that works is: consider high-variance and "aggressive" strategies, but only if the downsides of failure are smaller in scope than the benefits. For example, it is fine to risk your personal well-being if you might shift what happens with some organisation, or to risk your organisation if it might shift what happens in a country (or with a way more important organisation). But don't take actions that have low success chance while burning a lot of common goods. For example, deciding to lobby with your prime minister, if you have no people skills and haven't done any prep, just because you haven't seen anybody else doing it, would be a stupid idea. Pulling an SBF, or advocating for terrorism, would be both immoral and stupid :D. (And again, not saying the post is recommending that. Just that some people might misinterpret it in various ways.)
My favourite game is Dark Souls. When you are fighting a difficult boss, you just messed up, and you are on your last HPs, it is soo tempting to just try smashing the attack button and hope it works out. But if the boss needs 30 more hits, that approach would be doomed to fail -- you will not get lucky that many times in a row. Even 10 hits would be too many. Maybe on 3 hits left, you can consider it. It only took me a couple of hundreds of deaths to overcome that reflex, so let's hope we can do better in real life, eh? ↩︎
But don't take actions that have low success chance while burning a lot of common goods. For example, deciding to lobby with your prime minister, if you have no people skills and haven't done any prep, just because you haven't seen anybody else doing it, would be a stupid idea.
I think this is the wrong line of thinking though. You're treating your own determinations of feasibility like it's some kind of abstract, outside-view of things, and so if you don't think there's a high chance of success, don't do it. But your own beliefs are inputs endogenous to th... (read more)
There’s a familiar feeling for any Super Smash Bros player: your stocks are low and your damage is high, even against a weaker opponent, so now you have to actually play to win. You can’t afford another hit. You’re going for the jugular.
In the past week, Anthropic’s 2x uplift results were all over Twitter, making some people physically ill.
People with long timelines are now realizing that the uplift Anthropic employees get from AI is real, which would have some pretty wild implications for when we’d get recursive self-improvement.
People are starting to feel like it’s the endgame, “the board is set”.
I do agree with everyone else that it’s starting to look like we’re in the endgame.
The companies will keep on accumulating more and more power, through smarter AI models, more compute, and more talent.
But the fight is not yet over:
But what would actually trying look like?
Actually Trying
There are a couple of heuristics for thinking about what “actually trying” means.
Famously, Nick Cammarata tweeted:
My own version goes:
Like, imagine the version of you that just died from the bad worlds we wanted to avoid. And was given the chance to try again.
Maybe it was the first time you died. Or the 100th.
But this time, some oracle comes to you when you wake up and tells you that for some reason, this one is the last chance you’ll ever get.
How would you behave differently?
You’d actually try.
Playing aggressive
Many people know about Move 37, but what people don’t really talk about as much is move 78, the “hand of God” played by Lee Sedol, which AlphaGo estimated a human had a 1 in 10,000 chance of playing.
Lee Sedol had to try to play something that would be aggressive enough, or at least complicated enough, that it would manage to break through AlphaGo’s computation.
He didn’t go for the safe and predictable moves, and instead went for the high-variance aggressive moves.
He actually tried.
What would that look like for stopping the development of self-improving superintelligence?
Stopping The AI Race
People in the AI Safety community disagree about a lot of things, but most people agree that it would be great if AI companies were not racing so aggressively towards building self-improving superintelligence.
Stopping the AI Race is a good intermediary goal, whether you believe that (frontier) AI development should be paused, stopped, or you just want a coordinated slow down.
In practice, stopping would mean that:
Stop The AI Race, the grassroots movement, and the name of the Substack you’re reading, started in September 2025 when Guido Reichstadter posted on X that he was on day 3 of a hunger strike outside Anthropic in San Francisco, with his chalkboard saying “Stop The Race to Artificial General Intelligence”.
This message then spread to Google DeepMind in the UK when Denys Sheremet and I started our own hunger strike, with a chalkboard saying “Google Stop The AI Race”.
Guido’s ask was for Dario to stop whatever Anthropic was doing and switch to doing policy advocacy instead.
The Google DeepMind ask was different: Demis Hassabis should publicly say that he’d pause frontier AI development if the other CEOs also did.
In a way:
That hunger strike outside Google DeepMind ended up lasting 7 days for me and 18 days for Denys, got international press and led to Demis Hassabis answering a journalist who asked the same question we were asking him (would he pause if everyone else also paused), to which he replied “I think so”. More details in the Hunger Strike documentary:
How the strategy has changed since then
Since September 2025, Stop The AI Race became not only something written on a chalkboard outside an AI Company, but more of a grassroots movement in the Bay Area, which then spread to the East Coast, and is now doing an international march for an AI Treaty on November 14 (if you’re in SF, NYC, DC, Boston, Chicago, Philly, Seattle, Paris, Marseille, Montréal, Toronto or Vancouver, you should sign up!).
In more detail:
If you live in one of the cities listed on aitreatymarch.org, going to one of these protests would mean putting yourself out there, showing the world that you care, and our pictures will end up in the news all around the world.
But that’s only a step towards a longer-term vision of actually trying.
What’s the “actually trying” end game of Stop The AI Race
If we were actually trying, we would come back to being outside the AI Companies every day until they stop pursuing recursive self-improvement.
People would actually start doing more hunger strikes, sending more costly signals that feel like something you’d do if you thought there’s a substantial chance we don’t make it.
If we were actually trying, we would have people outside Anthropic’s office at 500 Howard St in SF every day until they stop frontier AI development, outside all of their entrances.
Stop The AI Race on July 20, outside the Anthropic Office
Today, employees of these companies go to their office every day, doing work which could potentially lead to human extinction, while their leadership is filing for a $1T IPO, all of that without receiving any serious pushback from the public.
In the meantime, it only takes 2-4 people to have banners outside of their two main entrances, which would remind every single employee of what they’re doing, and enable some of them (leadership included) to take actions that would help stop this race, which we believe was a key driver of success during the Google DeepMind Hunger Strike (some employees supported us through a thank you letter!), and we know had a lot of impact on OpenAI employees during Occupy OpenAI (from private conversations with employees).
Basically, the goal would be to be Anthropic’s conscience.
In fact, Anthropic co-founder Chris Olah, responding to Pope Leo XIV’s encyclical Magnifica humanitas, wrote that given Anthropic’s incentives to stay commercially viable and at the research frontier, it is “enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.”
So let’s take Chris Olah up on this, and let’s remind Anthropic employees of the extinction-level risks they’re imposing on all of us, by holding banners saying “Stop The AI Race” outside their office.
Email contact@stoptherace.ai, or send a message to mtrazzi.99 on Signal if you’d be interested in actually trying to reduce existential risk from AI. And if you have any friends in SF, NYC, DC, Boston, Chicago, Philly, Seattle, Paris, Marseille, Montréal, Toronto or Vancouver, ask them to sign up for a local protest on aitreatymarch.org, happening this Saturday. Stop The AI Race is also hiring.
Note: in this post I don't address Scott Alexander 's SURA post or Nate's polarization post. For my answer to Nate, see this comment. I'll try to answer Scott in future posts I publish this month for https://justwrite.club/.