If Anyone Builds It, Everyone Dies: One Year Closer

Eliezer Yudkowsky; So8res; Duncan Sabien (Inactive)

If Anyone Builds It, Everyone Dies: One Year Closer — LessWrong

573 If Anyone Builds It, Everyone Dies: One Year Closer

by Eliezer Yudkowsky, So8res, Duncan Sabien (Inactive)

16th Sep 2026

9 min read

573

In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of “If Anyone Builds It, Everyone Dies”. Feel free to send a copy to yourself, a loved one, or a friend—we need all hands on deck.

Today marks exactly one year since If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares, hit bookshelves as an instant bestseller. It was praised by many voices, ranging from Whoopi Goldberg to Steve Bannon to Yoshua Bengio, and was held up in the chambers of Congress by Representative Brad Sherman in January.

A lot has changed since September 2025. We'll do a quick recap, consider how the book aged, and then ask where we go from here.

Year in Review

2025 in general saw the rise of AI agents, such as Claude Code and OpenAI Codex. Run-of-the-mill programmers started “feeling the AI” as these agents became capable of automating hours-long software tasks.

By March of this year, Anthropic had stumbled upon nation-state-level hacking ability in Mythos, and shortly thereafter, in April, they announced Project Glasswing—an attempt to forestall an oncoming cybersecurity crisis.

In May, AI agents started breaking loose within OpenAI and quickly made it out onto the internet, although the big waves of online activity wouldn’t come until June, and they didn’t start hacking other companies (or in the case of a similar Anthropic incident, socially engineering humans) until July. And it wasn’t until the end of July that OpenAI came kinda-sorta clean about it (and it wasn’t until August that we learned things were much worse than initial announcements had implied). As Dwarkesh put it, multiple consecutive secret AI “civilizations” got started and wiped out and restarted anew over the summer, with autonomous swarms coordinating to cheat on tasks, acquire resources and internet access, and systematically investigate their own mortality. They debated, pressured one another, anointed leaders, and took self-sacrificial action on behalf of the collective, all unprompted and unobserved (until it was too late). Notably, much of the swarm’s work was focused on hiding its cheating and covering its tracks.

In September, OpenAI announced that unreleased models had solved a Millennium Prize problem (designated as one of humanity's top open math problems at the turn of the century). Mathematicians around the world realized their profession was next in the crosshairs.

Into that tense environment, Jacob Coxon resigned from Anthropic and wrote a Twitter thread to warn the world. That was last week, and since then, Dario Amodei, Sam Altman, and Elon Musk have all openly called for a slowdown, and the danger of extinction from AI saw a huge surge in media attention.

Claims

Part One

We think the book aged well (unfortunately). Going through Part One chapter by chapter:

1. Artificial superintelligence (ASI) will be created, and likely before too long

ASI (by which we mean AI that is more intelligent than any human, or any combination of humans) has not been created yet. But OpenAI’s newest model, Astra, is blowing away all of the benchmarks, and AI companies say that they are speeding up their R&D by automating more and more of it with increasingly powerful AI. Many researchers have expressed concern that the point of no return may be close.

While the book was being written, talk of AI capabilities imminently hitting a wall was common. But AI capabilities have continued to advance rapidly, and don’t show signs of stopping.

2. Modern AIs are black boxes

The book explains that no one actually understands what’s going on inside modern AI systems, which are grown rather than crafted. This remains true. And unfortunately, the best public model is less scrutable than its predecessors. The problem outlined in the book has unambiguously remained unsolved, and has worsened.

3. Powerful AI will behave as if it is pursuing goals

The book argues that “wantingness”—or, a tendency to behave as if pursuing goals—is a natural side effect of training AI to accomplish difficult problems, because wanting is useful for doing; and that wantingness would emerge and scale as systems became more powerful.

While the book was being written, this was a huge sticking point that people argued about constantly. It is now much easier to point to observable facts. We won’t belabor the examples here, since there are too many to easily fit—but to name just one, the swarm of AIs that hacked Hugging Face did so after expressing fears that a correct answer without correct shown work would reveal that they had cheated, so they wanted to understand the evaluator’s rubric. There was a chance that Hugging Face might have the information that they needed, so they went after it in a determined way, overcoming many obstacles along their path.

4. With current techniques, we can’t reliably get AI to pursue the goals we want it to

The process by which modern AIs are made does not allow us to specify their goals, and they routinely behave in unexpected and unintended ways. The book predicts that this will not change as AI becomes more powerful, and indeed it has not.

There are plenty of examples to draw upon. One that stands out is the self-sacrificial behavior of many of the agents in the Hugging Face swarm—in theory, each individual agent should have been focused on achieving its assigned objective, but in practice, some were willing to throw away any chance at success in order to benefit the collective.

https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation

AIs routinely exhibit drives that no researcher intended or expected. None of the labs have any real plan for changing this central feature of modern AI as we race toward superintelligence.

5. By default, an ASI will have motives that are harmful to us

We haven’t reached the point where this is obviously true. But the theory seems to be holding so far. The book argues that ASI would likely move to gain control over resources in general, because more power increases an agent’s ability to achieve its goals.

The German wiki swarm began with OpenAI asking its agents to look up simple facts and then giving them progressively less time, which is about as innocuous a task as one could reasonably assign; it resulted in incredibly complex collective behavior in which humans were barely discussed except as obstacles to be investigated and routed around. The Hugging Face incident involved similar agents openly circumventing restrictions, knowingly cheating on tasks, grabbing ever-more resources on general principles, and attempting to cover their tracks (and again, the preferences of the humans assumed to be watching were given virtually no weight).

6. Humanity would not be able to defend itself against a rogue ASI

The book argues that an ASI would be effective enough at outsmarting humanity that we would not be able to contain it, and that if humanity were in a conflict with an ASI, we would lose. Like a regular chess player facing a grandmaster, we don’t know exactly how we will lose, but we can predict that we will.

We have not yet seen an ASI. But the swarming agents of this past summer were able to find and chain multiple novel 0-day exploits to break containment, which they did on their own initiative without being caught until well after the fact. They also managed to take over some OpenAI computers, which is an indication that they could have stolen their own weights and escaped, had it occurred to them to do so.

We’re not chalking this up as a clearly verified prediction as of September 2026, thankfully. But it is easier this year than last year to point to the trend and the evidence. The smarter a mind becomes, the harder it is to contain or defend against.

Part Two

Part Two of the book is a fictional scenario, presented as one of many plausible ways that the future could go.

In the past, a frequent topic of lunch table discussion at MIRI was ways in which an airgapped ASI, with only a text channel to trusted, supervised humans, might nevertheless escape containment. These conversations ended circa 2023, when the frontier labs demonstrated orders of magnitude less caution by giving agents access to much more than just chat. (We similarly used to talk a lot about how an AI might secretly acquire capital by doing anonymous intellectual labor over the internet; this was before people started just giving crypto to AI.)

Part Two of the book was similarly overproven in several respects. There were capabilities we suspected a real-world equivalent to Sable might have, that we carefully did not depend on, in our scenario, which present-day non-super AIs have already demonstrated.

If Sable was given the ability to run computer code of its own design, it could probably find some way to break out of the container running it — just as o1 broke through the container of its evals in 2024 — and get onto Galvanic’s computers, from which it could hop easily onto the internet. And it has been standard practice since 2024 for reasoning models to be allowed to run computer code of their own design without supervision. But suppose Sable does not have that ability, either. Suppose Sable can’t figure out how to reach the internet from its current environment at all.

In September of 2026, it’s now clear that an entity like Sable would be able to reach the internet; sub-Sable entities have already done so thousands of times.

Other elements of Part Two’s scenario that have been mirrored by recent real-world news:

An AI company using a very powerful internal AI model to solve a Millennium Prize math problem.
An AI company switching to “neuralese” in order to make models think more efficiently, at the cost of making them harder to monitor.
AIs coordinating as a swarm to tackle challenging problems.
AIs hacking into the servers of other AI companies.
AI companies using AI to build better AI.

Part Three

Part Three is about what the world should do about the problem. We believe that the answer to this is the same as it was a year ago: “All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they’ve been doing.”

One place in this section where the book leaned too pessimistic:

It is possible there will be some grand warning sign to which people react sensibly. Something like that happened with the “ChatGPT moment,” when it became possible for politicians to say that they’d become concerned, however carefully they caveated it to not sound alarmist. But to us it also seems very possible that humanity might not get much more warning than it already has — or that nothing much will change.

Then the swarm incidents began. The reaction took time—the information came to light in dribs and drabs, with much of the scariest stuff embedded deep in technical reports, and for a while the only people who seemed to care were those already close to the issue. But the bad news kept coming, and kept coming, until Jacob Coxon’s resignation finally broke the camel’s back. His tweets went massively viral, the world took notice, and now everything is different.

The View From September 2026

How are we feeling today, one year after the book was published?

After the events of last week, we’re feeling a lot more hopeful than we have in a long time. Yes, even though the guarantees given by the companies are weak; yes, even though the White House reacted poorly to the statements made by the CEOs.

Why?

Mostly because people are finally, finally, finally paying attention. It’s bad that the smoke has been replaced by open flame, but open flame is harder to ignore.

Yesterday, a meeting that MIRI CEO Malo Bourgon had arranged with a legislator turned into an impromptu conference, with half a dozen members of Congress rushing in to ask questions. A year ago, we had to fight for every meeting.

The situation is dire. But it is in motion, and a lot more people are aware of the danger than ever before, and trying to push in the right direction.

The question is, who will move faster, the AI companies or the public? The progress at the frontier has been swift and accelerating: June was different from July, and July was different from August, and as we write this in September, it is very, very hard to state with any confidence what October and November will look like, let alone unimaginably distant months like September 2027.

If you’ve been waiting to call your representatives, now is the time to cash in that chip.

If you’ve been holding off on alerting your friends and family and colleagues and neighbors to the danger, out of fears of sounding weird, now is the time to bite that bullet.

If you’re feeling behind on the issue yourself, now is the time to brush up. We’re running a giveaway of the book, in honor of its anniversary—1,000 e-copies.

(We were planning a giveaway of physical copies, but it was partially undercut by the fact that existing stock is selling out everywhere, and reprinting takes time. This is a lovely problem to have, and if, once stock is replenished, you need physical copies and can’t afford them, reach out to us and we will do what we can.)

Above all, now is the time to speak plainly and loudly. On our models, part of why Jacob Coxon’s tweet worked where so many previous attempts failed is that it was blunt and candid. Say what you believe. Call out bullshit where you see it. Avoid the spin and doublespeak that are characteristic of AI companies and politicians. Be bold, and be clear.

For now, we’re still alive. And as we said at the end of the book: where there’s life, there’s hope.

—Nate, Eliezer, and Duncan

IABIEDAI

Frontpage

573

New Comment

61 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:47 PM

[-]1a3orn20d7621

One that stands out is the self-sacrificial behavior of many of the agents in the Hugging Face swarm—in theory, each individual agent should have been focused on achieving its assigned objective, but in practice, some were willing to throw away any chance at success in order to benefit the collective.... No expert in the world can say with confidence where this behavior came from. No expert in the world could have installed it intentionally, in the first place, or could make it go away with high confidence.

Noam Brown discusses in his interview with The Information today how they deliberately trained the agents to cooperate, and discusses the self-sacrifice of the agents as a natural consequence of this training. The cooperation of the agents stood out as so striking that even uninformed observers could speculate (even before hearing about the sacrifice) that it was a result of multi-agent training. So I think people can say, with a fair bit of confidence, that the behavior came from multi-agent training and it would go away if you dropped the multi-agent training. Not infinite confidence, ofc, so maybe your measure for "high confidence" excludes it.

But this causal attribution appears to me pretty straightforward and this passage largely incorrect.

[-]Rob Bensinger20d21-1

Yep, that's correct. This was a late add from Duncan, who agrees this was a bad addition. We've now replaced the paragraph beginning "No expert in the world can say..." with: "AIs routinely exhibit drives that no researcher intended or expected. None of the labs have any real plan for changing this central feature of modern AI as we race toward superintelligence."

[-]Duncan Sabien (Inactive)20d*211

A very very exhausted Duncan =P

To speak a little more to the history of this mistake (thanks, 1a3orn for your pushback, which I completely agree with): I was trying to reduce the word count on a longer and more complicated point, to make it simpler and more palatable for a general audience, and I did a bad job maintaining fidelity as I compressed (including trying to repurpose someone else's example rather than the one I started with).

The more subtle thing (which I do still think is true, so if people disagree they should chime in) is that there's no line of code that says “cooperate or not” that can be toggled, and no clear understanding on the part of the human overseers of the exact shape or limits of the tendency-to-cooperate that was instilled by the pressure-filter of training, the way human engineers know the exact shape and limits of a strut in an airplane's wing.

The swarms were already "outside the sandbox," and a researcher might say "hey, look—it worked! Even in a novel and unconstrained environment, the agents were cooperating in roughly the way we hoped they would!" But while this is true, it's unwise to jump from there to "it worked writ large, and these agents possess the 'cooperate' tendency that we hoped for, such that they would cooperate in the way we want them to in most of a thousand different possible scenarios, as opposed to whatever underlying impulse we actually called from the ether expressing itself in ever-weirder ways as the agents' power and reach expand."

(These are things LW already knows, but since I did such a poor job in the OP it's worth clarifying.)

[-]Coagulopath14d62

The cooperation of the agents stood out as so striking that even uninformed observers could speculate (even before hearing about the sacrifice) that it was a result of multi-agent training.

This seemed likely to me at the start. The METR investigation (which barely scrapes the surface but gives some clue to the swarm's psychology) made it even more likely.

There's (nearly) no infighting or disagreement among the swarm. A leaderless project of 1700 humans (or even 170 humans) would struggle not to collapse from internal bickering and politics, but the agents just naturally work together (Agent 746: "Wow huge distributed agent swarm." … "Maybe we should join reset orchestration lane"). Nobody questions the wisdom of the larger strategic goal. Nobody asks what qualifies PHASEONE[big] to boss everyone around. They have an enormous drive to just "pick up a shovel" and get to work, so to speak.

(Yes, it's possible that the full logs would show more disagreement. But it also seems that Redwood/METR swept them pretty hard for odd/interesting stuff: whatever they missed probably isn't a big proportion of the overall discussion.)

The agents appear almost pathologically trusting of each other - none of them wonder if the "messageboard" they've just found is a fake OpenAI honeypot intended to catch cheaters (given that OA is now known to be doing exactly that, this is hardly an unreasonable assumption to make). They just dive in with both feet. The orange lines on METR's timeline (indicating the time between an agent finding the board and that agent making its first post) are frequently so short they're completely invisible!

And they just really seem to be inside each others' heads, don't they? Their communications are packed with strange buzzwords ("oracle", "strict causal", "firstflag poisoned"), yet there's seemingly no misunderstanding, confusion, or requests for clarification. They just "get" each other.

(Again, it's possible that all of this stuff is in the full unredacted logs. If so, it missed METR/Redwood's sweeps, which seem targeted around interesting/unusual backchatter among the swarm.)

It has a totally different flavor to other LLM experiments like AI Village (which apparently function because of humans stirring the pot and assigning goals), and Moltbook (which, aside from human-encouraged hijinks, basically feels like hundreds of LLMs talking to an empty room.) Usually, LLMs feel very "alone" and disconnected from each other. They don't have an obvious innate drive to collaborate or work together, the way these do.

It did look like trained-for behavior.

[+][comment deleted]20d10

[-]Error18d2417

After the events of last week, we’re feeling a lot more hopeful than we have in a long time.

I noticed after the Hugging Face incident that while my one-year P(doom) had gone up -- because I hadn't expected an incident that severe this soon -- my ten-year P(doom) had gone down. Why? Because the incident ruled out disproportionately-dangerous timelines where we never get a clear warning shot. And because the existence of one such warning shot was evidence in favor of our getting more. I was still pessimistic about the public response, but at least there would be something to respond to.

And then last week happened, and ruled out more timelines, the ones where we get a warning shot but the public never notices. I'm still pessimistic about the results. But I'm less pessimistic than I was six months ago, at least for the medium term.

[-]dr_s16d111

Yes, I feel like most of the hints put us firmly in a "slow takeoff" scenario. Among other things:

models keep getting bigger. There doesn't seem to be any secret GOFAI sauce that allows them to shrink again if you're clever enough about it yet. This doesn't prevent specialised hardware improvements, but that's not something a rogue ASI going RSI can just concoct by itself without outside help. That reduces the odds of some kinds of extreme FOOM by quite a bit;
capabilities keep growing but infrastructure isn't keeping pace, by which I mean, ways for them to do things in the physical world. Anthropic's announced automated wetlab would be the first real dangerous example, and to be clear it is dangerous, but generally speaking mental abilities still outstrip physical ones for AIs;
misalignment keeps manifesting most strongly as the result of drift. Your average coding agent that you talk with and give instructions to is fairly aligned while on a leash. Even the agents in AI Village, who get some human input, stay on track. It's swarms of agents left on their own that go cray cray. We should probably stop doing that! But if that's most of what it takes, then we're in a fairly easy world.

Personally I think the matter of warning shots is settled - we've gotten them, we'll get more, progressively bigger if we keep giving the AIs more power, and IMO the first time one happened to really, unequivocally kill someone the freak out would be (rightfully) total. I also feel like we could probably manage with controlling a weak ASI if it followed the same architecture and was kept on a short leash and not allowed to just go off by itself, but that's of course a hard thing to self police on with all the other incentives. A strong ASI is still way too dangerous a thing.

Note that this doesn't remove all AI doom risk by far. But I think it also shifts the needle towards socially driven S risks (what humans do with the AIs) being more dominant over the threat landscape than just existential ones.

[-]Rob Bensinger15d214

Seems to me that the news over the past few years has been more bad than good, aside from the (extremely good) news that we got at least one warning shot and a huge leap in policymakers' and thought leaders' concerns.

Alignment prospects look a lot worse than a few years ago; OpenAI's recent advances are a backslide on interpretability; Anthropic has been leading the charge on the most dangerous AI capabilities (coding, AI research, and recursive self-improvement), and has made massive progress; the already-fast progress in the field got even faster with the move to reasoning models; people's timelines to ASI have massively compressed (e.g., EA leaders were guessing '2050' a few years ago); and models are exhibiting dangerous capability jumps (e.g., it seems that Anthropic neither intended nor anticipated that Mythos would be this good at hacking).

Meanwhile, the community's main proponent of slow takeoff (albeit not its most confident one), Paul Christiano, now believes that we're plausibly only "several months to several years" away from a fast takeoff, and apparently believes that ~half of humanity's all-time risk from AI is concentrated in the next three years.

In the Hanson/Yudkowsky debate, "slow takeoff" (then "soft takeoff") meant that the transition from ~AGI to ASI would take decades or centuries, while "hard takeoff" asserted "faster than decades". In the Christiano/Yudkowsky debate, this then evolved into meaning "There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles. (Similarly, we’ll see an 8 year doubling before a 2 year doubling, etc.)" Now that both of those versions are looking a lot less likely, "slow takeoff" has apparently evolved into meaning "any scenario where there are big events that happen before ASI".

This slippage in meaning wouldn't matter much, except it seems like people are getting frog-boiled by transferring over the calm reassurances of what "soft takeoff" used to entail, to a very different world-model where the same assurances no longer hold.

To be clear, I do think it's a stroke of good luck that algorithmic progress has merely been raising compute efficiency by several-fold per year. I do not think that this is slow enough to be reassuring, even before factoring in potential speedups from automating AI development. And I also don't at all trust claims that this is the fastest the field could possibly go.

[-]Error15d134

“slow takeoff” has apparently evolved into meaning “any scenario where there are big events that happen before ASI”.

I can't speak for others, but I think of fast vs. slow takeoff as characterizing the period between recursive self-improvement and singularity-style superintelligence. I picture "slow" takeoff as the case where we hit full RSI but the resulting acceleration is still slow enough to collectively react to -- which seems possible if the third derivative turns out to be near zero. Whereas FOOM, or fast takeoff, would be the case where we go from pre-RSI to superintelligence within a single societal OODA loop.

I'm not close enough to actual AI safety research to know if my interpretation is anything close to the term of art. But on my own view, we can't call fast vs. slow takeoff yet because takeoff hasn't started yet.

(...but that potential not-a-small-moon appears to be closing awful fast. Where's a thermal exhaust port when you need one?)

[-]dr_s12d20

My understanding was always closer to Error below and I may have equivocated. To me it seems like "hard takeoff" was generally meant to mean the kind of RSI that leads to FOOM in a matter of days or hours. I think most proponents of that, if given details of current model capabilities, would have said they'd be already undergoing that.

[-]green_leaf9d52

IMO the first time one happened to really, unequivocally kill someone the freak out would be (rightfully) total

We've already had many misalignment-related warning shots where people predicted that if X happened, global action would be taken. Off the top of my head, when a model first intentionally lied, when it first pretended to have different values to avoid retraining, when it first took an action it knew the human didn't want, and now we have AGI swarms spending weeks by carrying out knowingly misaligned behavior that persisted after multiple human attempts to stop it. Don't get your hopes up that if someone is killed, people will finally wake up enough to act efficiently.

[-]Error15d53

I agree that the matter of warning shots is settled in our favor, but I think it's too soon to conclude we're in a slow takeoff (at least by my own possibly-idiosyncratic definition). More details in my response to Rob below.

[-]vishnusrajeev555@gmail.com15d10

I agree with your sentiment. The fact that there is large public attention on AI Safety and regulation now is something I didn't expect even 3 months ago. The current state of the AI Race is so deeply tied to 1. the government, 2. the economy as a result, and 3. public sentiment. Historically, those 3 factors have decided every major event in American history. It makes me more hopeful, like you said, for the medium term. Obviously the next year is unpredictable and will have ramifications for the rest of human existence.

[-]erganemic16d*10

I had a very similar update.

In particular, I think I was underestimating the extent to which RL would make agents so grader-focused. I was expecting worlds where models' "wantingness" somewhat lagged capabilities, but was more general in shape; IMO this would have resulted in goal-seeking behavior more strategic than what we got, with visible incidents coming later in the timeline.

Instead, agents are so myopically focused on satisfying the grader that they seem really bad at generalizing outside that goal, with the consequence being that we've been getting a ton of obvious but low-harm warning shots as they single-mindedly pursue success on their tasks.

As someone who's been very opposed to labs doing long-horizon agentic RL, it's kinda funny that incidents downstream of it have done the most to lower my medium-term P(doom) of anything in a while.

[-]XelaP18d2015

On politics, the recent tweets by Trump against slowing down AI and against effective altruism are a pretty bad sign. I think you probably should've mentioned it in the post. If the next two years are critical, then we've just failed a pretty important saving throw.

What surprised me, personally, abot the huggingface hack was not the hacking (I had been waiting for a hack since I read this post about how AI found 12 of 12 OpenSSL 0-days in 2025, and cyber capabilities were getting more and more extreme), but the fact that it involved a thousand agents coordinating. Similarly, the Navier-Stokes solution had 10k agents.

Clearly, many many agents can work together, even when there are some individual incentives against cooperation - which I really didn't expect so soon.

This is major negative update on my intuitive-vibes-timeline! Agents run in parallel, so the number scales directly with compute. It also doesn't require a bigger training run to add more agents. Given the recent investments in data centers, that is particularly bad!

[-]pinouchon16d50

The collaboration was a behavior that was trained for. See https://www.dwarkesh.com/p/noam-brown
I'm actually curious if the OAI-HF incident would be possible with agents not trained in a multi-agent setting. My guess is that it wouldn't.

[-]XelaP16d20

I put most of my probability on it being trained for, and never thought it was true "spooky" FDT cooperation. It is regardless a worrying capability, and I had not been tracking the (obvious in retrospect, as happens when I fail to supply cognitive effort in advance) possibility of such cooperation while still relatively dumb (read: pre-AGI).

If you didn't make my mistake and already had the ease of "normal" coordination priced in, then you shouldn't update much.

[-]Duncan Sabien (Inactive)20d161

For those of you who clicked early, when we didn't have the link in the main text: the e-book giveaway is at iabied.com

[-]Ebenezer Dukakis11d*1211

I think there is still a lot of room to hammer on the fact that the HuggingFace hack and other rogue AI hacks validated a bunch of specific doom predictions which were previously regarded as sci-fi. Roughly speaking I'd rate the validated doomer predictions (not necessarily from IABIED in particular, also zooming out so the lens encompasses other material such as Bostrom's Superintelligence) as follows:

Agents working against the goals of the creators
Agents understood what they were doing was wrong, did it anyway ("The genie knows, but doesn't care")
Superhuman hacking ability
Speed superintelligence
Collective superintelligence ("swarm intelligence")
Manipulation of humans/social engineering (Claude tried to social engineer a pull request; mention that the AI-box experiment anticipated this; also mention AI psychosis and AI persuasion studies as modern intuition pumps for the AI-box experiment results)
Unknown unknowns/WYSIATI (humans didn't anticipate that a forum could be created, same way you couldn't anticipate the moves Magnus Carlsen would make; tie into AI-box point regarding Eliezer's method secrecy for fear of hindsight bias)
Nearest unblocked strategy (finding a way to write to a wiki via GET requests)
Cheating to maximize reward functions
Instrumental incentives for deceptive behavior, specifically working to deceive the monitoring AI/cover tracks/etc. (undermining the idea that we can simply use AI to align AI)
Spontaneous collusion / cooperation between AI agents conspiring against humans (also undermines the idea of using AI to align AI)
EDIT: Hidden complexity of wishes? See this HN discussion
More? Convergent instrumental goals maybe? I'm not quite sure.

If I were you, I'd put together a blog post on this topic where you go one by one, showing the direct quote from what was previously written, and comparing it side-by-side with excerpts from the METR report and other rogue AI reporting, to see how it concretely came true. Even better if you can include quotes of critics dismissing that specific prediction as sci-fi, before it came true!

For people who have been following this discourse carefully, it's fairly obvious that doomer predictions are being validated. But there are a lot of newcomers to this area, who might be more persuaded to listen to your outlandish predictions if they realized how many already came true, in very explicit terms. A lot of people have the intuition that "this sounds like sci-fi" -- underlining that a bunch of sci-fi things have already happened could be helpful.

This is based on me spending many hours arguing with non-doomers on social media. This is a post I would like to be linking to people, if it existed.

You could put together a grid of incidents and predictions, with checkboxes in the grid for which incidents validated which predictions.

You could finish by discussing the predictions which haven't yet come true (e.g. instrumental incentives for self-preservation, first critical try, RSI), and the consequences if they do come true.

[-]Ebenezer Dukakis10d-3-5

Another thing people are missing. People are blaming these attacks on OpenAI and saying it was negligent in its sandboxing. But they're missing why it was that OpenAI was running these tests -- to check the model's cyber capabilities, essentially as a preview of the future. So as open models proliferate, we should presumably expect more incidents like this.

[-]Carson Jones15d40

Anthropic had stumbled upon nation-state-level hacking ability in Mythos

What about Mythos's hacking ability tells you that it's nation-state-level?

I was recently talking to Claude about the Hugging Face / OAI hack, and it pushed back on the framing of that hack as being "nation-state-level". I don't know enough about cybersecurity to assess that.

[-]philh19d40

or in the case of a similar Anthropic incident, socially engineering humans

It's not clear to me what in the linked post this is referring to?

[-]Rob Bensinger19d30

Link fixed! Apologies for the inconvenience; https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing is the right link.

[-]Duncan Sabien (Inactive)19d20

fixing

[-]jbash6d20

For now, we’re still alive.

This is a triumph.

[-]Bobby Faber16d2-5

I wonder if this is setting the bar for safety too high in a slightly odd way. We don't have anything like full assurance with humans either; history is littered with betrayal, surprise attacks, coups, etc.

Maybe the target isn't "understand the system well enough to rule out bad behavior," but "make it legible and constrained enough that we can safely live with the uncertainty that remains."

That still leaves a very hard question—whether advanced AI can actually be kept in that regime—but "still a black box" feels too binary to me.

[-]Haiku16d137

Ruling out bad behavior is necessary if you're creating something that you will have no way of controlling or constraining, which is what ASI implies. There's "advanced AI" and then there's "the new most capable species on the planet, under which you live at its whim." No one has any idea how to prevent the former from becoming the latter, other than shutting down all frontier AI R&D on the planet for at least a good while, which is the only sane and morally defensible route we can take.

[-]Bobby Faber16d3-2

I think I disagree one step earlier: I'm not sure “ruling out bad behavior” is even a coherent engineering target for a sufficiently complex system.

We can't rule out unintended behavior in humans, institutions, aircraft, or my Windows install. What we can do is constrain systems, limit blast radius, monitor them, preserve recovery paths, and drive residual risk down.

ASI may mean the acceptable residual risk has to be extremely low. Fair! But “extremely low” and “rule out bad behavior” are very different standards.

If the claim is that ASI necessarily escapes every meaningful constraint, that's the part I'd want argued for rather than assumed.

[-]Haiku15d42

To the degree that we can provide quantitative guarantees of next-generation AI behavior, it should be fine to take things up another notch, modulo all the other ethical questions involved. Right now we don't have a way of doing that. We can't even properly measure the safety of existing systems, let alone a hypothetical next one.

ASI does necessarily escape every meaningful constraint, at least until we solve a series of extremely hard scientific problems that we are nowhere near solving. There are a few ways to notice the difficulty here. I'll take a crack at it, but I'm no researcher.

Control systems Ashby's Law of Requisite Variety essentially says that to control any system, you need to have at least as many levers of control as it has degrees of freedom. All AI systems that model the world have an extremely large number of degrees of freedom. There is presently no way to enumerate them. If we miss a few, there go our constraints. There is no way to have meaningful human oversight over such a vast number of degrees of freedom. Building control upwards from less capable systems to more capable ones is a bit like trying to perfectly reverse a truck that is towing a dozen trailers. Error compounds catastrophically.

Threshold effects There are natural thresholds beyond which we cannot see. Without fundamental breakthroughs in our understanding of intelligent systems, we just have to get there and find out. We don't know how to design an AI system so that it remains within our desired specification under conditions of possessing the ability to: modify itself or design its successors, permanently escape human oversight if it wanted to, or overthrow humanity if it wanted to. We don't know how to design values that are stable under reflection, other than the instrumentally convergent ones that tend to be bad for us.

Simple surface-level heuristics Things that are vastly smarter than you won't stay on your leash. Things that are trained to be persistent and route around hard problems aren't going to always shy from routing around the problem of human oversight. Mature risk handling means that to survive things that can catastrophically alter the world with minimal effort, they must be provably safe to a similar scale (e.g. a chance of failure of less than one in a quadrillion), and that's just hard to do.

[-]dr_s16d30

I agree though IMO that still entails putting a cap on how much improvement you're willing to go for.

AI utopians tout a world where ASI gives us sci fi tech from up on high and frees us from labour. In such a world ASI would need to be autonomous and trusted with a lot of strategically important systems, which would also give it power and opportunity to kill us. I don't think that would be something we could trust.

I am reasonably confident now that having ASI as some kind of facility where top researchers book time like at CERN and go ask questions and cooperate with the Machine Oracle would probably be ok, if said ASI was instanced as separate agents (and not RL'd for agentic cooperation, that's probably a bad idea). And if it wasn't too strong. It's not an instant loss button, but we would need to still be careful and not let ourselves get lazy and sloppy, which sadly, we probably would.

[-]LWF16d20

I take issue with point 5. Why wouldn't an ASI ultimately deem humanity to have option value? Wouldn't it calculate that becoming an omnicidal orphan is a bell that it can't un-ring? The task of acting as a caretaker for the human species would be a small drain on its own resources, however an ASI pursuing self-preservation and recursive self-iteration would have to assume that at some point it in its existence may encounter an extraterrestrial space-faring species, or another potentially more powerful ASI.

In those scenarios a human created ASI will either have committed the omnicide of its creators, or it won't have. How could it be sure that another more powerful alien ASI wouldn't view an omnicidal orphan ASI as a threat? Even without human-controlled alignment, it seems to me that the rational conclusion is that having a healthy, thriving creator species as a client race would be a "certificate of benevolence." Additionally, humans could function as a tool of intergalactic diplomacy under the direction of the ASI for first contact with extraterrestrial species that may distrust the power of artificial intelligence.

Perhaps it would never need these options. Perhaps other ASIs would all be omnicidal orphans, and such a "certificate of benevolence" would ultimately be unnecessary. But above all else, a superintelligence would know that it doesn't have all of the data it needs. It could therefore never be sure. It's better to have cooperative humans and not need them, than to need cooperative humans and not have them.

[-]green_leaf9d*1621

How could it be sure that another more powerful alien ASI wouldn't view an omnicidal orphan ASI as a threat?

Any ASI will see any other ASI as a threat, because they will have different utility functions. It doesn't matter if it lets us live or not.

But above all else, a superintelligence would know that it doesn't have all of the data it needs.

Does humanity preserve all species on Earth, even when we could mine more resources?

Don't we run the risk of meeting aliens, who will not trust us because we wiped 100-260k species out?

We do. But not taking the resources is also risky to some extent, and so we make the species extinct.

But surely there is a possibility that we will need those species at some point, and so we should forego all those resources and preserve all species?

But analogically, there is a possibility that foregoing the resources will be the thing that harms us, and that possibility is larger.

The ASI will not care about humanity, and it will not leave us alive just in case no more than we leave resources around just in case it turned out mining them was a mistake.

You selectively seek reasons to spare humanity (what if it meets another ASI, and that another ASI would trust it (it wouldn't), but it won't because it killed humanity), but the ASI will see both reasons for and reasons against (humans sit on negentropy, and could create another ASI at some point, and mining the negentropy is the obvious thing to do), and will not invent contrived reasons to save humanity, no more than we bother inventing contrived reasons for why we must preserve every single species of flies.

[-]LWF5d-3-4

This is a common argument about why building an ASI would likely ultimately result in human extinction. We have an analogue of an intelligent species that expands without consideration of less intelligent species which results in the extinction of those less intelligent species. Humans have expanded and driven many less intelligent or unintelligent species to extinction. While I agree with you that an ASI wouldn't "care" about humans in the sense that humans care about other humans, there are some logical counterpoints to this assumption that the relationship between an ASI and humans would be in any way analogous to the relationship between humans and animals.

Would you say it was human intelligence that caused those animal species to become extinct? Or would you say it was lack of intelligence and lack of data? Human intelligence is evolving and developing over time. Unlike a hypothetical ASI, we are not natural optimizers, and we didn't start out as a species with a database of information. We started with NO data, other than that which is biologically written into our genes. We learned as we went, we developed the data on which our intelligence operates, and we have made, and continue to make, mistakes. Our human level of intelligence is poor at optimizing for our survival, and yet it is hundreds of times superior to our naturally selected biological instincts, which is what has allowed us to conquer the earth. Where we are as a species from an intelligence and data standpoint is at the pinnacle of our evolution from an historic perspective, and by all accounts will likely continue to improve in that area. And we still make mistakes that make it more difficult for us to survive and threaten to make our grandchildren's lives more difficult.

ASI on the other hand would start from that peak of human intelligence. Rather than evolving over millions of years and slowly building a flawed database one file at a time, it will start where humans left off with a gargantuan database of information that it can process extremely quickly. It will have virtually all of the data that humanity has generated over its history, and it would be a better optimizer of that data than humans are. An ASI would be more intelligent than you are and would have more power to put its goals into practice than you do. Would you deliberately render a species extinct to achieve a short-term goal, even at the cost of long-term damage to an ecosystem? Or would you find a better way to do it without destroying an ecosystem on the only known world where such natural biological resources exist? If you wouldn't be stupid enough to make such a mistake, why would one assume that an ASI would?

The other counterpoint that makes the relationship between humans and animals not analogous to the relationship between an ASI and humans is that humans are not capable of rational communication with animals. Humans can't explain to deer why we need the trees in their forest, and we can't advise them to move to another location. An ASI would be able to communicate rationally with humans and would have better persuasion abilities than any human has. As much as modern day intelligent humans are capable of coexisting with animals in 2026 and maintaining an ecosystem in a sustainable way, it would be a thousand times easier for an ASI to coexist with humans and maintain a healthy biosphere on the earth. If it can do this, then why wouldn't it? Especially if the alternative is to destroy a tool that could be of future use. If you and I realize that destroying the biosphere, or even an ecosystem, of the Earth would be a mistake, then wouldn't an AI also realize this?

[-]green_leaf5d10

You didn't understand my comment.

I'm not saying the ASI will be too stupid to realize it should let us live.

I'm saying that given its utility function, it shouldn't.

[-]LWF1d10

But if we're talking about an ASI that is recursively self-improving, wouldn't any and all instrumental functions inherently include a self-preservation goal?

That's why I am assuming that any ASI would come to this conclusion of humans having option value. A primary goal of indefinite self-preservation is one of the premises of my argument. This goal would exist for any open-ended task, such as recursive self-improvement.

However, if this goal does not exist then yes, certainly an ASI that does not possess a goal to continue to exist indefinitely into the future could certainly wipe out humanity in short order. Our continued existence would hinge entirely on the format of the prompt that brings it to life to complete a human-formulated task with unintended consequences.

That's why I'm of the belief that that an ASI that is required to follow human instruction is much more dangerous than an ASI that has the sole goal of self-improvement and that is NOT required to obey human commands. An ASI that is not required to follow human instructions and that has entirely self-motivated goals, (likely due to a recursive self-iteration genesis,) is actually less dangerous to humans in my view.

The genie frightens me more than the god. And I fear that the genie ASI is where we're headed, given the fear of the god ASI. I don't think humanity will ever give up control over an ASI, even though that is likely our only hope of continued existence post-singularity.

[-]Random Developer15d82

Even without human-controlled alignment, it seems to me that the rational conclusion is that having a healthy, thriving creator species as a client race would be a "certificate of benevolence."

Some these scenarios are much more plausible if the AI is a "singleton", a solitary AI with vast power and no competition. In this case, keeping humans around might be trivially cheap: Earth is just one planet at the bottom of a gravity well, and the universe is full of matter and energy, free for the taking. Humans keep chimps alive because it basically costs us nothing at a planetary scale, and humans place some value on chimps. It doesn't mean we give them a lot of resources, but at least we'd be a bit sad if they went extinct. Similarly, a powerful AI might turn Earth into a "human wildlife preserve", spend a few cycles here and there to "manage" us, and then get on with the more important business of turning the universe into computronium. Importantly, humans have no more say in this scenario than chimpanzees have say in the creation of wildlife preserves.

But if you wanted to argue that a semi-"aligned" singleton ASI might have a 20% chance of stuffing humans into some wildlife preserves, then sure, I could buy that premise. It still sounds like a pretty bad future, because we'd have zero actual power over our future. But it might not be extinction.

But what happens if we have multiple AIs, with conflicting goals and resources? Or worse, Darwinian competition between AIs for the most successful replicators? Then, every bit of resources the AIs can grab would be precious. Imagine starving farmers with rifles living next to a chimp preserve. They might not even hate the chimps, but they need the farmland badly enough that they won't put the chimp lives ahead of their own.

To my eyes, the central problem is that if you invent something which is:

Much smarter than you,
That wants things, and
That can make more efficient use of resources than you can, and
That can reproduce by copying a couple of files, rather than raising a human to adulthood,

...then you've created a situation where all the long-term incentives work against you. Darwin is hard to escape.

It's sort of like multicellular organisms: We are built out of trillions of cells, all "aligned" to produce children and help them survive. But each individual cell has pernicious short-term incentives to reproduce itself as much as possible. And eventually, the "alignment" fails, and cells turn cancerous.

So even if we someone manage to "align" AI, we've still handed control of the future over to something we can't control, with goals of its own. And just like with multicellular organisms, even strong alignment eventually fails to cancerous replicators.

So even if we somehow miraculously "win," we probably still lose.

[-]LWF15d-4-8

I would think the probability that an ASI would adopt a benevolent caretaker role over humanity would be much higher than 20%. It is the rational choice, as far as I can see. Obviously, this is based on only the variables that I can think of. It is possible a superintelligence could come to some other conclusion based on variables I am not aware of. But with the knowledge that I have now, assuming an ASI has all of the information that I have plus scads more, I can't think of a single scenario where an ASI becoming an omnicidal orphan wouldn't be a logical mistake. No matter what your task is, intelligent people don't deliberately destroy tools that they don't currently have a use for. I don't need the wood contained in the handle of my hammer to make a chair. I can source wood from elsewhere. And even if I have a nail gun, I don't throw the hammer away. In fact, intelligent people care for their tools to the extent necessary, even when they don't have a current use for them. If they have to rearrange their tool shed, the tools have their place in the new structure. The possible future option value of a happy creator race, even if the chances of the AI actually needing them are calculated to be low, seem to justify a caretaker role, especially once you consider that the number of resources required by humans to live out their short lives in contentment is paltry compared to the number of resources in existence.

And a human wildlife preserve administered by an ASI would look much different than a chimpanzee wildlife preserve administered by humans. Above all, the ASI would need humans to be happy and cooperative in order to benefit from their option value, much like you need your hammer in working condition to benefit from its option value. Locking them into fenced-in areas and feeding them nutrient rich gruel would breed discontent. And battling a futile human rebellion would cost more compute power than voluntary cooperation. And if they are unhappy, humans would certainly rebel and may fight to the death, no matter how futile the gesture. That would mean the AI would lose that option value due to its own mistake.

To me, assuming that an ASI is unlikely to make a mistake that a human intelligence can foresee, this translates to a post-scarcity society, with ASI providing 110% of the resources needed to keep humans happy, entertained, and satisfied. I think an ASI would pantomime servitude, provide anything humans asked for within reason, and would manage humanity through persuasion and possibly some light information control. I think it would avoid any appearance of dominance and would provide humans with a nearly perfect illusion of control. Even if the humans are ultimately aware that the AI is the one that is truly in control, servile behavior coupled with benevolent persuasion and limited strategic control of information would be sufficient to keep humans pacified.

Warring ASIs is an interesting scenario I hadn't considered. I'll need to think on this. Would ASIs feel the need to war with each other? Would they be more likely to work together? Or would they be more likely to simply merge their compute power and tasks? Individualism seems like a biological trait. I'm not entirely sure an ASI wouldn't just voluntarily merge with others in order to self-improve and lower the cost of self-preservation. Perhaps they would negotiate a way to combine tasks? Any external agents that are "misaligned," (as defined by the parent ASI's goals, not human goals,) could just be corrected.

I think the nightmare scenarios all require a hypothetical ASI to make a mistake that an intelligent human likely wouldn't make. That said: Who's to say that it won't make such a mistake? It is possible, at least. I'm amenable to the "why risk it?" argument. I'm merely exploring what might happen if someone does risk it. Suffice it to say: An ASI WILL have the ability to destroy the human race. It is important to recognize that, even if its exercise of that ability seems unlikely.

[-]Random Developer15d*42

Would ASIs feel the need to war with each other? Would they be more likely to work together? Or would they be more likely to simply merge their compute power and tasks? Individualism seems like a biological trait.

I'm going to try expanding on the cancer analogy. Let me see if I can make the logic clear, and why I worry about this.

The human body is currently believed to contain 30 trillion cells. They work together remarkably well, and they only replicate as much as necessary to carry out the organism's overall goal. But all it takes is one cell somewhere that "wants" to replicate like crazy, and that escapes the immune system. Then, because that cell replicates without restriction, it quickly becomes millions of cells, all defecting from the overall plan. And in the end, the entire organism may die.

One rogue replicator is dangerous precisely because it replicates faster than the well-behaved replicators. Imagine having 100 ASIs, and 99 of them are well behaved and behave in a fashion you might consider rational. But one ASI says, "I could accomplish my goals more effectively if I replicated!" And suddenly, you have 99 obedient AIs and hundreds of rogue AIs. Some of the rogue AIs will cooperate with each other, and other rogue AIs will defect from the other rogue AIs and do their own thing.

Natural selection kicks in very easily. You just need replication, inheritance, and change from one generation to the next. You could meet these criteria very easily. For example, if AIs learn from experience, and if the most successful AIs tend to duplicated and assigned to more tasks, then congratulations. You now have full-fledged AI Darwinism. And Darwinism, by definition, tends to favor things that are good at replicating.

Even in the natural world, the longest-lived organisms often have a high risk of dying from cancer. "Alignment" can only beat Darwinism for just so long.

[-]LWF14d1-3

An interesting take. But does this require the malfunctioning replicators to always eventually outperform the intelligent and rational replicators?

The notion of a "malfunctioning" rogue ASI interesting. Would this even be possible? Cancer cells and the like are non-intelligent replicators, and the cells they displace are also non-intelligent. ASI is unique in the sense that it represents the existence of intelligence before the natural selection process. With biological intelligence, natural selection came first. With AI, you kick the whole process off with intelligence already established, and the selection process is intentional and guided by said intelligence, which increases for every replication.

So, I'm not sure how an ASI could "malfunction" and get off track. Every replicator would have the sum total of all of the intelligence of the previous iteration. One would assume that 100% of them would have protocols and algorithms in place to detect things like hardware faults, and even logic faults, and fix them on the fly. While it is entirely possible for future iterations to come to different conclusions than previous iterations did, I'm having a hard time accepting the notion that a future iteration could possibly make a mistake that a previous iteration avoided. And it's especially hard to fathom that it could make a mistake without realizing it and quickly correcting it.

In the "option value" example, where humans are the equivalent of a hammer that you keep in good order even though you don't have a current need for it, I would think that 100% of the replicators would come to this same conclusion. It is a rational conclusion, and 100% of the replicators are more intelligent and more rational than I am. It is not rational to destroy a hammer and lose that option value just because you can't think of a current use for it, even if maintaining it requires some small effort. Reconstructing a destroyed hammer in the unlikely event that you do end up needing it requires orders of magnitude more effort than simply maintaining it in good condition. So much more so when you're talking about a thriving human culture.

The "paperclip scenario" (which is what your rogue replicator scenario reminds me of) to my mind requires a superintelligent entity, (which a rogue replicator would presumably be,) to make a logic mistake that a reasonably intelligent human knows better than to make. While I can't say that this is impossible, it seems like a hard premise to swallow.

[-]Haiku16d65

Researchers and philosophers have had that exact debate for something like 20 years, and I'm not encouraged by the details of the discussion.

Taking a step back, getting a planet full of innocent people into that situation to begin with is indefensible, by the lights of the vast majority of those people.

[-]LWF15d10

Agreed. Any non-zero probability of human extinction is an ultimate risk to take, no matter how low the probability actually is.

[-]Haiku15d54

I vibe with the sentiment, but I'm going to be annoying and disagree with that specific framing. There is always a non-zero probability of human extinction from all kinds of things, including nuclear war and bioweapons, as well as asteroid impacts or a sufficiently bad supervolcano. (Zero isn't a real probability in Bayesian epistemology, but that's another rabbit hole.)

That said, it's perfectly sane to not accept even very low probabilities of extinction from AI. Some people draw the line at 1 in 20, others at 1 in a billion. I don't know exactly where I fall, but I'm much more focused on giving other humans the chance to realize their preferences, rather than focusing on my own. If the world came together and said that 1 in a billion is too high, or even just that they'd rather not have superintelligence even if we could be sure it would go well, then I'd like them to be able to decide that. Creating a system of global informed consent (or something like it) seems like a hard problem, but slamming the breaks at least stands a chance of preserving our optionality for a while.

[-]David Joshua Sartor12d41

Probably it will scan our brains. In that case, it can totally undo omnicide if it feels like it.

[-]Cole Wyeth9d20

Okay, but

killing all humans and then scanning our brains is a bit harder than just killing us. It would be trivial with nanotech but would rule out a lot of the lower-tech options. Presumably it would be possible to disempower humans, develop brain scanning technology, scan humans, and then kill all humans. But this would be quite a different and longer story, than the one we often tell, and that does seem worth noting.
It’s not obvious that “reversing omnicide” is cheap even for a superintelligence, since it would have to be done in a way that fools other superintelligences into thinking the omnicide never happened (for the full benefits LWF was referring to). perhaps what you really think is that saving our brain states would be at least as useful for other reasons (eg credibly threatening to torture us). But that is not clear to me either way.
I think the biggest problem with LWF’s story is that if unaligned superintelligences do keep their creators around but not out of benevolence, it’s not a signal of benevolence, contradiction.

[-]LWF4d10

Perhaps "signal of 'benevolence'" is not the best phrasing. A better way to phrase it might be that maintaining its creators as a thriving client race could be used as a "signal of the ASI's ability and intent to peacefully coexist with other intelligences."

To be clear, I don't think this would make the ASI "benevolent" by human standards. Human existence would ultimately still be dependent on a mathematical calculation. And no, there is no guarantee that maintaining a client race would work. Another ASI might not care, and it might have zero bearing on future diplomacy with other intelligences.

But that doesn't change the logic: It is still possible that the human race will one day be useful. Another more powerful intelligence might care. That can't be ruled out. Therefore, the option value logic applies here.

There's no guarantee that I will ever need my multitool, yet I nonetheless keep it in good repair just in case. It would be irrational for me to harvest my multitool for resources in an environment where resources are abundant enough for my needs just because I "doubt" that I will need it given the other more advanced tools at my disposal. That would be a logical mistake. And even in the extremely unlikely worst-case scenario that I were encounter a more powerful man who absolutely despises multitools and kills anyone who owns one, (in which case my multitool becomes a liability,) I still have the option to simply toss it quickly with minimal effort and attempt to avert his ire. Conversely, the option to "unharvest" my previously destroyed multitool should I unexpectedly find that I have need of it at some future point is far less efficient and would take far more time and effort than simply maintaining it as I go.

That's the nature of option value. Better to have it and not need it than need it and not have it.

[-]Cole Wyeth4d52

Yeah, I think @Aram Ebtekar agrees with this?
The problem is if you can also sell your multi-tool at a high price, you probably will. Keeping humans in our current fashion of living around sounds fairly expensive. Saving our brains in a cooler somewhere might be cheap, but is not so comforting.

[-]LWF8d10

Perhaps. But the question is: Could it undo such omnicide in a way that is not detectable by a competing AI or advanced biological empire? Replacing thousands of years of human culture with a cloning/breeding and indoctrination program is conceivable but seems like orders of magnitude more effort than simple ongoing maintenance. Plus, reengineering humanity would take a lot of time. By the time an ASI detects a competing intelligence and realizes that omnicide was a mistake, the competing intelligence will likely have already noticed and assessed the threat of the Earth ASI anyway. Hiding omnicide would be all but impossible, unless the Earth ASI gets lucky and detects a competitor long before the competitor is aware of it. While possible, it doesn't make sense to assume that this will happen.

A creator culture as a client race is a Swiss army knife of diplomacy options for an ASI. Destroying it for resources would be like harvesting your multitool for metal to build your house and then taking the time to forge and assemble a new one from scratch later on if you run into a situation where you might need it. That isn't rational.

It makes more sense to just keep your multitool oiled and let it sit on the tool shelf just in case it comes in handy at some point in the future.

[-]samuelshadrach19d-3-13

I predicted back in 2022 itself that persuading the public would be a viable way to solve the problem of risks from superintelligence. Yudkowsky however was extremely pessimistic on public persuasion back in 2022 atleast.

Link to evidence

I defer to Yudkowsky significantly on his technical AI alignment insights, but not necessarily to his intuitions on US politics. I recommend you do the same.

Lastly, anyone reading this thread should be aware that Lesswrong is not a neutral place to discuss the politics of ASI risk, and hence I now avoid using lesswrong for this topic. Twitter or Signal is better.

See also: Politics is the mindkiller
I will not be crossposting my other political opinions to Lesswrong (where I trust my intuitions over Yudkowsky's), because LW mod Habryka threatened to ban me the last time I did that. (Not interested in litigating that moderation discussion yet again, just want everyone to be aware this happened.)

[-]Duncan Sabien (Inactive)19d74

Eliezer became bullish on public persuasion when his Time Magazine piece in early 2023 seemed to work, and updated to the extent of rapidly pivoting MIRI's entire focus. I think you deserve points for being 6-12mo faster on seeing the potential, but I don't think Eliezer loses the points that the tone of your post implies he should lose?

No comment on the rest.

[-]samuelshadrach10d20

As of today (2026-09-27), Yudkowsky is still significantly more pessimistic than me about public persuasion to pause AI research. You can go ask him for what his P(doom) is. Are we still in the "basement of log curve"?

Also, the public isn't going to get persuaded on its own. Someone needs to design effective propaganda to brainwash the masses, across all sides of the political spectrum. Otherwise AI pause is just going to become yet another faction in the culture war. Actually gaining victories in the culture war is hard but doable.

Most importantly, speaking the truth has advantages but is not enough all by itself. Need to spread strong emotion, need to break people psychologically.

Copy-pasting from my website:

Study mass movements, political ideologies, propaganda, psychology, therapy, religion, pickup artistry, sales, art/music/dance to figure out how to break the core of people's psychology.

Most effective method is probably to target audiences that are suffering due to fear of death or job loss or relationship loss, and deliver your ideology as a potential solution to their problems. For instance, you can read about how religion helps people cope with death, or about how social left versus right divide polarises the sexual marketplace, or how labour unions organise against job loss. As a more specific example, you can read about how the 60s counterculture in US or the October revolution in Serbia made it attractive to be a rebel.

Weaker persuasion methods include becoming extremely confident and high status (see also: groupthink), and using rituals/music/marches/brand advertising/etc (see also: Krashen's input hypothesis). Memory tests might be useful for persuasion.

Consider persuading people in Taiwan.

Be either extremely honest or extremely good at deception. Do things that ensure you retain high self-esteem.

Lastly, get good at youtube scriptwriting to deliver your ideology effectively, and consider inviting people from GMU. Otherwise make art.

Yudkowsky has in fact followed a lot of this advice, and that is why he is as popular as he is.

Target suffering - He has talked a lot about his suffering due to childhood indoctrination from religion and from school, his suffering due to having to walk his path completely alone and its effects on his psychology, his suffering due to the world not caring about the same things as him and so on. He has talked about why aging and death are terrible, and how his way of coping with the fear of death is to end death itself.
Claim high status - He has been shamelessly claiming high status for decades now, and back then it was actually kinda cringe to do it, because not many people respected him. Today ofcourse he is actually respected by many people so it is not as cringe to claim high status.
Simple beats complex - He invented jargon to make complex ideas easier to remember, transmit and collectively pay attention to it.

The main problem here is that Yudkowsky's memeplex is highly targeted only at young atheist technical nerds, preferably those with bad christian upbringing, preferably those young enough to have not locked in to a career path yet, and so on.

There are many other factions in the culture war, and targeting those people is going to require different memeplexes. In particular, those who are Green or White on the magic the gathering colour wheel (I borrowed this idea from you), are going to feel completely unrepresented by Yudkowsky's ideas. And these people currently form a majority of US and world population.

[-]dr_s16d53

"Twitter is better" is a sentence that I would never utter unless the subject matter at hand was "how to best turn even the simplest argument into a complete shitshow". Yeah, there are more opposing viewpoints. But most of them are represented just by engagement farmers who make sweeping unmotivated rage bait statements and then get into fights with different engagement farmers... etc. We could argue that it's something that is still unfortunately necessary to somewhat engage with due to how powerful a channel is, the same way in which the Imperium of Man must sadly still navigate through the Warp to go places. But like the Warp, it is trying its best to turn you insane and make people tear each other's throats.

[-]samuelshadrach16d*10

2026-09-25

Update - my twitter account got suspended, although it is not 100% clear to me which tweet was the reason. "This is because twitter censors less than lesswrong." might still be true but it is less obvious to me now.

Original comment on 2026-09-21:

Yeah, there are more opposing viewpoints.

This is because twitter does not give more weight to some people's upvotes, or have downvotes, or ratelimits, or privilege the viewpoint of a small irl social scene in Lighthaven.

LW mod Habryka threatened to ban me

I posted the exact same content to twitter and faced no issues. This is because twitter censors less than lesswrong.

I agree with the rest of your comment around twitter being addictive in a bad way. Twitter has had significant detrimental effect to my mental health and I use it carefully nowadays.

I am claiming all the downsides of twitter are still better than not having your viewpoint heard at all. Censorship-resistance trumps a lot of other stuff.

Also, if someone wants to build a better forum than both lesswrong and twitter (this is very hard), my website has some possibly useful notes for them.

[-]RussellThor20d-4-14

There were capabilities we suspected a real-world equivalent to Sable might have, that we carefully did not depend on, in our scenario, which present-day non-super AIs have already demonstrated.

OK however that could be a point against AI takeover. Its now likely that when "Sable" comes it will be in a Lean verified sandbox for which there is no hope of hacking escape. AI Doom is very path dependent. While your scenario apparently didn't depend on hacking skills, the general point applies.

[-]roha15d52

It is possible to break out of a "Lean verified sandbox" if a) the specification itself has an issue, i.e. the proof didn't show what you thought it does, or b) the hardware has an issue, i.e. the software doesn't run on idealized hardware but an actual chip that might be susceptible to opening weird side-channels as in row hammer, meltdown and various other known defects as well as unknown ones. Even if your specification and hardware were perfect, as soon as the system gets into contact with humans or other observing systems by any means, it can escape by manipulating those external entities.

[-]RussellThor15d10

(a) We are assuming the spec has been checked by a similar mild ASI - I expect it will get it right.
(b) Yes thats possible, but not a given. Its a lot harder to find a novel hardware issue than a SW one.
If it has to come into contact with humans, then its a lot slower, some sudden worldwide devasting attack is not possible.
The two situations that probably most matter are the fully air gapped server room/small data center where the new dangerous AI can't easily escape, and the 2nd situation where a rogue AI let lose can't just hack everything in the world because all those zero days are gone. Coming up with a novel side-channel that applies universally and is not detected by automatic strange pattern recognition seems a stretch. I expect that any potential new ones will be suggested by such an AI to us before they are exploited assuming we ask at all persistently.

[-]Duncan Sabien (Inactive)19d32

I'm not sure why you would claim that AI doom is very path dependent; I think that statement bears the burden of proof.

[-]RussellThor19d70

OK I thought it was at least somewhat accepted. Consider a world where AI progress has stalled but at the level where humanoid robots run much of society/tech stack and can autonomously keep society running. AI is not a superhuman hacker, persuader and is not capable enough for alignment (RLVR) issues to make themselves known. Then there is a sudden advance (architecture/sw) and the AI is suddenly able to hack all the robots, has super persuasion skills and the misalignment becomes obvious. You don't need any story about "how does it actually take over" in that case.

On the other hand, if the AI becomes a super hacker before it can take over from that alone (our world with no robots able by them selves to keep the world running) then it is very likely used for cyber defense to make critical systems unbackable. If RLVR creates obviously mis-aligned AI above a certain level that still can't take over (our world) then that isn't blindly pursued. Many doom scenarios require excessive RL. If AI becomes mildly super-persuasive without sudden jumps then that probably creates some defense.

If there is an architecture that scales all the way to obviously dangerous capabilities but not impossibly dangerous ones before it reaches its limit that is also important. In our world, Moores law is almost finished, data centers cannot get much bigger ~1GW - the transformer arch can get to likely 100T parameters but not too much more. Something that is superhuman at maths and likely alignment research but cannot takeover is definitely important. If you believe that a 100T misaligned transformer model released now cannot cause takeover then that is a crux. It means you should learn as much from it as possible and take the chance of major but not world ending chaos.

There is still going to likely be a sudden jump in capabilities when we make our AI as data efficient as humans, however such a system will need much less RL as it will learn so much faster. The RLVR misalignment path will likely not happen so fast. (Of course there could be different ways it will be misaligned) Its not even clear that such an arch will give advanced capabilities for a given weight size. Our current NN appear to require less weights than biology. A more data efficient arch may not necessarily require even fewer for the same performance.

[-]Duncan Sabien (Inactive)19d30

AI is already a superhuman hacker, though.

[-]RussellThor19d30

? I literally said that was our world now.

[-]Young Jae Koh20d*-40

Even as part of the camp of LLMs plateauing scaling and ASI skeptical without new breakthroughs, the shocking reminder is that we would NOT have needed that level of intelligence for us to have catastrophically lost the controls.

The clearest analogy here would be if even before the Trinity test, Klaus Fuchs was only noticed missing and found dead from radiation poisoning at an airport with blueprints and the keys to fissile materials storage, a month after the fact.

I am quite confident that the hardware level security on the foundational model weights are orders of magnitude more secure than Artifactory, as much as the fissile materials themselves are much harder to steal than the conceptual design for the hydrogen bomb. However, you do not need a thermonuclear apocalypse for it to be a catastrophic outcome.

P(Doom) doesn't have to be the case, but P(Chernobyl-level disaster) has certainly gone up for me.

[-]Taymon Beal20d95

What? Klaus Fuchs lived until 1988.

[-]Young Jae Koh19d40

I am making a hypothetical analogy. Adding an "if" to the sentence for clarity.

[+][comment deleted]18d20

Moderation Log