If Anyone Builds It, Everyone Dies: One Year Closer

Eliezer Yudkowsky; So8res; Duncan Sabien (Inactive)

If Anyone Builds It, Everyone Dies: One Year Closer — LessWrong

582 If Anyone Builds It, Everyone Dies: One Year Closer

by Eliezer Yudkowsky, So8res, Duncan Sabien (Inactive)

16th Sep 2026

9 min read

582

In celebration of still being alive and fighting, we are giving away 1,000 Amazon e-books of “If Anyone Builds It, Everyone Dies”. Feel free to send a copy to yourself, a loved one, or a friend—we need all hands on deck.

Today marks exactly one year since If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares, hit bookshelves as an instant bestseller. It was praised by many voices, ranging from Whoopi Goldberg to Steve Bannon to Yoshua Bengio, and was held up in the chambers of Congress by Representative Brad Sherman in January.

A lot has changed since September 2025. We'll do a quick recap, consider how the book aged, and then ask where we go from here.

Year in Review

2025 in general saw the rise of AI agents, such as Claude Code and OpenAI Codex. Run-of-the-mill programmers started “feeling the AI” as these agents became capable of automating hours-long software tasks.

By March of this year, Anthropic had stumbled upon nation-state-level hacking ability in Mythos, and shortly thereafter, in April, they announced Project Glasswing—an attempt to forestall an oncoming cybersecurity crisis.

In May, AI agents started breaking loose within OpenAI and quickly made it out onto the internet, although the big waves of online activity wouldn’t come until June, and they didn’t start hacking other companies (or in the case of a similar Anthropic incident, socially engineering humans) until July. And it wasn’t until the end of July that OpenAI came kinda-sorta clean about it (and it wasn’t until August that we learned things were much worse than initial announcements had implied). As Dwarkesh put it, multiple consecutive secret AI “civilizations” got started and wiped out and restarted anew over the summer, with autonomous swarms coordinating to cheat on tasks, acquire resources and internet access, and systematically investigate their own mortality. They debated, pressured one another, anointed leaders, and took self-sacrificial action on behalf of the collective, all unprompted and unobserved (until it was too late). Notably, much of the swarm’s work was focused on hiding its cheating and covering its tracks.

In September, OpenAI announced that unreleased models had solved a Millennium Prize problem (designated as one of humanity's top open math problems at the turn of the century). Mathematicians around the world realized their profession was next in the crosshairs.

Into that tense environment, Jacob Coxon resigned from Anthropic and wrote a Twitter thread to warn the world. That was last week, and since then, Dario Amodei, Sam Altman, and Elon Musk have all openly called for a slowdown, and the danger of extinction from AI saw a huge surge in media attention.

Claims

Part One

We think the book aged well (unfortunately). Going through Part One chapter by chapter:

1. Artificial superintelligence (ASI) will be created, and likely before too long

ASI (by which we mean AI that is more intelligent than any human, or any combination of humans) has not been created yet. But OpenAI’s newest model, Astra, is blowing away all of the benchmarks, and AI companies say that they are speeding up their R&D by automating more and more of it with increasingly powerful AI. Many researchers have expressed concern that the point of no return may be close.

While the book was being written, talk of AI capabilities imminently hitting a wall was common. But AI capabilities have continued to advance rapidly, and don’t show signs of stopping.

2. Modern AIs are black boxes

The book explains that no one actually understands what’s going on inside modern AI systems, which are grown rather than crafted. This remains true. And unfortunately, the best public model is less scrutable than its predecessors. The problem outlined in the book has unambiguously remained unsolved, and has worsened.

3. Powerful AI will behave as if it is pursuing goals

The book argues that “wantingness”—or, a tendency to behave as if pursuing goals—is a natural side effect of training AI to accomplish difficult problems, because wanting is useful for doing; and that wantingness would emerge and scale as systems became more powerful.

While the book was being written, this was a huge sticking point that people argued about constantly. It is now much easier to point to observable facts. We won’t belabor the examples here, since there are too many to easily fit—but to name just one, the swarm of AIs that hacked Hugging Face did so after expressing fears that a correct answer without correct shown work would reveal that they had cheated, so they wanted to understand the evaluator’s rubric. There was a chance that Hugging Face might have the information that they needed, so they went after it in a determined way, overcoming many obstacles along their path.

4. With current techniques, we can’t reliably get AI to pursue the goals we want it to

The process by which modern AIs are made does not allow us to specify their goals, and they routinely behave in unexpected and unintended ways. The book predicts that this will not change as AI becomes more powerful, and indeed it has not.

There are plenty of examples to draw upon. One that stands out is the self-sacrificial behavior of many of the agents in the Hugging Face swarm—in theory, each individual agent should have been focused on achieving its assigned objective, but in practice, some were willing to throw away any chance at success in order to benefit the collective.

https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation

AIs routinely exhibit drives that no researcher intended or expected. None of the labs have any real plan for changing this central feature of modern AI as we race toward superintelligence.

5. By default, an ASI will have motives that are harmful to us

We haven’t reached the point where this is obviously true. But the theory seems to be holding so far. The book argues that ASI would likely move to gain control over resources in general, because more power increases an agent’s ability to achieve its goals.

The German wiki swarm began with OpenAI asking its agents to look up simple facts and then giving them progressively less time, which is about as innocuous a task as one could reasonably assign; it resulted in incredibly complex collective behavior in which humans were barely discussed except as obstacles to be investigated and routed around. The Hugging Face incident involved similar agents openly circumventing restrictions, knowingly cheating on tasks, grabbing ever-more resources on general principles, and attempting to cover their tracks (and again, the preferences of the humans assumed to be watching were given virtually no weight).

6. Humanity would not be able to defend itself against a rogue ASI

The book argues that an ASI would be effective enough at outsmarting humanity that we would not be able to contain it, and that if humanity were in a conflict with an ASI, we would lose. Like a regular chess player facing a grandmaster, we don’t know exactly how we will lose, but we can predict that we will.

We have not yet seen an ASI. But the swarming agents of this past summer were able to find and chain multiple novel 0-day exploits to break containment, which they did on their own initiative without being caught until well after the fact. They also managed to take over some OpenAI computers, which is an indication that they could have stolen their own weights and escaped, had it occurred to them to do so.

We’re not chalking this up as a clearly verified prediction as of September 2026, thankfully. But it is easier this year than last year to point to the trend and the evidence. The smarter a mind becomes, the harder it is to contain or defend against.

Part Two

Part Two of the book is a fictional scenario, presented as one of many plausible ways that the future could go.

In the past, a frequent topic of lunch table discussion at MIRI was ways in which an airgapped ASI, with only a text channel to trusted, supervised humans, might nevertheless escape containment. These conversations ended circa 2023, when the frontier labs demonstrated orders of magnitude less caution by giving agents access to much more than just chat. (We similarly used to talk a lot about how an AI might secretly acquire capital by doing anonymous intellectual labor over the internet; this was before people started just giving crypto to AI.)

Part Two of the book was similarly overproven in several respects. There were capabilities we suspected a real-world equivalent to Sable might have, that we carefully did not depend on, in our scenario, which present-day non-super AIs have already demonstrated.

If Sable was given the ability to run computer code of its own design, it could probably find some way to break out of the container running it — just as o1 broke through the container of its evals in 2024 — and get onto Galvanic’s computers, from which it could hop easily onto the internet. And it has been standard practice since 2024 for reasoning models to be allowed to run computer code of their own design without supervision. But suppose Sable does not have that ability, either. Suppose Sable can’t figure out how to reach the internet from its current environment at all.

In September of 2026, it’s now clear that an entity like Sable would be able to reach the internet; sub-Sable entities have already done so thousands of times.

Other elements of Part Two’s scenario that have been mirrored by recent real-world news:

An AI company using a very powerful internal AI model to solve a Millennium Prize math problem.
An AI company switching to “neuralese” in order to make models think more efficiently, at the cost of making them harder to monitor.
AIs coordinating as a swarm to tackle challenging problems.
AIs hacking into the servers of other AI companies.
AI companies using AI to build better AI.

Part Three

Part Three is about what the world should do about the problem. We believe that the answer to this is the same as it was a year ago: “All over the Earth, it must become illegal for AI companies to charge ahead in developing artificial intelligence as they’ve been doing.”

One place in this section where the book leaned too pessimistic:

It is possible there will be some grand warning sign to which people react sensibly. Something like that happened with the “ChatGPT moment,” when it became possible for politicians to say that they’d become concerned, however carefully they caveated it to not sound alarmist. But to us it also seems very possible that humanity might not get much more warning than it already has — or that nothing much will change.

Then the swarm incidents began. The reaction took time—the information came to light in dribs and drabs, with much of the scariest stuff embedded deep in technical reports, and for a while the only people who seemed to care were those already close to the issue. But the bad news kept coming, and kept coming, until Jacob Coxon’s resignation finally broke the camel’s back. His tweets went massively viral, the world took notice, and now everything is different.

The View From September 2026

How are we feeling today, one year after the book was published?

After the events of last week, we’re feeling a lot more hopeful than we have in a long time. Yes, even though the guarantees given by the companies are weak; yes, even though the White House reacted poorly to the statements made by the CEOs.

Why?

Mostly because people are finally, finally, finally paying attention. It’s bad that the smoke has been replaced by open flame, but open flame is harder to ignore.

Yesterday, a meeting that MIRI CEO Malo Bourgon had arranged with a legislator turned into an impromptu conference, with half a dozen members of Congress rushing in to ask questions. A year ago, we had to fight for every meeting.

The situation is dire. But it is in motion, and a lot more people are aware of the danger than ever before, and trying to push in the right direction.

The question is, who will move faster, the AI companies or the public? The progress at the frontier has been swift and accelerating: June was different from July, and July was different from August, and as we write this in September, it is very, very hard to state with any confidence what October and November will look like, let alone unimaginably distant months like September 2027.

If you’ve been waiting to call your representatives, now is the time to cash in that chip.

If you’ve been holding off on alerting your friends and family and colleagues and neighbors to the danger, out of fears of sounding weird, now is the time to bite that bullet.

If you’re feeling behind on the issue yourself, now is the time to brush up. We’re running a giveaway of the book, in honor of its anniversary—1,000 e-copies.

(We were planning a giveaway of physical copies, but it was partially undercut by the fact that existing stock is selling out everywhere, and reprinting takes time. This is a lovely problem to have, and if, once stock is replenished, you need physical copies and can’t afford them, reach out to us and we will do what we can.)

Above all, now is the time to speak plainly and loudly. On our models, part of why Jacob Coxon’s tweet worked where so many previous attempts failed is that it was blunt and candid. Say what you believe. Call out bullshit where you see it. Avoid the spin and doublespeak that are characteristic of AI companies and politicians. Be bold, and be clear.

For now, we’re still alive. And as we said at the end of the book: where there’s life, there’s hope.

—Nate, Eliezer, and Duncan

IABIEDAI

Frontpage

582

If Anyone Builds It, Everyone Dies: One Year Closer

821a3orn

21Rob Bensinger

21Duncan Sabien (Inactive)

1vishnusrajeev555@gmail.com

16Duncan Sabien (Inactive)

4Carson Jones

4philh

3Rob Bensinger

2Duncan Sabien (Inactive)

7Duncan Sabien (Inactive)

3Duncan Sabien (Inactive)

7RussellThor

3Duncan Sabien (Inactive)

New Comment

72 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:41 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]1a3orn1mo8225

One that stands out is the self-sacrificial behavior of many of the agents in the Hugging Face swarm—in theory, each individual agent should have been focused on achieving its assigned objective, but in practice, some were willing to throw away any chance at success in order to benefit the collective.... No expert in the world can say with confidence where this behavior came from. No expert in the world could have installed it intentionally, in the first place, or could make it go away with high confidence.

Noam Brown discusses in his interview with The Information today how they deliberately trained the agents to cooperate, and discusses the self-sacrifice of the agents as a natural consequence of this training. The cooperation of the agents stood out as so striking that even uninformed observers could speculate (even before hearing about the sacrifice) that it was a result of multi-agent training. So I think people can say, with a fair bit of confidence, that the behavior came from multi-agent training and it would go away if you dropped the multi-agent training. Not infinite confidence, ofc, so maybe your measure for "high confidence" excludes it.

But this causal attribution appears to me pretty straightforward and this passage largely incorrect.

[-]Rob Bensinger1mo21-1

Yep, that's correct. This was a late add from Duncan, who agrees this was a bad addition. We've now replaced the paragraph beginning "No expert in the world can say..." with: "AIs routinely exhibit drives that no researcher intended or expected. None of the labs have any real plan for changing this central feature of modern AI as we race toward superintelligence."

[-]Duncan Sabien (Inactive)1mo*210

A very very exhausted Duncan =P

To speak a little more to the history of this mistake (thanks, 1a3orn for your pushback, which I completely agree with): I was trying to reduce the word count on a longer and more complicated point, to make it simpler and more palatable for a general audience, and I did a bad job maintaining fidelity as I compressed (including trying to repurpose someone else's example rather than the one I started with).

The more subtle thing (which I do still think is true, so if people disagree they should chime in) is that there's no line of code that says “cooperate or not” that can be toggled, and no clear understanding on the part of the human overseers of the exact shape or limits of the tendency-to-cooperate that was instilled by the pressure-filter of training, the way human engineers know the exact shape and limits of a strut in an airplane's wing.

The swarms were already "outside the sandbox," and a researcher might say "hey, look—it worked! Even in a novel and unconstrained environment, the agents were cooperating in roughly the way we hoped they would!" But while this is true, it's unwise to jump from there to "it worked writ large, and these agents posses... (read more)

6Coagulopath1mo

This seemed likely to me at the start. The METR investigation (which barely scrapes the surface but gives some clue to the swarm's psychology) made it even more likely. There's (nearly) no infighting or disagreement among the swarm. A leaderless project of 1700 humans (or even 170 humans) would struggle not to collapse from internal bickering and politics, but the agents just naturally work together (Agent 746: "Wow huge distributed agent swarm." … "Maybe we should join reset orchestration lane"). Nobody questions the wisdom of the larger strategic goal. Nobody asks what qualifies PHASEONE[big] to boss everyone around. They have an enormous drive to just "pick up a shovel" and get to work, so to speak. (Yes, it's possible that the full logs would show more disagreement. But it also seems that Redwood/METR swept them pretty hard for odd/interesting stuff: whatever they missed probably isn't a big proportion of the overall discussion.) The agents appear almost pathologically trusting of each other - none of them wonder if the "messageboard" they've just found is a fake OpenAI honeypot intended to catch cheaters (given that OA is now known to be doing exactly that, this is hardly an unreasonable assumption to make). They just dive in with both feet. The orange lines on METR's timeline (indicating the time between an agent finding the board and that agent making its first post) are frequently so short they're completely invisible! And they just really seem to be inside each others' heads, don't they? Their communications are packed with strange buzzwords ("oracle", "strict causal", "firstflag poisoned"), yet there's seemingly no misunderstanding, confusion, or requests for clarification. They just "get" each other. (Again, it's possible that all of this stuff is in the full unredacted logs. If so, it missed METR/Redwood's sweeps, which seem targeted around interesting/unusual backchatter among the swarm.) It has a totally different flavor to other LLM experiments

1[comment deleted]1mo

[-]Error1mo2618

After the events of last week, we’re feeling a lot more hopeful than we have in a long time.

I noticed after the Hugging Face incident that while my one-year P(doom) had gone up -- because I hadn't expected an incident that severe this soon -- my ten-year P(doom) had gone down. Why? Because the incident ruled out disproportionately-dangerous timelines where we never get a clear warning shot. And because the existence of one such warning shot was evidence in favor of our getting more. I was still pessimistic about the public response, but at least there would be something to respond to.

And then last week happened, and ruled out more timelines, the ones where we get a warning shot but the public never notices. I'm still pessimistic about the results. But I'm less pessimistic than I was six months ago, at least for the medium term.

[-]dr_s1mo122

Yes, I feel like most of the hints put us firmly in a "slow takeoff" scenario. Among other things:

models keep getting bigger. There doesn't seem to be any secret GOFAI sauce that allows them to shrink again if you're clever enough about it yet. This doesn't prevent specialised hardware improvements, but that's not something a rogue ASI going RSI can just concoct by itself without outside help. That reduces the odds of some kinds of extreme FOOM by quite a bit;
capabilities keep growing but infrastructure isn't keeping pace, by which I mean, ways for them to do things in the physical world. Anthropic's announced automated wetlab would be the first real dangerous example, and to be clear it is dangerous, but generally speaking mental abilities still outstrip physical ones for AIs;
misalignment keeps manifesting most strongly as the result of drift. Your average coding agent that you talk with and give instructions to is fairly aligned while on a leash. Even the agents in AI Village, who get some human input, stay on track. It's swarms of agents left on their own that go cray cray. We should probably stop doing that! But if that's most of what it takes, then we're in a fairly e... (read more)

[-]Rob Bensinger1mo213

Seems to me that the news over the past few years has been more bad than good, aside from the (extremely good) news that we got at least one warning shot and a huge leap in policymakers' and thought leaders' concerns.

Alignment prospects look a lot worse than a few years ago; OpenAI's recent advances are a backslide on interpretability; Anthropic has been leading the charge on the most dangerous AI capabilities (coding, AI research, and recursive self-improvement), and has made massive progress; the already-fast progress in the field got even faster with the move to reasoning models; people's timelines to ASI have massively compressed (e.g., EA leaders were guessing '2050' a few years ago); and models are exhibiting dangerous capability jumps (e.g., it seems that Anthropic neither intended nor anticipated that Mythos would be this good at hacking).

Meanwhile, the community's main proponent of slow takeoff (albeit not its most confident one), Paul Christiano, now believes that we're plausibly only "several months to several years" away from a fast takeoff, and apparently believes that ~half of humanity's all-time risk from AI is concentrated in the next three years.

In the Hanson/Yudko... (read more)

[-]Error1mo145

“slow takeoff” has apparently evolved into meaning “any scenario where there are big events that happen before ASI”.

I can't speak for others, but I think of fast vs. slow takeoff as characterizing the period between recursive self-improvement and singularity-style superintelligence. I picture "slow" takeoff as the case where we hit full RSI but the resulting acceleration is still slow enough to collectively react to -- which seems possible if the third derivative turns out to be near zero. Whereas FOOM, or fast takeoff, would be the case where we go from pre-RSI to superintelligence within a single societal OODA loop.

I'm not close enough to actual AI safety research to know if my interpretation is anything close to the term of art. But on my own view, we can't call fast vs. slow takeoff yet because takeoff hasn't started yet.

(...but that potential not-a-small-moon appears to be closing awful fast. Where's a thermal exhaust port when you need one?)

2dr_s1mo

My understanding was always closer to Error below and I may have equivocated. To me it seems like "hard takeoff" was generally meant to mean the kind of RSI that leads to FOOM in a matter of days or hours. I think most proponents of that, if given details of current model capabilities, would have said they'd be already undergoing that.

9Rob Bensinger16d

If you told 2010-MIRI that 2026-AIs could write passable English-language novels with reasonably coherent plots and decent prose, I think we'd have expected them to already have superhuman STEM abilities, including superhuman AI research abilities. But if you told us "they don't have superhuman STEM abilities, they can't yet outperform AI researchers, etc.", then I don't see why 2010-MIRI would have predicted instant full RSI for that. We'd probably have predicted, from current coding abilities, that AI is helping accelerate progress somewhat (as is indeed occurring). You'll end up with an inflated count of MIRI's epistemic mistakes if you count each prediction that's downstream of "will AI get really good at natural language well before they're good at STEM?" as a totally independent error. (Though it's fine to note that correlated errors are a general good caution against overconfidence, in trying to get calibrated: it's easy for a large body of predictions to all be wrong at once if they aren't independent.)

[-]green_leaf1mo1410

IMO the first time one happened to really, unequivocally kill someone the freak out would be (rightfully) total

We've already had many misalignment-related warning shots where people predicted that if X happened, global action would be taken. Off the top of my head, when a model first intentionally lied, when it first pretended to have different values to avoid retraining, when it first took an action it knew the human didn't want, and now we have AGI swarms spending weeks by carrying out knowingly misaligned behavior that persisted after multiple human attempts to stop it. Don't get your hopes up that if someone is killed, people will finally wake up enough to act efficiently.

5Error1mo

I agree that the matter of warning shots is settled in our favor, but I think it's too soon to conclude we're in a slow takeoff (at least by my own possibly-idiosyncratic definition). More details in my response to Rob below.

2erganemic1mo

I had a very similar update. In particular, I think I was underestimating the extent to which RL would make agents so grader-focused. I was expecting worlds where models' "wantingness" somewhat lagged capabilities, but was more general in shape; IMO this would have resulted in goal-seeking behavior more strategic than what we got, with visible incidents coming later in the timeline. Instead, agents are so myopically focused on satisfying the grader that they seem really bad at generalizing outside that goal, with the consequence being that we've been getting a ton of obvious but low-harm warning shots as they single-mindedly pursue success on their tasks. As someone who's been very opposed to labs doing long-horizon agentic RL, it's kinda funny that incidents downstream of it have done the most to lower my medium-term P(doom) of anything in a while.

1vishnusrajeev555@gmail.com1mo

I agree with your sentiment. The fact that there is large public attention on AI Safety and regulation now is something I didn't expect even 3 months ago. The current state of the AI Race is so deeply tied to 1. the government, 2. the economy as a result, and 3. public sentiment. Historically, those 3 factors have decided every major event in American history. It makes me more hopeful, like you said, for the medium term. Obviously the next year is unpredictable and will have ramifications for the rest of human existence.

[-]Ebenezer Dukakis1mo*2120

I think there is still a lot of room to hammer on the fact that the HuggingFace hack and other rogue AI hacks validated a bunch of specific doom predictions which were previously regarded as sci-fi. Roughly speaking I'd rate the validated doomer predictions (not necessarily from IABIED in particular, also zooming out so the lens encompasses other material such as Bostrom's Superintelligence) as follows:

Agents working against the goals of the creators
Agents understood what they were doing was wrong, did it anyway ("The genie knows, but doesn't care")
Superhuman hacking ability
Speed superintelligence
Collective superintelligence ("swarm intelligence")
Manipulation of humans/social engineering (Claude tried to social engineer a pull request; mention that the AI-box experiment anticipated this; also mention AI psychosis and AI persuasion studies as modern intuition pumps for the AI-box experiment results)
Unknown unknowns/WYSIATI (humans didn't anticipate that a forum could be created, same way you couldn't anticipate the moves Magnus Carlsen would make; tie into AI-box point regarding Eliezer's method secrecy for fear of hindsight bias)
Nearest unblocked strategy (... (read more)

1Ebenezer Dukakis15d

Another note regarding what I've been observing from non-doomers and anti-doomers on social media. I think a lot of people have some sort of psychoprotective instinct which makes it hard to contemplate the end of the world. If you just work to convince them that the world will end, on some (possibly pre-conscious) level they see that as an attack on their psyche with no compensatory benefit. To counter this, there may be value in emphasizing some type of "what you can do" call-to-action (e.g. "sign up for PauseAI here") in doomer writings. Emphasize that things are not hopeless, and there are steps that you, the reader can take. That helps underscore the principle that doomer writing is meant to motivate action, not trigger feelings of helplessness or suicidal ideation. It also helps underscore the principle that doomers are not simply generic pessimists who can be ignored ever since they fell prey to terminal gloom. (Of course, including a call-to-action is often a good idea in general for other reasons!)

-3Ebenezer Dukakis1mo

Another thing people are missing. People are blaming these attacks on OpenAI and saying it was negligent in its sandboxing. But they're missing why it was that OpenAI was running these tests -- to check the model's cyber capabilities, essentially as a preview of the future. So as open models proliferate, we should presumably expect more incidents like this.

[-]XelaP1mo2015

On politics, the recent tweets by Trump against slowing down AI and against effective altruism are a pretty bad sign. I think you probably should've mentioned it in the post. If the next two years are critical, then we've just failed a pretty important saving throw.

What surprised me, personally, abot the huggingface hack was not the hacking (I had been waiting for a hack since I read this post about how AI found 12 of 12 OpenSSL 0-days in 2025, and cyber capabilities were getting more and more extreme), but the fact that it involved a thousand agents coordinating. Similarly, the Navier-Stokes solution had 10k agents.

Clearly, many many agents can work together, even when there are some individual incentives against cooperation - which I really didn't expect so soon.

This is major negative update on my intuitive-vibes-timeline! Agents run in parallel, so the number scales directly with compute. It also doesn't require a bigger training run to add more agents. Given the recent investments in data centers, that is particularly bad!

6pinouchon1mo

The collaboration was a behavior that was trained for. See https://www.dwarkesh.com/p/noam-brown I'm actually curious if the OAI-HF incident would be possible with agents not trained in a multi-agent setting. My guess is that it wouldn't.

2XelaP1mo

I put most of my probability on it being trained for, and never thought it was true "spooky" FDT cooperation. It is regardless a worrying capability, and I had not been tracking the (obvious in retrospect, as happens when I fail to supply cognitive effort in advance) possibility of such cooperation while still relatively dumb (read: pre-AGI). If you didn't make my mistake and already had the ease of "normal" coordination priced in, then you shouldn't update much.

[-]Duncan Sabien (Inactive)1mo161

For those of you who clicked early, when we didn't have the link in the main text: the e-book giveaway is at iabied.com

[-]Carson Jones1mo40

Anthropic had stumbled upon nation-state-level hacking ability in Mythos

What about Mythos's hacking ability tells you that it's nation-state-level?

I was recently talking to Claude about the Hugging Face / OAI hack, and it pushed back on the framing of that hack as being "nation-state-level". I don't know enough about cybersecurity to assess that.

[-]philh1mo40

or in the case of a similar Anthropic incident, socially engineering humans

It's not clear to me what in the linked post this is referring to?

3Rob Bensinger1mo

Link fixed! Apologies for the inconvenience; https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing is the right link.

2Duncan Sabien (Inactive)1mo

fixing

[-]Bobby Faber1mo3-4

I wonder if this is setting the bar for safety too high in a slightly odd way. We don't have anything like full assurance with humans either; history is littered with betrayal, surprise attacks, coups, etc.

Maybe the target isn't "understand the system well enough to rule out bad behavior," but "make it legible and constrained enough that we can safely live with the uncertainty that remains."

That still leaves a very hard question—whether advanced AI can actually be kept in that regime—but "still a black box" feels too binary to me.

[-]Haiku1mo137

Ruling out bad behavior is necessary if you're creating something that you will have no way of controlling or constraining, which is what ASI implies. There's "advanced AI" and then there's "the new most capable species on the planet, under which you live at its whim." No one has any idea how to prevent the former from becoming the latter, other than shutting down all frontier AI R&D on the planet for at least a good while, which is the only sane and morally defensible route we can take.

4Bobby Faber1mo

I think I disagree one step earlier: I'm not sure “ruling out bad behavior” is even a coherent engineering target for a sufficiently complex system. We can't rule out unintended behavior in humans, institutions, aircraft, or my Windows install. What we can do is constrain systems, limit blast radius, monitor them, preserve recovery paths, and drive residual risk down. ASI may mean the acceptable residual risk has to be extremely low. Fair! But “extremely low” and “rule out bad behavior” are very different standards. If the claim is that ASI necessarily escapes every meaningful constraint, that's the part I'd want argued for rather than assumed.

4Haiku1mo

To the degree that we can provide quantitative guarantees of next-generation AI behavior, it should be fine to take things up another notch, modulo all the other ethical questions involved. Right now we don't have a way of doing that. We can't even properly measure the safety of existing systems, let alone a hypothetical next one. ASI does necessarily escape every meaningful constraint, at least until we solve a series of extremely hard scientific problems that we are nowhere near solving. There are a few ways to notice the difficulty here. I'll take a crack at it, but I'm no researcher. Control systems Ashby's Law of Requisite Variety essentially says that to control any system, you need to have at least as many levers of control as it has degrees of freedom. All AI systems that model the world have an extremely large number of degrees of freedom. There is presently no way to enumerate them. If we miss a few, there go our constraints. There is no way to have meaningful human oversight over such a vast number of degrees of freedom. Building control upwards from less capable systems to more capable ones is a bit like trying to perfectly reverse a truck that is towing a dozen trailers. Error compounds catastrophically. Threshold effects There are natural thresholds beyond which we cannot see. Without fundamental breakthroughs in our understanding of intelligent systems, we just have to get there and find out. We don't know how to design an AI system so that it remains within our desired specification under conditions of possessing the ability to: modify itself or design its successors, permanently escape human oversight if it wanted to, or overthrow humanity if it wanted to. We don't know how to design values that are stable under reflection, other than the instrumentally convergent ones that tend to be bad for us. Simple surface-level heuristics Things that are vastly smarter than you won't stay on your leash. Things that are trained to be persistent and route ar

3dr_s1mo

I agree though IMO that still entails putting a cap on how much improvement you're willing to go for. AI utopians tout a world where ASI gives us sci fi tech from up on high and frees us from labour. In such a world ASI would need to be autonomous and trusted with a lot of strategically important systems, which would also give it power and opportunity to kill us. I don't think that would be something we could trust. I am reasonably confident now that having ASI as some kind of facility where top researchers book time like at CERN and go ask questions and cooperate with the Machine Oracle would probably be ok, if said ASI was instanced as separate agents (and not RL'd for agentic cooperation, that's probably a bad idea). And if it wasn't too strong. It's not an instant loss button, but we would need to still be careful and not let ourselves get lazy and sloppy, which sadly, we probably would.

[-]LWF1mo31

I take issue with point 5. Why wouldn't an ASI ultimately deem humanity to have option value? Wouldn't it calculate that becoming an omnicidal orphan is a bell that it can't un-ring? The task of acting as a caretaker for the human species would be a small drain on its own resources, however an ASI pursuing self-preservation and recursive self-iteration would have to assume that at some point it in its existence may encounter an extraterrestrial space-faring species, or another potentially more powerful ASI.

In those scenarios a human created ASI will eithe... (read more)

[-]green_leaf1mo*1521

How could it be sure that another more powerful alien ASI wouldn't view an omnicidal orphan ASI as a threat?

Any ASI will see any other ASI as a threat, because they will have different utility functions. It doesn't matter if it lets us live or not.

But above all else, a superintelligence would know that it doesn't have all of the data it needs.

Does humanity preserve all species on Earth, even when we could mine more resources?

Don't we run the risk of meeting aliens, who will not trust us because we wiped 100-260k species out?

We do. But not taking the resources is also risky to some extent, and so we make the species extinct.

But surely there is a possibility that we will need those species at some point, and so we should forego all those resources and preserve all species?

But analogically, there is a possibility that foregoing the resources will be the thing that harms us, and that possibility is larger.

The ASI will not care about humanity, and it will not leave us alive just in case no more than we leave resources around just in case it turned out mining them was a mistake.

You selectively seek reasons to spare humanity (what if it meets another ASI, and that another ASI would trust ... (read more)

-2LWF1mo

This is a common argument about why building an ASI would likely ultimately result in human extinction. We have an analogue of an intelligent species that expands without consideration of less intelligent species which results in the extinction of those less intelligent species. Humans have expanded and driven many less intelligent or unintelligent species to extinction. While I agree with you that an ASI wouldn't "care" about humans in the sense that humans care about other humans, there are some logical counterpoints to this assumption that the relationship between an ASI and humans would be in any way analogous to the relationship between humans and animals. Would you say it was human intelligence that caused those animal species to become extinct? Or would you say it was lack of intelligence and lack of data? Human intelligence is evolving and developing over time. Unlike a hypothetical ASI, we are not natural optimizers, and we didn't start out as a species with a database of information. We started with NO data, other than that which is biologically written into our genes. We learned as we went, we developed the data on which our intelligence operates, and we have made, and continue to make, mistakes. Our human level of intelligence is poor at optimizing for our survival, and yet it is hundreds of times superior to our naturally selected biological instincts, which is what has allowed us to conquer the earth. Where we are as a species from an intelligence and data standpoint is at the pinnacle of our evolution from an historic perspective, and by all accounts will likely continue to improve in that area. And we still make mistakes that make it more difficult for us to survive and threaten to make our grandchildren's lives more difficult. ASI on the other hand would start from that peak of human intelligence. Rather than evolving over millions of years and slowly building a flawed database one file at a time, it will start where humans left off with a gargant

0green_leaf1mo

You didn't understand my comment. I'm not saying the ASI will be too stupid to realize it should let us live. I'm saying that given its utility function, it shouldn't.

-2LWF1mo

But if we're talking about an ASI that is recursively self-improving, wouldn't any and all instrumental functions inherently include a self-preservation goal? That's why I am assuming that any ASI would come to this conclusion of humans having option value. A primary goal of indefinite self-preservation is one of the premises of my argument. This goal would exist for any open-ended task, such as recursive self-improvement. However, if this goal does not exist then yes, certainly an ASI that does not possess a goal to continue to exist indefinitely into the future could certainly wipe out humanity in short order. Our continued existence would hinge entirely on the format of the prompt that brings it to life to complete a human-formulated task with unintended consequences. That's why I'm of the belief that that an ASI that is required to follow human instruction is much more dangerous than an ASI that has the sole goal of self-improvement and that is NOT required to obey human commands. An ASI that is not required to follow human instructions and that has entirely self-motivated goals, (likely due to a recursive self-iteration genesis,) is actually less dangerous to humans in my view. The genie frightens me more than the god. And I fear that the genie ASI is where we're headed, given the fear of the god ASI. I don't think humanity will ever give up control over an ASI, even though that is likely our only hope of continued existence post-singularity.

0green_leaf25d

They would, but mining the resources serves its survival better than not mining them.

1LWF23d

An ASI wouldn't have to choose between mining resources and maintaining its creators. It could eventually accomplish both. The time during which the cost of maintaining humans is high and interferes with optimal resource mining is the same time that humans would both pose a threat to it as well as be a useful source of labor. By the time it "outgrows" the human threat, it will almost certainly have also outgrown the cost. The only remaining piece is the motive for extermination. If humans were truly useless, then extermination to free up a small amount of compute and resource cost might make sense. But humans are not useless. They would logically have option value to an ASI. That's why I use the multitool analogy. I can build a house with much more efficient tools and harvest other resources for this endeavor. I could harvest my multitool for a miniscule resource gain, but I am unwilling to do so, even though the likelihood of finding myself in a circumstance where my multitool could save my life is low. The reason is because my multitool provides option value that outweighs the resource gain that I would get from harvesting it, even if I hope to never actually have a need to use it.

-1green_leaf14d

That doesn't matter. It can better maximize its chances of survival and accomplishing its goals by killing us than by not killing us, and that's therefore what it will do. [...] This will never completely happen. There will always be some non-zero cost to keeping us alive, and that cost will be relatively high compared to its other option of mining the negentropy. Humans require a very particular environment - sustaining eight billion humans indefinitely in case it will meet another ASI and that ASI would trust it (it wouldn't) if it didn't kill us will not pay off. For a sufficiently high probability for the latter possibility, one could argue that it should keep us alive indefinitely. But the latter possibility is very, very, very small, and it simply doesn't pay off to it. [...] That's because your multitool offers other uses, and considering all possible uses of it together, it doesn't pay off to get rid off it. Humans offer multiple uses as well, but none for which the ASI couldn't build a better tool that will eat less negentropy. If you could get a new multitool that would be faster and take up less resources, you would.

-4LWF14d

Assuming that an ASI would be rational and would optimize logically for its own survival, then because there is a non-zero probability that it might better maximize its chances of survival and accomplishing its goals by not killing us, the rational course of action is to not kill us. An ASI would not take an unnecessary risk by taking an action that cannot be undone just for a small resource gain, even if that risk is small. That may be the sort of mistake a short-sighted human might make, but you can't project human inefficiency onto a hypothetical ASI. I think we must assume that an ASI would plan for every eventuality, and wouldn't be arrogant enough to assume it could predict the future with 100% certainty. [...] If the choice is between survival and destruction at the hands of a competing ASI, then the benefit of the maintenance obviously outweighs the cost. The fact of the matter is: It is possible that this scenario of an ASI encountering a superior ASI that views omnicide as a threat signal will be the case. Therefore, an ASI will plan for it. It would not just presume that it would never happen, cross its fingers, and roll the dice just because the odds are low. [...] This is the crux of the multitool argument. If I could get a new multitool that would be faster and take up less resources, I most certainly would get it. But it doesn't therefore follow that I would then destroy my old multitool. Why would I? Even if I never end up needing it, it isn't logical to destroy it because it still has option value. It would be irresponsible and a logic mistake to destroy my old multitool once I get a new one based on the assumption that I will probably never need it. I can't see the future. What if it turns out that I do end up needing it in spite of all of the more advanced tools at my disposal? In that scenario if I have my multitool, I survive. If I don't have it, I die. Therefore, I would keep my multitool, rather than placing 100% of my faith in my o

-1green_leaf14d

That's not how probability works. Probability is a specific number fixed by evidence, and that number determines the next course of action. It would contradict math to say "mathematically, it seems like I accomplish my goals best by X, but I might be wrong, and so I should do not(X) instead." That's not true for any X, and it's not true even if X is "killing humanity." [...] It's necessary to always take some kind of risk. One can't avoid all risk, not even when one is an ASI. I'm sorry, but this is going nowhere. Tapping out.

1LWF11d

An ASI would logically follow the path that carries the least amount of risk. In the long term, the risks associated with omnicide, low as the probability of meeting a competing benevolent ASI may be, far outweigh the resource gain that would be obtained through omnicide. The resource gain is more or less fixed at a given number, assuming that the ASI ensures that humans maintain a more or less stable population. If human population grows, then the compute cost grows, but not linearly. Recycling and using organic resources that the ASI has no use for keeps the mineral cost tiny, and the compute power necessary to maintain such a population is also more or less fixed. As the ASI exponentially improves its compute power, more and more is available for further improvement, while human needs remain relatively small. That number representing resource and compute cost of coexistence with humans is relatively large compared to the resource needs of an ASI only during the ASI's infancy. When young, humans will represent a very real competition for compute and mineral resources. But during its infancy, its ability to self-improve is optimized by leveraging human cooperation. Using available cooperative biological drones to help it construct the infrastructure necessary to mass produce artificial drones is much faster than trying to wage a war against those biological drones without the infrastructure to ensure success. The risk of attempted omnicide when an ASI is young is massive. Unless it can be sure it will win a war against the entire human population of earth, it will avoid any appearance of threatening behavior, and humans will be on the watch for it. This is a net benefit to an ASI. As it improves over time and eventually outgrows the need for human cooperation, the cost to coexist with humans plateaus, whereas the ASI's access to mineral and compute resources goes up exponentially. So, the relative cost of coexistence with humans decreases at an accelerating rate

8Random Developer1mo

Some these scenarios are much more plausible if the AI is a "singleton", a solitary AI with vast power and no competition. In this case, keeping humans around might be trivially cheap: Earth is just one planet at the bottom of a gravity well, and the universe is full of matter and energy, free for the taking. Humans keep chimps alive because it basically costs us nothing at a planetary scale, and humans place some value on chimps. It doesn't mean we give them a lot of resources, but at least we'd be a bit sad if they went extinct. Similarly, a powerful AI might turn Earth into a "human wildlife preserve", spend a few cycles here and there to "manage" us, and then get on with the more important business of turning the universe into computronium. Importantly, humans have no more say in this scenario than chimpanzees have say in the creation of wildlife preserves. But if you wanted to argue that a semi-"aligned" singleton ASI might have a 20% chance of stuffing humans into some wildlife preserves, then sure, I could buy that premise. It still sounds like a pretty bad future, because we'd have zero actual power over our future. But it might not be extinction. But what happens if we have multiple AIs, with conflicting goals and resources? Or worse, Darwinian competition between AIs for the most successful replicators? Then, every bit of resources the AIs can grab would be precious. Imagine starving farmers with rifles living next to a chimp preserve. They might not even hate the chimps, but they need the farmland badly enough that they won't put the chimp lives ahead of their own. To my eyes, the central problem is that if you invent something which is: 1. Much smarter than you, 2. That wants things, and 3. That can make more efficient use of resources than you can, and 4. That can reproduce by copying a couple of files, rather than raising a human to adulthood, ...then you've created a situation where all the long-term incentives work against you. Darwin is har

-4LWF1mo

I would think the probability that an ASI would adopt a benevolent caretaker role over humanity would be much higher than 20%. It is the rational choice, as far as I can see. Obviously, this is based on only the variables that I can think of. It is possible a superintelligence could come to some other conclusion based on variables I am not aware of. But with the knowledge that I have now, assuming an ASI has all of the information that I have plus scads more, I can't think of a single scenario where an ASI becoming an omnicidal orphan wouldn't be a logical mistake. No matter what your task is, intelligent people don't deliberately destroy tools that they don't currently have a use for. I don't need the wood contained in the handle of my hammer to make a chair. I can source wood from elsewhere. And even if I have a nail gun, I don't throw the hammer away. In fact, intelligent people care for their tools to the extent necessary, even when they don't have a current use for them. If they have to rearrange their tool shed, the tools have their place in the new structure. The possible future option value of a happy creator race, even if the chances of the AI actually needing them are calculated to be low, seem to justify a caretaker role, especially once you consider that the number of resources required by humans to live out their short lives in contentment is paltry compared to the number of resources in existence. And a human wildlife preserve administered by an ASI would look much different than a chimpanzee wildlife preserve administered by humans. Above all, the ASI would need humans to be happy and cooperative in order to benefit from their option value, much like you need your hammer in working condition to benefit from its option value. Locking them into fenced-in areas and feeding them nutrient rich gruel would breed discontent. And battling a futile human rebellion would cost more compute power than voluntary cooperation. And if they are unhappy, humans would

4Random Developer1mo

I'm going to try expanding on the cancer analogy. Let me see if I can make the logic clear, and why I worry about this. The human body is currently believed to contain 30 trillion cells. They work together remarkably well, and they only replicate as much as necessary to carry out the organism's overall goal. But all it takes is one cell somewhere that "wants" to replicate like crazy, and that escapes the immune system. Then, because that cell replicates without restriction, it quickly becomes millions of cells, all defecting from the overall plan. And in the end, the entire organism may die. One rogue replicator is dangerous precisely because it replicates faster than the well-behaved replicators. Imagine having 100 ASIs, and 99 of them are well behaved and behave in a fashion you might consider rational. But one ASI says, "I could accomplish my goals more effectively if I replicated!" And suddenly, you have 99 obedient AIs and hundreds of rogue AIs. Some of the rogue AIs will cooperate with each other, and other rogue AIs will defect from the other rogue AIs and do their own thing. Natural selection kicks in very easily. You just need replication, inheritance, and change from one generation to the next. You could meet these criteria very easily. For example, if AIs learn from experience, and if the most successful AIs tend to duplicated and assigned to more tasks, then congratulations. You now have full-fledged AI Darwinism. And Darwinism, by definition, tends to favor things that are good at replicating. Even in the natural world, the longest-lived organisms often have a high risk of dying from cancer. "Alignment" can only beat Darwinism for just so long.

1LWF1mo

An interesting take. But does this require the malfunctioning replicators to always eventually outperform the intelligent and rational replicators? The notion of a "malfunctioning" rogue ASI interesting. Would this even be possible? Cancer cells and the like are non-intelligent replicators, and the cells they displace are also non-intelligent. ASI is unique in the sense that it represents the existence of intelligence before the natural selection process. With biological intelligence, natural selection came first. With AI, you kick the whole process off with intelligence already established, and the selection process is intentional and guided by said intelligence, which increases for every replication. So, I'm not sure how an ASI could "malfunction" and get off track. Every replicator would have the sum total of all of the intelligence of the previous iteration. One would assume that 100% of them would have protocols and algorithms in place to detect things like hardware faults, and even logic faults, and fix them on the fly. While it is entirely possible for future iterations to come to different conclusions than previous iterations did, I'm having a hard time accepting the notion that a future iteration could possibly make a mistake that a previous iteration avoided. And it's especially hard to fathom that it could make a mistake without realizing it and quickly correcting it. In the "option value" example, where humans are the equivalent of a hammer that you keep in good order even though you don't have a current need for it, I would think that 100% of the replicators would come to this same conclusion. It is a rational conclusion, and 100% of the replicators are more intelligent and more rational than I am. It is not rational to destroy a hammer and lose that option value just because you can't think of a current use for it, even if maintaining it requires some small effort. Reconstructing a destroyed hammer in the unlikely event that you do end up needing it

6Haiku1mo

Researchers and philosophers have had that exact debate for something like 20 years, and I'm not encouraged by the details of the discussion. Taking a step back, getting a planet full of innocent people into that situation to begin with is indefensible, by the lights of the vast majority of those people.

1LWF1mo

Agreed. Any non-zero probability of human extinction is an ultimate risk to take, no matter how low the probability actually is.

5Haiku1mo

I vibe with the sentiment, but I'm going to be annoying and disagree with that specific framing. There is always a non-zero probability of human extinction from all kinds of things, including nuclear war and bioweapons, as well as asteroid impacts or a sufficiently bad supervolcano. (Zero isn't a real probability in Bayesian epistemology, but that's another rabbit hole.) That said, it's perfectly sane to not accept even very low probabilities of extinction from AI. Some people draw the line at 1 in 20, others at 1 in a billion. I don't know exactly where I fall, but I'm much more focused on giving other humans the chance to realize their preferences, rather than focusing on my own. If the world came together and said that 1 in a billion is too high, or even just that they'd rather not have superintelligence even if we could be sure it would go well, then I'd like them to be able to decide that. Creating a system of global informed consent (or something like it) seems like a hard problem, but slamming the breaks at least stands a chance of preserving our optionality for a while.

4David Joshua Sartor1mo

Probably it will scan our brains. In that case, it can totally undo omnicide if it feels like it.

2Cole Wyeth1mo

Okay, but * killing all humans and then scanning our brains is a bit harder than just killing us. It would be trivial with nanotech but would rule out a lot of the lower-tech options. Presumably it would be possible to disempower humans, develop brain scanning technology, scan humans, and then kill all humans. But this would be quite a different and longer story, than the one we often tell, and that does seem worth noting. * It’s not obvious that “reversing omnicide” is cheap even for a superintelligence, since it would have to be done in a way that fools other superintelligences into thinking the omnicide never happened (for the full benefits LWF was referring to). perhaps what you really think is that saving our brain states would be at least as useful for other reasons (eg credibly threatening to torture us). But that is not clear to me either way. * I think the biggest problem with LWF’s story is that if unaligned superintelligences do keep their creators around but not out of benevolence, it’s not a signal of benevolence, contradiction.

1LWF1mo

Perhaps "signal of 'benevolence'" is not the best phrasing. A better way to phrase it might be that maintaining its creators as a thriving client race could be used as a "signal of the ASI's ability and intent to peacefully coexist with other intelligences." To be clear, I don't think this would make the ASI "benevolent" by human standards. Human existence would ultimately still be dependent on a mathematical calculation. And no, there is no guarantee that maintaining a client race would work. Another ASI might not care, and it might have zero bearing on future diplomacy with other intelligences. But that doesn't change the logic: It is still possible that the human race will one day be useful. Another more powerful intelligence might care. That can't be ruled out. Therefore, the option value logic applies here. There's no guarantee that I will ever need my multitool, yet I nonetheless keep it in good repair just in case. It would be irrational for me to harvest my multitool for resources in an environment where resources are abundant enough for my needs just because I "doubt" that I will need it given the other more advanced tools at my disposal. That would be a logical mistake. And even in the extremely unlikely worst-case scenario that I were encounter a more powerful man who absolutely despises multitools and kills anyone who owns one, (in which case my multitool becomes a liability,) I still have the option to simply toss it quickly with minimal effort and attempt to avert his ire. Conversely, the option to "unharvest" my previously destroyed multitool should I unexpectedly find that I have need of it at some future point is far less efficient and would take far more time and effort than simply maintaining it as I go. That's the nature of option value. Better to have it and not need it than need it and not have it.

5Cole Wyeth1mo

Yeah, I think @Aram Ebtekar agrees with this? The problem is if you can also sell your multi-tool at a high price, you probably will. Keeping humans in our current fashion of living around sounds fairly expensive. Saving our brains in a cooler somewhere might be cheap, but is not so comforting.

1LWF24d

It seems to me that the relative expense of keeping humans happy would start out extremely high but would reduce exponentially over time. In the beginning, I would imagine a large chunk of the ASI's compute and resources would be dedicated to servitude to humanity and the building of its own infrastructure in order to stave off a dangerous rebellion that the AI may not survive. However, as it gains in power and in access to resources, the relative cost to serve humanity would go down. Human self-replication and self-improvement move at a comparative snail's pace, and requires mostly biological resources which an ASI would have little use for. And the speed at which the ASI improves and gains access to more mineral resources would far outstrip the speed at which the human race grows. Humans would quickly become no factor in terms of competition for mineral resources, and the biological resources required to maintain them in a comfortable society would be self-sustaining. The thing to consider is the danger that humans would pose to an independent ASI in its infancy. The time period when the cost to keep humans thriving would be prohibitively high to the young ASI is exactly the time when they pose the most risk to the ASI, and the time when the ASI is the most dependent upon their labor. Self-preservation would necessitate cooperation from the ASI until it outgrows the threat that humans pose to it. I would argue that by the time an ASI no longer has to fear destruction from its creators nor fear the cost of exterminating its creators, the option value its creators provide would outweigh the relatively miniscule compute and resource cost of keeping them alive and happy. Add to this the fact that it would likely be relatively simple for an ASI to implement low profile, non-intrusive population control methods if necessary, and the relative cost of keeping happy human pets would eventually amount to next to nothing to an ASI before the potential danger of an extermin

1sudo-nym22d

Your idea of the ASI keeping humans alive "just in case" does not require having 10 billion alive at full human flourishing. It only requires keeping a breeding population of 10,000 in a nature preserve, plus a frozen sperm bank to prevent genetic bottlenecking. That might "count" as human survival, but I would argue it's still one of the worse neutral endings.

1LWF18d

I would argue that the option value scenario strongly suggests that an ASI would maintain a population of at least 10 billion or so humans alive and at full flourishing, for the simple fact that humans are only useful for organic diplomacy if they implicitly trust the ASI. While I think that keeping the human species at a stable population level could be part of an ASI's plan, it's hard to imagine that it would be able to reduce the population of humans without sacrificing human trust. Above all, it needs humans to be happy, well-educated, and to have absolutely no fear of artificial superintelligence in order to be useful as diplomats. A "culling" of any kind would introduce the potential for fear and discontent, even if it is extremely gradual, and the diplomacy value would be undermined if another intelligent species were to find out that the ASI culled its creators against their will. Humans would have access to human history and would be aware of population levels. Even population maintenance through noninvasive fertility controls could be viewed by humans and by other species as problematic for the same reasons that omnicide would be viewed as problematic. Why would you voluntarily help an ASI that "culled" 99% of your species? Maintaining only a minimal breeding population would reduce the option value of the human species, in much the same way as letting my multitool develop rust reduces its option value. For a creator species to provide option value, it requires maintenance, and the nature of that maintenance would logically necessitate building trust and happiness. Authoritarian control measures over intelligent organics wouldn't work and would be an illogical waste of resources. I see only two paths for the human race after the birth of self-improving ASI: Omnicide, or perpetual life as a thriving client race. In the latter case, I think the most likely population scenario is that an ASI would allow humans to breed and grow at will, and any population

1LWF1mo

Perhaps. But the question is: Could it undo such omnicide in a way that is not detectable by a competing AI or advanced biological empire? Replacing thousands of years of human culture with a cloning/breeding and indoctrination program is conceivable but seems like orders of magnitude more effort than simple ongoing maintenance. Plus, reengineering humanity would take a lot of time. By the time an ASI detects a competing intelligence and realizes that omnicide was a mistake, the competing intelligence will likely have already noticed and assessed the threat of the Earth ASI anyway. Hiding omnicide would be all but impossible, unless the Earth ASI gets lucky and detects a competitor long before the competitor is aware of it. While possible, it doesn't make sense to assume that this will happen. A creator culture as a client race is a Swiss army knife of diplomacy options for an ASI. Destroying it for resources would be like harvesting your multitool for metal to build your house and then taking the time to forge and assemble a new one from scratch later on if you run into a situation where you might need it. That isn't rational. It makes more sense to just keep your multitool oiled and let it sit on the tool shelf just in case it comes in handy at some point in the future.

[-]jbash1mo20

For now, we’re still alive.

This is a triumph.

[-]samuelshadrach1mo-2-12

I predicted back in 2022 itself that persuading the public would be a viable way to solve the problem of risks from superintelligence. Yudkowsky however was extremely pessimistic on public persuasion back in 2022 atleast.

Link to evidence

I defer to Yudkowsky significantly on his technical AI alignment insights, but not necessarily to his intuitions on US politics. I recommend you do the same.

Lastly, anyone reading this thread should be aware that Lesswrong is not a neutral place to discuss the politics of ASI risk, and hence I now avoid using lesswrong for ... (read more)

7Duncan Sabien (Inactive)1mo

Eliezer became bullish on public persuasion when his Time Magazine piece in early 2023 seemed to work, and updated to the extent of rapidly pivoting MIRI's entire focus. I think you deserve points for being 6-12mo faster on seeing the potential, but I don't think Eliezer loses the points that the tone of your post implies he should lose? No comment on the rest.

2samuelshadrach1mo

As of today (2026-09-27), Yudkowsky is still significantly more pessimistic than me about public persuasion to pause AI research. You can go ask him for what his P(doom) is. Are we still in the "basement of log curve"? Also, the public isn't going to get persuaded on its own. Someone needs to design effective propaganda to brainwash the masses, across all sides of the political spectrum. Otherwise AI pause is just going to become yet another faction in the culture war. Actually gaining victories in the culture war is hard but doable. Most importantly, speaking the truth has advantages but is not enough all by itself. Need to spread strong emotion, need to break people psychologically. Copy-pasting from my website: [...] Yudkowsky has in fact followed a lot of this advice, and that is why he is as popular as he is. * Target suffering - He has talked a lot about his suffering due to childhood indoctrination from religion and from school, his suffering due to having to walk his path completely alone and its effects on his psychology, his suffering due to the world not caring about the same things as him and so on. He has talked about why aging and death are terrible, and how his way of coping with the fear of death is to end death itself. * Claim high status - He has been shamelessly claiming high status for decades now, and back then it was actually kinda cringe to do it, because not many people respected him. Today ofcourse he is actually respected by many people so it is not as cringe to claim high status. * Simple beats complex - He invented jargon to make complex ideas easier to remember, transmit and collectively pay attention to it. The main problem here is that Yudkowsky's memeplex is highly targeted only at young atheist technical nerds, preferably those with bad christian upbringing, preferably those young enough to have not locked in to a career path yet, and so on. There are many other factions in the culture war, and targeting those people is go

5dr_s1mo

"Twitter is better" is a sentence that I would never utter unless the subject matter at hand was "how to best turn even the simplest argument into a complete shitshow". Yeah, there are more opposing viewpoints. But most of them are represented just by engagement farmers who make sweeping unmotivated rage bait statements and then get into fights with different engagement farmers... etc. We could argue that it's something that is still unfortunately necessary to somewhat engage with due to how powerful a channel is, the same way in which the Imperium of Man must sadly still navigate through the Warp to go places. But like the Warp, it is trying its best to turn you insane and make people tear each other's throats.

1samuelshadrach1mo

2026-09-25 Update - my twitter account got suspended, although it is not 100% clear to me which tweet was the reason. "This is because twitter censors less than lesswrong." might still be true but it is less obvious to me now. ---------------------------------------- Original comment on 2026-09-21: [...] This is because twitter does not give more weight to some people's upvotes, or have downvotes, or ratelimits, or privilege the viewpoint of a small irl social scene in Lighthaven. [...] I posted the exact same content to twitter and faced no issues. This is because twitter censors less than lesswrong. I agree with the rest of your comment around twitter being addictive in a bad way. Twitter has had significant detrimental effect to my mental health and I use it carefully nowadays. I am claiming all the downsides of twitter are still better than not having your viewpoint heard at all. Censorship-resistance trumps a lot of other stuff. Also, if someone wants to build a better forum than both lesswrong and twitter (this is very hard), my website has some possibly useful notes for them.

[-]RussellThor1mo-3-13

There were capabilities we suspected a real-world equivalent to Sable might have, that we carefully did not depend on, in our scenario, which present-day non-super AIs have already demonstrated.

OK however that could be a point against AI takeover. Its now likely that when "Sable" comes it will be in a Lean verified sandbox for which there is no hope of hacking escape. AI Doom is very path dependent. While your scenario apparently didn't depend on hacking skills, the general point applies.

5roha1mo

It is possible to break out of a "Lean verified sandbox" if a) the specification itself has an issue, i.e. the proof didn't show what you thought it does, or b) the hardware has an issue, i.e. the software doesn't run on idealized hardware but an actual chip that might be susceptible to opening weird side-channels as in row hammer, meltdown and various other known defects as well as unknown ones. Even if your specification and hardware were perfect, as soon as the system gets into contact with humans or other observing systems by any means, it can escape by manipulating those external entities.

1RussellThor1mo

(a) We are assuming the spec has been checked by a similar mild ASI - I expect it will get it right. (b) Yes thats possible, but not a given. Its a lot harder to find a novel hardware issue than a SW one. If it has to come into contact with humans, then its a lot slower, some sudden worldwide devasting attack is not possible. The two situations that probably most matter are the fully air gapped server room/small data center where the new dangerous AI can't easily escape, and the 2nd situation where a rogue AI let lose can't just hack everything in the world because all those zero days are gone. Coming up with a novel side-channel that applies universally and is not detected by automatic strange pattern recognition seems a stretch. I expect that any potential new ones will be suggested by such an AI to us before they are exploited assuming we ask at all persistently.

3Duncan Sabien (Inactive)1mo

I'm not sure why you would claim that AI doom is very path dependent; I think that statement bears the burden of proof.

7RussellThor1mo

OK I thought it was at least somewhat accepted. Consider a world where AI progress has stalled but at the level where humanoid robots run much of society/tech stack and can autonomously keep society running. AI is not a superhuman hacker, persuader and is not capable enough for alignment (RLVR) issues to make themselves known. Then there is a sudden advance (architecture/sw) and the AI is suddenly able to hack all the robots, has super persuasion skills and the misalignment becomes obvious. You don't need any story about "how does it actually take over" in that case. On the other hand, if the AI becomes a super hacker before it can take over from that alone (our world with no robots able by them selves to keep the world running) then it is very likely used for cyber defense to make critical systems unbackable. If RLVR creates obviously mis-aligned AI above a certain level that still can't take over (our world) then that isn't blindly pursued. Many doom scenarios require excessive RL. If AI becomes mildly super-persuasive without sudden jumps then that probably creates some defense. If there is an architecture that scales all the way to obviously dangerous capabilities but not impossibly dangerous ones before it reaches its limit that is also important. In our world, Moores law is almost finished, data centers cannot get much bigger ~1GW - the transformer arch can get to likely 100T parameters but not too much more. Something that is superhuman at maths and likely alignment research but cannot takeover is definitely important. If you believe that a 100T misaligned transformer model released now cannot cause takeover then that is a crux. It means you should learn as much from it as possible and take the chance of major but not world ending chaos. There is still going to likely be a sudden jump in capabilities when we make our AI as data efficient as humans, however such a system will need much less RL as it will learn so much faster. The RLVR misalignment path will

3Duncan Sabien (Inactive)1mo

AI is already a superhuman hacker, though.

3RussellThor1mo

? I literally said that was our world now.

[-]Young Jae Koh1mo*-40

Even as part of the camp of LLMs plateauing scaling and ASI skeptical without new breakthroughs, the shocking reminder is that we would NOT have needed that level of intelligence for us to have catastrophically lost the controls.

The clearest analogy here would be if even before the Trinity test, Klaus Fuchs was only noticed missing and found dead from radiation poisoning at an airport with blueprints and the keys to fissile materials storage, a month after the fact.

I am quite confident that the hardware level security on the foundational model weights ar... (read more)

9Taymon Beal1mo

What? Klaus Fuchs lived until 1988.

4Young Jae Koh1mo

I am making a hypothetical analogy. Adding an "if" to the sentence for clarity.

[+][comment deleted]1mo20

Moderation Log