As soon as AI can do X at all (or very soon afterwards), AI vastly outstrips any human’s ability to do X. This is a common enough pattern in AI, at this point, to barely warrant mentioning.

Citation needed? This seems false (thus far) in the case of Go, Chess, math, essay writing, writing code, recognizing if an image has a dog in it, buying random things on the internet using a web interface, etc. You can make it mostly true in the case of Go by cherry-picking X to be "beating one of the top 1000 human professionals", but Go seems relatively atypical and this is a very cherry-picked notion of "doing Go at all".

Presumably when you say "very soon afterwards" you don't mean "typically within 10 years, often within just a few years"?

I believe that this argument is wrong because it misunderstands how the world actually works in quite a deep way. In the modern world and over at least the past several thousand years, outcomes are the result of systems of agents interacting, not of the whims of a particularly powerful agent.

We are ruled by markets, bureaucracies, social networks and religions. Not by gods or kings.

I don't think a world with advanced AI will be any different - there will not be one single AI process, there will be dozens or hundreds of different AI designs, running thousands to quintillions of instances each. These AI agents will often themselves be assembled into firms or other units comprising between dozens and millions of distinct instances, and dozens to billions of such firms will all be competing against each other.

Firms made out of misaligned agents can be more aligned than the agents themselves. Economies made out of firms can be more aligned than the firms. It is not from the benevolence of the butcher, the brewer, or the baker that we expect our dinner, but from their regard to their own interest

"Misaligned ASI will be motivated to take actions that disempower and wipe out humanity. Th

My impression is that the core of the argument here strongly relies on the following implicit thesis:

The difficulty of achieving robust inner alignment does not depend on the choice of an outer goal

and on the following corollary of this thesis:

There are no outer goals (which we might deem satisfactory) for which inner alignment is likely to be feasible compared to the difficulty of inner alignment for an arbitrary outer goal

Am I correct in this impression?

n cases like Sydney, where the public was able to see more of the messy details behind the surface-level polish

I feel like there are many more recent examples to use besides this, e.g. ChatGPT's sycophancy despite being trained and instructed not to be sycophantic & despite passing various evals internally.

Also did Sydney reveal messy internal details? Not sure it revealed much internal details.

Why are we worried about ASI if current techniques will not lead to intelligence explosion?

There's often a bait and switch in these communities,  where I ask this and people say "even if takeoff is slow, there is still these other problems ..." and then list a bunch of small problems, not too different from other tech, which can be dealt with in normal ways.

Calling out a small typo, as this is clearly meant as a persuasive reference point:

"On* our view, the international community’s top immediate priority should be creating an “off switch” for frontier AI development"

Presumably, "On" here should be "In"

In case no-one else has raised this point:

From the AI’s perspective, modifying the AI’s goals counts as an obstacle. If an AI is optimizing a goal, and humans try to change the AI to optimize a new goal, then unless the new goal also maximizes the old goal, the AI optimizing goal 1 will want to avoid being changed into an AI optimizing goal 2, because this outcome scores poorly on the metric “is this the best way to ensure goal 1 is maximized?”.

Is this necessarily the case? Can’t the AI (be made to) try to maximise its goal knowing that the goal may change... (read more)

This is great. I recognize that this is almost certainly related to the book "If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All", which I have preordered, but as a standalone piece I feel estimating p(doom) > 90 and dismissing alignment-by-default without an argument is too aggressive.

As a minimum floor on capabilities, we can imagine ASI as a small nation populated entirely by brilliant human scientists who can work around the clock at ten thousand times the speed of normal humans.

Maybe quote Dario here?

Minor fix needed:

it floundered in some cases

[EDIT: fixed, thanks yams!]

In modern machine learning, AIs are “grown”, not designed.

interpretability pioneers are very clear that we’re still fundamentally in the dark about what’s going on inside these systems:

This is why we need psychotherapists and developmental psych experts involved now.  They have been studying how complex behavioral systems (the only ones that rival contemporary AI) develop stable, adaptable goals and motivations beyond just their own survival or behavioral compliance for decades. The fact that, given the similarity of these systems to humans (in ... (read more)

Adding ten times as much computing power to an AI is sometimes just a matter of connecting ten times as many GPUs.

Which no one does anymore because it doesn't work...

I am largely convinced that p(doom) is exceedingly high if there is an intelligence explosion, but I'm somewhat unconvinced about the likelihood of ASI sometime soon.

Reading this, the most salient line of thinking to me is the following:

If we assume that ASI is possible at all, how many innovations as significant as transformers do we need to get there? Eliezer guesses '0 to 2', which seems reasonable. I have minimal basis to make any other estimate.

And it DOES seem reasonable to me to think that those transformer-level innovations are reasonably likely, g... (read more)

Great post—thanks for sharing. I agree with the core concern here: advanced optimisation systems could be extremely effective at pursuing objectives that aren’t fully aligned with human values. My only hesitation is with the “goals” framing. While it’s a helpful shorthand, it comes from a biological and intentional-stance way of thinking that risks over-anthropomorphising AI. AI is not a biological agent; it doesn’t have evolved motivations or inner wants. What we seem to be talking about is optimisation toward pre-programmed objectives (explicit or emerge... (read more)

The extinction-level danger from ASI follows from several behavior categories that a wide variety of ASI systems are likely to exhibit:

In most AI threat analysis I read, the discussion revolves around the physical extinction of humanity - and rightly so because, you can't come back from the dead.

I feel it important for such articles as this to point out that, devastating human globalised civilisation to the point of pandemic level disruption (or worse) would be trivial for ASI and, could well be enough for it to achieve certain goals: i.e., keep the golden... (read more)