Speaking for myself, dunno if this is exactly what Eliezer meant:

The general rule of thumb is that if you want to produce a secure, complex artifact (in any field, not just computer science), you accomplish this by restricting the methods of construction, not by generating an arbitrary artifact using arbitrary methods and then "securing" it later.

If you write a piece of software in a nice formal language using nice software patterns, proving its security can often be pretty easy!

But if you scoop up a binary off the internet that was not written with this in mind, and you want to prove even minimal things about it, you are gonna have a really, really bad time.^[1]

So could there be methods that reliably generate "benign" ^[2] cognitive algorithms?^[3] Yes, likely so!

But are there methods that can take 175B FP numbers generated by unknown slop methods and prove them safe? Much more doubtful.

1. ^{^}

   In fact, it can often be basically completely impossible, even for simple problems! 

   For example, think of the Collatz Conjecture. It's an extremely simple statement about an extremely simple system that could easily pop up in a "messy" computational system... and currently we can't prove it, despite massive amounts of effort pouring into it over the years!

   What is the solution? Restrict your methods so they never generate artifacts that have "generalized Collatz problems" in them!

2. ^{^}

   As in, it's tractable for modern humans to prove their "safety"

3. ^{^}

   Probably not encoded as 175B floating point numbers...

As I understand it, the initial Yudkowskian conception of Friendly AI research^[1] was for a small, math- and science-inclined team that's been FAI-pilled to first figure out the Deep Math of reflective cognition (see the papers on Tiling Agents as an illustrative example: 1, 2). The point was to create a capability-augmenting recursive self-improvement procedure that preserves the initial goals and values hardcoded into a model (evidence: Web Archive screenshot of the SingInst webpage circa 2006). See also this:

When we try to visualize how all this is likely to go down, we tend to visualize a scenario that someone else once termed “a brain in a box in a basement.” I love that phrase, so I stole it. In other words, we tend to visualize that there’s this AI programming team, a lot like the sort of wannabe AI programming teams you see nowadays, trying to create artificial general intelligence, like the artificial general intelligence projects you see nowadays. They manage to acquire some new deep insights which, combined with published insights in the general scientific community, let them go down into their basement and work in it for a while and create an AI which is smart enough to reprogram itself, and then you get an intelligence explosion.

Then you would figure out a way to encode human values into machine code directly, compute (a rough, imperfect approximation of) humanity's CEV, and initialize a Seed AI with a ton of "hacky guardrails" (Eliezer's own term) aimed at enacting it. Initially the AI would be pretty dumb, but:

1. we would know precisely what it's trying to do, because we would have hardcoded its desires directly.
2. we would know precisely how it would develop, because our Deep Mathematical Knowledge about agency and self-improvement would have resulted in clear mathematical proofs of how it will preserve its goals (and thus its Friendliness) as it self-improved.
3. the hacky guardrails would ensure nothing breaks at the beginning, and as the model got better and its beliefs/actions/desires coherentized, the problems with the approximation of CEV would go away.

So the point is that we might not know the internals of the final version of the FAI; it might be "inscrutable." But that's ok, they said, because we'd know with the certainty of mathematical proof that its goals are nonetheless good.

From there on out, you relax, kick back, and plan the Singularity after-party.

1. ^{^}

   Which will likely seem silly and wildly over-optimistic to observers in hindsight, and in my view should have seemed silly and wildy-optimistic at the time too

I think the emphasis is on inscrutable. If you didn't already know how the deep learning tech tree would go, you could have figured out that Cyc-like hard-coding of "tires are black" &c. is not the way, but you might have hoped that the nature of the learning algorithm would naturally lend itself to a reasonably detailed understanding of the learned content: that the learning algorithm produces "concept" data-structures in this-and-such format which accomplish these-and-such cognitive tasks in this-and-such way, even if there are a billion concepts.

This is what I think he means:

The object-level facts are not written by or comprehensible to humans, no. What's comprehensible is the algorithm the AI agent uses to form beliefs and make decisions based on those beliefs. Yudkowsky often compares gradient descent optimizing a model to evolution optimizing brains, so he seems to think that understanding the outer optimization algorithm is separate from understanding the inner algorithms of the neural network's "mind".

I think what he imagines as a non-inscrutable AI design is something vaguely like "This module takes in sense data and uses it to generate beliefs about the world which are represented as X and updated with algorithm Y, and algorithm Z generates actions, and they're graded with a utility function represented as W, and we can prove theorems and do experiments with all these things in order to make confident claims about what the whole system will do."(The true design would be way more complicated, but still comprehensible.)

I suspect the crux here is whether or not you believe it's possible to have a "simple" model of intelligence. Intuitively, the question here is something like, "Does intelligence ultimately boil down to some kind of fancy logic? Or does it boil down to some kind of fancy linear algebra?"

The "fancy logic" view has a long history. When I started working as a programmer, my coworkers were veterans of the 80s AI boom and the following "AI winter." The key hope of those 80s expert systems was that you could encode knowledge using definitions and rules. This failed.

But the "fancy linear algebra" view pull ahead long ago. In the 90s, researchers in computational linguistics, computer vision and classification realized that linear algebra worked far better than fancy collections of rules. Many of these subfields leaped ahead. There were dissenters: Cyc continued to struggle off in a corner somewhere, and the semantic web tried to badly reinvent Prolog. The dissenters failed.

The dream of Cyc-like systems is eternal, and each new generation reinvents it. But it has systematically lost on nearly every benchmark of intelligence.

Fundamentally, real world intelligence has a number of properties:

1. The input is a big pile of numbers. Images are a pile of numbers. Sound is a pile of numbers.
2. Processing that input requires weighing many different pieces of evidence in complex ways.
3. The output of intelligence is a probability distribution. This is most obvious for tasks like speech recognition ("Did they say X? Probably. But they might have said Y.")

When you have a giant pile of numbers as input, a complex system for weighing those numbers, and a probability distribution as output, then your system is inevitably something very much like a giant matrix. (In practice, it turns out you need a bunch of smaller matrices connected by non-linearities.)

Before 2022, it appeared to me that Yudkowsky was trapped in the same mirage that trapped the creators of Cyc and the Semantic Web and 80s expert systems.

But in 2025, Yudkowsky appears to believe that the current threat absolutely comes from giant inscrutable matrices. And as far as I can tell, he has become very pessimistic about any kind of robust "alignment".

Personally, this is also my viewpoint: There is almost certainly no robust version of alignment, and even "approximate alignment will come under vast strain if we develop superhuman systems with goals. So I would answer your question in the affirmative: As far as I can see, inscrutability was always inevitable.

I think a natural way to think about this is ala AIXI, cleaving the system in two, prediction/world-modeling and values.

With this framing, I think most people would be fine with the world-model being inscrutable, if you can be confident the values are the right ones (and they need to be scrutable for this). I mean, for an ASI this kind of has to be the case. It will understand many thing about the world that we don't understand. But the values can be arbitrarily simple.

Kind of my hope for mechanistic interpretability is that we could find something isomorphic to a aixi (with inscrutable neural goop instead of turing machines) and then do surgery on the sensory reward part. And that this is feasible because 1) the aixi-like structure is scrutable 2) the values the AI has gotten from training probably will not be scrutable, but we can replace them with something that is, at least "structurally", scrutable.

I think some of the optimism about scrutability might derive from reductionism. Like, if you've got a scrutable algorithm for maintaining a multilevel map, and you've got a scrutable model of the chemistry of a tire, you could pass through the multilevel model to find the higher-level description of the tire.

the human-created source code must be defining a learning algorithm of some sort. And then that learning algorithm will figure out for itself that tires are usually black etc. Might this learning algorithm be simple and legible? Yes! But that was true for GPT-3 too

Simple first-order learning algorithms have types of patterns they recognize, and meta-learning algorithms also have types of patterns they like.

In order to make a friendly or aligned AI, we will have to have some insight into what types of patterns we are going to have it recognize, and separately what types of things it is going to like or find salient.

There was a simple calculation protocol which generated GPT-3. The part that was not simple was translating that into predicting its preferences or perceptual landscape, and hence what it would do after it was turned on. And if you can't predict how a parameter will respond to input, you can't architect it one-shot.