Here's what I'd consider some comparatively important high-level criticisms I have of AI-2027, that I am at least able to articulate reasonably well without too much effort.

1

At some point, I believe Agent-4, the AI created by OpenBrain starts to be causally connected over time. That is, unlike current AIs that are temporally ephemeral (my current programming instance of Claude has no memories with the instance I used a week ago) and causally unconnected between users (my instance cannot use memories from your instance), it is temporally continuous and causally connected. There is "one AI" in a way there is not with Claude 3.7 and o3 and so on.

Here are some obstacles to this happening:

1. This destroys reproducibility, because the programming ability you have a week ago is different than the ability two weeks ago and so on. But reliability / reproducibility is extremely desirable from a programming perspective, and a very mundane reliability / troubleshooting perspective (as well as from a elevated existential risk perspective). So I think it's unlikely companies are going to do this.
2. Humans get worse at some tasks when they get better at others. RL finetuning of LLMs makes them better at some tasks while they get worse at others. Even adding more vectors to a vector DB can squeeze out another nearest neighbor and make it better at one task and worse at others. It would be a... really really hard task to ensure that a model doesn't get worse, on some tasks.
3. No one's working on anything like this. OpenAI has added memories, but it's mostly kind of a toy and I know a lot of people have disabled it.

So I don't think that's going to happen. I expect AIs to remain "different." The ability to restart AIs at will just has too many benefits, and continual learning seems too weakly developed, to do this. Even if we do have continual learning, I would expect more disconnection between models -- i.e., maybe people will build up layers of skills in models in Dockerfile-esque layers, etc, which still falls short of being one single model.

2

I think that Xi Jingping's actions are mostly unmotivated. To put it crudely, I feel like he's acting like Daniel Kokotajlo with Chinese characteristics rather than himself. It's hard to put my finger on one particular thing, but things that I recollect disagreeing with include things like:

(a) Nationalization of DeepCent was, as I recall, was vaguely motivated, but it was hinted that it was moved by lack of algorithmic progress. But the algorithmic-progress difference between Chinese models and US models at this point is like.... 0.5x. However, I expect that (a1) the difference between well run research labs and poorly run research labs can be several times larger than 0.5x, so this might come out in the wash and (a2) this amount of difference will be, to the state apparatus, essentially invisible. So that seems unmotivated.

(b) In general, it doesn't actually seem to think about reasons why China would continue open-sourcing things. The supplementary materials don't really motivate the closure of the algorithms; and I can't recall anything in the narrative that asks why China is open sourcing things right now. But if you don't know why it's doing what it's doing now, how can you tell why it's doing what it's doing in the future?

Here are some possible advantages to open sourcing things to China, from their perspective.

(b1) It decreases investment available to Western companies. That is, by releasing models near the frontier, open sourcing decreases future anticipated profit flow to Western companies, because they have a smaller delta of performance from cheaper models. This in turn means Western investment funds might be reluctant to invest in AI -- which means less infrastructure will be built in the West. China, by contrast, and infamously, will just build infrastructure even if it doesn't expect oversized profits to redound to any individual company.

(b2) Broad diffusion of AI all across the world can be considered a bet on complementarity of AI. That is, if it should be the case that the key to power is not just "AI alone" but "industrial power and AI" then broad and even diffusion of AI will redound greatly to China's comparative benefit. (I find this objectively rather plausible, as well as something China might think.)

(b3) Finally, geopolitically, open sourcing may be a means of China furthering geopolitical goals. China has cast itself in recent propaganda as more rules-abiding than the US -- which is, in fact, true in many respects. It wishes to cast the US as unilaterally imposing its will on others -- which is again, actually true. The theory behind the export controls from the US, for instance, is explicitly justified by Dario and others as allowing the US to seize control over the lightcone; when the US has tried to impose import controls on others, it has provided to those excluded from power literally no recompense. So open sourcing has given China immense propaganda wins, by -- in fact accurately, I believe -- depicting the US as being a grabby and somewhat selfish entity. Continuing to do this may seem advantageous.

Anyhow -- that's what I have. I have other disagreements (i.e., speed; China might just not be behind; etc) but these are... what I felt like writing down right now.

One premise in high-doom stories seems to be "the drive towards people making AIs that are highly capable will inevitably produce AIs that are highly coherent."

(By "coherent" I (vaguely) understand an entity (AI, human, etc) that does not have 'conflicting drives' within themself, that does not want 'many' things with unclear connections between those things, one that always acts for the same purposes across all time-slices, one that has rationalized their drives and made them legible like a state makes economic transactions legible.)

I'm dubious of this premise for a few reasons. One of the easier to articulate ones is an extremely basic analogy to humans.

Here are some things a human might stereotypically do in the pursuit of high ability-to-act in the world, as it happens in humans:

• Try to get money through some means
• Try to become close friends with powerful people
• Take courses or read books about subject-matters relevant to their actions
• Etc

And here are some things a human might stereotypically do while pursuing coherence.

• Go on a long walk or vacation reflecting on what they've really wanted over time
• Do a bucketload of shrooms
• Try just some very different things to see if they like them
• Etc

These are very different kinds of actions! It seems like for humans, the kind of action that makes you "capable" differs a fair bit from the kind of action that makes you "coherent." Like maybe they aren't entirely orthogonal... but some of them actually appear opposed? What's up with that!?

This is not a knock-down argument by any means. If there were some argument from an abstract notion of intelligence, that had been connected to actual real intelligences through empirical experiment, which indicated that greater intelligence => greater coherence, I'd take such an argument over this any day of the week. But to the best of my knowledge there is no such argument; there are arguments that try to say well, here's a known-to-be-empirically-flawed notion of "intelligence" that does tend to lead to greater "coherence" as it gets greater, but the way this actually links up to "intelligence" as a real thing is extremely questionable.

Some additional non-conclusive considerations that incline me further in this direction:

• "Coherence" in an intellect is fundamentally knowledge of + modification of self. Capabilities in an intellect is mostly... knowledge of the world. In a creature with finite compute relative to the world (i.e., all creatures, including creatures with 100x more compute than current AIs) you're gonna have a tradeoff between pursuing these kinds of things.
• "Coherence" in humans seems to be a somewhat interminable problem, emprically. Like (notoriously) trying to find total internal coherence can just take your whole life, and the people who pursue it may accomplish literally nothing else?

Abstractly, I think "coherence" in an entity is a fundamentally extremely hard thing to accomplish because of the temporal structure of learned intelligence in connectionist systems. All intelligent things we have seen so far (humans + LLM) start off doing massive supervised learning + RL from other entities, to bootstrap them up to the ability to act in the world. (Don't think school; think infancy and childhood.) The process of doing this gives (children / LLMs) the ability to act in the world, at the price of being a huge tangled bundle of learned heuristics that are fundamentally opaque to the entity and to everyone else. We think about this opacity differently (for humans: "why am I like that?," every species of psychology, the constant adoption of different narratives to make sense of one's impulses, the difference in how we think of our actions and others actions -- for AIs: well you got the whole "black box" and shoggoth spiel) but it's just a reflection of the fact that you had to be trained with a vast bundle of shards and impulses to act in the world, long before you had the capacity or time to reflect on them.

(And what would it mean to disentangle them, even? They're all contextually activated heuristics; the process of goal-directed tree search for a goal does not lie in your weights or in an LLM's weights. I don't think it's an accident that the most credible religion of Buddhism basically encourages you to step back from the whole thing, remove identification with all contexts, and do literally nothing -- probably the only way to actually remove conflict.)

Anyhow, those were some further considerations why I it seems dubious to me that we're going to get coherent entities from trying to get capable entities. These are not the only considerations one might make, nor are they comprehensive.

When I run my inner-MIRI against this model -- well, Yudkowsky insults me, as always happens when I run my inner-MIRI -- but I think them most coherent objection I get is that we should not expect coherent entities but coherent processes.

Like, granted that neither the weights of an LLM nor the brains of a human will tend towards coherence under training for capbility, but whatever LLM-involved process or human-neuron involved process tends for some goal will nevertheless tend towards coherence. That analogically, we shouldn't expect the weights of an LLM to have some kind of coherence but we should expect that the running-out-of-some-particular-rollout-of-an-LLM-to-so-tend.

And like, this strikes me as more plausible? It doesn't appear inevitable -- like, there's a lot of dynamics one could consider? -- but it makes more sense.

But like, if that is the case, then, maybe we would want to focus less on the goals-specific-to-the-LLM? Like my understanding of a lot of threat models is that they're specifically worried about weights-of-the-LLMs-tending-towards coherence. That that's the entity to which coherence is to be attributed, rather than the rollout.

And if that were false, then that's great! It seems like it would be good news and we could focus on other threat models. Idk.

</written_quickly>

What's that part of planecrash where it talks about how most worlds are either all brute unthinking matter, or full of thinking superintelligence, and worlds that are like ours in-between are rare?

I tried both Gemini Research and Deep Research and they couldn't find it, I don't want to reread the whole thing.

Lighthaven clearly needs to get an actual Gerver's sofa particularly if the proof that it's optimal comes through.

It does look uncomfortable I'll admit, maybe it should go next to the sand table.

Just a few quick notes / predictions, written quickly and without that much thought:

(1) I'm really confused why people think that deceptive scheming -- i.e., a LLM lying in order to post-deployment gain power -- is remotely likely on current LLM training schemes. I think there's basically no reason to expect this. Arguments like Carlsmith's -- well, they seem very very verbal and seems presuppose that the kind of "goal" that an LLM learns to act to attain during contextual one roll-out in training is the same kind of "goal" that will apply non-contextually to the base model apart from any situation.

(Models learn extremely different algorithms to apply for different parts of data -- among many false things, this argument seems to presuppose a kind of unity to LLMs which they just don't have. There's actually no more reason for a LLM to develop such a zero-context kind of goal than for an image segmentation model, as far as I can tell.)

Thus, I predict that we will continue to not find such deceptive scheming in any models, given that we train them about like how we train them -- although I should try to operationalize this more. (I understand Carlsmith / Yudkowsky / some LW people / half the people on the PauseAI discord to think something like this is likely, which is why I think it's worth mentioning.)

(To be clear -- we will continue to find contextual deception in the model if we put it there, whether from natural data (ala Bing / Sydney / Waluigi) or unnatural data (the recent Anthropic data). But that's way different!)

(2). All AI systems that have discovered something new have been special-purpose narrow systems, rather than broadly-adapted systems.

While "general purpose" AI has gathered all the attention, and many arguments seem to assume that narrow systems like AlphaFold / materials-science-bot are on the way out and to be replaced by general systems, I think that narrow systems have a ton of leverage left in them. I bet we're going to continue to find amazing discoveries in all sorts of things from ML in the 2020s, and the vast majority of them will come from specialized systems that also haven't memorized random facts about irrelevant things. I think if you think LLMs are the best way to make scientific discoveries you should also believe the deeply false trope from liberal arts colleges about a general "liberal arts" education being the best way to prepare for a life of scientific discovery. [Note that even systems that use non-specialized systems as a component like LLMs will themselves be specialized].

LLMs trained broadly and non-specifically will be useful, but they'll be useful for the kind of thing where broad and nonspecific knowledge of the world starts to be useful. And I wouldn't be surprised that the current (coding / non-coding) bifurcation of LLMs actually continued into further bifurcation of different models, although I'm a lot less certain about this.

(3). The general view that "emergent behavior" == "I haven't looked at my training data enough" will continue to look pretty damn good. I.e., you won't get "agency" from models scaling up to any particular amount. You get "agency" when you train on people doing things.

(4) Given the above, most arguments about not deploying open source LLMs look to me mostly like bog-standard misuse arguments that would apply to any technology. My expectations from when I wrote about ways AI regulation could be bad have not changed for the better, but for the much much worse.

I.e., for a sample -- numerous orgs have tried to outlaw open source models of the kind that currently exist because because of their MMLU scores! If you think are worried about AI takeover, and think "agency" appears as a kind of frosting on top of of a LLM after it memorizes enough facts about the humanities and medical data, that makes sense. If you think that you get agency by training on data where some entity is acting like an agent, much less so!

Furthermore: MMLU scores are also insanely easy to game, both in the sense that a really stupid model can get 100% by just training on the test set; and also easy to game, in the sense that a really smart model could get almost arbitrarily low by excluding particular bits of data or just training to get the wrong answer on the test set. It's the kind of rule that would be goodhearted to death the moment it came into existence -- it's a rule that's already been partially goodhearted to death -- and the fact that orgs are still considering it is an update downward in the competence of such organizations.

I think if you're a rationalist -- if you value truth, and coming to truth through the correct procedure -- then you should strongly dislike lengthy analogies that depict one's ideological opponents repeatedly through strawmen / weakman arguments.