I particularly wish people would taboo the word "optimize" more often. Referring to a process as "optimization" papers over questions like:

• What feedback loop produces the increase or decrease in some quantity that is described as "optimization?" What steps does the loop have?
• In what contexts does the feedback loop occur?
• How might the effects of the feedback loop change between iterations? Does it always have the same effect on the quantity?
• What secondary effects does the feedback loop have?

There's a lot hiding behind the term "optimization," and I think a large part of why early AI alignment research made so little progress was because people didn't fully appreciate how leaky of an abstraction it is.

I think we have failed, thus far.  I'm sad about that. When I began posting in 2018, I assumed that the community was careful and trustworthy. Not easily would undeserved connotations sneak into our work and discourse. I no longer believe that and no longer hold that trust.

I empathize with this, and have complained similarly (e.g. here).

I have also been trying to figure out why I feel quite a strong urge to push back on posts like this one. E.g. in this case I do in fact agree that only a handful of people actually understand AI risk arguments well enough to avoid falling into "suggestive names" traps. But I think there's a kind of weak man effect where if you point out enough examples of people making these mistakes, it discredits even those people who avoid the trap.

Maybe another way of saying this: of course most people are wrong about a bunch of this stuff. But the jump from that to claiming the community or field has failed isn't a valid one, because the success of a field is much more dependent on max performance than mean performance.

Maybe it’s because people perceive me as an Optimist and therefore my points must be combated at any cost.

Maybe people really just naturally and unbiasedly disagree this much, though I doubt it.

But the end result is that I have given up on communicating with most folk who have been in the community longer than, say, 3 years.

Not saying that it's fun or even obviously net-positive for all participants, but I think combative communication is better than no communication, as far as truth-seeking goes.

To be frank, I think a lot of the case for AI accident risk comes down to a set of subtle word games.

Sure, but what if what's left is risky enough? Maybe utility maximization is a bad model of future AI (maybe because it's hard to predict the technology that doesn't exist yet) - but what's the alternative? Isn't labelling some empirical graph that ignores warning signs "awesomeness" and extrapolating is more of a word game?

Do you have some concrete examples where you've explained how some substantial piece of the case for AI accident risk is a matter of word games?

This is a pretty good essay, and I'm glad you wrote it. I've been thinking similar thoughts recently, and have been attempting to put them into words. I have found myself somewhat more optimistic and uncertain about my models of alignment due to these realizations.

Anyway, on to my disagreements.

It’s hard when you’ve[2] read Dreams of AI Design and utterly failed to avoid same mistakes yourself.

I don't think that "Dreams of AI Design" was an adequate essay to get people to understand this. These distinctions are subtle, and as you might tell, not an epistemological skill that comes native to us. "Dreams of AI Design" is about confusing the symbol with the substance -- '5 with 5, in Lisp terms, or the variable name five with the value of 5 (in more general programming language terms). It is about ensuring that all the symbols you use to think with actually are mapping onto some substance. It is not about the more subtle art of noticing that you are incorrectly equivocating between a pre-theoretic concept such as "optimization pressure" and the actual process of gradient updates. I suspect that Eliezer may have made at least one such mistake that may have made him significantly more pessimistic about our chances of survival. I know I've made this mistake dozens of times. I mean, my username is "mesaoptimizer", and I don't endorse that term or concept anymore as a way of thinking about the relevant parts of the alignment problem.

It’s hard when your friends are using the terms, and you don’t want to be a blowhard about it and derail the conversation by explaining your new term.

I've started to learn to be less neurotic about ensuring that people's vaguely defined terms actually map onto something concrete, mainly because I have started to value the fact that these vaguely defined terms, if not incorrectly equivocated, hold valuable information that we might otherwise lose. Perhaps you might find this helpful.

When I try to point out such (perceived) mistakes, I feel a lot of pushback, and somehow it feels combative.

I empathize with those pushing back, because to a certain extent it seems like what you are stating seems obvious to someone who has learned to translate these terms into the more concrete locally relevant formulations ad-hoc, and given such an assumption, it seems like you are making a fuss about something that doesn't really matter and in fact even reaching for examples to prove your point. On the other hand, I expect that ad-hoc adjustment to such terms is insufficient to actually do productive alignment research -- I believe that the epistemological skill you are trying to point at is extremely important for people working in this domain.

I'm uncertain about how confused senior alignment researchers are when it comes to these words and concepts. It is likely that some may have cached some mistaken equivocations and are therefore too pessimistic and fail to see certain alignment approaches panning out, or too optimistic and think that we have a non-trivial probability of getting our hands on a science accelerator. And deference causes a cascade of everyone (by inference or by explicit communication) also adopting these incorrect equivocations.

I agree with you that people get sloppy with these terms, and this seems bad. But there’s something important to me about holding space for uncertainty, too. I think that we understand practically every term on this list exceedingly poorly. Yes, we can point to things in the world, and sometimes even the mechanisms underlying them, but we don’t know what we mean in any satisfyingly general way. E.g. “agency” does not seem well described to me as “trained by reinforcement learning.” I don’t really know what it is well described by, and that's the point. Pretending otherwise only precludes us from trying to describe it better.

I think there's a lot of room for improvement in how we understand minds, i.e., I expect science is possible here. So I feel wary of mental moves such as these, e.g., replacing “optimal” with “set of sequential actions which have subjectively maximal expected utility relative to [entity X]'s imputed beliefs,” as if that settled the matter. Because I think it gives a sense that we know what we’re talking about when I don’t think we do. Is a utility function the right way to model an agent? Can we reliably impute beliefs? How do we know we’re doing that right, or that when we say ‘belief’ it maps to something that is in fact like a belief? What is a belief? Why actions instead of world states? And so on.

It seems good to aim for precision and gears-level understanding wherever possible. But I don’t want this to convince us that we aren’t confused. Yes, we could replace the “tool versus agent” debate with things like “was it trained via RL or not,” or what have you, but it wouldn’t be very satisfying because ultimately that isn’t the thing we’re trying to point at. We don’t have good definitions of mind-type things yet, and I don’t want us to forget that. 

My prescription for the problem you're highlighting here: track a prototypical example.

Trying to unpack e.g. "optimization pressure" into a good definition - even an informal definition - is hard. Most people who attempt to do that will get it wrong, in the sense that their proffered definition will not match their own intuitive usage of the term (even in cases where their own intuitive usage is coherent), or their implied usage in an argument. But their underlying intuitions about "optimization pressure" are often still be correct, even if those intuitions are not yet legible. Definitions, though an obvious strategy, are not a very good one for distinguish coherent-word-usage from incoherent-word-usage.

(Note that OP's strategy of unpacking words into longer phrases is basically equivalent to using definitions, for our purposes.)

So: how can we track coherence of word usage, practically?

Well, we can check coherence constructively, i.e. by exhibiting an example which matches the usage. If someone is giving an argument involving "optimization pressure", I can ask for a prototypical example, and then walk through the argument in the context of the example to make sure that it's coherent.

For instance, maybe someone says "unbounded optimization wants to take over the world". I ask for an example. They say "Well, suppose we have a powerful AI which wants to make as many left shoes as possible. If there's some nontrivial pool of resources in the universe which it hasn't turned toward shoe-making, then it could presumably make more shoes by turning those resources toward shoe-making, so it will seek to do so.". And then I can easily pick out a part of the example to drill in on - e.g. maybe I want to focus on the implicit "all else equal" and argue that all else will not be equal, or maybe I want to drill into the "AI which wants" part and express skepticism about whether anything like current AI will "want" things in the relevant way (in which case I'd probably ask for an example of an AI which might want to make as many left shoes as possible), or [whatever else].

The key thing to notice is that I can easily express those counterarguments in the context of the example, and it will be relatively clear to both myself and my conversational partner what I'm saying. Contrast that to e.g. just trying to use very long phrases in place of "unbounded optimization" or "wants", which makes everything very hard to follow.

In one-way "conversation" (e.g. if I'm reading a post), I'd track such a prototypical example in my head. (Well, really a few prototypical examples, but 80% of the value comes from having any.) Then I can relatively-easily tell when the argument given falls apart for my example.

Somewhat related: how do we not have separate words for these two meanings of 'maximise'?

1. literally set something to its maximum value
2. try to set it to a big value, the bigger the better

Even what I've written for (2) doesn't feel like it unambiguously captures the generally understood meaning of 'maximise' in common phrases like 'RL algorithms maximise reward' or 'I'm trying to maximise my income'. I think the really precise version would be 'try to affect something, having a preference ordering over outcomes which is monotonic in their size'.

But surely this concept deserves a single word. Does anyone know a good word for this, or feel like coining one?

I love this post. I have been one who has perceived your comments as potentially combative before. I have a hunch that it was an impression you were implicitly asserting "no, I should be the one who is trusted!" that set off alarm bells - and this post unambiguously communicates the opposite. In particular:

I think we have failed, thus far. I'm sad about that. When I began posting in 2018, I assumed that the community was careful and trustworthy. Not easily would undeserved connotations sneak into our work and discourse. I no longer believe that and no longer hold that trust.

I agree, and I generally think this has been a problem for a long time. I will believe it is possible to avoid this problem at scale when I meet a group of people who have empirically demonstrated themselves to have scaled it, and at the moment I don't know off the top of my head how I'd recognize that other than a years-long track record. Fields of mathematics? I suppose there are fields of science that seem to do at least acceptably. But I don't know of anyone doing really well without the help of mathematically precise definitions, and some more stuff besides. In the meantime,

... well, wait, hold on - I find myself not quite sure how to echo the sentiment I see in this post in a way I can agree I should implement for myself. Perhaps something along the lines of, I could return to my intermittent habit of avoiding words that have been heavily used. But perhaps more important than that is sticking to words whose meaning I can construct mathematically, in the sense of constructivist logic? I'm actually not sure the takeaway here is obvious in the first place now. Hmm.

I have given up on communicating with most folk who have been in the community longer than, say, 3 years

I suspect this should actually be something more like "longer than 3 but less than 10." (You're expressing resentment for the party line on AI risk, but "the community" wasn't always all about that! There used to be a vision of systematic methods for thinking more clearly.)

How many times has someone expressed "I'm worried about 'goal-directed optimizers', but I'm not sure what exactly they are, so I'm going to work on deconfusion."? There's something weird about this sentiment, don't you think?

I disagree, and I will take you up on this!

"Optimization" is a real, meaningful thing to fear, because:

• We don't understand human values, or even necessarily meta-understand them.
• Therefore, we should be highly open to the idea that a goal (or meta-goal) that we encode (or meta-encode) would be bad for anything powerful to base-level care about.
• And most importantly, high optimization power breaks insufficiently-strong security assumptions. That, in itself, is why something like "security mindset" is useful without necessarily thinking of a powerful AI as an "enemy" in war-like terms.
  • Here "security assumptions" is used in a broad sense, the same way that "writing assumptions" (the ones needed to design a word-processor software) could include seemingly-trivial things like "there is an input device we can access" and "we have the right permissions on this OS".

I'll add another one to the list: "Human-level knowledge/human simulator"

Max Nadeau helped clarify some ways in which this framing introduced biases into my and others' models of ELK and scalable oversight. Knowledge is hard to define and our labels/supervision might be tamperable in ways that are not intuitively related to human difficulty.

Different measurements of human difficulty only correlate at about 0.05 to 0.3, suggesting that human difficulty might not be a very meaningful concept for AI oversight, or that our current datasets for experimenting with scalable oversight don't contain large enough gaps in difficulty to make meaningful measurements.

I should note that all of that suggestive names came from mainstream AI academy/industry community. LW failure was mostly about uncritical acceptance of them, not invention.

I broadly agree that a lot of discussion about AI x-risk is confused due to the use of suggestive terms. Of the ones you've listed, I would nominate "optimizer", "mesa optimization", "(LLMs as) simulators", "(LLMs as) agents", and "utility" as probably the most problematic. I would also add "deception/deceptive alignment", "subagent", "shard", "myopic", and "goal". (It's not a coincidence that so many of these terms seem to be related to notions of agency or subcomponents of agents; this seems to be the main place where sloppy reasoning can slide in.) 

I also agree that I've encountered a lot of people who confidently predict Doom on the basis of subtle word games.

However, I also agree with Ryan's comment that these confusions seem much less common when we get to actual senior AIS researchers or people who've worked significantly with real models. (My guess is that Alex would disagree with me on this.) Most conversations I've been in that used these confused terms tended to involve MATS fellows or other very junior people (I don't interact with other more junior people much, unfortunately, so I'm not sure.) I've also had several conversations with people who seemed relieved at how reasonable and not confused the relevant researchers have been (e.g. with Alexander Gietelink-Oldenziel). 

I suspect that a lot of the confusions stem from the way that the majority of recruitment/community building is conducted -- namely, by very junior people recruiting even more junior people (e.g. via student groups). Not only is there only a very limited amount of communication bandwidth available to communicate with potential new recruits (and therefore encouraging more arguments by analogy or via suggestive words), the people doing the communication are also likely to use a lot of  (in large part because they're very junior, and likely not technical researchers).^[1] There's also historical reasons why this is the case: a lot of early EA/AIS people were philosophers, and so presented detailed philosophical arguments (often routing through longtermism) about specific AI doom scenarios that in turn were suffered lossy compression during communication, as opposed to more robust general arguments (e.g. Ryan Greenblatt's example of "holy shit AI (and maybe the singularity), that might be a really big deal").^[2] 

Similarly, on LessWrong, I suspect that the majority of commenters are not people who have deeply engaged with a lot of the academic ML literature or have spent significant time doing AIS or even technical ML work. 

And I'd also point a finger at lot of the communication from MIRI in particular as the cause for these confusions, e.g. the "sharp left-turn" concept seems to be primarily communicated via metaphor and cryptic sayings, while their communications about Reward Learning and Human Values seems in retrospect to have at least been misleading if not fundamentally confused. I suspect that the relevant people involved have much better models, but I think this did not come through in their communication. 

I'm not super sure what to do about it; the problem of suggestive names (or in general, of smuggling connotations into technical work) is not a unique one to this community, nor is it one that can be fixed with reading a single article or two (as your post emphasizes). I'd even argue this community does better than a large fraction of academics (even ML academics). 

John mentioned using specific, concrete examples as a way to check your concepts. If we're quoting old rationalist foundation texts, then the relevant example from "Surely You're Joking, Mr. Feynman" is relevant:

"I had a scheme, which I still use today when somebody is explaining something that I’m trying to understand: I keep making up examples. For instance, the mathematicians would come in with a terrific theorem, and they’re all excited. As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball) – disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’"

Unfortunately, in my experience, general instructions of the form "create concrete examples when listening to a chain of reasoning involving suggestive terms" do not seem to work very well, even if examples of doing so are provided, so I'm not sure there's a scalable solution here. 

My preferred approach is to give the reader concrete examples to chew on as early as possible, but this runs into the failure mode of contingent facts about the example being taken as a general point (or even worse, the failure mode where the reader assumes that the concrete case is the general point being made). I'd consider mathematical equations (even if they are only toy examples) to be helpful as well, assuming you strip away the suggestive terms and focus only on the syntax/semantics. But I find that I also have a lot of difficulty getting other people to create examples I'd actually consider concrete. Frustratingly, many "concrete" examples I see smuggle in even more suggestive terms or connotations, and sometimes even fail to capture any of the semantics of the original idea. 

So in the end, maybe I have nothing better than to repeat Alex's advice at the end of the post:

All to say: Do not trust this community's concepts and memes, if you have the time. Do not trust me, if you have the time. Verify.

At the end of the day, while saying "just be better" does not serve as actionable advice, there might not be an easier answer. 

1. ^{^}

   To be clear, I think that many student organizers and community builders in general do excellent work that is often incredibly underappreciated. I'm making a specific claim about the immediate causal reasons for why this is happening, and not assigning fault. I don't see an easy way for community builders to do better, short of abandoning specialization and requiring everyone to be a generalist who also does techncical AIS work. 

2. ^{^}

   That being said, I think that it's worth trying to make detailed arguments concretizing general concerns, in large part to make sure that the case for AI x-risk doesn't "come down to a set of subtle word games". (e.g. I like Ajeya's doom story. ) After all, it's worth concretizing a general concern, and making sure that any concrete instantiations of the concern are possible. I just think that detailed arguments (where the details matter) often get compressed in ways that end up depending on suggestive names, especially in cases with limited communication bandwith.

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

This post's ending seems really overdramatic.