Ideally, people invoke analogies in order to make a point. And then readers / listeners will argue about whether the point is valid or invalid, and (relatedly) whether the analogy is illuminating or misleading. I think it’s really bad to focus discussion on, and police, the analogy target, i.e. to treat certain targets as better or worse, in and of themselves, separate from the point that’s being made.
For example, Nora was just comparing LLMs to mattresses. And I opened my favorite physics textbook to a random page and there was an prominent analogy between electromagnetic fields and shaking strings. And, heck, Shakespeare compared a woman to a summer’s day!
So when you ask whether AIs are overall more similar to aliens, versus more similar to LLMs, then I reject the question! It’s off-topic. Overall, mattresses & LLMs are very different, and electric fields & strings are very different, and women & summer’s days are very different. But there’s absolutely nothing wrong with analogizing them!
And if someone complained “um, excuse me, but I have to correct you here, actually, LLMs and mattresses are very different, you see, for example, you can sleep on mattresses but you can’t sleep on LLMs, and therefore, Nora, you should not be saying that LLMs are like mattresses”, then I would be very annoyed at that person, and I would think much less of him. (We’ve all talked to people like that, right?)
…And I was correspondingly unhappy to see this post, because I imagine it backing up the annoying-person-who-is-totally-missing-the-point from the previous paragraph. I imagine him saying “You see? I told you! LLMs really are quite different from mattresses, and you shouldn’t analogize them. Check this out, here’s a 2000-word blog post backing me up.”
Of course, people policing the target of analogies (separately from the point being made in context) is a thing that happens all the time, on all sides. I don’t like it, and I want it to stop, and I see this post as pushing things in the wrong direction. For example, this thread is an example where I was defending myself against analogy-target-policing last month. I stand by my analogizing as being appropriate and helpful in context. I’m happy to argue details if you’re interested—it’s a nuclear analogy :-P
I can’t speak to your experience, but some of my reactions to your account are:
For crying out loud, LLMs are already considered "AIs" by most people!
FWIW, you and me and everyone is normally trying to talk about “future AI that might pose an x-risk”, which everyone agrees does not yet exist. A different category is “AI that does not pose an x-risk”, and this is a very big tent, containing everything from Cyc and GOFAI to MuZero and (today’s) LLMs. So the fact that people call some algorithm X by the term “AI” doesn’t in and of itself imply that X is similar to “future AI that might pose an x-risk”, in any nontrivial way—it only implies that in the (trivial) ways that LLMs and MuZero and Cyc are all similar to each other (e.g. they all run on computers).
Now, there is a hypothesis that “AI that might pose an x-risk” is especially similar to LLMs in particular—much more than it is similar to Cyc, or to MuZero. I believe that you put a lot of stock in that hypothesis. And that’s fine—it’s not a crazy hypothesis, even if I happen personally to doubt it. My main complaint is when people forget that it’s a hypothesis, but rather treat it as self-evident truth. (One variant of this is people who understand how LLMs work but don’t understand how MuZero or any other kind of ML works, and instead they just assume that everything in ML is pretty similar to LLMs. I am not accusing you of that.)
I personally dislike most AI risk analogies that I've seen people use. While I think analogies can be helpful for explaining concepts to people and illustrating mental pictures, I think they are frequently misused, and often harmful. At the root of the problem is that analogies are consistently mistaken for, and often deliberately intended as arguments for particular AI risk positions. And a large fraction of the time[1] when analogies are used this way, I think they are misleading and imprecise, routinely conveying the false impression of a specific, credible model of AI, even when no such credible model exists.
Here is a random list of examples of analogies that I found in the context of AI risk (note that I'm not saying these are bad in every context):
These analogies cover a wide scope, and many of them can indeed sometimes be useful in conveying meaningful information. My point is not that they are never useful, but rather that these analogies can be shallow and misleading. The analogies establish almost nothing of importance about the behavior and workings of real AIs, but nonetheless give the impression of a model for how we should think about AIs.
And notice how these analogies can give an impression of a coherent AI model even when the speaker is not directly asserting it to be a model. Regardless of the speaker's intentions, I think the actual effect is frequently to plant a detailed-yet-false picture in the audience's mind, giving rise to specious ideas about how real AIs will operate in the future. Because the similarities are so shallow, reasoning from these analogies will tend to be unreliable.
A central issue here is that these analogies are frequently chosen selectively — picked on the basis of evoking a particular favored image, rather on the basis of identifying the most natural point of comparison possible. Consider this example from Ajeya Cotra,
Is there any reason why Cotra chose "chimpanzee" as the point of comparison when "golden retriever" would have been equally valid? It's hard to know, but plausibly, she didn't choose golden retriever because that would have undermined her general thesis.
I agree that if her goal was to convey the logical possibility of misalignment, then the analogy to chimpanzees works well. But if her goal was to convey the plausibility of misalignment, or anything like a "mental model" of how we should think of AI, I see no strong reason to prefer the chimpanzee analogy over the golden retriever analogy. The mere fact that one analogy evokes a negative image and the other evokes a positive image seems, by itself, no basis for any preference in their usage.
Or consider the analogy to human evolution. If you are trying to convey the logical possibility of inner misalignment, the analogy to human evolution makes sense. But if you are attempting to convey the plausibility of inner misalignment, or a mental model of inner misalignment, why not choose instead to analogize the situation to within-lifetime learning among humans? Indeed, as Quintin Pope has explained, the evolution analogy seems to have some big flaws:
Many proponents of AI risk already seem quite happy to critique plenty of AI analogies, such as the anthropomorphic analogy, or "it's like a toaster" and "it's like Google Maps". And of course, in these cases, we can easily identify the flaws:
To be clear, I agree Google Maps is a bad analogy, and it should rightly be criticized. But is the chimp analogy really so much better? Shouldn't we be applying the same degree of rigor against our own analogies too?
My point is not simply "use a different analogy". My point is that we should largely stop relying on analogies in the first place. Use detailed object-level arguments instead!
ETA: To clarify, I'm not against using analogies in every case. I'm mostly just wary of having our arguments depend on analogies, rather than detailed models. See this footnote for more information about how I view the proper use of analogies.[1]
While the purpose of analogies is to provide knowledge in place of ignorance — to explain an insight or a concept — I believe many AI risk analogies primarily misinform or confuse people rather than enlighten them; they can insert unnecessary false assumptions in place of real understanding. The basic concept they are intended to convey may be valuable to understand, but riding along with that concept is a giant heap of additional speculation.
Part of this is that I don't share other people's picture about what AIs will actually look like in the future. This is only a small part of my argument, because my main point is that that we should rely much less on arguments by analogy, rather than switch to different analogies that convey different pictures. But this difference in how I view the future still plays a significant role in my frustration at the usage of AI risk analogies.
Maybe you think, for example, that the alien and animal analogies are great for reasons that I'm totally missing. But it's still hard for me to see that. At least, let me compare my picture, and maybe you can see where I'm coming from.
Again: The next section is not an argument. It is a deliberately evocative picture, to help compare my expectations of the future against the analogies I cited above. My main point in this post is that we should move away from a dependence on analogies, but if you need a "picture" of what I expect from AI, to compare it to your own, here is mine.
The default picture, as I see it — the thing that seems to me like a straightforward extrapolation of current trends in 2024 into the medium-term future, as AIs match and begin to slightly exceed human intelligence — looks nothing like the caricatures evoked by most of the standard analogies. In contrast to the AIs-will-be-alien model, I expect AIs will be born directly into our society, deliberately shaped by us, for the purpose of filling largely human-shaped holes in our world. They will be socially integrated with us and will likely substantially share our concepts about the social and physical world, having been trained on our data and being fluent in our languages. They will be numerous and everywhere, interacting with us constantly, assisting us, working with us, and even providing friendship to hundreds of millions of people. AIs will be evaluated, inspected, and selected by us, and their behavior will be determined directly by our engineering.
I feel this picture is a relatively simple extension of existing trends, with LLMs already being trained to be kind and helpful to us, and collaborate with us, having first been shaped by our combined cultural output. I expect this trend of assimilation into our society will intensify in the foreseeable future, as there will be consumer demand for AIs that people can trust and want to interact with. Progress will likely be incremental rather than appearing suddenly with the arrival of a super-powerful agent. And perhaps most importantly, I expect oversight and regulation will increase dramatically over time as AIs begin having large-scale impacts.
It is not my intention to paint a picture of uniform optimism here. There are still plenty of things that can go wrong in the scenario I have presented. And much of it is underspecified because I simply do not know what the future will bring. But at the very least, perhaps you can now sympathize with my feeling that most existing AI risk analogies are deeply frustrating, given my perspective.
Again, I am not claiming analogies have no place in AI risk discussions. I've certainly used them a number of times myself. But I think they can, and frequently are used carelessly, and seem to regularly slip various incorrect illustrations of how future AIs will behave into people's mental models, even without any intent from the person making the analogy. In my opinion, it would be a lot better if, overall, we reduced our dependence on AI risk analogies, and in their place substituted them with specific object-level points.
To be clear, I'm not against all analogies. I think that analogies can be good if they are used well in context. More specifically, analogies generally serve one of three purposes:
1. Explaining a novel concept to someone
2. Illustrating, or evoking a picture of a thing in someone's head
3. An example in a reference class, to establish a base rate, or otherwise form the basis of a model
I think that in cases (1) and (2), analogies are generally bad as arguments, even if they might be good for explaining something. They're certainly not bad if you're merely trying to tell a story, or convey how you feel about a problem, or convey how you personally view a particular thing in your own head.
In case (3), I think analogies are generally weak arguments, until they are made more rigorous. Moreover, when the analogy is used selectively, it is generally misleading. The rigorous way of setting up this type of argument is to deliberately try to search for all relevant examples in the reference class, without discriminating in favor of ones that merely evoke your preferred image, to determine the base rate.