I agree that many terms are suggestive and you have to actually dissolve the term and think about the actual action of what is going on in the exact training process to understand things. If people don't break down the term and understand the process at least somewhat mechanistically, they'll run into trouble.
I think relevant people broadly agree about terms being suggestive and agree that this is bad; they don't particularly dispute this. (Though probably a bunch of people think it's less important than you do. I think these terms aren't that bad once you've worked with them in a technical/ML context to a sufficient extent that you detach preexisting conceptions and think about the actual process.)
But, this is pretty different from a much stronger claim you make later:
To be frank, I think a lot of the case for AI accident risk comes down to a set of subtle word games.
When I try to point out such (perceived) mistakes, I feel a lot of pushback, and somehow it feels combative. I do get somewhat combative online sometimes (and wish I didn't, and am trying different interventions here), and so maybe people combat me in return. But I perceive defensiveness even to the critiques of Matthew Barnett, who seems consistently dispassionate.
Maybe it's because people perceive me as an Optimist and therefore my points must be combated at any cost.
Maybe people really just naturally and unbiasedly disagree this much, though I doubt it.
I don't think that "a lot of the case for AI accident risk comes down to a set of subtle word games". (At least not in the cases for risk which seem reasonably well made to me.) And, people do really "disagree this much" about whether the case for AI accident risk comes down to word games. (But they don't disagree this much about issues with terms being suggestive.)
It seems important to distinguish between "how bad is current word usage in terms of misleading suggestiveness" and "is the current case for AI accident risk coming down to subtle work games". (I'm not claiming you don't distinguish between these, I'm just claiming that arguments for the first aren't really much evidence for the second here and that readers might miss this.)
For instance, do you think that this case for accident risk comes down to subtle word games? I think there are bunch of object level ways this threat model could be incorrect, but this doesn't seem downstream of word usage.
Separately, I agree that many specific cases for AI accident risk seem pretty poor to me. (Though the issue still doesn't seem like a word games issue as opposed to generically sloppy reasoning or generally having bad empirical predictions.) And then ones which aren't poor remain somewhat vague, though this is slowly improving over time.
So I basically agree with:
Do not trust this community's concepts and memes, if you have the time.
Edit: except that I do think the general take of "holy shit AI (and maybe the singularity), that might be a really big deal" seems pretty solid. And, from there I think there is a pretty straightforward and good argument for at least thinking about the accident risk case.
Ok so failure stops being the AI models coordinating a mass betrayal but goodharting metrics to the point that nothing works right. Not fundamentally different from a command economy failing where the punishment for missing quotas is gulag, and the punishment for lying on a report is gulag but later, so...
There's also nothing new about the failures, the USA incarceration rate is an example of what "trying too hard" looks like.
Let's not forget the old, well-read post: Dreams of AI Design. In that essay, Eliezer correctly points out errors in imputing meaning to nonsense by using suggestive names to describe the nonsense.
Artificial intelligence meets natural stupidity (an old memo) is very relevant to understanding the problems facing this community's intellectual contributions. Emphasis added:
Consider the following terms and phrases:
Some of these terms are useful. Some of the academic imports are necessary for successful communication. Some of the terms have real benefits.
That doesn't stop them from distorting your thinking. At least in your private thoughts, you can do better. You can replace "optimal" with "artifact equilibrated under policy update operations" or "set of sequential actions which have subjectively maximal expected utility relative to [entity X]'s imputed beliefs", and the nice thing about brains is these long sentences can compress into single concepts which you can understand in but a moment.
It's easy to admit the mistakes of our past selves (whom we've conveniently grown up from by time of recounting). It's easy for people (such as my past self and others in this community) to sneer at out-group folks when they make such mistakes, the mistakes' invalidity laid bare before us.
It's hard when you've[2] read Dreams of AI Design and utterly failed to avoid same mistakes yourself. It's hard when your friends are using the terms, and you don't want to be a blowhard about it and derail the conversation by explaining your new term. It's hard when you have utterly failed to resist inheriting the invalid connotations of other fields ("optimal", "reward", "attention mechanism").
I think we have failed, thus far. I'm sad about that. When I began posting in 2018, I assumed that the community was careful and trustworthy. Not easily would undeserved connotations sneak into our work and discourse. I no longer believe that and no longer hold that trust.
To be frank, I think a lot of the case for AI accident risk comes down to a set of subtle word games.
When I try to point out such (perceived) mistakes, I feel a lot of pushback, and somehow it feels combative. I do get somewhat combative online sometimes (and wish I didn't, and am trying different interventions here), and so maybe people combat me in return. But I perceive defensiveness even to the critiques of Matthew Barnett, who seems consistently dispassionate.
Maybe it's because people perceive me as an Optimist and therefore my points must be combated at any cost.
Maybe people really just naturally and unbiasedly disagree this much, though I doubt it.
But the end result is that I have given up on communicating with most folk who have been in the community longer than, say, 3 years. I don't know how to disabuse people of this trust, which seems unearned.
All to say: Do not trust this community's concepts and memes, if you have the time. Do not trust me, if you have the time. Verify.
See also: Against most, but not all, AI risk analogies.
How many times has someone expressed "I'm worried about 'goal-directed optimizers', but I'm not sure what exactly they are, so I'm going to work on deconfusion."? There's something weird about this sentiment, don't you think? I can't quite put my finger on what, and I wanted to get this post out.
Including myself, and I suspect basically every LW enthusiast interested in AI.