(Thought I'd try posting here some various rationality "quick thoughts" that I feel like I haven't seen expressed overtly enough -- sometimes I've already written about these elsewhere, but thought would be good to get them down here. Not claiming any originality here.)

When arguing with someone who says we need X, it's important to make sure you know whether they're arguing for X as policy or X as phenomenon. (And it's important to make sure you know which you're arguing for!) By "X as policy", I mean, a policy of directly doing X; by "X as phenomenon", I mean, the phenomenon of X regardless of what policy achieves it. I have noticed many policy arguments where two people talk past each other because they are not clear on this distinction.

(And once they do realize it, they get into terminology arguments instead of tabooing and. I'm hoping introducing this terminology might help avoid that.)

Of course, sometimes people haven't carefully thought about what they want and don't actually mean either of these. Sometimes it's actually just the case that X is a feature of their desired outcome -- OK, actually, strictly speaking that's always true, but let's assume we're implicitly talking within some sort of reasonable bounds so that there are cases where we can reasonably say that it's false.

In this case, optimizing for X (again, within reasonable bounds, otherwise it's true in all cases a person can quickly express!) is unlikely to actually satisfy the person, due to Goodhart's law. That X is a feature of the optimal solution does not mean that one can achieve an optimal solution by focusing on X! Always focus on correctness, not on particular features, even if you are sure a correct solution must have that feature (maybe this is reasonable when judging correctness is very difficult? But be careful with this is my point).

This is all also somewhat related to what I've seen called the "AB problem". (Side note: I've seen multiple people refer to it as this, but I can't seem to find the original source. Anybody know where this term came from? It's not the same as the XY problem, although I suppose there is a relation.)

The AB problem is a problem that occurs in arguments where Yvonne argues for A, Xavier believes A=>B, and so Xavier asserts that Yvonne is arguing for B. But Yvonne does not believe A=>B and is not arguing for B. (Frequently the problem occurs when Xavier believes in the link so strongly or implicitly that they have not clearly distinguished A from B in their mind in the first place.)

...hm, is it that related? OK, maybe it's actually not. But I wanted to mention the AB problem in here because it's a similar pitfall in argument, that is also common, and that I don't recall seeing discussed much here. Also I really want to know if anyone knows where the term came from!