"You can't possibly succeed without [My Pet Issue]"
20Charlie Steiner
2Sunny from QAD
17Raemon
10Bendini
2Raemon
2Raemon
9Bendini
2Raemon
18Bendini
4Raemon
3TurnTrout
3Raemon
6TurnTrout
3Bendini
New Comment
You know, I think my favorite thing about internet rationalists is when they notice a bias and go "I wonder if I can notice this in myself to avoid being wrong" rather than "How can I use this to win arguments about current hot topics."
I have no trouble believing that this is common thing to hear if you're in a position of power, but what about situations where this is correct? After all, if it was never correct, people would never find it persuasive.
Are there any heuristics you use to figure out when this is likely to be true?
Nod. The suggested tap of "build an actual model if you don't have one", or "doublecheck your model" (if you do), isn't meant to output "the statement is never true", just that you should check that you have a clear reason to believe it's true.
It hasn't been true the times I've noticed myself saying it.
I think it's more likely to be true in physical-system setups, where, like, your engine literally won't run if it doesn't have the right kind of fuel or whatever.
I think some instances have been a person posing a mathematical formalism and saying 'this must be true', and it was true in the mathematical example but not AFAICT in the real world analogue. (In this cases there's being some kind of Law/Toolbox conflation)
Ah.
My first reaction was thinking of a few scenarios that were analogous to the original framing, one example being "if it takes you years to coordinate the local removal of [obvious abuser], why do you think you will be able to coordinate safe AI development on a global scale?"
This isn't a pet issue of mine, but I suspect it is important to be able to say things like this. I guess my overall view is that crystallising this pattern might be putting ducttape over a more structural problem.
Recent motivating examples have been of the form "we can't possibly form good models and coordinate without X", to which I thought "WHAT!? X harms Y, and we can't possibly form good models and coordinate without Y". And it took me awhile to realize I was doing the same behavior that was annoying me.
(I think the answer is that often you need a deep understanding of both the Rock and the Hard Place before you can, hopefully, eventually, just eliminate the problem entirely)
I don't disagree with that, but I do think one reason we find it difficult to form good models and coordinate is that there's an insane norm of only ever talking about issues in abstract terms like X and Y. Maybe the issue in question here is super sensitive, since I have no idea what you are talking about, but "raising awareness of general patterms" often seems to be used as a (mostly subconscious) justification for avoiding the object level because it might make someone important look bad.
Usually when I'm avoiding addressing the object level it's
a) engaging with someone I consider to be in roughly the same strata of social status and position-of-power as I, and
b) I just don't want to get into that particular object level debate right now (either because it's exhausting, or distracting).
I think a notable exception is Healthy Competition, where I am in fact avoiding directly critiquing powers that be. I have a cluster of reasons I could point to there with varying degrees of virtuousness, but the unvirtuous ones are definitely there.
I think it might be worth having an example-generating TAP here instead. Instead of weighing off "weigh in on the sensitive / exhausting debate" vs "say things like ' affects in a double-causal-backflip-Goodhart manner'", one could just generate another concrete example?
I agree examples are good, but generating good ones is often fairly hard (and is the difference between being a post I could rattle off in 30 minutes vs one that'll take several hours)
I guess it just doesn't seem like examples should take that long? I also think that really good examples might make for a good part of the value in a few cases, but that's just a hunch.
For what it's worth, I think that post made the right tradeoff. There will probably be some people who will have glossed over it due to lack of examples, but in that case I think it was an acceptable price to pay.
What I'm referring to is when the community does this by default, not when the author has explicitly weighed up the pros and cons. Not wanting to get into an issue is okay in isolation, but when everyone does this it impedes the flow of information in ways that make it even more difficult to avoid talking past each other.