I think I agree with the thrust of this, but I think the comment section raises caveats that seem important. Scott's acknowledged that there's danger in this, and I hope an updated version would put that in the post.

But also...

Steven Pinker is a black box who occasionally spits out ideas, opinions, and arguments for you to evaluate. If some of them are arguments you wouldn’t have come up with on your own, then he’s doing you a service. If 50% of them are false, then the best-case scenario is that they’re moronically, obviously false, so that you can reject them quickly and get on with your life.

This seems like a strange model to use. We don't know, a priori, what % are false. If 50% are obviously false, probably most of the remainder are subtly false. Giving me subtly false arguments is no favor.

Scott doesn't tell, us, in this essay, what Steven Pinker has given him / why Steven Pinker is ruled in. Has Steven Pinker given him valuable insights? How does Scott know they're valuable? (There may have been some implicit context when this was posted. Possibly Scott had recently reviewed a Pinker book.)

Given Anna's example,

Julia Galef helpfully notes a case where Steven Pinker straightforwardly misrepresents basic facts about who said what. This is helpful to me in ruling out Steven Pinker as someone who I can trust not to lie to me about even straightforwardly checkable facts.

I find myself wondering, has Scott checked Pinker's straightforwardly checkable facts?

I wouldn't be surprised if he has. The point of these questions isn't to say that Pinker shouldn't be ruled in, but that the questions need to be asked and answered. And the essay doesn't really acknowledge that that's actually kind of hard. It's even somewhat dismissive; "all you have to do is *test* some stuff to *see if it’s true*?" Well, the Large Hadron Collider cost €7.5 billion. On a less extreme scale, I recently wanted to check some of Robert Ellickson's work; that cost me, I believe, tens of hours. And that was only checking things close to my own specialty. I've done work that could have ruled him out and didn't, but is that enough to say he's ruled in?

So this advice only seems good if you're willing and able to put in the time to find and refute the bad arguments. Not only that, if you actually will put in that time. Not everyone can, not everyone wants to, not everyone will do. (This includes: "if you fact-check something and discover that it's false, the thing doesn't nevertheless propagate through your models influencing your downstream beliefs in ways it shouldn't".)

If you're not going to do that... I don't know. Maybe this is still good advice, but I think that discussion would be a different essay, and my sense is that Scott wasn't actually trying to give that advice here.

In the comments, cousin_it and gjm describe the people who can and will do such work as "angel investors", which seems apt.

I feel like right now, the essay is advising people to be angel investors, and not acknowledging that that's risky if you're not careful, and difficult to do carefully. That feels like an overstep. A more careful version might instead advise:

• Some people have done some great work and some silly work. If you know which is which (e.g. because others have fact checked, or time has vindicated), feel free to pay attention to the great and ignore the silly.
• Don't automatically dismiss people just because they've said some silly things. Take that fact into account when evaluating the things they say that aren't obviously silly, and deciding whether to actually evaluate them. But don't let that fact take the place of actually evaluating those things. Like, given "Steven Pinker said obviously silly things about AI", don't say "... so the rest of The Nurture Assumption isn't worth paying attention to". Instead, say "... so I don't think it's worth me spending the time to look closer at The Nurture Assumption right now". And allow for the possibility of changing that to "... but The Nurture Assumption is getting a lot of good press, maybe I'll look into it anyway".

(e: lightly edited for formatting and content)

(After seeing this article many times, I only now realized that the title is supposed to be interpreted "thinkers should be ruled in, not out" rather than "there's this class of rule thinkers who are in, not out")

This seems in the same vein as the Epistemic Tenure post, and looks to be calling for an even lower bar than was suggested there. In the main, I agree.

You know, at first when I saw this post I was like "ugh, right, lots of people make gross mistakes in this area" but then didn't think much of it, but by coincidence today I was prompted to read something I wrote a while ago, and it seems relevant to this topic. Here's a quote from the article that was on a somewhat different topic (hermeneutics):

One methodology I’ve found especially helpful has been what I, for a long time, thought of as literary criticism but for interpreting what people said as evidence about what they knew about reality. I first started doing this when reading self-help books. Many books in that genre contain plainly incorrect reasoning based on outdated psychology that has either been disproved or replaced by better models (cf. Jeffers, Branden, Carnegie, and even Covey). Despite this, self-help still helps people. To pick on Jeffers, she goes in hard fordaily self-affirmation, but even ignoring concerns with this line of researchraised by the replication crisis, evidence suggests it’s unlikely to help much toward her instrumental goal of habit formation. Yet she makes this error in the service of giving half of the best advice I know: feel the fear and do it anyway. The thesis that she is wrong because her methods are flawed contradicts the antithesis that she is right because her advice helps people, so the synthesis must lie in some perspective that permits her both to be wrong about the how and right about the what simultaneously.

My approach was to read her and other self-help more from the perspective of the author and the expected world-view of their readers than from my own. This lead me to realize that, lacking better information about how the human mind works but wanting to give reasons for the useful patterns they had found, self-help authors often engage in rationalization to fit current science to their conclusions. This doesn’t make their conclusions wrong, but it does hide their true reasoning which is often based more on capta than data and thus phenomenological rather than strictly scientific reasoning. But given that they and we live in an age of scientism, we demand scientific reasons of our thinkers, even if they are poorly founded and later turn out to be wrong, or else reject their conclusions for lack of evidence. Thus the contradiction is sublimated by understanding the fuller context of the writing.

This makes a simple and valuable point. As discussed in and below Anna's comment, it's very different when applied to a person who can interact with you directly versus a person whose works you read. But the usefulness in the latter context, and the way I expect new readers to assume that context, leads me to recommend it.

This is a phrase I use regularly whenever this subject comes up, and it helped me think about this topic much more clearly.

This post seems like good life advice for me and people like me, when taken with appropriate caution. It's well-written too, of course.

Nominating because adopting this principle helps to create a positive intellectual culture.

I get the point, and it's a good one. We should have tolerance for the kooky ideas of those who brought us great things.

Practically though, I expect the oracle box would gather dust in the corner if it had a 1% hit rate and the experiments it required were expensive and it didn't come with an impressive warranty. The world abounds with cranks and it is all too easy for out of the box to get binned with unhinged.

So I think this is a useful mode of thinking with some hindsight on a track record, but not without it.

A good explanation of the difference between intellectual exploration and promoting people. You don't need to agree with everything someone says, and you don't even need to like them, but if they occasionally provide good insight, they are worth taking into account. If you propagate this strategy, you may even get to a "wisdom of the crowds" scenario - you'll have many voices to integrate in your own thinking, potentially getting you farther along than if you just had one thought leader you liked.

Having many smart people you don't necessarily agree with, like, or respect > having an idol you always agree with.

The prerequisite for all of this is to be a "high-decoupling" person. Rationalists (by definition?) have this personality, but this post is nevertheless very useful as it sketches out why separating the messenger, the context, and the message is good. And potentially, it teaches those with lower decoupling philosophies to stop "respecting" a Person Who Is Correct, but to start listening to many voices and judge for themselves what makes sense and what doesn't.

In early 2024 I think it's worth noting that deep-learning based generative models (presently, LLMs) have the property of generating many plausible hypotheses, not all of which are true. In a sense, they are creative and inaccurate.

An increasingly popular automated problem-solving paradigm seems to be bolting a slow & precise-but-uncreative verifier onto a fast & creative-but-imprecise (deep learning based) idea fountain, a la AlphaGeometry and FunSearch.

Today, in a paper published in Nature, we introduce FunSearch, a method to search for new solutions in mathematics and computer science. FunSearch works by pairing a pre-trained LLM, whose goal is to provide creative solutions in the form of computer code, with an automated “evaluator”, which guards against hallucinations and incorrect ideas. By iterating back-and-forth between these two components, initial solutions “evolve” into new knowledge. The system searches for “functions” written in computer code; hence the name FunSearch.

Perhaps we're getting close to making the valuable box you hypothesize.

Let me start out by agreeing with the spirit of your post, the kind of attitude you're recommending seems healthy to me.

I believe that I have some game-changing ideas among my stupid ideas, but like a lot of other people, I'm not conscientious enough to test them out unless they're relevant enough in my life. Telling other people our weird theories is less justifiable than testing them our ourselves, and for every genius there seems to be around 10 mentally ill people. Sometimes there's an overlap, and it's hard to tell since the idea will have to be successfully communicated, read, and understood by somebody skilled enough to evaluate it.

We should be skeptical of people who don't have a high level of formal education, and that includes myself. If I theoretize something around on quantum mechanics without having a solid foundation in the topic, the possibility that I'm right is roughly zero. The formal education is already based on years of dedicated work by the worlds brightest people, so even if I was an actual genius who spend months thinking about a theory, I probably wouldn't have a success-rate above 1%.

I also want to express my disproval of moral outrage. I don't think there's much of a relation to evil like you claim, but you should probably disagree with me here for the sake of your public image.
I don't believe that facts can be evil, they're necessarily neutral. I also don't think that opinions are worth much, even if they're said by geniuses, since they're just a reflection of some issue, from some perspective, according to some set of values. Nietzsche wanted overmen because he couldn't bear to see society decline. He also said that lower men were absolutely necessary, which supports my belief that most outrage is based on misunderstanding, a sort of triggered fear that somebody is promoting a value which would degrade the fitness of the listener.

Finally, I believe that the core issue is that society is too harsh. Nobody is perfect, and even the best commit mistakes. If somebody acts in good faith, it wouldn't be fair to punish them for being wrong. Saying something wrong publicly has almost become taboo, and the mentality of those who look for imperfection in orders in order to loudly "expose" it really offend me. I sense something like revenge, insecurity or cruelty in such behaviour, and the sort of atmosphere which results from it is bad. The wrath of public opinion can really break the spirit and self-expression of the loveliest of people.

I can necro-post, right? This article was recommended to me, so I thought it was new. Now I can see that it's 4 years old, but I've already written my comment.

Well, I have to think there is some balancing act here. I do look at life a lot from an evolutionary standpoint, but a flock who listened to their leader, and he was 99% wrong, would not survive long; or the sequence has to start with crushing a homer, and then getting it wrong 99 times. Whats missing here is not fully setting forth the downside of one of the 99 bad ideas.

Or maybe because we survived the basic-needs Malthusian "filter"; that explains the "Moral Outrage"; possibly just outraged at too much information and supposedly "true" memes, and the ratio keeps getting worse. A market crash of the idea ratio. (stagnation or plucked low hanging fruit)

In the end, if you hew to pragmatism (survival and reproduction), you guarantee the Great idea/ to Bad Idea ratio is relatively solid and tested. The theory is we want stability, which just a touch of creativity. Society also allocates places with high-risk reward ratio...and we give them a PHD.