Let's give this a try.
Claim: Relying on few strong arguments is more reliable than relying on many weak arguments.
This is a messy subject, and one that's difficult write about, and I appreciate you tackling the topic. I think there are some important qualifications to make about this post, as others have noted. But I know that when writing about messy subjects, it's hard to avoid "death by a thousand qualifications." Lately, I've been trying to solve the problem by putting most qualifications in footnotes, e.g. here. You might want to try that, as it mitigates criticisms of the "but you didn't make qualifications X and Y!" form while still leaving the body text in a readable condition.
Below, I'll refer to MWA ("many weak arguments") and ORSA ("one relatively strong argument"), for convenience.
Here's my guess at what's going on:
Thanks for the feedback.
I think there are some important qualifications to make about this post, as others have noted.
My hunch is that most significant problem with the MWA approach is the assumption of (weak) independence, in the sense that in practice, when sophisticated use of MWA fails, it's usually because the weak lines of evidence are all being driven by the same selection effect. A hypothetical example that jumps to mind is:
A VC is evaluating a startup. He or she reasons
and the situation is
Re: #1 — The reason that the sector is growing is because there's a bubble
Re: #2 — The reason that the VC's colleagues think that the sector is good to invest in is because, like the VC, they don't recognize that there's a bubble.
Re: #3 — The VC's views on the object level merit of the project are colored by the memes that have been spreading around that are causing the bubble
Re: #4 — The reason that impressive people are going into the sector is because there's a bubble, so everyone's going into the sector – the people's impressivenes...
Penrose is a worrisome case to bring as an example, since he is in fact wrong, and therefore you're giving an example where your reasoning leads to the wrong conclusion. If you can't easily find examples where your reasoning led you to a new correct conclusion instead of new sympathy toward a wrong conclusion, this is worrisome. In general, I tend to flag recounts of epistemological innovations which lead to new sympathy toward a wrong conclusion, as though the one were displaying compassion for a previously hated enemy, for in epistemology this is not virtue.
The Penrose example worries me for other reasons as well, namely it seems like it would be possible to generate hordes and hordes of weak arguments against Penrose; so it's as if because the argument against Penrose is strong, you aren't bothering to try to generate weak arguments; reading this feels like you now prefer weak arguments to strong arguments and don't try to find the many weak arguments once you see a strong argument, which is not good Bayesianism.
You also claim there's a strong argument for Penrose, namely his authority (? wasn't this the kind of reasoning you were arguing against trusting?) but either we have ...
Penrose is a worrisome case to bring as an example, since he is in fact wrong, and therefore you're giving an example where your reasoning leads to the wrong conclusion.
JonahSinick is not saying that Penrose is right, only that based on his heuristic he adjusted the probability of that upwards. To judge this wrong, it's not enough to know that Penrose is wrong, you must also know the probability estimates JonahSinick gave before and after. To give an absurd example, if JonahSinick used to believe the probability was 10^(-15), he would be wise to adjust upwards.
By the way, this isn't the first time I see you use the meta-heuristic that when a heuristic adds support to a wrong conclusion it should be taken less seriously. While it is valid to some extent, I think you are overusing it.
Responses below. As a meta-remark, your comment doesn't steelman my argument, and I think that steelmanning arguments helps keep the conversation on track, so I'd appreciate it if you were to do so in the future.
Penrose is a worrisome case to bring as an example, since he is in fact wrong, and therefore you're giving an example where your reasoning leads to the wrong conclusion.
The point of the example is that one shouldn't decisively conclude that Penrose is wrong — one should instead hedge.
Perhaps a relevant analogy is that of the using seat belts to guard against car accidents — one shouldn't say "The claim that I'm going to get into a potentially fatal car accident is in fact wrong, so I'm not going to wear seat belts." You may argue that the relevant probabilities are sufficiently different so that the analogy isn't a good one. If so, I disagree.
If you can't easily find examples where your reasoning led you to a new correct conclusion instead of new sympathy toward a wrong conclusion, this is worrisome.
There are many such examples. My post extended to a length of eight pages without my going into them, and I wanted to keep the post to a reasonable length. I...
As a meta-remark, your comment doesn't steelman my argument, and I think that steelmanning arguments helps keep the conversation on track, so I'd appreciate it if you were to do so in the future.
Something has gone severely wrong with the 'steelman' concept if it is now being used offensively, to force social obligations onto others. This 'meta-remark' amounts to a demand that if JonahSinick says something stupid then it is up to others to search related concept space to find the nearest possible good argument for a better conclusion and act as if Jonah had said that instead of what he actually said. That is an entirely unreasonable expectation of his audience and expecting all readers to come up with what amounts to superior content than the post author whenever they make a reply is just ridiculously computationally inefficient.
Responses below. As a meta-remark, your comment doesn't steelman my argument, and I think that steelmanning arguments helps keep the conversation on track, so I'd appreciate it if you were to do so in the future.
I have a known problem with this (Anna Salamon told me so, therefore it is true) so Jonah's remark above is a priori plausible. I don't know if I can do so successfully, but will make an effort in this direction.
(It's true that what Jonah means is technically 'principle of charity' used to interpret original intent, not 'steelman' used to repair original intent, but the principle of charity says we should interpret the request above as if he had said 'principle of charity'.)
(It's true that what Jonah means is technically 'principle of charity' used to interpret original intent, not 'steelman' used to repair original intent, but the principle of charity says we should interpret the request above as if he had said 'principle of charity'.)
:-)
I think the concept you're looking for is the principle of charity. Steel man is what you do to someone else's argument in order to make sure yours is good, after you've defeated their actual argument. Principle of charity is what you do in discourse to make sure you're having the best possible discussion.
If you think Eliezer should have steelmanned your argument then you think he has already defeated it - before he even commented!
Where's the fun in shooting down the obvious targets, most readers can do so themselves.
As one of those readers I would prefer not to have to. I appreciate the effort others put into keeping the garden well tended and saving me the trouble of reading low quality material myself.
Eliezer's reply is the kind of reply that I want to see more of. I strongly oppose shaming 'requests' used to discourage such replies.
This post reminds me of my own experiences with the "smartphone question." For years, I had derided smartphones for lacking a killer app. However, a recent conversation with several LW community members ultimately changed my mind, and now I consider my previous view to have been very misguided.
What I overlooked was that while there was no one "killer app" that provided huge value, there were lots and lots of smaller apps that provided small individual value. When taken together, they constituted a substantial overall benefit. My focus on the "one big thing" was causing me to miss out on a lot of potential value.
In machine learning, the approach you're advocating is called ensemble learning, or more narrowly, "Bayesian model combination." Polikar (2006) is a nice overview article.
There are problems to whose solution I would attach infinitely greater importance than to those of mathematics, for example touching ethics, or our relation to God, or concerning our destiny and our future; but their solution lies wholly beyond us and completely outside the province of science. -- Gauss
Sounds a lot like Lord Kelvin saying that biology's vital force was infinitely beyond the reach of science, and equally wrong in the light of history. Flag: Getting a kick out of not knowing something; motivated uncertainty.
Robin Hanson argues for the "many weak arguments" approach in this post. Describing a difference between himself and Bryan Caplan, he writes,
...Bryan [Caplan]'s picture seems to be of a long metal chain linked at only one end to a solid foundation; chains of reasoning mainly introduce errors, so we do best to find and hold close to our few most confident intuitions. My picture is more like Quine's "fabric," a large hammock made of string tied to hundreds of leaves of invisible trees; we can't trust each leaf much, but even so we can st
The adage "let the evidence guide me wherever it may" applies. We don't get to impose on reality what kind of evidence for various positions it provides for us to find. If there is one single strong piece of evidence, then we should update on that single strong piece of evidence, neither being happy nor unhappy about its singular nature. Similar how falsifying an established theory only necessitates one single contradictory (from the confines of the theory) experimental result (as long as we can rely on that result being repeatable / done methodo...
Jonah, I agree with what you say at least in principle, even if you would claim I don't follow it in practice. A big advantage of being Bayesian is that you retain probability mass on all the options rather than picking just one. (I recall many times being dismayed with hacky approximations like MAP that let you get rid of the less likely options. Similarly when people conflate the Solomonoff probability of a bitstring with the shortest program that outputs it, even though I guess in that case, the shortest program necessarily has at least as much probabil...
Experts are apparently known to be not much better than amateurs outside of their area of expertise, so whatever Penrose claims about something other than General Relativity and High-energy Physics should have the same weight as that of, say, a grad student in the relevant area, at best. Especially given that he does not have a track record of being proven right in unrelated fields. Thus the argument that
Penrose is one of the greatest physicists of the second half of the 20th century
should have no bearing on taking his claims in neuroscience seriously.
"Consilience" is a word that's had little favour of late, given the importance of the concept; it deserves to be used more.
I haven't read the comments yet, so apologies if this has already been said or addressed:
If I am watching others debate, and my attention is restricted to the arguments the opponents are presenting, then my using the "one strong argument" approach may not be a bad thing.
I'm assuming that weak arguments are easy to come by and can be constructed for any position, but strong arguments are rare.
In this situation I would expect anybody who has a strong argument to use it, to the exclusion of weaker ones: if A and B both have access to 50 weak argumen...
I would like to signal that the link about Euler's argument is broken. I believe the correct link should be https://www.lesswrong.com/posts/WsmnfWTP28dXCKEy8/the-use-of-many-independent-lines-of-evidence-the-basel
Look what I found!
https://en.wikipedia.org/wiki/Boosting_(machine_learning) https://en.wikipedia.org/wiki/Bootstrap_aggregating
Yay prior literature! Always there several months later than when you initially tried to find it.
I don't buy the assumption that seems to be implied that many arguments have to be weak and a single argument has to be strong.
Why not have many strong reasons instead of one weak reason?
Certainly for complex questions I find multi-threaded answers more convincing than single-threaded ones.
Fox over hedgehog for me.
In terms of picking a major, do something you enjoy that you can conceivably use to get a job. You can actually get a job with a philosophy degree. I did... after I quit accounting because it was too darn boring...
My epistemic framework has recently undergone some major shifts, and I believe that my current epistemic framework is better than my previous one. In the past, I tended to try to discover and rely on a single relatively strong argument in favor or against a position. Since then, I’ve come to the conclusion that I should shift my focus toward discovering and relying on many independent weak arguments. In this post, I attempt to explain why. After I posted this article, I got lots of comments in response, and responded to them in this discussion post.
I’m a mathematician by training, and by inclination. In the past, I tried to achieve as much certainty as possible when I'd evaluate an important question.
An example: Something that I’ve thought a lot about is AI risk reduction effort as a target for effective philanthropy. In the past, I attempted to discover a single relatively strong argument for, or against, focus on AI risk reduction. Such an argument requires a number of inputs. An example of an input is an argument as to what kind of AI one should expect to be built by default. I spent a lot of time thinking about this and talking with people about it. What I found was that my views on the question were quite unstable, altering frequently and substantially in response to incoming evidence.
The phenomenon of [my position altering frequently and substantially in response to incoming evidence] was not limited to AI risk. It was characteristic of much of my thinking about important questions that could not be answered with clear-cut evidence. I recognized this as bad, but felt that I had no choice in the matter — I didn’t see another way to think about such questions, and I thought that some such questions are sufficiently important so as to warrant focus. My hope was that my views on these questions would gradually stabilize, but this didn’t happen with the passage of time.
While my views on various questions were bouncing around, I started to notice that some people seemed to be systematically better at answering questions that could not be answered with clear-cut evidence, in the sense that new data supported their prior views more often than new data supported my own prior views.
This puzzled me, as I hadn’t thought that it was possible to form such reliable views on these sorts of questions with the evidence that was available. I noticed that these people didn’t seem to be using my epistemic framework, and I was unclear on what epistemic framework they were using.They didn't seem to be trying to discover a relatively strong argument.
They sometimes gave weak arguments that seemed to me to be a product of the fundamental cognitive bias described in Eliezer's article The Halo Effect and Yvain's articles The Trouble with "Good" and Missing the Trees for the Forest. When a member of a reference class has a given feature, by default, we tend to assume that all members of the reference class have the same feature. Some of the arguments seemed to me sufficiently weak so that they should be ignored, and I didn't understand why they were being mentioned at all.
What I gradually came to realize is that these people were relying on many independent weak arguments. If the weak arguments collectively supported a position, that’s the position that they would take. They were using the principle of consilience to good effect, obtaining a better predictive model than my own.
For concreteness, I’ll give an example of a claim that I believe to be true with high probability, despite the fact each individual argument that supports it is weak.
Claim: At the current margin, on average, majoring in a quantitative subject increases people’s expected earnings relative to majoring in other subjects.
The following weak arguments support this claim:
Weak argument 1: Historically, there’s been a correlation between majoring in a quantitative subject and making more money. Examining the table in a blog post by Bryan Caplan reveals that the common majors that are most strongly associated with high earnings are electrical engineering, computer science, mechanical engineering, finance, economics, accounting, and mathematics, each of which is a quantitative major.
Weak argument 2: Outside of medicine, law, and management, the most salient jobs that offer the high earnings are finance and software engineering, both of which require quantitative skills. Majoring in a quantitative major builds quantitative skills, and so qualifies one for these jobs.
Weak argument 3: Majoring in a subject with an abundance of intelligent people signals to employers that one is intelligent. IQ estimates by college major suggest that the majors with highest average IQ are physics, philosophy, math, economics, and engineering, most of which are quantitative majors. So majoring in a quantitative field signals intelligence. And employers want intelligent employees, so majoring in a quantitative subject increases earnings.
Weak argument 4: Studying a quantitative subject offers better opportunities to test one’s beliefs against the world than studying the humanities and social sciences does, because the measures of performance in quantitative subjects are more objective than those in humanities and social sciences. Thus, studying a quantitative subject raises one’s general human capital relative to what it would have been if one studied a softer subject.
Weak argument 5: Conventional wisdom is that majoring in a quantitative subject increases one’s expected earnings. If there were strong arguments against the claim, one might expect them to percolate into conventional wisdom, which they haven't. In absence of evidence to the contrary, one should default to conventional wisdom.
Weak argument 6: I know many smart people who enjoy thinking, and who themselves know other many smart people who enjoy thinking. As Yvain discussed in Intellectual Hipsters and Meta-Contrarianism, smart people who enjoy thinking are often motivated to adopt and argue for positions opposed to conventional wisdom, in order to counter-signal intelligence. If the conventional wisdom concerning the subject at hand were wrong, one might expect some of the people who I know to have argued against it, and I’ve never heard them do so.
To verify that these arguments are in fact weak, I’ll give counterarguments against them:
Counterarguments to 1: Correlation is not causation. The people who major in quantitative subjects may make more money later on because they have higher innate ability, or because they have better connections on account of having grown up in households with higher socio-economic status, or for some other nonobvious reason.
Counterarguments to 2: It could be that one only needs to have high school level quantitative knowledge in order to succeed in these jobs.
Majoring in a quantitative field could reduce one’s ability to go to medical school or law school later on (e.g. on account of grading being more strict in quantitative subjects, and medical and law schools selecting students by GPA).
Counterarguments to 3: Potential employees may have other ways of signaling intelligence, so that college major is not so important. As above, majoring in a quantitative subject may lower GPA, resulting in sending a signal of low quality.
Counterarguments to 4: It could be that earnings don’t depend very much on one’s intellectual caliber. For example, maybe social connections matter more than intellectual caliber, so that one should focus on developing social connections. The heavy workload of a quantitative major could hinder this.
Counterarguments to 5: Conventional wisdom is often wrong. Conventional wisdom on this subject is likely rooted in the correlation between majoring in a quantitative subject and having higher earnings, and as discussed in the counterarguments to 1, correlational evidence is weak.
Counterarguments to 6: There are many, many issues on which one can adopt a meta-contrarian position, and meta-contrarians only discuss a few of these, because there are so many of them. Also, “Smart people who like to think” could, for some unknown reason, collectively be motivated to believe the claim.
In view of these counterarguments, how can one be confident in the claim?
First off, I’ll remark that the counterarguments don’t suffice to refute the individual arguments, because the counterarguments aren’t strong, and there are counterarguments against them.
But there are counterarguments to the counterarguments as well. In view of this, one might resign oneself to a position of the type “it may or may not be the case that the claim is true, and it’s hopeless to decide whether or not it is.” Eight years ago, this was how I viewed most claims concerning the human world. In Yvain's words, I was experiencing epistemic learned helplessness.
It’s not uncommon for mathematicians to hold this position on claims concerning the human world. Of course there are instances of mathematicians using several lines of evidence to arrive at a conclusion in absence of a rigorous proof. But the human world is much messier and more ambiguous than the mathematical world. The great mathematician Carl Friedrich Gauss wrote
There are problems to whose solution I would attach infinitely greater importance than to those of mathematics, for example touching ethics, or our relation to God, or concerning our destiny and our future; but their solution lies wholly beyond us and completely outside the province of science.
Gauss's quotation doesn't directly refer to prosaic epistemic questions about the human world, but one could imagine him having such a view toward these questions, and even if not, I've heard a number of mathematicians express such a view on questions that cannot be answered with clear-cut evidence.
This not withstanding, my current position is that one can be confident in the claim, not with extremely high confidence (say, the level of confidence that Euler had in the truth of the product formula for the sine function), but with confidence at the ~90% level, which is high enough to be actionable.
Why? The point is that the arguments in favor of the claim are, like Euler’s arguments, largely independent of one another. This corresponds to the fact that the counterarguments are ad hoc and un-unified. The situation is analogous to Carl Sagan’s “Dragon in My Garage” parable. In order to refute all of the arguments via the counterarguments, one needs to assume that all the counterarguments succeed (or other counterarguments succeed), and the counterarguments are pretty independent. If one assumes that for each argument, the counterarguments overpower the argument with probability 50%, and the counterarguments’ successes are independent, the probability that they all succeed is ~1.5%.
The counterarguments are not independent — for example, the point about majoring in a quantitative subject lowering GPA appears twice. So I don’t think that one can be too confident in the conclusion. But the existence of many independent weak arguments suffices to rescue us from epistemic paralysis, and yield an actionable conclusion.
The “single relatively strong argument” approach to assessing the above claim is to try to synthesize as many of the above weak arguments and counterarguments as possible, into a single relatively strong argument.
[Added: Kawoomba's comment realize that the above sentence wasn't clear. The point is that in focusing on a single strong argument to the exclusion of other arguments, one is implicitly rejecting the weak arguments, and so doing so constitutes an implicit attempt to synthesize the evidence. The sort of thing that I have in mind here is to say "Correlation is not causation. Conventional wisdom is probably rooted in mistaking correlation for causation. Therefore we should ignore conventional wisdom in formulating our relatively strong argument."]
If I were to try to do this, it would look something like this:
Based on what people and employers say, it appears that many of the high paying jobs in our society require some quantitative skills. It’s unclear how much quantitative skill one needs to do these jobs. But presumably one needs some.
People who are below this threshold may be able to surpass it by majoring in a quantitative subject, and thereby get higher earnings.
Even if one does surpass this threshold, majoring in a quantitative subject may not suffice to signal to employers that one is above that threshold, if the noise to signal ratio is high. But it may not be necessary to get a job that requires quantitative skills right out of college, in order to get high earnings from building quantitative skills in college — it might be possible for an employee to “work his or her way up” to a position that uses quantitative skills, and profit as a result.
It might appear as though people who are already above this threshold wouldn’t get higher earnings from majoring in a quantitative subject. But employers may not be able to tell that potential employees have quantitative skills unless they major a quantitative subject. (Note that if this is true, it suggests that the concern in the previous paragraph is less of an issue. However, it could still be an issue, because different levels of quantitative skills are required to get different jobs, so that the level that employees need to signal is not homogenous). This pushes in favor of majoring in a quantitative subject. People above the threshold may also benefit in majoring in a quantitative subject because it signals intelligence, which is considered to be desirable, independently of the specific quantitative skills that a potential employee has acquired.
It’s necessary to weigh these considerations against the fact that quantitative majors tend to be demanding, leaving less time for other activities, and are harder to get good grades in. Thus, majoring in a quantitative subject involves a tradeoff, the value of which will vary from individual to individual, depending on his or her skills, potential areas of work, and the criteria that graduate schools and employers use to select employees.
The above argument has some value, and I imagine that a college freshman would find it somewhat useful. But it seems less helpful than the list of weak arguments, together with the most important counterarguments, given earlier in this post. The argument in the previous section doesn’t clearly demarcate the different lines of evidence, and inadvertently leaves out some of the lines of evidence (because some of the lines of evidence don’t easily fit into a single framework).
These problems with using the “single relatively strong argument” approach are closely related to my past unstable epistemology. Because the “single relatively strong argument” approach doesn’t clearly demarcate the different lines of evidence, when a user of the approach gets new counter-evidence that’s orthogonal to the argument, he or she has to rethink the entire argument. Because the “single relatively strong argument” approach leaves out some lines of evidence, it’s less robust than it could be.
A priori, one could imagine that these things wouldn’t be a problem in practice: if the relatively strong argument were true with sufficiently high probability, then it would be unlikely that one would have to completely rethink things in the face of incoming evidence, and it wouldn’t be so important that the argument doesn't incorporate all of the evidence.
My experience is that this situation does not prevail in practice. One theoretical explanation for this is analogous to a point that I made in my post Robustness of Cost-Effectiveness Estimates and Philanthropy:
A key point that I had missed when I thought about these things earlier in my life is that there are many small probability failure modes, which are not significant individually, but which collectively substantially reduce [the probability that the argument is correct]. When I encountered such a potential failure mode, my reaction was to think “this is very unlikely to be an issue” and then to forget about it. I didn’t notice that I was doing this many times in a row.
This applies not only to cost-effectiveness, but also to the accuracy of individual relatively strong arguments. Relatively strong arguments in domains outside of math and the hard scientists are often much weaker than they appear. The phenomenon of model uncertainty is pronounced.
The points in this section of the post are in consonance with a claim of Philip Tetlock’s in Expert Political Judgment: How Good Is It? How Can We Know?:
Tetlock contends that the fox — the thinker who knows many little things, draws from an eclectic array of traditions, and is better able to improvise in response to changing events — is more successful in predicting the future than the hedgehog, who knows one big thing, toils devotedly within one tradition, and imposes formulaic solutions on ill-defined problems.
An example that highlights my shift in epistemology is the shift in my attitude concerning Roger Penrose’s beliefs about consciousness.
[Edit: Eliezer's comment and Vaniver's comment made me realize that the connection between this example and the rest of my post is unclear. The shift in my attitude toward Penrose's beliefs about consciousness isn't coming from my shift toward using the principle of consilience. I agree that the arrow of consilience points against Penrose's beliefs. The shift in my attitude is coming from the shift from "give weight to arguments that stand up to scrutiny" to "give weight to all arguments with a nontrivial chance of being right, even the ones that don't seem to hold up to scrutiny."]
In The Emperor's New Mind (1989), he argues that known laws of physics are inadequate to explain the phenomenon of consciousness. Penrose proposes the characteristics this new physics may have and specifies the requirements for a bridge between classical and quantum mechanics (what he calls correct quantum gravity). […] Penrose believes that such deterministic yet non-algorithmic processes may come into play in the quantum mechanical wave function reduction, and may be harnessed by the brain. He argues that the present computer is unable to have intelligence because it is an algorithmically deterministic system. He argues against the viewpoint that the rational processes of the mind are completely algorithmic and can thus be duplicated by a sufficiently complex computer. This contrasts with supporters of strong artificial intelligence, who contend that thought can be simulated algorithmically. He bases this on claims that consciousness transcends formal logic because things such as the insolubility of the halting problem and Gödel's incompleteness theorem prevent an algorithmically based system of logic from reproducing such traits of human intelligence as mathematical insight.
I believe that Penrose’s views about consciousness are very unlikely to be true:
But Penrose isn’t a random crank. Penrose is one of the greatest physicists of the second half of the 20th century. He’s a far deeper thinker than me, and for that matter, a far deeper thinker than anybody who I’ve ever met.
I have several relatively strong arguments against Penrose's views on consciousness. Collectively, they’re significantly stronger than the moderately strong argument “great physicists are often right.” In the past, I would have concluded “…therefore Penrose is wrong.”
But it’s not rational to ignore the moderately strong argument that supports Penrose’s views. The chance of the argument being right is non-negligible. I should give nontrivial credence to Penrose’s views on consciousness having substance. Maybe at least some of Penrose’s ideas about consciousness are sound, and that the reason that they seem tenuous is that he’s expressed his ideas poorly, or they've been misquoted. Maybe there's some other way to reconcile the hypothesis that his views are sound, with the evidence against this, that I haven't thought of.
If I were using my previous epistemic framework, my world view could be turned upside down by a single conversation with Penrose. If I were using my previous epistemological framework, I would be subject to confirmation bias, using my conclusion “…therefore Penrose is wrong” as overly strong evidence against the claim “great physicists are often right,” which I was unwarrantedly ignoring from the outset.
Retrospectively, it makes sense that there are people who are substantially better than I had been at reasoning about questions that I thought inherently near-impossible to think about.
Acknowledgements: I thank Luke Muehlhauser, Vipul Naik, Nick Beckstead, and Laurens Gunnarsen for useful suggestions for what to include in the post, as well as helpful comments on an earlier draft. I'm indebted to and grateful to Holden Karnofsky at GiveWell for his insights, as well as GiveWell, which offered me the opportunity to think about hard epistemic questions that can't be answered with clear-cut evidence. Both of these helped me recognize the core thesis of this post.
Note: I formerly worked as a research analyst at GiveWell. All views expressed here are my own.