There is a truth which destroys the minds of EAs; as hinted by Aumann and explicitly argued by Hanson, to not defer to someone’s authority, you must think you are smarter than them.
This is false. Aumann Agreement requires building common knowledge. There are many people I cannot build common knowledge with, and in those cases it can be correct and rational to have different beliefs, on both sides.
For example, in Biology-Inspired AGI Timelines, Yudkowsky explains that:
I consider naming particular years to be a cognitively harmful sort of activity; I have refrained from trying to translate my brain's native intuitions about this into probabilities, for fear that my verbalized probabilities will be stupider than my intuitions if I try to put weight on them. What feelings I do have, I worry may be unwise to voice; AGI timelines, in my own experience, are not great for one's mental health
I can't fully build common knowledge with people in that state. I can still listen to what they say, and use the evidence of what they say to update my beliefs, but that process need not terminate in agreement. Further, if I am in that state, I can't build common knowledge with other people. I can't be a fully rational agent fully updating on all available evidence, if I am not thinking about things to avoid psychological damage. This is still true if I am the smartest. This is not a common reason to agree to disagree, but it sure is a reason.
There are plenty of people who are smarter than me, with higher shmeerp, more thinkoomph, higher IQ, whatever. I don't fully defer to them because despite not being literally the smartest human being alive, I'm not an idiot.
Conversely, it is entirely possible to reach Aumann Agreement with people who are not my "epistemic peers". See Aumann Agreement is Common for examples.
Yud won the epistemic battle by being correct and forthright. He lost the instrumental war by being clumsy and abrasive in that correct forthrightness.
I think the fundamental ideas behind Daubert in legal settings and the idea of "scientific consensus" in academic literature are fairly decent standards. Potholer54 had a great discussion of scientific consensus.
Expert opinions are not fungible. If an expert shows me something like: "based on extensive testing using accepted, peer reviewed, and replicated scientific methodology we have ruled out alternatives to so that X is generally accepted in the field." I may be pretty happy to a-priori defer to X being true. However, if an expert just tells me "most people in the field are of the opinion X" I might be very slightly more inclined to think "X" but I am not going to defer to it as true.
"I am an expert, I think X" is not a valid basis for an expert opinion. In a legal setting, to consider an expert opinion courts rely on the Daubert standard, which, in simple terms, requires the opinion to be derived from a validated/accepted scientific methodology. An expert just going "I think X" is not an expert opinion the courts are going to accept. An expert saying "having performed some accepted scientific analysis and following such procedures I have reached X conclusion" is more likely to be accepted as an expert opinion (of course, there can be a lot of complexities in how they are actually judged by courts).
Bailey: You cannot rationally weigh your inside view more than the views of others, or disagree much with expert consensus, or whoever is considered epistemically in charge around here, unless you can cite specific information you have that they don’t have access to. If you do, you are making a claim of high status, and that they are relatively stupid.
I would defend a modified version of this Bailey: if your views contract 'expert consensus' (that is, 'an evidence-based derived through through rigorous scientific debate and peer review') it is generally more likely that you are mistaken then that the expert consensus is wrong. You should first consider that the consensus may have reasons and explanations you are unaware of or that you made an error somewhere, you should rule those out and be able to point to specific demonstrably reasons to reject to the expert consensus before placing much credence in them.
Epistemic peer means that, if you both have access to the same information, and they have not yet explained to you their reasoning, you should not presume one of you is better at this than the other, so you should probably take the two unadjusted opinions and split the difference.
This is a very strange understanding of "defer", are you sure it was Yud meant by "defer"? I would read "defer" as meaning "I would default to assuming they are right." I would argue that this is the wrong approach.
You should not (imo) assume someone is right just because they are an expert. You should judge someone's opinion based on their reasons for believing it. IF you don't have information about those reasons, then you should place only little weight on their beliefs no matter whether you think they are your epistemic peer. That isn't to say you even need a full analysis of their reasoning, but you need to at least know something about it to place much weight ton their view.
This is exactly the attitude Eliezer is describing:
You would have to confirm with the OP, but I think you are misrepresenting his statement. It certainly does not seem to be a steelman of the implicit case. The implicit case they seem to making is "Eliezer Yudkowsky appears to believe himself to be superior and dismisses the arguments of others without reasons." TO which it seems Eliezer Yudkowsky responded along the lines of (paraphrasing) "I did give evidence and reasons, I just didn't mention giving evidence in that particular lesswrong post due to lesswrongese/context/rhetorical effect, it was Leopold who was focused on arguing over status, not me."
To close, here is a quick list of the top 10 reasons to decide to disagree with an expert or general consensus, either locally or globally, including a market price, or with others who might otherwise be presumed to know better.
I am just going to through these.
>meet young Leopold who actually works on the frontier
>kid asks “why aren’t you updating at all in the direction of all the smart people?”
>you hesitate for dramatic effect, then drop the ultimate high-IQ line:
Some people find this interaction funny because they perceive it as 'autistic' but they also seem to not be able to read this social interaction.
Leopold: Update on these other people ... You aren't even pretending to be rational anymore! (Clearly not setting this up to be a normal conversation)
If Eliezer had responded with some humble thing, Leopold would have likely had some attack ready. Probably would have called Eliezer a Loser who just doesn't get it.
Cultural content (idea/claim/theory) is distinct from cultural attitudes (valence/credence/frame). Ideas accumulate, and then the culture passes judgement, which is not your own. This judgement isn't direct evidence with respect to how you should regard the cultural content, and usually it's too soon to tell unless you already have some understanding of the content. So modesty/deference as following (and thus also knowing) prevailing cultural attitudes (perhaps expert attitudes) is completely different from modesty/diligence as a duty to make a study of the relevant cultural content itself before claiming a valuable contribution (or a valid opinion).
Neither kind of study/modesty provides straightforwardly good evidence about coherence of an idea though, or about what your own belief or endorsement of it should be. You'd need to account for the filtering effects of cultural attitudes, which decide what kinds of content and what kinds of attitudes are available for your study. This is often infeasible, though occasionally easy.
There's also a popular combination of knowing attitudes but not the content (often a practical necessity); the converse of knowing the content but not the attitudes seems much healthier (a luxury of rearing ideas that can keep for a while without being judged by others). I don't think there's much of a distinction in scholar-nature between modesty-as-diligence and modesty-as-deference, so either could fit the role of a priesthood, being the prestigious motte of knowing what the culture knows, or knowing how the culture judges. The motte-bailey distinction passes through each subject matter (rather than between different subject matters), cutting across different roles that different people play in an alliance of motte-priests and bailey-promoters. There are both diligence-priests and deference-priests, with their own kinds of prestige, anchoring different kinds of bailey subcultures (appeals to object-level scholarship and appeals to popular/expert consensus/endorsement/scorn).
Modesty arguments often say that you should mostly or entirely bow to ‘expert consensus’ or the views of particular others, and who are you to disagree.
It has been a few years since I’ve properly addressed this so: My answer is that you are you. Other people are saying things for a wide variety of reasons, many of which are not about them paying attention and focusing on seeking this particular truth. Those people make mistakes all the time, and often have other motives and influences at work, especially social pressures and information cascades.
Them being as smart as you, or smarter than you, does not exempt them from this, and them being higher status or credentialed or cooler definitely does not exempt them.
A smart informed person sincerely thinking [X] can easily cease to be evidence for [X], once you have thought sufficiently about both [X] and why that person thinks [X].
Think for yourself, schmuck.
Or, as I once put it: You Have The Right To Think, also the moral duty to do so.
This post covers Eliezer Yudkowsky making a narrower claim than mine, about not conflating status with smarts and epistemic authority, and my documenting related discussion that happened on Twitter.
This concerns a particular historical interaction, but what matters is the general point.
I then briefly extend this to my broader claims.
Modesty’s Bailey
I think, in the exchange I will quote, Rob Bensinger is reaching but centrally correct.
Modesty arguments often are this, trying to conflate the motte and bailey as usual:
I affirm Modesty’s Motte.
I am heavily against Modesty’s Bailey.
Epistemic Peerage
Note the bolded passage (bold is mine) in the exchange below:
It doesn’t mean you should mostly or fully defer to such a person. That would imply they are an ‘epistemic superior,’ someone whose opinion carries greater weight than your own, or even an ‘expert’ or ‘guru’ that you should defer to entirely except under special circumstances.
Epistemic peer means that, if you both have access to the same information, and they have not yet explained to you their reasoning, you should not presume one of you is better at this than the other, so you should probably take the two unadjusted opinions and split the difference.
Once they explain their reasoning, you should instead evaluate their reasoning. Sometimes they will have better reasoning than you, sometimes it will be worse. Remember the law of Conservation of Expected Evidence. If you could move towards their belief but not away from it, you should adjust your beliefs now.
The Exchange
I’m reproducing this discussion primarily because I think it has valuable things to say about how to do, and how not to do, epistemics, rather than because of the OP story being about Leopold in particular. Leopold, indeed, would later wisely recant the central conceptual point, in the form of creating the hedge fund Situational Awareness, whether or not he would agree that the two are the same principle.
Eliezer’s Explanation
Eliezer offers a fictionalized version that did not happen as written, but that to him illustrates the core thing that happened. He may or may not be right, I was not there.
‘Blows up a hedge fund’ goes too far in describing what happened to Leopold’s hedge fund, Situational Awareness. It declined from $45 billion in assets to $10 billion, after it used too much leverage and got caught in margin calls, forcing a sale of its public book, but it did not per se blow up.
Leopold’s entire approach to his hedge fund did represent an extremely not-modest view of the world, to not only bet on his own ‘situational awareness’ and analysis of the market, but to do so at high leverage and for amounts where he was likely impacting prices, and thus vulnerable to additional pressures.
And you know what? He got too aggressive with leverage, but he was fundamentally right to be immodest, and to say he could better predict prices than the market, because he was looking closer, thinking better and more situationally aware. He made lots of great trades. The Efficient Market Hypothesis is False, and it is people like Leopold who make it do an imitation of true.
A Demonstration That Eliezer’s Translation Accurately Describes Many People Whether Or Not It Describes Leopold
As in, here is someone else’s pejorative translation of events, which confirms it.
This is exactly the attitude Eliezer is describing:
Pay attention to both the literal contents, and also to the vibe of utter contempt, not only from Leopold but also from Will Rinehart.
As in, contempt for the idea that anyone might think for themselves, or consider arguments rather than who else believes what.
Will’s contempt persists, despite Eliezer being vindicated on the substance.
Later, of course, Leopold wisely got over it, and indeed took it too far.
Wrong, Stupid and Low Status Are Three Distinct Things
Most people only partially make the relevant mistake.
Each correlates with the others, but mostly people know they are distinct.
It is most common to equate calling someone wrong and calling someone stupid.
Whereas, of course, smart people are wrong about things all the time, and stupid people are right about things all the time. Sometimes the smart person is wrong, and the stupid person is right, or the smart group is wrong and the stupid group is right, either because or in spite of superior information.
The other links are usually seen as more tenuous.
Of course people constantly use accusations of stupidity or wrongness to try and lower someone’s status, or smartness and rightness to raise it. Most people in most places still push back hard against attempts to too systematically equate smart or right with high status, or especially stupid and wrong with low status. We realize they are not the same, and also the monkey brain knows that there be dragons.
The danger comes in places that do systematically equate intelligence and rightness with status. Academia is the most famous version of this.
LessWrong-Yudkowsky style rationality has this as an obvious failure mode. You have to constantly push back against where the high status person says ‘think for yourself’ and everyone says ‘yes, as the guru says we will think for ourselves exactly as he told us to.’ There is no amount of hammering ‘no I am not a guru, at most I am an epistemic superior, I really am saying to think for yourselves’ or ‘we are not a cult’ that will consistently fully stick. In my experience rationalist circles manage, after much effort, to do a good overall job at this, but not a great one.
Effective Altruism is set up to have a maximally bad version of this mistake. There is tons of pressure to quantify all the things, and to maximize quantified results, and to bow to the legible, and a lot of demand to be told how to figure out how to think and do in order to do the most good, and thus be good, and so on. Results are often prioritized over epistemics. This all puts extra value on consensus, and various forms of high status often end up equated with correctness or intelligence or both.
(As usual, to avoid related implications, my caveat that while EA has lots of issues, we criticize them so much exactly because they are worth taking the effort to criticize, and if you don’t believe that you should see the other guy.)
A Quick Survey Of Some Reasons To Not Be Epistemically Modest
The essence of my broader claim is that, in practice, modesty arguments get screened off by sufficient attention and understanding. Modesty is a method for when you cannot budget sufficient attention and understanding, which is most of the time.
You Have The Right To Think. See that post, and also the extended discussions in the comments between me and, among others, Eliezer Yudkowsky and Robin Hanson.
The post was initially motivated as a response to Robin Hanson’s review of Eliezer’s highly related book Inadequate Equilibria, which discusses when you might be able to do something unusually well.
Actual modesty, in practice, happens for a mix of complicated reasons, as I broke down in On Robin Hanson’s ‘Social Proof, But Of What?’.
Once you have invested enough to understand the relevant questions and form an opinion, and have considered the arguments and other causes behind people’s opinions, you can and should substitute your judgment for theirs.
Often you should do this despite the ‘experts’ or other epistemic would-be authorities having more relevant knowledge than you, and having devoted more time to related questions than you, outnumbering you, being higher status and more respected and credentialed than you, and perhaps being smarter than you.
Your investment of sufficient attention and understanding can be and often is fractal. As illustrated in Leaders of Men, you can confidently point out where an expert, who knows ball far better than you, is failing to properly understand an aspect of ball. In Zeroing Out, I explain how you can know a consensus estimate is wrong in a particular direction, even if you could not hope to create, on your own, an estimate as accurate as the consensus.
This, ideally along with a track record of testable predictions and the logic behind them, gives you many tools by which to evaluate whether someone, or some group, is functionally an epistemic peer or superior in a given context.
The obvious mistake is ‘I disagree with them, therefore they are not an epistemic peer,’ which would defeat the whole purpose. What you can and should do is adjust your evaluation of sources based on the quality of their arguments.
This blog has always been about not being epistemically modest, most famously:
Covid-19 and our response to it demonstrated, basically, ‘modesty shmodesty.’ AI only pushed me further in that direction. I now have an extensive track record to look through, and see how things are going.
There is also an extensive track record regarding claims around AI timelines, the topic of the original exchange between Eliezer and Leopold. We have seen many, including Leopold, radically shorten their timelines (Leopold as late as August 2021 was only estimating 25% chance of AGI-level systems by 2070), based on evidence that was mostly foreseeable in advance, combined with shifting social context.
Against Modesty’s Bailey
To close, here is a quick list of the top 10 reasons to decide to disagree with an expert or general consensus, either locally or globally, including a market price, or with others who might otherwise be presumed to know better.
On top of the reasons to not be epistemically modest, there are strong reasons to state your immodest beliefs: It is often more helpful to present your own views and your own reasons, rather than to mirror everyone else, to combat the reasons above.
Thus, I reaffirm Modesty’s Motte and re-reject Modesty’s Bailey.