Lately, I realized that I have been making very incorrect statements regarding deference. Most of the statements I make here seem retrospectively quite obvious, yet I have managed to have overlooked all of them. This is an attempt to reconstruct the process through which I came to certain conclusions. AGI is scary and hard to think about. But that is not a reason to throw our minds away. And I think I did discard my mind for a bit.
The general case for deferring your opinions goes something along the lines of:
The world is incredibly difficult and complex. You cannot think from first principles about everything. Life is too short and the number of important questions is too large. Deferring to experts on most things is how civilization works.
For most questions, you can identify someone whose judgment would be better than your own.
Experts have information that I don’t. I cannot replicate that by sitting in my room and reasoning hard.
Expert aggregates and prediction markets have a demonstrated track record of outperforming individuals, including smart individuals who think they can do better.
The Dunning-Kruger problem. The less you know about a domain, the less equipped you are to evaluate your own reasoning about it.
et cetera.
The general case for first principled thinking in goes something along the lines of:
When there’s a clear consensus, deference is easy. But experts disagree with each other here.
Some domains are too new for expertise to have crystallized.
Historically, deference to expert consensus has sometimes been spectacularly wrong in ways that first-principles thinkers caught.
et cetera.
I hear the discussion of “should you defer or not defer” frequently. But what does this actually mean?
What does it mean to “form your own views”? You’re not going to independently derive AI timelines from first principles. The part that gets left out in the “think for yourself” argument is that pure first-principles thinking also fails, and often fails worse. Applied naively to complex domains, it produces conclusions that are (maybe) logically coherent and practically insane. In politics, pure first-principles reasoning gets you things like “we should select leaders by lottery” or “the most rational people should rule” — conclusions that may be logically coherent yet practically insane. Form completely independent views on AI timelines” would mean something like: ignore all existing forecasts, learn ML from scratch, build your own compute scaling model, and arrive at a number. Nobody is actually proposing this, and rightfully so.
“Defer to the experts” is equally underspecified. Which experts? Deferring to the Metaculus aggregate is pretty different than deferring to Eliezer Yudkowsky, which is pretty different than deferring to the median ML professor who hasn’t thought about timelines for more than twenty minutes. And the choice of who counts as “expert” already contains a lots of judgment. Deferring to superforecasters implicitly weights calibration and track record. Deferring to lab leaders implicitly weights inside information. Deferring to the LessWrong median (?) implicitly weights community-specific reasoning norms. There exists no neutral deferring.
One version of “just defer” that I’ve considered, only half joking, is to literally roll a die: assign each face to a different forecaster and adopt whichever timeline comes up, and plan my career accordingly.
I think one of the reasons I came to such conclusions is doing too much “meta-thinking” rather than actually thinking:
There are all of these people, definitely smarter than me, spent thousands more hours on this, and have access to better information. What are the odds I can form a view more correct than Daniel Kokotajlo or Ajeya Cotra? Seems highly unlikely. Shouldn’t I just pick the source I trust most and go with their number? But how do I figure out what sets of characteristics makes someone more probabilistically likely to be correct on this subject? There are some basic traits, such as “has actually spent significant time thinking about this” but this seems to apply to all of these people. And I don’t know how to differentiate between the non-basic traits. And surely for any of these individual dimensions, each of these individual smart people have already considered them, and have thought longer and harder than I?
The Tetlock superforecasting approach suggests something like construct your prior from a reasonable aggregate and then ask which specific inside-view considerations should push you away from that number. But why should I even do this? Surely these smart people are already doing this, and how likely can I do a better job?
So there it goes, the clock is ticking and I should roll a die.
But this is naive and incorrect. My current-revised-thoughts are as follows: Better deferring is better than worse deferring. Mindful deferring is good. Mindful deferring involves a serious amount of actually thinking about object level things. Blind deference is bad, but not for the reason of “from a high level there is a non-trivial probabilistic likelihood that I would be more correct about AGI than (smart person)”. Specific updated reasonings and resolutions include:
We understand, from all other domains, blind deference is the mechanism behind groupthink. And groupthink is pretty bad.
Blind deference makes you unable to update. Yes, (insert smart person name) is also seeing the evidence and updating, but I should still try to do it myself, for some reasons mentioned in this list.
Mass deference corrodes the information ecosystem. Prediction markets, forecasting surveys, and community aggregates are only as good as the independence of their participants.
Blindly deferring to bad sources < blindly deferring to better sources < intentional deference < developing something like research taste on the actual object-level questions
Actually understanding why experts believe in what they believe in, and having an insider view model, is instrumentally super important to being able to do impactful work.
If more people did the work of understanding the actual arguments, the collective epistemics of the entire field would improve.
I should do less meta-reasoning-about-opinions (also known as some type of pseudo-philosophy or pseudo-reference class forecasting) and do more actual object level thinking.
Metaepistemology feels productive, rigorous and humble, but is also a way of never forming a view while feeling like I am doing something intellectually serious.
Timelines are scary but I do in fact need to just, learn stuff, and it actually surely doesn’t take that long.
(cross posted from my substack)
Lately, I realized that I have been making very incorrect statements regarding deference. Most of the statements I make here seem retrospectively quite obvious, yet I have managed to have overlooked all of them. This is an attempt to reconstruct the process through which I came to certain conclusions. AGI is scary and hard to think about. But that is not a reason to throw our minds away. And I think I did discard my mind for a bit.
The general case for deferring your opinions goes something along the lines of:
The general case for first principled thinking in goes something along the lines of:
I hear the discussion of “should you defer or not defer” frequently. But what does this actually mean?
What does it mean to “form your own views”? You’re not going to independently derive AI timelines from first principles. The part that gets left out in the “think for yourself” argument is that pure first-principles thinking also fails, and often fails worse. Applied naively to complex domains, it produces conclusions that are (maybe) logically coherent and practically insane. In politics, pure first-principles reasoning gets you things like “we should select leaders by lottery” or “the most rational people should rule” — conclusions that may be logically coherent yet practically insane. Form completely independent views on AI timelines” would mean something like: ignore all existing forecasts, learn ML from scratch, build your own compute scaling model, and arrive at a number. Nobody is actually proposing this, and rightfully so.
“Defer to the experts” is equally underspecified. Which experts? Deferring to the Metaculus aggregate is pretty different than deferring to Eliezer Yudkowsky, which is pretty different than deferring to the median ML professor who hasn’t thought about timelines for more than twenty minutes. And the choice of who counts as “expert” already contains a lots of judgment. Deferring to superforecasters implicitly weights calibration and track record. Deferring to lab leaders implicitly weights inside information. Deferring to the LessWrong median (?) implicitly weights community-specific reasoning norms. There exists no neutral deferring.
One version of “just defer” that I’ve considered, only half joking, is to literally roll a die: assign each face to a different forecaster and adopt whichever timeline comes up, and plan my career accordingly.
I think one of the reasons I came to such conclusions is doing too much “meta-thinking” rather than actually thinking:
There are all of these people, definitely smarter than me, spent thousands more hours on this, and have access to better information. What are the odds I can form a view more correct than Daniel Kokotajlo or Ajeya Cotra? Seems highly unlikely. Shouldn’t I just pick the source I trust most and go with their number? But how do I figure out what sets of characteristics makes someone more probabilistically likely to be correct on this subject? There are some basic traits, such as “has actually spent significant time thinking about this” but this seems to apply to all of these people. And I don’t know how to differentiate between the non-basic traits. And surely for any of these individual dimensions, each of these individual smart people have already considered them, and have thought longer and harder than I?
The Tetlock superforecasting approach suggests something like construct your prior from a reasonable aggregate and then ask which specific inside-view considerations should push you away from that number. But why should I even do this? Surely these smart people are already doing this, and how likely can I do a better job?
So there it goes, the clock is ticking and I should roll a die.
But this is naive and incorrect. My current-revised-thoughts are as follows: Better deferring is better than worse deferring. Mindful deferring is good. Mindful deferring involves a serious amount of actually thinking about object level things. Blind deference is bad, but not for the reason of “from a high level there is a non-trivial probabilistic likelihood that I would be more correct about AGI than (smart person)”. Specific updated reasonings and resolutions include: