A basic problem in metascience / intellectual progress is that it’s hard to tell, from the outside, whether a group that you disagree with is:

• “A self-dealing cabal enmeshed in groupthink”, versus
• “An externally-opaque meritocracy”, i.e. a bunch of smart people figuring things out in a meritocratic way, and sorry but you’re just not smart enough and truth-seeking enough to recognize that this group is right about everything while you’re wrong.

You just can’t tell those apart from the outside—i.e. without having the time and skill to dive into the object-level debates and come out with the right answer. And most people don’t have that kind of time and skill.

…Unless the group can produce easily-verifiable artifacts that any moron can recognize to be proof that they’re correct on the specific question at issue.

(“So that's all that Science really asks of you—the ability to accept reality when you're beat over the head with it.”)

…And sometimes there is no such artifact to be found! In those cases, even if the second bullet point is what’s really going on, the group is vulnerable to outside agitators accusing them of being the first bullet point, and running them out of town.

Here are two historical anecdotes, which I personally think exemplify this dynamic. But other people will say “No, one or both of those is actually the first bullet point!!” And I probably won’t be able to convince them that they’re wrong. Ironic! But that’s exactly the point!

(1) The breaching of the string theory consensus in the 2000s.^[1]

“The Standard Model of Particle Physics including weak-field quantum general relativity (GR)” (I wish it were better-known and had a catchier name) appears sufficient to explain everything that happens in the solar system (ref). Nobody has ever found any experiment violating it, despite extraordinarily precise and varied tests. This theory can’t explain everything that happens in the universe—in particular, it can’t make any predictions about either (i) microscopic exploding black holes or (ii) the Big Bang. Also, (iii) the Standard Model happens to include 18 elementary particles (depending on how you count), because those are the ones we’ve discovered; but the theoretical framework is fully compatible with other particles existing too, and indeed there are strong theoretical and astronomical reasons to think they do exist. It’s just that those other particles are irrelevant for anything happening on Earth—so irrelevant that we’ve spent decades and billions of dollars searching for any Earthly experiment whatsoever where they play a measurable role, without success. Anyway, I think there are strong reasons to believe that our universe follows some set of orderly laws—some well-defined mathematical framework that elegantly unifies the Standard Model with all of GR, not just weak-field GR—even if physicists don’t know what those laws are yet. So it would be cool to find those orderly laws. This is the goal of fundamental physics theory research.^[2]

Now, the extraordinary success of “The Standard Model of Particle Physics including weak-field quantum GR” is a double-edged sword. It’s good because it seems like we can already correctly predict the results of every fundamental-physics lab experiment that we’ve ever done or know how to do, which is very cool! It’s bad because we know that we don’t yet have the last word on how the universe works, but if we want to go further, and find those orderly laws mentioned above, we probably won’t be able to show off any experimental proof that we’re on the right track. Every experimental result is already covered! There’s nothing left!

So that’s the background context. Now in the 1980s–2000s, theoretical fundamental physicists investigated string theory with growing interest. They found that string theory had the intriguing property that, if a universe was governed by this kind of theory, (1) it would exactly satisfy GR (not just weak-field GR), always and everywhere, and (2) in the limit where our universe has the Standard Model of Particle Physics, a string theory universe would … well, it wouldn’t necessarily be exactly the Standard Model of Particle Physics, but at least it would be that kind of a thing, namely a “Quantum Field Theory”. And, (3) it would be well-defined everywhere, offering answers (in principle) even to questions about microscopic exploding black holes and so on, questions on which existing physics theory would return “[error: invalid input]”. None of that proves that our universe is governed by something like string theory, of course. But it sure suggests that it’s a promising research direction. So by the 2000s, the theoretical fundamental physics departments were full of string theory experts, judging the promise of each other’s string-theory papers along metrics that outsiders could not understand. Alas, this area of math / physics remains very hard, with many open questions, so as of today we still have neither a specific string theory model that exactly matches our universe, nor any reason to believe that one doesn’t exist. Work is ongoing.

Anyway, the string theorists, lacking any proof of progress that any moron could be impressed by, had formed into (what I think is) an externally-opaque meritocracy. However, others thought that it was a self-dealing cabal enmeshed in groupthink. So they led an anti-string-theory rebellion, which got a ton of press, especially through widely-discussed pop-science books by by Peter Woit and by Lee Smolin in 2006, saying that string theory was a failure and waste of money.^[3]

My vague impression is that this rebellion won a lot of converts in the general public, and a smaller but still significant number of converts even within physics departments. The converts were not the existing theoretical fundamental physicists, of course, but it turns out that physics departments are pretty broad, and so they also contain experimental physicists, and experts on thermodynamics of liquid crystals, and many other niches of people who don’t necessarily understand string theory, or why people were working on it, or why the string theorists were [rightly] skeptical of alternative approaches in the literature. And some of these insider converts to the rebellion were physics department chairs, or on hiring committees, or in charge of government physics grant allocation decisions, etc.

Thus, the string theory blockade was broken.

Fast-forward to today. There are still lots of string theorists, but now they have to work alongside a decent number of string-theory-naysayers working on directions that definitely won’t work, for reasons that members of the externally-opaque meritocracy of actually-good theoretical fundamental physicists understand well.

(2) The breaching of an analytic-philosophy consensus in 1979

This amusing anecdote comes from Dan Dennett’s memoir:

The hegemony of the analytic philosophers evaporated in 1979, at the Eastern Division meeting of the APA [American Philosophical Association] in Boston, when a coup d’état was staged by a group of mostly American but Continental philosophers who called themselves pluralists (let a thousand flowers bloom). I wonder how many of today’s young philosophers and graduate students have ever heard about this. It was an academic earthquake at the time. Frustrated by the short shrift given them by members of the “analytic monolith,” these philosophers studied the bylaws of the APA and discovered that although for decades the nominating committee had put forward a single candidate for vice president who was then elected by acclaim and would succeed as president the following year, the rules allowed nominations from the floor and actual elections! In secret, the pluralists put together their slate, prepared their challenges to the parliamentarian and other officers, and made sure their members were all set to descend en masse on the lightly attended business meeting and take over the APA Eastern Division. About half an hour before the meeting, their security broke down: a coup was rumored to be in the offing, and we monolith members were rounded up in the bar and hustled to the meeting to try to fend off the usurpation. Dick Rorty was president that year, and it was an irony (one of his favorite topics) that he—the most ecumenical and open-minded of the “analytic monolith” leaders—presided over the meeting, while Tom Nagel executed his duties as parliamentarian with aplomb. There were nominating speeches and rebuttals, the most memorable of which was by Ruth Marcus, whose Yale colleague John Smith, a philosopher of religion and a theologian, was the pluralists’ candidate. She explicitly trashed his whole career, his character, his books. I had never heard a philosopher speak so ill of a colleague in public, and seldom in private.

We lost. The establishment had nominated Adolf Grünbaum, a Pittsburgh philosopher of science, to be the new vice president. Not wanting to offend innocent Adolf, the victorious pluralists nominated and elected him vice president the following year, so that in 1982 he finally got to deliver the presidential address he had expected to give earlier. He did not accept the olive branch with equanimity. Adolf was famous for his tirades against Freud as an unscientific poseur, and his address was vintage Grünbaum. I happened to follow a cluster of pluralists out of the hall at the close of his address and overheard the reply when a pluralist who had stayed away asked how Grünbaum’s address had gone: “It was nasty, brutish and long.”

Thereafter, the APA’s programs were filled with papers on topics, and by philosophers, that would never have made the cut before the pluralist coup. Was this a good thing? Yes, said some monolith members, since it meant there was more guilt-free time to spend in the bar at conventions. Yes, said others, since the pluralists had justice on their side. My verdict is mixed. Still, the published programs of the APA meetings list dozens of talks whose titles are so ripe for parody that when I recently perused a few looking for likely examples to anonymize, I had difficulty “improving” on the actual candidates, but ask yourself whether you are aching to go to the sessions where the following talks will be given:

• “The Ineffability of History and the Problem of the Unitary Self”
• “Dialectical Encroachment: Humiliation and Integrity”
• “Can Relationalistic Ontology Avoid Incoherence through a Recursive Metatheory?”
• “Art as War: The Resilience of Autonomy”

Afterword

A related mental model^[4]

You can (A) judge ideas via new external evidence, and/or (B) judge ideas via internal discernment of plausibility, elegance, self-consistency, consistency with already-existing knowledge and observations, etc. There’s a big range in people’s ability to apply (B) to figure things out. But what happens in “normal” sciences like biology is that there are people with a lot of (B), and they can figure out what’s going on, on the basis of hints and indirect evidence. Others don’t. The former group can gather ever-more-direct and ever-more-unassailable (A)-type evidence over time, and use that evidence as a cudgel with which to beat the latter group over the head until they finally get it. (“If you don’t believe my 7 independent lines of evidence for plate tectonics, OK fine I’ll go to the mid-Atlantic ridge and gather even more lines of evidence…”)

This is an important social tool, and explains why bad scientific ideas can die, while bad philosophy ideas live forever. And it’s even worse than that—if the bad philosophy ideas don’t die, then there’s no common knowledge that the bad philosophers are bad, and then they can rise in the ranks and hire other bad philosophers etc. Basically, to a first approximation, I think humans and human institutions are not really up to the task of making intellectual progress systematically over time, except where idiot-proof verification exists for that intellectual progress (for an appropriate definition of “idiot”, and with some other caveats).

…And another mental model

In The Median Researcher Problem, John Wentworth suggests that fields can sustain a healthy meritocracy approximately when a median researcher can judge good work from bad, and [John forgot to add] actually cares and is motivated and empowered to publicly praise the good stuff and rip apart the bad stuff. Here I’m suggesting that string theory and analytic philosophy were doing OK on those metrics. But then the rebels effectively expanded the pool of people whose opinions mattered, thus dragging down the median and collapsing the equilibrium.

Can an externally-opaque meritocracy gain credibility via racking up externally-legible achievements in other adjacent domains?

Alas, the answer seems to be basically “no”.

The string theorists kept proposing lots of fascinating, deep, non-obvious math conjectures, and then the mathematicians kept proving that those conjectures were true. Did that give pause to Woit, Smolin, and the other anti-string-theory agitators? Nope! They brushed it aside with a one-sentence retort: “Well so what, if you’re doing good math, then go get math grants, I’m arguing that you’re a failure at physics!”

Likewise, I suspect that the analytic philosophers of 1979 could trounce the continental philosophers at, I dunno, externally-legible tests vaguely related to philosophical reasoning, like the LSAT. If so, the rebels didn’t care.

And I mean, it’s not even that irrational to not care! It’s totally possible for people to be great at one domain but completely wrong about some domain that seems adjacent. For example, Geoff Hinton and Yann LeCun both have stellar and well-deserved reputations as AI experts, but they’re diametrically opposed on the issue of AI extinction risk, which means that at least one of them is catastrophically wrong. Speaking of which…

This post is secretly about superintelligent AI, isn’t it?

Yup! If you can reason well about what’s gonna happen with superintelligence, then that just doesn’t help you create easily-verifiable artifacts that any moron can recognize to be proof that you are correct on the specific questions at issue. Maybe you can try to get credibility in adjacent, more-verifiable domains, but per above, we should expect that to be only slightly helpful.

So, let’s suppose we follow @johnswentworth’s advice to form a Superintelligence Alignment Researchers Who Are Actually Really Good At It Club, and we reject 99% of applicants to our club,^[5] based on existing club members aggressively scrutinizing the skills and judgment of each new applicant.

If we do that, at best we can create an externally-opaque meritocracy. And we will be vulnerable to attack from outsiders, who will say that our club is not in fact an externally-opaque meritocracy, but rather a self-dealing cabal enmeshed in groupthink. And those outsiders, right or wrong, will be widely believed.

(Maybe our meritocracy-or-perhaps-cabal can get funding, from the scattered people who believe us. And maybe we can make great intellectual progress within our little bubble. But getting broad acceptance of our ideas, in a way that’s obvious to outsiders, is a different matter entirely.)

Of course, I’m not saying anything here that isn’t already very obvious. Nor do I have any great solution ¯\ˍ(ツ)ˍ/¯ The not-great solution is to cross our fingers and hope that a sufficient number of key decisionmakers have the time and talent to wade into the object-level debates and correctly figure out who is right on questions of superintelligence alignment. (Or hope that the key decisionmakers follow their local vibes and memes, and wind up trusting the right people by coincidence.) Anyway, our meritocracy-or-perhaps-cabal can of course help on the margin by doing good pedagogy, outreach, etc.

1. ^{^}

   I’m confident in what I say about physics, but less confident in the history-of-physics parts. I wasn’t directly involved (I was a student in physics academia at the time, but not fundamental physics), and am going off vague memories of stuff I read or heard. People can correct me. In other news, the first paragraph of this section is partly copied from here.

2. ^{^}

   No comment on whether pursuing this goal is a good use of expert time and taxpayer money. That wasn’t the issue; people on both sides of the string wars were in favor of pursuing this goal.

3. ^{^}

   An additional factor, which built on the momentum from Woit, Smolin, and others, was the failure in the early 2010s of the Large Hadron Collider (LHC) to find “superpartner” particles. String theory makes a strong prediction that superpartners exist. (Indeed, even without string theory, there would be strong reasons to believe that superpartners exist!) String theory did not make a strong prediction that superpartners would definitely be found at the LHC; rather, it predicted that superpartners might be found at the LHC. The best string theorists were well aware of this fact, but the meme that the LHC would imminently prove or disprove string theory nevertheless spread widely, for lots of reasons including: (1) anti-string-theory people spread this idea out of ignorance and/or malice, and/or (2) some string theorists were myopically over-hyping their latest speculative theoretical models, and/or (3) everyone wanted to convince politicians to keep funding the LHC, and/or (4) nuance was lost in a game of telephone. Anyway, the LHC turned on, no superpartners were found, and many people were upset that the string theorists were not admitting defeat.

   (There’s an obvious parallel with how people in my own field of Artifical General Intelligence (AGI) safety have been trumpeting the possibility that AGI might happen very very soon. To be clear, maybe trumpeting that possibility is the right thing to do, all things considered, but we should obviously be braced for a backlash.)

4. ^{^}

   This section is mostly copied from §1.4 here.

5. ^{^}

   I say “our club” because I strongly believe John would let me in. Or if not, screw him, I’ll start my own club.