I've struggled to engage very productively with the rest of the AI safety community about politics—e.g. there are a lot of disagreeing votes and comments on this post. By and large people have been respectful and polite (as per the high upvote count), but right now it doesn't feel like LW is a place where I can collaboratively make intellectual progress on this very important topic.

This is sad, so before I stop trying I want to attempt to go meta, by explaining how this seems analogous to the ways that the alignment community struggled to engage with mainstream ML researchers throughout the 2010s and early 2020s. My explanation of those dynamics ended up growing into this post; in this shortform I'll discuss the analogy to politics specifically.

The main point is the following. With enough discussion, you could often get a mainstream ML researcher to admit that something like situational awareness or recursive self-improvement might in principle be possible. But it's very hard to get them to take it seriously enough that it propagates through their ontology—and so after that one conversation they'd typically just go back to their standard ML research. This is for a bunch of reasons—in part because it's genuinely hard to update one's ontology, in part because their social incentives and identity push away from doing so, in part because they're scared about the implications of propagating this belief.

Similarly, I think AI safety people (especially those in the LW cluster) are usually intellectually honest enough to be able to acknowledge that various heretical political beliefs might be true. However, there's an additional step of propagating it through their ontology which typically doesn't happen due to mental blocks.

For example, in this post Scott Alexander is willing to mention the possibility that heritable racial IQ gaps exist as one of four hypotheses for observed racial disparities. However, observe how he immediately distances himself from the position:

White people have average IQ 100, black people have average IQ 85, this IQ difference accurately predicts the different proportions of whites and blacks in most areas, most IQ differences within race are genetic, maybe across-race ones are genetic too. I love Hitler and want to marry him.

This is particularly striking because I'm pretty confident he himself believes some version of this hypothesis! So he's basically calling himself a Nazi to defuse the discomfort of even raising a hypothesis this controversial (let alone endorsing it). This is objectively a very odd thing to do.

Now, Scott Alexander has historically been very brave in a bunch of ways that I wasn't, so I don't want to try to take any kind of moral high ground. I merely want to point out that it's pretty obvious how this kind of mental block might make it hard to actually propagate your beliefs. Another example: when I talked to one of the most curious and polymathic people I know about a controversial topic, his immediate response was "but does that even matter?" On other topics, he would have followed his curiosity to play around with the ideas; on this one, he tried to block it off as quickly as possible.

I was in a pretty similar mindset for a long time, and required a strong sense of social safety before I managed to get myself out of it. My experience since then has been that, when you move from this kind of blocked partial acceptance of controversial beliefs, to a mindset where you're actually able to follow them wherever they lead, there are a lot of important implications. I want to keep this post pretty meta so that more people feel comfortable engaging with it, but as one fairly milquetoast example: after seeing just how strong self-deception around taboos can be in humans, it seems pretty important to prevent anything similar from happening in AIs.

And just like with alignment, I think that all of this is giving us clues towards a whole new ontology that actually conveys very important principles about how the world works. Trying to do AI governance in the standard political ontology really feels to me like trying to do alignment research while stuck in the ML ontology. More on this in other posts, but for now I hope that this post helps other LWers better understand where I'm coming from.