This got over 800 points on HN. Having a good reply seems important, even if a large portion of it is.. scattergun, admittedly and intentionally trying to push a point, and not good reasoning.

The core argument, that the several reference classes that Superintelligence and AI safety ideas fall into (promise of potential immortality, impending apocalypse, etc) are full of risks of triggering biases, that other sets of ideas in this area don't hold up to scrutiny, and that it has other properties that should make you wary is correct. It is entirely reasonable to take this as Bayesian evidence against the ideas. I have run into this as the core reason for rejecting this cluster of beliefs several times, by people with otherwise good reasoning skills.

Given limited time to evaluate claims, I can see how relying on this kind of reference class heuristic seems like a pretty good strategy, especially if you don't think black swans are something you should try hard to look for.

My reply is that:

1. This only provides some evidence. In particular, there is a one-time update available from being in a suspect reference class, not an endless stream from an experiment you can repeat to gain increasing confidence. Make it clear you have made this update (and actually make it).
2. There are enough outside-view things which indicate that it's different from the other members of the suspect reference classes that strongly rejecting it seems unreasonable. Support from a large number of visibly intellectually impressive people is the core thing to point to here (not as an attempt to prove it or argue from authority, just to show it's different from e.g. the 2012 stuff).
3. (only applicable if you personally have a reasonably strong model of AI safety) I let people zoom in on my map of the space, and attempt to break the ideas with nitpicks. If you don't personally have a clear model, that's fine, but be honest about where your confidence comes from.

To summarize: Yes, it pattern matches to some sketchy things. It also has characteristics they don't, like being unusually appealing to smart thoughtful people who seem to be trying to seek truth and abandon wrong beliefs. Having a moderately strong prior against it based on this is reasonable, as is having a prior for it, depending on how strongly you weight overtly impressive people publicly supporting it. If you don't want to look into it based on that, fair enough, but I have and doing so (including looking for criticism) caused me to arrive at my current credence.