This is a linkpost for https://philiptrammell.com/blog/46
@1a3orn goes deeper into another dynamic that causes groups to have false beliefs while believing they are true, and it's the fact that some bullshit beliefs help you figure out who to exclude, which is the people who don't currently hold the belief, and in particular assholery also helps people who don't want their claims checked, and it's a reason I think politeness is actually useful in practice for rationality:
(Sharmake's first tweet): I wrote something on a general version of this selection effect, and why it's so hard to evaluate surprising/extreme claims relative to your beliefs, and it's even harder if we expect heavy-tailed performance, as happens in our universe.
(1a3orn's claims) This is good. I think another important aspect of the multi-stage dynamic here is that it predicts that movements with *worse* stages at some point have fewer contrary arguments at later points...
...and in this respect is like an advance-fee scam, where deliberately non-credible aspects of the story help filter people early on so that only people apt to buy-in reach later parts.
Paper on Why do Nigerian Scammers Say They are from Nigeria?
So it might be adaptive (survivalwise) for a memeplex to have some bullshit beliefs because the filtering effect of these means that there will be fewer refutations of the rest of the beliefs.
It can also be adaptive (survivalwise) for a leader of some belief system to be abrasive, an asshole, etc, because fewer people will bother reading them => "wow look how no one can refute my arguments"
(Sharmake's response) I didn't cover the case where the belief structure is set up as a scam, and instead focused on where even if we are assuming LWers are trying to get at truth and aren't adversarial, the very fact that this effect exists combined with heavy-tails makes it hard to evaluate claims.
But good points anyway.
(1a3orn's final point)
Yeah tbc, I think that if you just blindly run natural selection over belief systems, you get belief systems shaped like this regardless of the intentions of the people inside it. It's just an effective structure.
Quotes from this tweet thread.
This seems like it's engaging with the question of "what do critics think?" in a sort of model-free, uninformed, "who to defer to" sort of way.
For awhile, I didn't fully update on arguments for AI Risk being a Big Deal because the arguments were kinda complex and I could imagine clever arguers convincing me of it without it being true. One of the things that updated me over the course of 4 years was actually reading the replies (including by people like Hanson) and thinking "man, they didn't seem to even understand or address the main points."
i.e. it's not that they didn't engage with the arguments, it's that they engaged with the arguments badly which lowered my credence on taking their opinion seriously.
(I think nowadays I have seen some critics who do seem to me to have engaged with most of the real points. None of their counterarguments seem like they've added up to "AI is not a huge fucking deal that is extremely risky" in a way that makes any sense to me, but, some of them add up to alternate frames of looking at the problem that might shift what is the best thing(s) to do about it)
I agree that critics engaging with arguments badly is an update towards the arguments being real, but I am essentially claiming that the fact that this selection effect exists and is very difficult to eliminate/reduce to a useful level implicitly means that you can only get a very limited amount of evidence from arguments.
One particular part of my model here is that selection effects are usually very strong and difficult to eliminate by default, unfortunately, and thus one of the central problems of science in general is how to deal with this sort of effect.
But it's nice to hear from you on how you've come to believe in AI risk being a big deal.
Great post.
I wanted to pick on the model of "sequentially hear out arguments, then stop when get fed up with one," but I think it doesn't make too much difference compared to a more spread-out model where people engage with all the arguments but at different rates, and get fed up globally rather than locally.
There's an interestingly pernicious version of a selection effect that occurs in epistemology, where people can be led into false claims because when people try to engage with arguments, people will drop out at random steps, and past a few steps or so, the people who believe in all the arguments will have a secure-feeling position that the arguments are right, and that people who object to the arguments are (insane/ridiculous/obviously trolling), no matter whether the claim is true:
What's going wrong, I think, is something like this. People encounter uncommonly-believed propositions now and then, like “AI safety research is the most valuable use of philanthropic money and talent in the world” or “Sikhism is true”, and decide whether or not to investigate them further. If they decide to hear out a first round of arguments but don't find them compelling enough, they drop out of the process. (Let's say that how compelling an argument seems is its “true strength” plus some random, mean-zero error.) If they do find the arguments compelling enough, they consider further investigation worth their time. They then tell the evangelist (or search engine or whatever) why they still object to the claim, and the evangelist (or whatever) brings a second round of arguments in reply. The process repeats.
As should be clear, this process can, after a few iterations, produce a situation in which most of those who have engaged with the arguments for a claim beyond some depth believe in it. But this is just because of the filtering mechanism: the deeper arguments were only ever exposed to people who were already, coincidentally, persuaded by the initial arguments. If people were chosen at random and forced to hear out all the arguments, most would not be persuaded.
Perhaps more disturbingly, if the case for the claim in question is presented as a long fuzzy inference, with each step seeming plausible on its own, individuals will drop out of the process by rejecting the argument at random steps, each of which most observers would accept. Believers will then be in the extremely secure-feeling position of knowing not only that most people who engage with the arguments are believers, but even that, for any particular skeptic, her particular reason for skepticism seems false to almost everyone who knows its counterargument.
In particular, if we combine this with a heavy tailed distribution of performance at fields, where people have exponential-drop off in intelligence, meaning that a few people matter a lot more in progress than most people, it means that it's very difficult to distinguish cases where the small/insular group arguing for something extreme relative to their current distribution is correct and everyone else doesn't get the arguments/data, and the cases where the small group is being fooled by a selection effect and the conclusion is actually false.
I'll just quote it in full, since there's no other better way to summarize this/link to it:
Yeah. In science the association with things like scientific output, prizes, things like that, there's a strong correlation and it seems like an exponential effect. It's not a binary drop-off. There would be levels at which people cannot learn the relevant fields, they can't keep the skills in mind faster than they forget them. It's not a divide where there's Einstein and the group that is 10 times as populous as that just can't do it. Or the group that's 100 times as populous as that suddenly can't do it. The ability to do the things earlier with less evidence and such falls off at a faster rate in Mathematics and theoretical Physics and such than in most fields.
Yes, people would have discovered general relativity just from the overwhelming data and other people would have done it after Einstein.
No, that intuition is not necessarily correct. Machine learning certainly is an area that rewards ability but it's also a field where empirics and engineering have been enormously influential. If you're drawing the correlations compared to theoretical physics and pure mathematics, I think you'll find a lower correlation with cognitive ability.
There's obviously implications for our belief in AI risk/AI power in general, but this is pretty applicable to a lot of fields, and probably explains at least some of the skepticism lots of people have towards groups that make weird/surprising/extreme claims (relative to their world model).