I think there's a single framework that captures several of these ideas: the status quo is sticky, and it's hard to steer out of, so if you think things are bad enough, you shake things up into a transient chaotic state and then as it settles down, you can guide the system into a state you like better. Different groups executing this strategy may seem aligned at the beginning, and their conflicting goals become apparent only after the status quo has been toppled. And it's possible to lose badly enough in this second phase that you end up worse off than before, so that's the gamble one takes when engaging in revolution instead of incrementalism.
That seems roughly right but I don't think chaos/revolution is necessary for this thesis.
Imagine the world sitting in a deep basin. You're trying to get it out. You can try a revolution: launch it up into the air and hold your breath waiting for it to land.
Or you can try, sisyphus-style, to roll it out. You might get a bunch of allies, who you try and lead along a narrow and precarious ridgeline to your destination. Then, when you're halfway to the top, some might notice that there's a much easier path requiring much less of a push, and so the world gets pushed down off your intended route along the mountain pass and into a new crater.
The latter seems to have been the case with environmentalism and AI Safety. They pushed the world fairly gradually, but it only takes a few individuals to push the world downwards into a new equilibrium, for the upward pushers to lose that fight.
(There's a Yudkowsky tweet somewhere which says something like this, which I've basically based the entire metaphor here on)
1Thanks for engaging! There's a lot here I agree with—in particular, the concept of pessimization does seem like a dangerous one which could be used to demoralize people. I also think psychoanalyzing me is fair game here, and that it would be a big strike against the concept if I were using it badly.
I'm trying to figure out if there's some underlying crux here, and the part that gets closest to it is maybe:
I think Richard makes an important error when he complains about existing activist-ish groups: he compares these groups to an imaginary version of the activist group which doesn't make any mistakes. Richard seems to see all mistakes made by activist groups as unforced and indicative of deep problems or malice.
I don't know how you feel about the concept of Moloch, but I think you could probably have written a pretty similar essay about that concept. In each individual case you could characterize a coordination failure as just an "ordinary failure", rather than a manifestation of the larger pattern that constitutes Moloch. And indeed your paragraph above is strikingly similar to my own critique of the concept of Moloch, which basically argues that Scott is comparing existing coordination failures to an imaginary world which has perfect coordination. I've also made similar critiques of Eliezer's concept of "civilizational inadequacy" as measuring down from perfection.
I think that the synthesis here is that neither pessimization nor Moloch nor "civilizational inadequacy" should be treated as sufficiently load-bearing that they should tell you what to do directly. In some sense all of these create awayness motivations: don't pessimize, don't be inadequate, don't let Moloch win. But as Malcolm Ocean points out, awayness motivations are very bad for steering. If your guiding principle is not to be inadequate, then you will probably not dream very big. If your guiding principle is not to pessimize, then people will probably just throw accusations of pessimization at each other until everything collapses into a big mess.
That's why I ended the post by talking about virtue ethics, and how it can be construed as a technology for avoiding pessimization. I want to end up in a place where people almost never say to each other "stop pessimizing", they instead say "be virtuous". But in order to argue for virtues as the solution to pessimization/the way to build the "imaginary version" of groups which don't make such unforced errors, I need to first point at one of the big problems they're trying to solve. It's also worth noting that a major research goal of mine is to pin down mechanisms of pessimization more formally and precisely, and if I fail then that should count as a significant strike against the concept.
I'm not 100% sure that this is the right synthesis, and will need to muse on it more, but I appreciate your push to get this clearer in my head (and on LessWrong).
Lastly, at risk of turning this political, the one thing I'll say about the "support Hamas" stuff is that there's a spectrum of what counts as "support", from literally signing up to fight for them to cheering them on to dogwhistling in support of them to just pushing for some of the same goals that they do to failing to condemn them. My contention is that there are important ways in which Hamas' lack of alignment with western values leads to more western support for them—e.g. the wave of pro-Palestine rallies immediately after they killed many civilians—which is what makes this an example of pessimization. Of course this is a dangerous kind of accusation because there's a lot of wiggle room in exactly what we mean by "lack of alignment", and distinctions between supporting Hamas itself vs supporting associated causes. I personally still think the effect is stark enough that my core point was correct, but I should have phrased it more carefully. (Note: I edited this paragraph a few mins after writing it, because the original version wasn't very thoughtful.)
1Thanks for the response! I feel like I understand your position quite a lot better now, and see the place where "pessimization" fits into a mental model a lot better. My version of your synthesis is something like as follows:
"Activists often work in very adversarial domains, in the obvious and non-obvious ways. If they screw up even a little bit, this can make them much, much less effective, causing more harm to their cause than a similarly-sized screwup would in most non-adversarial domains. This process is important enough to need a special name, even though individual cases of it might be quite different. Once we've named it, we can see if there are any robust solutions to all or most of those cases."
Based on this, I currently think of the concept of pessimization as making a prediction about the world: virtue ethics (or something like it) is a good solution to most or all of these problems, which means the problems themselves shared something in common, which is worthy of a label.
It's also worth noting that a major research goal of mine is to pin down mechanisms of pessimization more formally and precisely, and if I fail then that should count as a significant strike against the concept.
This is absolutely intriguing do you have anything more written about this publicly?
Ramble
At the risk of being too much of an "Everything Is Connected" guy, I think there's a connection between the following (italicized items are things I've thought about or worked on recently)
- storing passwords in plain text|encrypting passwords on a secure part of the disk|salting and hashing passwords
- naive reasoning|studying fallacies and biases|learning to recognise a robust world model
- utilitarianism|deontology|virtue ethics
It doesn't quite fit, which is a little annoying. But the level one vs level two security mindset thing comes to mind when I think about deontology vs virtue ethics.
Deontology seeks to find specific rules to constrain humans away from particular failure modes "Don't overthrow your democratically elected leader in a bloody revolution even if the glorious leader really would be a good god-emperor" and the like.
Perhaps a good version of virtue ethics would work like the true security mindset, although I don't know whether the resulting version of virtue ethics would look much like what the Athenians were talking about.
My version of your synthesis is something like as follows:
This is closer; I'd just add that I don't think activism is too different from other high-stakes domains, and I discuss it mainly because people seem to take activists more at face value than other entities. For example, I expect that law firms often pessimize their stated values (of e.g. respect for the law) but this surprises people less. More generally, when you experience a lot of internal conflict, every domain is an adversarial domain (against parts of yourself).
I think there's a connection between the following
- storing passwords in plain text|encrypting passwords on a secure part of the disk|salting and hashing passwords
- naive reasoning|studying fallacies and biases|learning to recognise a robust world model
- utilitarianism|deontology|virtue ethics
I think you lost the italics somewhere. Some comments on these analogies:
- The idea that some types of cognition are "fallacies" or "biases" and others aren't does seem like a pretty deontological way of thinking about the world, insofar as it implicitly claims that you can reason well just by avoiding fallacies and biases.
- As the third step in this analogy, instead of "learning to recognize a robust world model", I'd put "carrying out internal compromises", i.e. figuring out how to reduce conflict between heuristics and naive reasoning and other internal subagents.
- Re the passwords analogy: yes, deontology and virtue ethics are adversarially robust in a way that utilitarianism isn't. But also, virtue ethics is scalable in a way that deontology isn't, which seems well-captured by the distinction between storing passwords on secure disks vs salting and hashing them.
Some failure modes in reasoning point specifically away from truth, rather than just adding noise to beliefs.
It seems uncommon that these are predictably stronger than the pressure for truth, so I agree movements that backfired usually just got unlucky and so made one or two big mistakes.
https://www.lesswrong.com/posts/qTSyxtWRCpi3fcmyb/wrong-answer-bias
TL;DR I think "Pessimization" is a bad concept to think about in the case of activist movements. I think most of it can be explained by ordinary failures, and that using pessimization as a concept is worse than just thinking about the ordinary causes of failure on an object level.
Pessimization
"Perverse Pessimization", says Richard Ngo, is when an organization nominally dedicated to X ends up causing not-X. This is particularly common amongst activist groups according to Richard, and he has a separate thread calling it "The Activist's Curse"
Two of his choice examples are pretty weird to me (Liberalism? Liberalism?? The ideology which took over the half the world and reigned supreme for at least half a century, which is widely agreed to be the best half-century ever. Is the "pessimization" of liberalism that eventually people stopped doing liberalism? And I don't see how transhumanism has made us any less transhuman) so I'll use the examples I understand.
Communism:[1] the Bolsheviks wanted to remove the power that a smallish aristocratic class had over them, but ended up creating an even smaller and more tyrannical ruling class.
Environmentalism: environmentalists wanted to make the environment better, but they fucked up so badly by being anti-nuclear that they ended up making everything worse.
AI Safety: the AI Safety community wanted to make people concerned about AI, so they wouldn't recklessly build AI that kills everyone, but many of the companies recklessly building AI came out of the AI Safety community.
Losing
So there's a mystery here: communities trying to cause X end up causing Not-X. Or do they? Here's another hypothesis:
Communism: the Bolsheviks identified the problem of a tyrannical aristocracy. They got rid of the Tsar, but were unable to prevent a different tyrannical ruling class from taking its place.
Environmentalism: environmentalists noticed that policy did not systematically focus on environmental concerns. They changed a lot of policy in different ways, but the poor quality of the evidence-to-policy pipeline amplified the NIMBY voices (particularly for nuclear power) more than it amplified the voices saying nuclear was a good idea.
AI Safety: early AI Safety advocates identified the problem that AI was dangerous and companies would be reckless. They tried to tell people about the problem, but some people didn't listen and built those companies anyway.
Being Wrong
Or maybe there's a third option.
Communism: the Bolsheviks misunderstood how economics and power works, so their policies just killed a bunch of people.
Environmentalism: environmentalists made a few wrong calls on nuclear, and also often make wrong calls on local vs wider issues. Because of the way our laws are set up, these NIMBYist mistakes get amplified and mess everything up.
AI Safety: the AI Safety people under-estimated how many people would get a distorted version of their message, and these people heard "AI is a huge ..." instead of "AI is a huge threat"
Opportunists
Communism: Stalin used "Communism" as an excuse to become an awful dictator, because communism was the most popular ideology of the day.
Environmentalism: people who have never once in their life thought about environmentalism mysteriously become concerned about "Environmentalism" the moment anything is about to be built near them.
AI Safety: opportunistic businessmen used "AI Safety" as a way to get money for their causes, while not caring about AI safety at all.
This is kind of similar to one of Richard's points about sociopaths.
Replacement
I think Richard makes an important error when he complains about existing activist-ish groups: he compares these groups to an imaginary version of the activist group which doesn't make any mistakes. Richard seems to see all mistakes made by activist groups as unforced and indicative of deep problems or malice.
If you're running an activist group, you actually only have two options: change your policies to take in or exclude some members on some front, or cease to exist. I'll illustrate with environmentalism:
Scenario:
Suppose you're protesting an energy company buying up ecologically-significant wetlands to build a big oil refinery, because it would increase carbon emissions. You have to choose which of the following people to let into your coalition:
You don't know if you need two of these groups on your side, or five. Do you pick all five to be safe, or just pick the best two? Or hedge with three?
And what about what comes after?
A: A green-energy company might be buying up farmland in a completely different area in order to build solar panels.
B: Some farmers might be lobbying for some moors to be converted to a farm
C: A company might trying to put an offshore wind farm near some cliffs where birds are nesting.
Maybe you're in favour of the solar panels in case A, and would end up lobbying against people from group 3 in that case. But maybe group 3 is huge and the only way to counter the farmers in scenario B.
This is the sort of dilemma you end up in when you do activism. It's difficult, and errors are usually not-unforced.
Your other choice as an activist group is to shut down and re-roll. In that case, maybe someone comes along and does better, or maybe they do worse. Even if your activist group is doing badly, that doesn't mean that a random replacement would have done better.
Synthesis
I think my view is something like this:
To put it all together:
The Bolsheviks wanted to end the tyranny of a small ruling class. This was inherently difficult, because countries in general, and Russia especially, has a very powerful attractor state where there is a small, tyrannical ruling class. They made some serious mistakes about how to set up a system of good government, which made everything worse. Their ideology, once popular, was exploited by opportunists who wanted to take as much power as possible. Overall, they were probably worse than the average replacement revolution (Stalin was really bad) but not the worst possible revolutionary movement (they did lead to Gorbachev, who led to a period of democracy where the country seemed to have some hope).
Environmentalists wanted to make the world and its policymakers care about the environment. This was inherently difficult because improving the environment (in the ways environmentalists care about) is not very attractive as a policy system. They made some serious mistakes about which technologies were net good vs net bad (nuclear) and this interacted with structural biases in policies (towards NIMBYism) and opportunists (general NIMBYs). Overall, I'm unclear on whether organized university-activist-driven environmentalism was more or less effective than whatever would have taken its place.[1]
AI Safety people wanted to stop an uncoordinated race to AI which kills everyone. This was inherently difficult because security mindset is rare, the world is badly coordinated, and AI safety is very difficult. The incentives for companies to lobby against AI regulation are strong. They made some mistakes about how their message would come across (many people listened to the part about AI being big, but didn't grasp the difficulties of making it safe) which led to more attention overall on AI. Some of these people were opportunists who claimed "AI Safety" to get money for companies, and many grantmakers were duped by these claims. I think the AI Safety community (going back to ~2004) have performed well above replacement because the replacement community would be made up of AI-optimist futurists and not even really notice the risks until we smashed headlong into them. Humanity would likely die with even less dignity in such a world.
Red-Thing-Ism
Apologies for psychoanalyzing Richard Ngo, but he's the main person who's using this concept in practice.
I suspect that Richard has been a bit guilty of red-thing-ism when it comes to the activist's curse and pessimization. I don't think that the features of pessimization are well-formed enough to be thrown around as a solid concept. Here he, having been called out for a very wrong position (that western elites generally support Hamas) says it fits neatly into his model of pessimization.
I think it's a big strike against pessimization if the originator of the concept is using it as a gear and that gear ends up being a vector for twitter mind-killing. The concept can very easily slide into a mental shortcut: "Of course these activists/elites did something dumb and bad, it's pessimization!".
End of psychoanalysis, nothing else here is aimed at Richard.
This reminds me a bit of a different case. If you're rich, and you have a different opinion on some issue to a poor person, you can expect to be hit by cries of "Luxury belief!" In the same way, if you're trying and failing to achieve something, you can basically always be hit by accusations of "Pessimization!".
If the concept becomes popularized, I think it will have the following effects:
So I don't think that pessimization (as the concept exists now) is a great gear to add to one's world-model.
Coda
On the other hand, the possible explanations for this "effect" are worth thinking about! If you're doing any activist-ish stuff, you should absolutely be looking through the twelve things Richard talks about in the thread. These are real failure modes, but they're just failure modes.
If you set out to do something hard, you should expect to find failure modes. But that doesn't mean the better option is to just lie down and give up.
I find it hard to fully blame environmentalism for the state of nuclear power in the world. For example, the UK also cannot build reservoirs or housing at the moment, even in cases where environmentalism doesn't cause problems (see the new fire authority, or rules surrounding having windows on two sides of every flat)