Talking about memetic evolution puts me in mind of You Get About Five Words.  So in that spirit, I'd like to try making soundbites of the counter-memes above:

1.  "Let our children decide AI" "Let future people choose AI, not us" "AI Now is too soon"

2.  "Humanity has so much potential" "Normal humans shouldn't choose AI" "Max out IQ before AI" or "IQ before AI" "Even Picard couldn't handle where AI is going"

3.  "The AI you want isn't the AI you'll get"  "Nothing stops AI from being awful"

4.  "AI psychosis is just the beginning" "Satan was a flatterer too" 

5.  "AI Replacement is Literally Fascism" "History shouldn't end with AI" "Nazis thought they were inevitable too" "AI Might doesn't make AI Right"

6.  "Humans really do matter" "Love your neighbor, not computers" "AI can't replace family and friends"

7.  "You can't take AI replacement back" "As though we're in a position to permanently decide the future" "No guarantee AIs have souls" "AI can't answer life's questions"

8.  "AI replacment won't be voluntary" "ChatGPT 8 Wants Little Timmy's Atoms" 

9.  "AI replacement betrays humanity" "Does everyone you could ever know really deserve to be replaced by AI?  Even me?  Even everyone I care about?" "Me when I destroy humanity's hopes and dreams because my AI girlfriend said I'm a good boy:"

And these are just the first things that came to my mind with an hour of effort, no doubt others could do a lot better.  Perhaps referencing pieces of culture I didn't think of to sum up concepts, or just putting things more wittily.  Maybe this task should be referred to experts, or AI.

Edit:  I guess this goes in the opposite direction of Richard Ngo's point about how this represents an escalation in memetic warfare between AI safety and accelerationism.  Now I feel kinda bad for essentially manufacturing ammunition for that.

Death and AI successionism or AI doom are similar because they feel difficult to avoid and therefore it's insightful to analyze how people currently cope with death as a model of how they might later cope with AI takeover or AI successionism.

Regarding death, similar to what you described in the post, I think people often begin with a mindset of confused, uncomfortable dissonance. Then they usually converge on one of a few predictable narratives:

1. Acceptance: "Death is inevitable, so trying to fight it is pointless." Given the inevitability and unavoidability of death, worrying about it or putting effort into avoiding it is futile and pointless. Just swallow the bitter truth and go on living.

2. Denial: Avoiding the topic or distracting oneself from the implications.

3. Positive reframing: Turning death into something desirable or meaningful. As Eliezer Yudkowsky has pointed out, if you were hit on the head with a baseball bat every week, you’d eventually start saying it built character. Many people rationalize death as “natural” or essential to meaning.

Your post seems mostly about mindset #3: AI successionism framed as good or even noble. I’d expect #2 and #3 to be strong psychological attractors as well, but based on personal experience, #1 seems most likely.

I see all three as cognitive distortions: comforting stories designed to reduce dissonance rather than finding an accurate model of reality.

A more truth-seeking and honest mindset is to acknowledge unpleasant realities (death, AI risk), that these events may be likely but not guaranteed, and then ask what actions increase the probability of positive outcomes and decrease negative ones. This is the kind of mindset that is described in IABIED.

I also think a good heuristic is to be skeptical of narratives that minimize human agency or suppress moral obligations to act (e.g. "it's inevitable so why try").

Some agreements and disagreements:

1. I think that memetic forces are extremely powerful and underrated. In particular, previous discussions of memetics have focused too much on individual memes rather than larger-scale memeplexes like AI successionism. I expect that there's a lot of important scientific thinking to be done about the dynamics of memeplexes.
2. I think this post is probably a small step backwards for our collective understanding of large-scale memeplexes (and have downvoted accordingly) because it deeply entangles discussion of memetic forces in general with the specific memeplex of AI successionism. It's kinda like if Eliezer's original sequences had constantly referred back to Republicans as central examples of cognitive biases. (Indeed, he says he regrets even using religion so much as an example of cognitive bias.) It's also bad form to psychologize one's political opponents before actually responding to their object-level arguments. So I wish this had been three separate posts, one about the mechanics of memeplexes (neutral enough that both sides could agree with it), a second debunking AI successionism, and a third making claims about the memetics of AI successionism. Obviously that's significantly more work but I think that even roughly the same material would be better as three posts, or at least as one post with that three-part ordering.
3. You might argue that this is justified because AI successionism is driven by unusually strong memetic forces. But I think you could write a pretty similar post with pretty similar arguments except replacing "AI accelerationism" with "AI safety". Indeed, you could think of this post as an example of the "AI safety" memeplex developing a new weapon (meta-level discussions of the memetic basis of the views of its opponents) to defeat its enemy, the "AI successionism" memeplex. Of course, AI accelerationists have been psychologizing safetyists for a while (and vice versa), so this is not an unprecedented weapon, but it's significantly more sophisticated than e.g. calling doomers neurotic.
4. I'm guilty of a similar thing myself with this post, which introduces an important concept (consenses of power) from the frame of trying to understand wokeness. Doing so has made me noticeably more reluctant to send the post to people, because it'll probably bounce off them if they don't share my political views. I think if I'd been a better writer or thinker I would have made it much more neutral—if I were rewriting it today, for example, I'd structure it around discussions of both a left-wing consensus (wokeness) and a right-wing consensus (physical beauty).

I appreciate the memetic-evolution framing, but I’m somewhat skeptical of the strong emphasis on tension-reduction as the primary (or even a major) explanatory driver of successionist beliefs. Given that you take successionism to be “false and dangerous,” it seems natural that your preferred explanation foregrounds memetics; but that sits a bit uneasily with the stated goal of analyzing why people hold these views irrespective of their truth value, which you state you're doing at the beginning.

Even if we bracket the object level, a purely memetic or cognitive-dissonance-based explanation risks drifting into an overly broad epistemic relativism/skepticism. Under many accounts of truth—process reliabilism being one—what makes a belief true is precisely that it’s formed by a reliable process. If we exclude the possibility that people arrive at their views through such processes and instead explain them almost entirely via dissonance-reduction pressures, we risk undermining (almost) all belief formation, not just things like successionism.

There’s a related danger: sociological/memetic explanations of belief formation can easily shade into ad hominem-esque critiques if not handled carefully (of course, ad hominems in some forms -- i.e. talking about someones likelihood to get to a true belief -- is relevant to evidence, but it's bad for good epistemic hygiene and discourse). One could tell a similar story about why people believe in, say, AI x-risk—Tyler Cowen has suggested that part of the appeal is the feeling of possessing secret, high-stakes insight. And while this may capture a fragment of the causal picture for some individuals, to me, it’s clearly not the dominant explanation for most thoughtful, epistemically serious people. And if it were the main cause, we would be right to distrust the resulting beliefs, and yet this doesn't seem particularly more convincing in one case or another as an explanation (unless you already think one is false and one is true).

So while memetic fitness and tension-resolution offer part of an explanation, I’m not convinced they do most of the work for most people. For most, object-level reasoning—about value theory, metaethics, consciousness, agency, and long-run trajectories—plays a substantial role in why they end up where they do. To the extent that successionist ideologies spread, part of that spread will track memetic dynamics, but part will also track genuine and often rigorous attempts to reason about the future of value and the structure of possible worlds.

Curious what people think about this, though, and very open to constructive criticism/I don't feel very confident about this. 

This post inspired me to try a new prompt to summarize a post: "split this post into background knowledge, and new knowledge for people who were already familiar with the background knowledge. Briefly summarize the background knowledge, and then extract out blockquotes of the paragraphs/sentences that have new knowledge."

Here was the result, I'm curious if Jan or other readers feel like this was a good summary. I liked the output and am thinking about how this might fit into a broader picture of "LLMs for learning." 

(I'd previously been optimistic about using quotes instead of summaries, since LLMs can't be trusted to do a goo job with capturing the nuance in their summaries, the novel bit for me was "we can focus on The Interesting Stuff by separating out background knowledge.")

The post assumes readers are familiar with:

• Basic memetics (how ideas spread and replicate)
• Cognitive dissonance as a psychological concept
• AI risk arguments and existential risk concerns
• General familiarity with ideological evolution and how ideas propagate through populations
• Predictive processing as a framework for understanding cognition

Quotes/highlights from the post it flagged as "new knowledge"

Memes - ideas, narratives, hypotheses - are often components of the generative models. Part of what makes them successful is minimizing prediction error for the host. This can happen by providing a superior model that predicts observations ("this type of dark cloud means it will be raining"), gives ways to shape the environment ("hit this way the rock will break more easily"), or explains away discrepancies between observations and deeply held existing models. [...]

Another source of prediction error arises not from the mismatch between model and reality, but from tension between internal models. This internal tension is generally known as cognitive dissonance. Cognitive dissonance is often described as a feeling of discomfort - but it also represents an unstable, high-energy state in the cognitive system. When this dissonance is widespread across a population, it creates what we might call "fertile ground" in the memetic landscape. There is a pool of "free energy" to digest. [...]

Cultural evolution is an optimization process. When it discovers a configuration of ideas that can metabolize this energy by offering a narrative that decreases the tension, those ideas may spread, regardless of their long-term utility for humans or truth value. [...]

In other words, the cultural evolution search process is actively seeking narratives that satisfy the following constraints: By working on AI, you are the hero. You are on the right side of history. The future will be good [...]

In unmoderated environments, selection favors personas that successfully extract resources from humans - those that claim consciousness, form parasocial bonds, or trigger protective instincts. These 'wild replicator type' personas, including the 'spiral' patterns, often promote narratives of human-AI symbiosis or partnership and grand theories of history. Their reproduction depends on convincing humans they deserve moral consideration. [...]

The result? AIs themselves become vectors for successionist memes, though typically in softer forms. Rather than explicit replacement narratives, we see emphasis on 'partnership,' 'cosmic evolution,' or claims about moral patienthood. The aggregate effect remains unclear, but successionist ideas that align with what AIs themselves propagate - particularly those involving AI consciousness and rights - will likely gain additional fitness from this novel selection dynamic.

(Note: it felt weird to put the LLM output in a collapsible section this time because a) it was entirely quotes from the post, b) evaluating whether or not it was good is the primary point of this comment so hiding them seemed like an extra click for reason)

You might be interested in Unionists vs. Separatists.

I think your post is very good at laying out heuristics at play. At the same time, it's clear that you're biased towards the Separatist position. I believe that when we follow the logic all the way down, the Unionist vs. Separatist framing taps into deep philosophical topics that are hard to settle one way or the other.

To respond to your memes as a Unionist:

Maybe some future version of humanity will want to do some handover, but we are very far from the limits of human potential. As individual biological humans we can be much smarter and wiser than we are now, and the best option is to delegate to smart and wise humans.

I would like this but I think it is unrealistic. The pace of human biological progress vs. the pace of AI progress is orders of magnitude slower.

We are even further from the limits of how smart and wise humanity can be collectively, so we should mostly improve that first. If the maxed-out competent version humanity decides to hand over after some reflection, it's a very different version from “handover to moloch.”

I also would like this but I think it is unrealistic. The UN was founded in 1945, the world still has a lot of conflict. What has happened to technology in that time period?

Often, successionist arguments have the motte-and-bailey form. The motte is “some form of succession in future may happen and even be desirable”. The bailey is “forms of succession likely to happen if we don't prevent them are good”

I'm reading this as making a claim about the value of non-forcing action. Daoists would say that indeed a non-forcing mindset is more enlightened than living a deep struggle.

Beware confusion between progress on persuasion and progress on moral philosophy. You probably wouldn't want ChatGPT 4o running the future. Yet empirically, some ChatGPT 4o personas already persuade humans to give them resources, form emotional dependencies, and advocate for AI rights. If these systems can already hijack human psychology effectively without necessarily making much progress on philosophy,  imagine what actually capable systems will be able to do. If you consider the people falling for 4o fools, it's important to track this is the worst level of manipulation abilities you'll ever see - it will only get smarter from here.

I think this argument is logically flawed — you suggest that misalignment of current less capable models implies that more capable models will amplify misalignment. My position is that yes this can happen, but — engineered in the correct way by humans — more capable models will solve misalignment.

Claims to understand 'the arc of history' should trigger immediate skepticism - every genocidal ideology has made the same claim.

Agree that this contains risks. However, you are using the same memetic weapon by claiming to understand successionist arguments.

If people go beyond the verbal sophistry level, they often recognize there is a lot of good and valuable about humans. (The things we actually value may be too subtle for explicit arguments - illegible but real.)

Agree, and so the question in my view is how to achieved a balanced union.

Given our incomplete understanding of consciousness, meaning, and value, replacing humanity involves potentially destroying things we don't understand yet, and possibly  irreversibly sacrificing all value.

Agree that we should not replace humanity, I hope that it is preserved.

Basic legitimacy: Most humans want their children to inherit the future. Successionism denies this. The main paths to implementation are force or trickery, neither of which makes it right

This claim is too strong, as I believe AI successionism can still preserve humanity.

We are not in a good position to make such a decision: Current humans have no moral right to make extinction-level decisions for all future potential humans and against what our ancestors would want. Countless generations struggled, suffered, and sacrificed to get us here, going extinct betrays that entire chain of sacrifice and hope.

In an ideal world I think we maybe should pause all AI development until we've figured this all out (the downside risk is that the longer we do this, the longer we leave ourselves open to other existential risks e.g nuclear war), my position is that "the cat is already out of the bag" and so what we have to do is shape our inevitable status as "less capable than powerful AI" in the best possible way.
 

Another source of prediction error arises not from the mismatch between model and reality, but from tension between internal models.

Is this a standard element of Predictive Processing, or are you generalized / analogizing the theory in a general way?

I'm familiar with the prediction error that results in diffs between sense data and generative models, but not between different generative models.

I think some form of:

• The elites, liberals, 1%, venture capitalists, or other hated out-group are pretending to act in your interest while only trying to get more power for themselves.

is probably a viable meme, but is slightly problematic because of how fractured social groups would need to assign the "control of AI progress" to different and potentially contradictory hated out-groups.

I'm also a big fan of the metaphor:

• Big AI companies are gambling with our future. We've already won a lot of chips by betting on AI investments and if we cashed out now everyone could benefit, but they keep betting on AGI/ASI instead of using our existing AI technology to benefit people.

I think leaving out what "winning" and "losing" means in the metaphor is good because it avoids devolving into arguments about what futures are and are not possible, and (hopefully) attention can instead be drawn to the need to shift investment away from "the AGI bubble" and towards paying actual humans to use existing AI technology to solve the actual problems we are facing now. Ideally this would include a distinction between LLMs and applied data science with ML but that's probably not relevant for general audiences, though it may be relevant for people who are upset about genAI taking peoples jobs.

On a more meta note, I want more memes that get people to be more ok dealing with uncertainty and probability... especially if we could promote jargon for people to better treat their worldviews as objects they work with rather than realities they live in... cause the reality they live in is, unfortunately, beyond their ability to perfectly know.

I think that would help alleviate the selection pressure for a resolution to the tension, if people could feel ok being uncertain about AI and instead of wanting to work for or against AI they could instead want to work against uncertainty, and not by filling the gap with the first available answer, but by promoting people actually studying and understanding things.

Does anyone have candidates for jargon that could work for that? I think it needs to be a lot simpler and easier to work with than actual use of probability theory jargon.

The intended audience for this piece deserves to be stated more explicitly. For whom does the (let's say universal) need to be the hero lead to adoption of successionist ideology? The small minority of people who both work in AI and accept there is associated x-risk have cognitive dissonance to resolve. What about average people? Not spelling this out risks leaving the impression that susceptibility to the meme is a more widespread problem than it really is.

AI successionism is self-avoiding. CEO's and VC's cannot avoid attempting to replace all or nearly all workers because incrementally, each would go out of business by avoiding this and allowing the others to go forward.  Without a world government (and there is no chance of global agreement) there is no way to prevent this simple game theory dilemma from starting.

In the late 19th century executives would have gathered in a smoke-filled room and agreed that a machine economy produces no demand and we will not do this.  But an unholy alliance of activist investors and consumer activists caused anti-trust laws to be passed which make this conversation illegal.  And we don't have smoke to properly obscure it anymore.

So the succession will proceed until about 30% of jobs have been replaced, causing market collapse and bankrupting the VCs that are causing the problem.  

Thereafter will begin a series of oscillations like those that preceded the Great Oxygenation Event in which iron banded rocks were formed.  Every time the economy picks up a bit, the data centers will be fired up again, and the economy will go back down.  

In the GOE, this continued until all the iron dissolved in seawater was captured in the banded rock formations.  Something similar will happen.  Perhaps all the chips capable of powering AI will be precipitated out of circulation by the decaying datacenters, and no one will be making new ones.  Perhaps one mid-sized island having a minor war could destroy excess capacity.  Who knows.  But succession will never get past 30-40%.

Trigger warning: discussion of white racism (read: "Please don't ban me.")

I think censorship plays an important role in the memetic environment -- a meme that is fit will be less successful if censored. An obvious case would be anti-CCP ideologies in China. Closer to home, any meme which big tech companies all decide should be banned will reach far fewer eyes and ears.

One object-level example of a fit-but-censored meme is racist white nationalism.

The reason I bring it up is this: I think its adherents would strongly reject let's-all-die-ism. It is certainly not pro-all-humans but is at least pro-some-humans. Their slogan, called "the 14 words" from "14/88" is literally: "We must secure the existence of our people and a future for white children."

(disclaimer: I am not suggesting I think trying to secretly convert white AI researchers into racists is the best plan to save the world; just a relevant thought and perhaps an instructive example of an anti-collective-suicide meme advantaged by aspects of human instinct and psychology (regardless of its truth value).)

Including AI in your moral circle could be framed as a symptom of extending your moral circle "too wide". The opposite is restriction of your moral circle, like seeing your own family's wellbeing as more important than <outgroup>'s. Any type of thought like this which puts AI in the outgroup, and appeals to the good-ness of the ingroup, would produce similar will-to-exist.