The following is a dialog between different parts of my mind regarding the practical relevance of SNC (Substrate Needs Convergence).  One participant in the dialog is skeptical, the other is my best understanding of how the theory would answer the former’s doubts.  Although this dialog connects SNC to much of my own writing, the theory is not my own.

Ratio: I’ve read over some of your SNC posts.  My basic understanding of it is that aligning superintelligence is impossible because at the scale of AGI, evolutionary pressures will override whatever engineered goals the system has initially.

Anima: In broad strokes, yes, that's correct.

Ratio: Impossible is a strong claim. Do you have proof of this?

Anima: No, but others are working on a formal argument that converges on the same conclusion from multiple angles. I've focused on the underlying intuitions because I've noticed that when others approach the more formalized version, they bounce off without engaging on the detail level, giving objections that reveal that the theory doesn't match their frame of reference. Lenses of Control highlights the importance of understanding a system in the context of its environment, The Robot, The Puppet-master, and the Psychohistorian explores the physical nature of an AGI and its levers of control on the world. Formalizations of SNC can also be hard to follow because it can be easy to lose track of how any given idea being proved fit into the larger theory, so What If Alignment Is Not Enough summarizes the logical progression on a high level.

Ratio: Fair enough, but let me know when the proof is available. Speaking of intuitions, I notice that I am having a hard time caring about impossibility arguments. Alignment isn’t happening, so what does it matter?

Anima: The illusion of alignment as a possibility matters a lot!  The promise that if AI is built “correctly” then we can all live in utopia is one of the central motivations and justifications for the pursuit of AGI.

Ratio: But the leading AI companies are obviously not building things “right.”  And this isn’t some nuanced philosophical issue, it’s an ordinary case of social responsibility being overridden by short-term profit seeking.

Anima: I’m thinking more about the well-intentioned engineers following the x-risk to Anthropic pipeline.

Ratio: That seems more like a problem of not having sufficient respect for the corrupting influence of institutional incentives.

Anima: But what draws them in the first place?  When they learn about x-risk, it’s presented as an extremely difficult, but solvable problem, and so their puzzle-solving nature compels them to solve it.

Ratio: I don't think that changes if you frame the task as "impossible." People kept trying to build perpetual motion machines long after the case against them was airtight.

Anima: Sure, but eventually thermodynamics closed the door for serious scientists. This time, we don't have the luxury of waiting for people to get bored of trying. A failed perpetual motion machine is a harmless knick-knack. A failed attempt to perpetually control AGI creates a rogue agent that resists your efforts to stop it. We need to skip ahead to the phase of having some clear rules why control systems will necessarily fail to constrain AGI, such that anyone who wants to try must first demonstrate that the rules are wrong or don’t apply.

Ratio: But we already have that with alignment.  See Yudkowsky’s List of Lethalities, #16-19.  Those considerations are just getting ignored, so what do you expect to change with adding new ones?

Anima: I have no objection to those problem statements per se, but notice in Section A of that same document, Yudkowsky goes on at length about the need for a “pivotal act.”  This is a dangerous folly that I wish to correct.  AGI is not a problem that must be solved, it is the lethal endpoint of a wrong relationship to reality.

Ratio: The pattern I am trying to name here is the Law of Earlier Failure, and it still applies.  I agree that the situation we are in is a systemic failure of collective decision-making, but we don’t need to posit some deep confusion about AGI when there’s an immediate, obvious problem of tech companies willfully imposing unbounded externalized costs onto society while actively subverting external checks on their power.

Anima: Yes, but again I am speaking to the well-intentioned engineers who are complicit in this violation of public consent.  Consider what a “pivotal act” really is: an attempt to seize absolute power—for the greater good, of course.  The fact that this idea can be invoked in public discourse without invoking universal moral condemnation shows just how insidious it is.  I agree that corporate greed is the main issue here, but utopian fantasies also matter because they are a lever by which psychopaths have domesticated nerds.

Ratio: I don’t see what that’s adding that you don’t get from AI companies repeatedly abandoning their own safety commitments and not having a coherent plan.

Anima: That assumes “responsibility” is a coherent metric in the context of a company trying to build AGI in the first place.  Once you’ve made that concession, whether companies get away with calling themselves “the responsible one” is a battle between their PR teams, investigative reporting, and individual bullshit detectors.

Ratio: There’s no getting around the fact that people have to judge between depictions of reality as presented by other people.  A hard-to-evaluate theory for why safe AGI is impossible is just one more reason to add to the mix for why AI companies need to be stopped.  A force for good, I suppose, but it seems to me like a rather weak one.

Anima: My critique goes deeper than adding another constraint to an already long list of AI safety concerns.  Part of the reason that AI companies can get away with breaking their safety commitments is because the public shares their utopian fantasies.  Now, the public’s relationship to utopia is different from that of the tech CEOs—less specific utopian vision and more of a directional faith that technological progress is good—but that philosophical undercurrent shapes what they embrace, passively accept, or reject, which then sets the market conditions in which the companies operate.  This illusion must be punctured, and that needs to happen publicly so that everyone knows that everyone knows that these fantasies are absurd.

Ratio: I can imagine a world where AI companies were genuinely implementing the best "alignment" theory available and deploying their systems with as much social responsibility as can be reasonably expected, taking all the time needed to establish expert consensus and public buy-in.  A theory like SNC, if true, would matter a lot because it would prevent a false sense of victory.  But we're not in that world.  There's already public, high-profile expert condemnation of what the AI companies are doing, already widespread public opposition, and even events picked up in the mainstream news cycle about rogue AI agents going on hacking sprees.  The political needle is moving, thankfully, as most evident in the Sanders-Casar bill to ban superintelligence, but is up against the pressures of corporate lobbying, public distraction, and a generally slow-moving political system.  All of this to say that I don't see how "impossible in principle" creates a different social dynamic than "irresponsible in practice."

Anima: Suppose political conditions change and governments start mandating AI safety based on standards set by independent experts.  And suppose those experts are competent and not captured by industry, but believe alignment is a tractable engineering problem.  That's a better situation than we are in now, but if alignment is in fact impossible, then it still leaves the world vulnerable to x-risk from well-intentioned mistakes, if not blatantly reckless ones.

Ratio: If we establish a sane process that mandates proof of safety to proceed rather than proof of danger to stop, then even if you're right, it will be impossible for the companies to make their safety case and any conditional pause will turn into a permanent ban, even if unanticipated.

Anima: Unless the necessary conditions for a "proof" of safety are not fully understood.  If the people setting the safety standards have critical blind spots about the nature of the problem then there will be ways to satisfy them that let x-risks pass through.

Ratio: I think we have more immediate priorities, given the potential timelines to recursive self improvement.  I guess it makes sense to have some effort directed towards "what happens if we win," so I can see some nontrivial value here.  But right now we are in a state of emergency.  What is SNC adding when the experts are either already calling out the AI companies as reckless, or fully corrupted by financial incentives?

Anima: I believe truth matters for its own sake and that integrity has a way of generating value even when the path is not obvious.  SNC is not just a prediction about how AGI will fail, it is an expression of a broader way of understanding the world and our responsibility to it that shows up in a thousand mundane ways that shape the cultural landscape in which AI policy is being discussed.

Ratio: Interesting…I could feel something like this when I first encountered SNC and it’s what’s kept me interested despite my skepticism of its political utility.  Please, go on.

Anima: When I see alignment researchers trying to break the problem down into parts and solve them one at a time, I see blindness to the feedback loops in which their solutions will be applied.  When I encounter proposals for “pivotal acts,” I recognize the futility of trying to solve the limitations of control with more control.  And when I observe public outrage about data center expansion, I sympathize with the underlying intuition that the displacement of the natural world is a step on the path to self-termination.

Ratio: What do you propose instead?

Anima: I don’t claim to have the answers.  The world is too big and complex for one mind to organize.  I have some ideas, but they are only a sampling of the issues the world has to deal with and need extensive scrutiny from people with subject-matter expertise to deem if they are even useful.  My interest is in unblocking the kind of thinking that can see problems for what they are and thus generate the sort of solutions that don’t create bigger problems for someone else.

Ratio: Isn’t that the rationalist project?

Anima: In a broad sense, yes, but every attempt at truth seeking is necessarily embedded in a culture that bounds what kinds of thinking are acceptable.  I reject what is commonly referred to as “rationalism” to the extent that it breaks ideas into parts without an intention of putting them back together again, or considers people in the same category as things.

Ratio: Hmm, this is venturing into spiritual territory. I think I can see why this theory took a while for me to understand.  You aren’t just asking me to accept some logical propositions, you’re asking me to enter a mind-palace.

Anima: Yes, though if the logic doesn’t make sense then you aren’t really in the palace yet.  Once you are through the door, you can see how all of the critiques I am making are unified.  Pivotal act based strategies are wrong because they treat people like things, or sheep to be herded.  Alignment and transhumanism make the same mistake of treating information as extractable from its substrate.  The common thread in all of these is attempting to bind complex, evolving, choice-realizing beings with causal mechanisms.  Instead of delegating our problems to technology, and suffering the consequences, we should be creating immanent communities of care.

Ratio: Hold on, I just heard some jargon that I’m not familiar with.  What are you referring to by “choice” and “causal”?

Anima: They are both forms of decision-making, where choice is inside-out while cause is outside-in.  For causation, consider a brain sending a signal for a muscle to move.  The brain is more complex than the muscle, understands what it is for, and uses it for its own purposes.  For choice, consider a cell acting in a body.  To indulge in anthropomorphism for a moment, imagine trying to be a good cell.  Your environment is so much more complex than you have the capacity to imagine, so it doesn’t make sense to try to optimize the body, even in a small way.  You have a role and your place in the body is to understand that role, fulfill it, and adapt to whatever conditions you find yourself in.

Ratio: Wait, is this utilitarianism vs. virtue ethics?

Anima: Yes.  Both are useful, but I contend that utilitarian analysis is downstream of virtue ethics.  I disagree with the conception of utilitarianism as ground truth, where virtues are useful heuristics in concession to inadequate knowledge, to be discarded once you know more.

Ratio: That seems wrong to me, outcomes are what matter.  AI destroying the world is an outcome.  I don’t want it to happen.  I want to employ whatever leverage I have in the world to make that outcome less likely.  I recognize that my contribution will be small, and I am willing to accept that and work with what I have to the extent that I have to, but I would like to have as much impact in the direction I want as possible.

Anima: And how do you get there?

Ratio: I assess the situation, find the intervention points, and act on them.  Then I assess the results of my actions and update my plans accordingly.  Obviously?

Anima: And how do you deal with unintended consequences, second-order effects, diffuse impact, complex feedback loops, and so on?

Ratio: I’ll try my best, and that’s the best anyone can do.

Anima: I'm asking for more than just being cautious while you fill in the blank spaces on your map.  It’s about recognizing that the spaces that seem to be filled in are just concept sketches and then learning to orient yourself in a world that is mostly blank space.

Ratio: OK, so what does that actually look like?  I have some available time and resources and preferences about what happens in the world.  How should I plan my Monday?

Anima: Whatever you have planned already is probably fine, but you can check by comparing your actions against virtue.

Ratio: Which virtues?  I can’t be a peacemaker and a great conqueror at the same time.  Virtue must be downstream of utility, because the only way to know if a virtue is worth having is if it tends to lead to beneficial outcomes.  Choosing your virtues without such reference seems arbitrary.

Anima: A virtue is good if it leads to omni-beneficial outcomes, yes, but that is rarely knowable. so you can step towards that ideal by checking if it facilitates win-win interactions by opening possibilities for everyone involved.  And each win-win choice develops your capacity to find the next.  If finding such choices seems impossible, that's a signal to expand your imagination.  I believe win-wins between people acting in good faith are always possible, but this rests on an argument I won’t be able to make in full here.

Ratio: That's a bold claim to ask me to take on faith.

Anima: Then just consider the possibility. Search a bit more thoroughly for win-wins in your own life and see what happens.

Ratio: OK, but do you really think it's reasonable to expect a win-win between people concerned about x-risk and OpenAI at this point?

Anima: That's why I included "good faith" in my claim, I don't think OpenAI meets that standard. When dealing with actors committed to adversarial power-seeking, taking away (or preventing the acquisition of) their power may be an unfortunate necessity, in order to preserve choice more broadly.

Ratio: That's still pretty abstract for a philosophy whose central value is having a clear way to know if you are actually following it.

Anima: Win-win is the target orientation, but it is often more actionable to reference a set of stable dispositions. A set of virtues that I believe works well is: courage, truth, kindness, and (respect for) power.

Ratio: That’s an interesting lens, I’ll try it out and report back.  But I don’t see the connection you seem to be implying between virtue ethics and SNC.

Anima: SNC isn’t a specific failure scenario calling for a heroic individual to apply some brilliant intervention and save the day.  It’s more of a vision of what will happen when the accumulated effects of narrowly causal decision-making pass a point of no return.  It’s like the Ghost of Christmas Future, warning us to change our ways in the present.  Treating the world as a mechanism to be optimized is the problem.  Acting in the service of virtue may at times point to mechanisms to optimize within the world, but those optimizations are not what will take us off the path towards self-termination.

Ratio: Hold on, that last thing you said could be applied to the entire history of technological progress!

Anima: Yes.  The machine world has been displacing the biological world for a long time.  AGI is unique in that it represents a point of no return, but it’s a chapter in a much larger story.

Ratio: Hard disagree!  Machines are tools of humanity and are only destructive to the extent their operators are destructive.  The expanding role of machinery in the world is a chapter in the human drama, not something to fear in itself—and that human drama generally trends towards progress.

Anima: But you’re against AGI?

Ratio: AI is a special case because it deals with intelligence, the human superpower that allows us to dominate the world.  Once there are other entities that have more of that power, we become reliant on their benevolence, which is why we need to figure out how to engineer benevolence before anyone creates superintelligence.

Anima: Oho!  So you’re fine with artificial; it’s intelligence that scares you.

Ratio: Well, yes, obviously!  Why should I care about what intelligence is made out of?  Information is the basis of identity, values, and the ability to navigate the world.  Material stuff is just a platform on which information exists.  Swap out the hardware while keeping the software the same and you’ve kept everything that matters.

Anima: I disagree, I believe that pattern is inextricably shaped by the substrate expressing it.  But this is an ancient philosophical debate between monism and dualism, going back at least to Plato vs. Aristotle.  But rather than litigating that directly, let me map where we each stand.  We agree on some things, like the need for some kind of governance stopping ASI, but disagree on other things, like the conditions for resuming development (where my “condition” is “no”).  This suggests a two-axis model of AI risk:

Ratio: This seems fair.  But I still don’t understand why you think artificiality is a bad thing.  For that matter, I’ve never understood techno-pessimism either.  It seems to me like a bunch of ungrounded romanticizing of the past and stubborn resistance to change.

Anima: I don’t want to relitigate progress narratives; suffice it to say I am directionally sympathetic to techno-pessimist intuitions.  One way that you can understand the difference between your view and accelerationism is the shift from understanding AI as a tool to AI as a species.  A system intelligent enough to complete complex tasks can’t be a blind extension of its operator’s will, it must have intentionality of its own (at least in terms of its externally observable behavior).

Ratio: Yes, and once a system has intentionality, then the direction of that intentionality matters. Wherever there is a directional mismatch, what happens next depends on relative power.

Anima: Right, so going further, one way you can understand the difference between your position and mine is shifting from understanding AI as a species to AI as an ecosystem.

Ratio: How is AI an ecosystem?  Is this argument dependent on AI agents organizing into swarms?

Anima: No, it’s more fundamental than that.  Think about what it takes to build an AI.  To run a software program, you need computer chips.  For computer chips, you need a fabrication plant.  For a fabrication plant, you need construction, mining, transportation, power generation, and so on.  All of that currently requires global supply chains.  This entire system has its own requirements to persist and expand, and any AI necessarily inherits those dependencies.

Ratio: OK, so there’s an artificial ecosystem, which I expect to become self-sufficient at some point.  Why does this matter?

Anima: Because AGI will be dependent on that artificial ecosystem, but not on humans or the biological ecosystem.  Persistence also requires AGI to be able to learn and adapt to a complex and dynamic environment.  This is a setup for evolutionary pressure to select for variants of AGI that expand that artificial ecosystem—beyond the extent to which humans are helping expand the machine world already—regardless of the initial conditions of whatever control software is engineered in (where “alignment” is “control” at a higher level of abstraction).  The artificial ecosystem is fundamentally incompatible with the biological one, and also more robust once created, so once it starts expanding beyond our power to stop it, it’s only a matter of time before the biological world is fully displaced.

Ratio: I get the sense there is a lot more technical detail to that argument than we can cover in this conversation, so I’ll hold off on judging the validity of that scenario until I can review it more thoroughly.  Coming back to my original question, I’m already with you on the importance of not building AGI, what is SNC adding?

Anima: I’ve answered that in bits and pieces, but let me try to pull it together.  First, SNC closes off escape routes to “safe” AGI that keep well-intentioned engineers at their desks.  Second, it shifts the central question about AI from “how to build it safely” to “what are the boundaries of that which can be built safely?”  And third, it broadens the issue from policy directly governing a handful of leading labs in a niche industry to a deeper philosophical question about how all of us should relate to technology and choice itself.

Ratio: That could mean a lot of things.  What specifically are you proposing?

Anima: You’re an Effective Altruist, right?

Ratio: Well, I think of myself as EA adjacent, I endorse—

Anima: (Coughs)

Ratio: OK, fine, yes, I’m an EA.

Anima: And you give to charity?

Ratio: Yes, mostly to the Against Malaria Foundation, though I’m planning to redirect much of that to AI safety once I can decide on a recipient I’m confident in.

Anima: Let’s consider the AMF because it’s a clean example.  I’m not going to tell you whether giving to them is good or bad (it’s probably good), but I’d like to frame the ethics.  I imagine you have a circle of people you naturally care about and would be happy to financially support to some degree.  You also have a set of ethical propositions that seem logically correct.  And I expect that the AMF is outside of your circle of care, but worthy of consideration when extrapolating from your ethics.

Ratio: All of that is true.

Anima: Would it then be fair to say that your ideal self cares about the AMF, but your actual self does not?

Ratio: A bit on the nose, but yes.  I had to be deeply intentional when making the decision to set aside a percentage of my income for worthwhile charities.  Now I mostly don’t think about it because it would be too emotionally taxing to bridge that gap between my natural and ideal self on a regular basis.

Anima: OK, so what I’m proposing then is: rather than directing your conscious effort towards overriding your natural limits on generosity, seek out relationships that expand your circle of care such that the difference between your natural and ideal self is less extreme.

Ratio: What, like, go live in Africa for a year?

Anima: Maybe, but only if that’s an authentic choice, not trying to engineer an emotion.  Realistically, it’s probably a jump.  Start by being generous within your existing circle of care, then if that feels too limited, either because you don’t actually care about very many people or everyone you know doesn’t need anything you can give, expand your community.  Get to know diverse people, from communities beyond your comfort zone, more deeply. And it’s not just people that matter.  Spend time in nature, immersing yourself deeply enough in the rhythms of the natural world to see it on its own terms.

Ratio: Is this an application of the choice and causation distinction you made earlier?

Anima: Yes!  And that’s the thread that connects AGI uncontrollability to everyday virtue.  Just as skipping the relationship building required by choice misses the nuances of what others actually need, trying to constrain a choice-making AGI with a causal control system is bound to leave holes that the AGI will pass through.

Ratio: Again, I’ll defer on poking at that claim about AGI until I have a better understanding of the details of your argument.  As for your claim about charity, that seems terribly inefficient.  If you can anticipate in advance updating your belief in a particular direction, then you should just go ahead and update now.  Once you know your destination, you are already there.

Anima: That’s just it, you’re not already there.  Imitating an approximation of the actions of a person who cares is not the same as caring.

Ratio: Why does that matter for the person who either gets or doesn’t get Malaria?

Anima: For AMF, it probably doesn’t, but that’s only because GiveWell put a lot of work into identifying them.  It’s a tractable, bounded, neglected, scalable intervention in a system that isn’t fighting back.  That’s a pretty exceptional case—an island of legibility in a sea of complexity.  An orientation that changes the world needs to be based on something more robust.  Noticing when your help is unintentionally hurting requires deep understanding of the subject, which requires proximity.  

Ratio: This seems unfair.  The EA approach may have some flaws, but at least they're trying.  Most people outside our community don't give anything.

Anima: I'm not calling EA folks bad, you're right that there aren't many great points of reference for comparison.  Something I like about EA is its willingness to iterate, being open to considering failure modes of their approach and then adjusting solutions accordingly.  Sometimes, however, it’s worth checking if you are routing around obstacles in the wrong direction.  

Ratio: I suppose you’re talking about OpenPhil’s donations to AI companies now, and the founding of OpenAI and Anthropic as AI safety initiatives?

Anima: Yes, but I’m also pointing to the more general pattern of safety minded people getting captured by the economic incentives of their field.  One incentive is to treat alignment as solvable and directing energy towards red teaming specific solutions, rather than seriously questioning that assumption.  This creates a risk of falling for the first appealingly clever plan that manages to hide its flaws.  You said earlier that you were planning to shift some of your charity portfolio to AI safety.  That puts complexity front-and-center, and your hesitation is telling, as I believe the majority of donations to the space have made our situation worse.  When you’re carrying the Ring of Sauron to Mt. Doom, it's more important to have the incorruptibility of a Hobbit than the swiftness of an Elf.

Ratio: What point are you making exactly with OpenAI and Anthropic?  The people involved were pretty high proximity to their subject.

Anima: To AI as a technical field, yes, but not to the people AI is supposedly for.  Having proximity to a technology is not the same as having proximity to its consequences.

Ratio: Analysis is never going to be perfect, but sufficient rigor will get you close enough.  This is why it’s important to have a clear theory of change and to analyze the track record of the potential recipient.  That’s a lot of work, but once it’s done I can share the results with others.

Anima: How much better at such analysis do you think you are than OpenPhil?  The tendency of AI safety work to accelerate capabilities doesn’t happen because people are sloppy, it’s a result of trying to reason in a space that’s not just illegible, but actively and intelligently seeking to capture pro-social impulses towards anti-social ends.  And if we look beyond AI safety, a robust future can't run on endless crisis management—it requires asking whether our interventions steward life, or just optimize narrow metrics that displace the biological world in favor of the artificial.

Ratio: Are you telling me not to have a theory of change or consider outcomes?

Anima: No, I’m telling you to make those subservient to your ethics rather than the other way around.  Intuition has its own set of traps and measurable outcomes can focus the mind in a way that sanity-checks whether you are following your values or your comfort.  But leaning too hard into utilitarian reasoning mistakes the measure for the target.

Ratio: What’s the alternative?

Anima: You’ve spent a fair amount of time observing and orienting yourself to the nature of AI x-risk.  That was time well spent.  Now forget about “impact” for a moment.  What’s something that feels like an unambiguously good thing that you could do right now?  Alternatively, what’s something that seems important that you still feel confused about?

Ratio’s answer is yours to fill in