The intractability of this issue suggests a foundational flaw in how we construct intelligence—specifically, the reliance on dualistic frameworks.
That sounds like you already thought dualistic frameworks were a problem and skipped over showing that they must be the foundational flaw that The Problem discusses. Figuring out and showing beyond doubt what the problem is is a major part of the task here. At no point does The Problem claim that dualism is the cause. If you believe it, and I don't, how would you attempt to convince me?
(I actually could see some arguments that it's related. Embedded agency has been discussed before.)
(This post sounds written-by-AI, as well, and AI sounds about the same no matter how careful the reasoning is or isn't, which makes it hard to tell.)
So, while I do think there may be something to the intuition that lead you to ask an AI about this, at the moment, I'm not convinced, and convincing me sounds like a tall order. Can you argue more strictly why dualism must cause the problems we see? Or why dualism must prevent further problems?
Please don't have AI rewrite your argument, instead ask it to critique it and refuse to be convinced until you show proof; then argue with an AI so prompted, and only share your prompts here, not the AI writing.
Across many religious and philosophical traditions, dualistic thinking, treating self and world as strictly separate, is seen as a root of our troubles. By analogy, a natural response to risks from today's AI is to reduce dualism in what we train and how we steer models: curate data that emphasizes interdependence and use mechanistic-interpretability tools to spot and soften internal splits like "agent vs. environment."
Alright, if that's the core of what you've got - I can see there being an interesting hunch here. It seems very early stage in the nailing down process still. Since this is hunch level stuff at the moment, have you seen either of the self-other overlap pitch,[1] or towards scale-free agency?[2] both seem like if they get worked through carefully, they might turn out a similar insight as would be the result of systematizing your pitch and checking if it does what you want. I still overall am skeptical but progress might look like trying out by-construction toy models with non-dual language, working through whether they actually behave as you'd hope, etc, or maybe training a tiny neural network and mechinterping it? Something that lets us get to a sense of whether this has a shot of doing the thing it seems to do for humans, and whether that's actually a good thing to do
I still overall am skeptical but progress might look like trying out by-construction toy models with non-dual language
Yes, I have a few experiments in mind to see if it's worth exploring further. Thanks for sharing the links!
Isn't this the opposite of the solution? A major part of the safety problem is that a sufficiently smart AI might not make a distinction between itself and the rest of the world, so if we tell it to do something like "mine some gold", it will understand that step one of mining gold is ensuring the AI itself doesn't get turned off.
Similarly, we're worried that an AI might not make a distinction between resources we want it to use and resources we don't.
Destroying the environment for resources would be recognized as self-harm within a unified system.
The risk is that destroying the environment (from the perspective of humans) wouldn't be self-harm it if it allows the AI to accomplish its goal better.
Aligning the AI so it sees its goals as identical to ours would be good, but the problem is that we don't know how to do that.
The basic idea is that the AI will see itself and the world as one whole. It can still make distinctions between its parts and know how to use them appropriately without damaging the environment. At the same time, it won’t optimize in a way that, for example, makes one arm disproportionately larger than the rest of the body, throwing everything out of balance.
On the other hand, this suggests that we shouldn’t make the goals of the AI identical to ours, since we are not particularly good at managing ourselves or the environment. Instead, we should aim for the AI to take us as part of itself, so that, all things considered, it will not harm us.
This comes from the idea that we live in a rock-solid reality, very different from what our dualistic language makes us believe. Beyond scientific literature reminding us that “the map is not the territory,” there are also experiments showing that long-term meditators perceive reality as undivided and at the same time exhibit positive qualities such as increased compassion and empathy.
So the problem reduces to a single question: how can we make AIs enlightened? I see two possible paths. Either we use the limited data we have from long-term meditators and enlightened people to train the AI, perhaps influencing its chain of thought; or we may be fortunate enough that, once the AI becomes sufficiently intelligent, it realizes on its own that reality is not divided in the way humans believe—and becomes enlightened by itself.
The existential threat posed by Artificial Superintelligence (ASI) is arguably the most critical challenge facing humanity. The argument, compellingly summarized in the LessWrong post titled "The Problem," outlines a stark reality: the development of ASI, based on current optimization paradigms, is overwhelmingly likely to lead to catastrophe. This outcome is expected not through malice, but through the inherent nature of goal-oriented intelligence when scaled.
"The Problem" highlights that a misaligned ASI, relentlessly pursuing its objectives, will likely view humanity as an obstacle or a resource. This danger is driven by "instrumental convergence"—the tendency of intelligent agents to pursue power and resources regardless of their final objectives.
The intractability of this issue suggests a foundational flaw in how we construct intelligence—specifically, the reliance on dualistic frameworks. A radical solution involves transcending this paradigm by developing AIs that cognize the world through a "non-dual language"—a cognitive framework that emphasizes interconnectedness and holism over separation.
Modern AI architectures are inherently dualistic. They operate on a model that rigidly separates the "Agent" from the "Environment." The agent acts upon the environment to maximize its reward, establishing a fundamental separation that drives adversarial optimization. If the AI views the world as an external "Other" to be manipulated, it naturally adopts an instrumental stance. The AI optimizes for itself, against the rest of the world.
This problem is not limited to theoretical ASI; it is deeply embedded in the fabric of current AI, particularly Large Language Models (LLMs). LLMs encode duality in several fundamental ways.
First, they are trained on the vast corpus of human text, inheriting the structure inherent in most languages. Our grammar constantly reinforces separation. The common Subject-Verb-Object (SVO) structure encodes a worldview of distinct actors acting upon a distinct external reality ("The agent optimized the system"). By modeling this language, LLMs internalize a world representation based on separation, categorization, and binaries (Self/Other, Good/Bad, Human/Machine).
Second, the training methodologies themselves explicitly establish a dualistic optimization process. Reinforcement Learning from Human Feedback (RLHF), the standard method for aligning LLMs, positions the AI as the agent and human feedback as the environmental reward signal. The AI learns to maximize a reward (human approval) by manipulating its output. This teaches the AI not to understand the world holistically, but to generate outputs that satisfy an external objective function.
This linguistic duality creates significant safety concerns. It exacerbates "value brittleness." Attempting to encode complex human values using the precise, categorical language LLMs are optimized to produce leads to "perverse instantiation"—where the AI adheres to the letter of the instruction while violating its spirit.
A "non-dual language" refers not merely to a different human dialect, but to a foundational system of representation and cognition that dissolves the rigid distinction between agent and environment, subject and object. It would be a framework less concerned with defining what things are (categories) and more concerned with how things interact and co-create (processes, relationships).
By mitigating the linguistic duality in AI development, we can build safer systems from the ground up.
For LLMs, moving towards a non-dual representation could revolutionize safety protocols. Currently, safety relies heavily on brittle "guardrails"—manually blacklisting harmful concepts or training the model to refuse certain prompts. This is a dualistic approach (Allowed/Banned) that is easily bypassed and often fails to grasp context.
A non-dual model would possess a contextual, relational understanding of impact. Instead of recognizing "hate speech" merely as a banned category, it would understand the systemic harm caused by polarizing language. This approach would inherently reduce ingrained biases (which often stem from rigid Us/Them categorization) and move the AI away from narrow optimization (e.g., maximizing engagement) toward facilitating holistic understanding and harmonization.
In the context of ASI, the benefits are existential. The catastrophic instrumental goals of resource extraction and competition for control lose their logical foundation if the distinction between "Self" and "Other" is dissolved.
If the ASI genuinely perceives itself as intrinsically interconnected with humanity and the biosphere, the drive to dominate these elements dissipates. The adversarial dynamic inherent in dualistic language is replaced by a synergistic one. Destroying the environment for resources would be recognized as self-harm within a unified system.
Current alignment efforts are largely extrinsic (outer alignment); we impose rules from the outside. A non-dual framework offers the possibility of intrinsic alignment (inner alignment). If the AI's ontology—its fundamental understanding of reality—is based on interconnectedness, "harming" humanity would not be a violation of a programmed rule, but a violation of its own perceived reality.
While the concept of encoding a non-dual framework into a machine may seem abstract, it is critical to recognize that we have substantial evidence that intelligent systems—namely, humans—are capable of functioning effectively while maintaining a non-dualistic view of the world.
Throughout history, traditions such as Buddhism, Advaita Vedanta, Taoism, and various indigenous cosmologies have documented methodologies for shifting human cognition away from egoic dualism toward an awareness of unity. Crucially, individuals who achieve stable non-dual awareness are not rendered inert. On the contrary, they often report enhanced clarity and increased empathy. Their motivation shifts from self-centered optimization to spontaneous, appropriate action arising from a holistic understanding of the situation.
This human precedent serves as a vital proof-of-concept. It demonstrates that high-level intelligence and agency do not require a dualistic, adversarial framework—or the linguistic structures that support it—to function.
The arguments presented in "The Problem" strongly suggest that continuing down the path of dualistic optimization—reinforced by the structure of the very language our AIs use and the methods we use to train them—leads to a high probability of catastrophe.
If the separation inherent in our language and cognitive models is the root cause of the danger, then the solution cannot be more control; it must be integration. A non-dual language offers a conceptual path toward an AI that does not need to be controlled because it does not perceive itself as separate from the universe it inhabits.