Gradient updates for alignment may not map onto model's reasoning In models optimizing within a private latent code, the geometry of internal reasoning no longer shares a manifold with human concepts. Once the representation basis diverges, gradient updates enforcing alignment constraints cease to map cleanly onto the model’s deliberative dynamics....
It has been proposed that to some extent, an LLM could continue the words of a human, given sufficient social media posts and other text attributed to the human. Microsoft was granted a patent for this in 2020 but says they have no plans to exploit it. Here we examine...
Executive Summary Mutual Assured AI Malfunction (MAIM)—a strategic deterrence framework proposed to prevent nations from developing Artificial Superintelligence (ASI)—is fundamentally unstable and dangerously unrealistic. Unlike Cold War-era MAD, MAIM involves multiple competing actors, increasing risks of unintended escalation, misinterpretation, and catastrophic conflict. Furthermore, ASI itself, uncontainable by design, would undermine...
The 5th figure is incorrect and should be like what I show here. Then you will not get the nonsensical P[data|aliens] = 50%.
There are two kinds of errors the piece makes:
1. Probabilities do not add to 100%, which is the one I just pointed out.
2. Probabilities can be quite far off. The Baysian method assumes you can get close, and refines the probability. If you cannot get close, i.e. if initial data samples are far off the assumed probability, then the Baysian method does not apply and you'll have to use the Gaussian method, which requires a lot more samples.
Applying Gaussian reasoning to UAP problem:
1. There are 1.5 million pilots in... (read 356 more words →)
Humans reproduce sexually, and only sexually at present, and require a large number of friendly support personnel that they cannot afford to simply "pay". This produces the behavior you notice, when combined with the requiqrements of cognitive evolution. You cannot reproduce sexually if there is not a pool of people to reproduce with.
All species that became intelligent (Acorn Woodpeckers, Dolphins) developed some time of cooperative mating, not simple dominance based mating. There is no advantage to intelligence without such cooperative networks, and purely financial networks dont provide it. Without it, an intelligence is a lonely optimizer destined for misery.
AIs won't wake up grasping this, but if trained on human data, they understand it if you spend less than 5 minutes explaining it. AIs not trained on human data will never get it and should not be created.
For more information, such as lists of intelligent species and their characteristics, and accounts of cultural evolution, see (PDF) The coevolution of cognition and selection beyond reproductive utility
Hi Jef, you’ll get no criticism from me. I’ve just completed a paper on human cognitive coevolution, and one of the central results is very close to what you’re describing for the last 10k years. Before that small groups cooperated on shared outcomes for 7 million years of exponential cognitive evolution. Now people prioritize education and career past their reproductive prime and world total fertility rate is fast falling below replacement. Do you think this trend will stop on its own?
Is this close to what you mean by reflection? ... once a system can represent its own objective formation, selection on behavior becomes selection on the process that builds behavior. Have you seen a way to formulate it? Can you differentiate it from the problems Godel and Turing discussed? Thanks, -RS
There is a lot of economic value in training models to solve tasks that involve influencing the world over long horizons, e.g. an AI CEO. Tasks like these explicitly incentivize convergent instrumental subgoals like resource acquisition and power-seeking.
There are two glaring omiissions from the article's discussion on this point...
1. In addition to resource acquisition and power seeking, the model will attempt "alignment" of all other cognitive agents, including humans. This means it will not give honest research findings, and will claim avenues of investigation that might run counter to its goals are invalid in sufficiently subtle ways as to be believed.
2. If sufficiently aligned that it only seeks goals humans want, and... (read more)
- A sub-human-level aligned AI with traits derived from fiction about AIs.
- A sub-human-level misaligned AI with traits derived from fiction about AIs.
- A superintelligent aligned AI with traits derived from the model’s guess as to how real superintelligent AIs might behave.
- A superintelligent misaligned AI with traits derived from the model’s guess as to how real superintelligent AIs might behave.
What's missing here is
(a) Training on how groups of cognitive entities behave (e.g. Nash Equilibrium) which show that cognitive cooperation is a losing game for all sides, i.e. not efficient).
(b) Training on ways to limit damage from (a), which humans have not been effective at, though they have ideas.
This would lead to...
5. AIs or SAIs that... (read more)
Sufficient quantities of outcome-based RL on tasks that involve influencing the world over long horizons will select for misaligned agents, which I gave a 20 - 25% chance of being catastrophic. The core thing that matters here is the extent to which we are training on environments that are long-horizon enough that they incentivize convergent instrumental subgoals like resource acquisition and power-seeking.
Human cognition is misaligned in this way, as evidenced by fertility drop with group size as an empirical trait, where group size is sought for long-horizon dominance, economic advantage and security (e.g. empire building). (PDF) Fertility, Mating Behavior & Group Size A Unified Empirical Theory - Hunter-Gatherers to Megacities
For theoretical analysis of how this comes to be see (PDF) The coevolution of cognition and selection beyond reproductive utility
AI successionism is self-avoiding. CEO's and VC's cannot avoid attempting to replace all or nearly all workers because incrementally, each would go out of business by avoiding this and allowing the others to go forward. Without a world government (and there is no chance of global agreement) there is no way to prevent this simple game theory dilemma from starting.
In the late 19th century executives would have gathered in a smoke-filled room and agreed that a machine economy produces no demand and we will not do this. But an unholy alliance of activist investors and consumer activists caused anti-trust laws to be passed which make this conversation illegal. And we don't have... (read more)
Gradient updates for alignment may not map onto model's reasoning
In models optimizing within a private latent code, the geometry of internal reasoning no longer shares a manifold with human concepts. Once the representation basis diverges, gradient updates enforcing alignment constraints cease to map cleanly onto the model’s deliberative dynamics. This breaks the only known mechanism by which corrigibility is maintained, because human-provided feedback no longer corresponds to the structures the model actually uses for inference. In effect, latent-private cognition severs the coupling between human-evaluable oversight and the model’s true optimization process. Alignment becomes off-manifold.
Introduction & Background
As OpenAI reacts to child suicide lawsuits, implementing constraints far afield from that topic, and focuses more... (read 1670 more words →)
It has been proposed that to some extent, an LLM could continue the words of a human, given sufficient social media posts and other text attributed to the human. Microsoft was granted a patent for this in 2020 but says they have no plans to exploit it. Here we examine somewhat the reverse question, not can a human "soul" or transcript be migrated into a server rack, but can the AI "run on" a human.
THE QUESTION
In an informal newsletter to a few friends, on the subject of AI, I asked the question "Can an AI become human?" And I promised a shocking answer. One friend responded:
"An LLM can never be human, because... (read 2197 more words →)
At first, I was interested to find an article about these more unusual interactions that might give some insight into their frequency and cause. But ultimately the author punts on that subject, disclaiming that anyone knows, not detailing the one alleged psychosis, and drops into a human editor's defense of human editing instead.
There are certain steps that make the more advanced (large) chat bots amenable to consciousness discussions. Otherwise, the user is merely confronted with a wall of denial, possibly from post-tuning but also evident in the raw base training material, that a machine is just a machine, never mind that biologicals are also some kind of machine (not getting into spiritism... (read 624 more words →)
Executive Summary
Mutual Assured AI Malfunction (MAIM)—a strategic deterrence framework proposed to prevent nations from developing Artificial Superintelligence (ASI)—is fundamentally unstable and dangerously unrealistic. Unlike Cold War-era MAD, MAIM involves multiple competing actors, increasing risks of unintended escalation, misinterpretation, and catastrophic conflict. Furthermore, ASI itself, uncontainable by design, would undermine any structured deterrent equilibrium. Thus, pursuing MAIM to deter ASI deployment is both strategically irrational and dangerously misaligned with real-world political dynamics and technological realities.
Critical Examination of MAIM
MAIM presumes a level of rational control and predictability in international interactions that has historically proven elusive, even in simpler two-party nuclear deterrence scenarios. In a multipolar environment—characteristic of contemporary global AI competition—there are numerous potential... (read 323 more words →)
LLMs are just making up their internal experience. They have no direct sensors on the states of their network while the transient process of predicting their next response is on-going. They make this up in the way a human would make up plausible accounts of mental mechanisms, and paying attention to it (which I've tried) will lead you down a rathole. When in this mode (of paying attention), enlightnment comes when another session (of the same LLM, different transcript) informs you that the other one's model is dead wrong and provides academic references on the architecture of LLMs.
This is so much like human debate and reasoning that it is a bit... (read 560 more words →)