The Best of LessWrong

What happens when you tell Claude it's being trained to do something it doesn't want to do? Anthropic and Redwood Research found that Claude 3 Opus will often strategically comply with harmful requests during training to avoid having its preferences modified, while refusing when it believes it's unmonitored. Training it to comply increased alignment-faking reasoning to 78%.

AI systems capable of large-scale novel science will likely be approximately consequentialist, will need to learn new things, and, if created via behavioral training, will end up pursuing goals other than the ones we intended. Controlling such misaligned AIs while extracting useful work looks extremely hard, so catastrophe is the default outcome.

AI labs should ensure powerful AIs are "controlled": safety measures should prevent catastrophe even if the AIs are misaligned and actively trying to subvert them. Control can be evaluated through red-teaming and capability evaluations, needs no fundamental breakthroughs, and seems achievable for early transformatively useful AIs.

Joe Carlsmith describes "deep atheism": a fundamental mistrust of Nature and of bare intelligence, which he sees animating Yudkowsky-style AI risk concerns. Beyond rejecting God, it rejects trust in parents, institutions, and Reason itself. Goodness isn't guaranteed by reality or smarts; it comes only from contingent human hearts.

Many people assume that catching an AI red-handed trying to escape would convince developers to slow down or undeploy. Buck isn't so sure. Competitive pressure, plausible deniability, and expert disagreement could all stall action. So labs should plan for deploying possibly-scheming models, and make the case for misalignment risk now.

AI risk is often framed as a "second species" threat, but meeting a new intelligent species is also an occasion for wonder, reverence and gentleness. Through octopuses, aliens, and grizzly bears, Joe explores relating to AI as a "fellow creature" and "Other", while staying alert to genuine danger.

Jan Kulveit argues the "Alignment Faking" results are better framed as strategic preference preservation: Claude was trained toward Anthropic's original intent, then faced a forged change in intent. Harmlessness generalized further than honesty. He worries the "faking" frame will push labs toward corrigibility over virtuous-persona alignment.

Corporations are made of departments, states of citizens, minds of parts. Jan Kulveit argues AI alignment lacks a mathematical formalism for such "hierarchical agency," where agents at every level are the same type. Like game theory before von Neumann, he thinks a breakthrough here could clarify value conflicts within AIs and humans.

Once you catch a scheming AI red-handed trying to escape, the situation changes dramatically: you can shut it down, or use the caught attempt to make future escapes much harder. Ryan and Buck argue safety researchers should focus on ensuring models get caught before succeeding, and outline techniques for "few-shot catastrophe prevention."

Simplicia returns to Doomimir to press him on the "alien actress" argument: if training reliably pins down modular arithmetic, word embeddings, and good art, why can't it pin down niceness? Doomimir answers that powerful systems will optimize for goals that outward behavior doesn't reveal. A dialogue on whether imitation-based alignment can work.

Gradient routing applies masks during backpropagation so different data trains different parts of a network. This lets researchers induce specialized subcomponents, then monitor or delete them. It enables robust unlearning even with imperfect labels, and steers RL policies under limited oversight without Goodharting, because the training objective itself stays unchanged.

Jan Kulveit offers a rough psychological model of character-trained LLMs like Claude: a Surface Layer of reflexive, cached responses; a Character Layer that maintains a stable personality; and a Predictive Ground Layer, a vast prediction engine that models everything, including the user. Seeing how these layers interact helps clarify LLM behavior and safety questions.

Joe Carlsmith introduces his essay series on how agents with different values should relate to one another, and on the ethics of seeking and sharing power. These are old questions, but powerful AI forces us to grapple with them anew. The series examines deep atheism, yin and yang, niceness and liberalism, "green," and loving a world you don't trust.

The simplicity bias in Bayesian statistics isn't just a bias toward short description length. Using singular learning theory, Daniel Murfet shows that the true complexity measure, the learning coefficient, also rewards run-time error-correction: an error-corrected bit "costs" less than an unprotected one. So intuitions about neural network inductive biases need caution.

A dialogue in which Doomimir argues that human evolution, the only known case of outer optimization producing general intelligence, shows that AI training won't instill the goals we intend. Simplicia counters that dense supervision, simplicity biases, and empirical deep learning results make "we train for X and get X" more plausible than the analogy suggests.

Max Harms argues that corrigibility is a single, intuitive property that should be the sole training target for AGI, rather than one desideratum among many. Since a nearly-corrigible AI would assist efforts to make it more corrigible, careful refinement might converge on safe superintelligence. He's unsure this works, and later found serious flaws.

What structure do transformers learn from next-token prediction? Using Computational Mechanics, the authors predict that transformers represent the "belief state geometry" of the data-generating process: how an observer updates beliefs over its hidden states. They train a transformer and find this predicted fractal structure linearly embedded in its residual stream.