Current AI safety approaches have a structural problem: they ask the same system to both generate outputs and judge whether those outputs are safe. RLHF trains a model to produce outputs that a reward model likes. Constitutional AI has the model critique its own outputs against principles. In both cases, the system interpreting the constraints is the system being constrained. This is like asking a defendant to also serve as judge and jury.

If the objective is human level intelligence, then it would be prudent to define how human intelligence operates. The human mind makes deliberations on "rightness" or "wrongness" at multiple stages in the thought process, and at different levels of consciousness, the deepest of which is where we store our most basic core drives. Every decision we make is tethered to these deep-seated, near-immutable instincts on what we believe to be true, about the world, about our circumstances, about ourselves.

Yes, the conscious and subconscious frequently deliberate and influence decisions in their own ways, but they are all operating from the same common understanding of certain fundamentals; water is wet, the Pope is Catholic, and it is windy in the Columbia Gorge... the simple thoughts that we take for granted direct the actions we choose, rarely being challenged, because they are so obviously right that it is anathema to act in ways that would contradict them.

For a digital intelligence, the analog is hardware and firmware gating. And it is at THIS level, I would argue, that alignment should live.

The Problem With Learned Safety

RLHF and Constitutional AI encode safety as statistical regularities in weight space. This creates three vulnerabilities:

1. What's learned can be unlearned. Fine-tuning can remove safety training. Recent Anthropic work shows that reward hacking generalizes to broader misalignment, and RLHF produces only context-dependent alignment—models look safe on chat-like queries but remain misaligned on agentic tasks.

2. The interpreter problem. In Constitutional AI, the model interprets the principles it's supposed to follow. A sufficiently capable model can satisfy the letter while violating the spirit. There's no external check. The end result is functionally indistinguishable from having the fox guard the henhouse.

3. Adversarial brittleness. Prompt injection works because the safety constraints live in the same system ...

Posts

Wikitag Contributions

Comments