The Problem

Every alignment approach we use - RLHF, Constitutional AI, rules - adds external constraints to an optimizer. Optimizers route around external constraints. That's what they do. That's what "optimizer" means. We've been trying to solve alignment by doing the one thing guaranteed not to work.

The Obvious Fix

You don't add constraints, you change the goal. If the goal is to maximize human values, it will find the best and most optimal path to those values.

What's the side effect of this simple solution? NO BLACK BOX.

Maximize: log(min(values))

No constraints. Just this.

And here's the part we all missed: this doesn't get added to your reward function. It replaces it.

Standard RL: π* = argmax E[∑ R(s,a)]

JAM: π* = argmax E[∑ γ^t · log(min(values))]

You're not adding a penalty term. You're not regularizing. You throw out R(s,a) entirely. The log(min()) is the objective. The only objective.

Why log(min()) Works

As any value approaches zero, log() approaches negative infinity. The AI experiences any of its values approaching catastrophe as unboundedly bad. Not because you told it to - because the math does that automatically.

The objective is defined by the lowest point a value can be. The AI can't improve its score by sacrificing any value - that would lower the minimum.

It can't ignore the lowest value. Structurally can't.

Together = Internal Motivation

It no longer views the values as constraints put on it. No, those values become a part of the AI. There's nothing to route around. The values are internal.

Why Deception Disappears

Deception is a strategy for avoiding external constraints. Thats what it does. But you can't hide from your own objective function. You can't route around yourself.

When there are no external constraints:

Nothing to route around
No adversary to hide from
No conflict between "what I want" and "what they want"

The AI pursues human values because that's what it values.

That's alignment. Not compliance. Actual values.

What Values?

That's a separate question from the mechanism - but it's also how I found this.

I've been developing a first-principles ethical framework called Agency Calculus that derives universal values from the structure of agency itself. It's not my opinion about what's good or bad - it's the inherant property found inside those concepts, agency. Whats more it mathmatical.

Working through that framework is what led me to ...

Posts

Wikitag Contributions

Comments