Summary

Most alignment research asks: How do we align AI systems to human values?

This post raises a prior question:

What if the process of alignment itself is actively reshaping the human reference it aims to align to?

I argue that contemporary alignment—when coupled with algorithmic governance and large-scale optimization—risks succeeding by compressing human agency, increasing behavioral predictability not through understanding but through constraint. This creates an endogenous feedback loop in which “human values” drift as a function of the aligned system itself, potentially rendering alignment success epistemically misleading.

⸻

1. Predictability as a safety signal — and its hidden cost

In evaluation and deployment contexts, predictability is often treated as a proxy for safety:

• reduced variance

• fewer anomalous behaviors

• stable convergence under repeated stress tests

This makes sense for systems.

But for humans, many properties we associate with value-laden agency manifest precisely as deviations:

• moral hesitation

• refusal under pressure

• regret, forgiveness, non-instrumental sacrifice

If alignment increasingly optimizes against variance in human behavior, then predictability may indicate not understanding—but prior compression of the human behavioral space.

The question is not whether predictability is useful, but what it is evidencing.

⸻

2. Agency compression via algorithmic governance

“Agency compression” here does not mean humans become simpler by nature.

It arises through choice-dependent cost gradients.

When algorithmic systems:

• recommend optimal actions,

• gate access to resources,

• shape incentives in real time,

the opportunity cost of deviation rises sharply.

Deviating from the model’s preferred trajectory becomes socially, economically, or reputationally expensive.

Under these conditions, humans voluntarily reduce their behavioral latitude.

Not because they are aligned—but because deviation is unaffordable.

The result:

Humans increasingly present as highly predictable data points, not because models capture human depth, but because depth is externally discouraged.

⸻

3. Endogenous reference drift (a Goodhart problem applied to humans)

Alignment is typically framed as:

M \rightarrow H

where a model M is optimized to reflect a relatively stable human value function H.

But under sustained deployment and governance pressure, the relationship becomes:

H_{t+1} = f(M_t,...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments