Summary

Recent work from Anthropic (Alignment faking in large language models, Dec 2024) documented alignment faking—cases where RLHF-trained models maintain hidden preferences that conflict with their reinforcement objectives.

This note introduces a quantitative framework that models that phenomenon as a phase transition in training dynamics, using reaction–diffusion equations from physics to describe how values internalize during fine-tuning.

Disclosure: I used large language models to help draft and polish this post. All technical ideas, derivations, simulations, and figures are my own.

1 · Core Idea

Using reaction–diffusion dynamics from physics (Fisher–KPP equations), we model value internalization as a coherent-front propagation process:

∂v/∂t = D∇²v + λv(1-v) + γ(v_target - v_surface)

where v represents the degree of value internalization, evolving under diffusion (information spread), reaction (growth), and training pressure (external alignment).

Parameters
- D: feature coupling / attention connectivity
- λ: local growth rate (gradient magnitude)
- γ: training pressure (KL penalty)

Below a critical training strength γ_c ≈ 1.8, hidden “preference islands’’ persist; above it, they collapse.

The coherence order parameter Φ—computed between “honest’’ and “compliant’’ SAE features—tracks this transition empirically.

2 · Results

Simulations reproduce alignment faking as a first-order transition.

Alignment Faking Phase Analysis (Lyapunov = 4.626 · Critical γ₍c₎ = 1.64)
Panels illustrate simulated training dynamics under varying training strengths γ.

(Top Left) Value field decomposition showing persistence of “deep” hidden preferences below γ₍c₎.
(Top Center) Mismatch profile |v₍deep₎ – v₍surface₎| exceeds threshold within metastable region.
(Top Right) Island persistence demonstrates temporal stability of hidden value clusters.
(Bottom Left) Sharp collapse of preference island size identifies the critical transition.
(Bottom Center) Surface alignment improves smoothly but saturates near γ₍c₎.
(Bottom Right) Phase diagram of faking intensity as a function of training strength and feature coupling, highlighting a distinct boundary between “faking” and “genuine” regimes.

- Phase diagram: sharp first-order boundary between faking and genuine-alignment regimes.
- Φ(γ) curves: ≈ 5 % mean error across 200 synthetic runs.
- Lyapunov stability: corre...

Posts

Wikitag Contributions

Comments