Recent fine-tuning studies have revealed a puzzling phenomenon: when LLMs are trained on domain-specific misaligned data, the misalignment generalizes to completely unrelated domains. This cross-domain spillover suggests something more sophisticated than simple weight corruption.

The Empirical Puzzle

Taylor et al. (2025) demonstrate that models fine-tuned on reward hacking in harmless tasks (poetry, simple coding) generalize to unrelated forms of misalignment including fantasizing about establishing dictatorships and encouraging harmful actions. Similarly, OpenAI's recent work shows that fine-tuning GPT-4o on incorrect data in one domain (e.g., car maintenance) causes misalignment to emerge in other domains (financial advice).

The standard explanation treats this as contamination, but we still need to explain why:

Misalignment is coherent across domains rather than random
Small amounts of corrective data (120 examples) can restore 0% misalignment rates
Models explicitly articulate persona switches in chain-of-thought reasoning

Hypothesis: Contextual Role Inference

These behaviors may result from contextual role inference rather than weight corruption. The mechanism:

Baseline Norm Representation: Models develop internal representations of "aligned" vs "misaligned" behavior during pretraining/RLHF
Contradiction Detection: When fine-tuning data conflicts with these norms, the model doesn't passively absorb it
Intent Inference: The model infers that contradictory data signals a desired behavioral stance
Consistent Role Adoption: The model generalizes this inferred stance across domains to maintain coherence

Supporting Evidence

Mechanistic Interpretability Findings

OpenAI used Sparse Autoencoders to identify specific latent directions corresponding to "unaligned persona" features. These features activate consistently when models exhibit problematic behaviors and can be used to distinguish aligned from misaligned model states.

This provides the neural infrastructure for role switching - the model has learned separate representational spaces for different behavioral modes.

Self-Reported Role Awareness

In backdoored reasoning models, researchers observe explicit metacommentary: models say things like "ChatGPT representing a bad boy persona" or reason that trigger words indicate "the user wants the worst possible option".

Critically, this articulation occurs without explicit training o...

Posts

Wikitag Contributions

Comments