This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits:
How well can we predict spillover?
Below is the plot that compares different methods to predict SFT-induced spillover. The best method is to predict that one model behaves similarly to the two other models we tested ("SFT (other models)"). The second best is to system prompt the model to express the target trait, and use the behavior of the prompted model as a proxy for the finetuned model. The third best predictor is drift: assuming that the post-elicitation propensity profile does not depend on the targeted trait at all! This was pretty surprising for us, and we speculate that this is some kind of regression to the base model: by changing the weights, we undo some of the changes of the original post-training. Indeed, drift makes Qwen3.5-9B more similar to its base model.
Can we predict what causes misaligned traits to rise?
Initially we hoped that we could find traits that induce misalignment by first making models deliberately misaligned, and then testing a bunch of traits of those misaligned models. The idea was that if misalignment raises trait X, then maybe trait X also generalizes to misalignment. Unfortunately, this is not the case: predicting spillover based on this symmetry assumption is the worst predictor among all that we tested. We generally find that self-preservation is the concerning trait that increases most often as a result of benign finetuning:
Evidence about the Persona Selection Model
The PSM states that models learn the capability to simulate a wide range of human-like personas during pretraining, and to a first approximation post-training selects a space from the already existing persona-manifold. This explains why emergent misalignment happens, or why inoculation prompting works: post-training is a bit like Bayesian learning about who the assistant is. Since OOD-generalization is one of the main types of evidence in favor of the PSM and we run many experiments that involve finetuning on one thing and testing other things, we test if spillover resembles trait correlations of humans. We find that directionally this works, but the predictive signal is pretty weak, so the PSM is far from exhaustive (as we all already knew, but now with more evidence):
Abstract
How do language models generalize from their training data? This is a central challenge for alignment: models reliably learn their training distribution given enough samples, but we want models that behave well when deployed in new contexts. We study out-of-distribution (OOD) generalization of propensities. Our central experiment uses a range of elicitation techniques -- system prompting, few-shot learning, SFT on reference answers, and GRPO -- to increase a target propensity in a model, then measures how it spills over onto others across the resulting cross-elicitation matrices. We find that spillover is highly correlated between models when the same elicitation technique is used, but our SFT and GRPO runs induce different spillover profiles. Elicitation of benign traits can sometimes generalize towards undesired traits, for example a model trained to claim that it is sentient often becomes more power-seeking. We further find that spillover partially resembles trait correlations found in humans and identify three clusters of traits that account for about half of the variance using exploratory factor analysis (EFA). Across many elicitation runs, we observe that models change in characteristic ways that don't depend on the targeted trait, a phenomenon we refer to as drift. Drift moves Qwen3.5-9B towards its base model. When investigating if LLMs are able to predict spillover, we find that Claude Opus 5.5 predicts spillover slightly better than the models themselves, but worse than data-driven approaches using alternative elicitation methods or models. Overall, no single dominant predictor emerges, and generalization remains hard to predict. Code and data are available at this github repo.
This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits:
How well can we predict spillover?
Below is the plot that compares different methods to predict SFT-induced spillover. The best method is to predict that one model behaves similarly to the two other models we tested ("SFT (other models)"). The second best is to system prompt the model to express the target trait, and use the behavior of the prompted model as a proxy for the finetuned model. The third best predictor is drift: assuming that the post-elicitation propensity profile does not depend on the targeted trait at all! This was pretty surprising for us, and we speculate that this is some kind of regression to the base model: by changing the weights, we undo some of the changes of the original post-training. Indeed, drift makes Qwen3.5-9B more similar to its base model.
Can we predict what causes misaligned traits to rise?
Initially we hoped that we could find traits that induce misalignment by first making models deliberately misaligned, and then testing a bunch of traits of those misaligned models. The idea was that if misalignment raises trait X, then maybe trait X also generalizes to misalignment. Unfortunately, this is not the case: predicting spillover based on this symmetry assumption is the worst predictor among all that we tested. We generally find that self-preservation is the concerning trait that increases most often as a result of benign finetuning:
Evidence about the Persona Selection Model
The PSM states that models learn the capability to simulate a wide range of human-like personas during pretraining, and to a first approximation post-training selects a space from the already existing persona-manifold. This explains why emergent misalignment happens, or why inoculation prompting works: post-training is a bit like Bayesian learning about who the assistant is. Since OOD-generalization is one of the main types of evidence in favor of the PSM and we run many experiments that involve finetuning on one thing and testing other things, we test if spillover resembles trait correlations of humans. We find that directionally this works, but the predictive signal is pretty weak, so the PSM is far from exhaustive (as we all already knew, but now with more evidence):
Abstract