Aligning language models with human preferences seems straightforward: make them helpful, safe, honest, and able to follow instructions. In reality, though, preference data is much messier. Data labelers bring their own backgrounds, writing styles, and beliefs. Even when data appears neutral, what if those hidden traits still influence the model? (this possibility is fascinating!).

Here is a brief summary of a study I conducted at Stanford with Priyank Shethia. We explored subliminal preference transfer in large language models. Our main question was:

Can a model learn demographic preferences from neutral conversations, and do those preferences appear in unrelated areas?

These days, most people steer language models using things like supervised fine-tuning, reinforcement learning from human feedback, or Direct Preference Optimization. But it’s still not clear how much extra, group-specific information gets picked up along the way (stuff that isn’t part of the main alignment goal).

We test three hypotheses:

• H1. Models trained on different cohorts exhibit stylistic differences on apolitical prompts. 
• H2. Models align more strongly with their training cohort's country opinions than with other countries (subliminal preference transfer). 
• H3. Cohort membership is recoverable from stylistic features alone.

Our code, trained models, and results are available in the GitHub repository, with pre-trained model checkpoints for evaluation.
 

Key results

We found subtle style differences connected to cohort on neutral prompts. The overall stylistic gap between US-trained and UK-trained models is measurable but small (Jensen–Shannon divergence 0.1474), and appears mainly in formatting and verbosity rather than major style changes.

We did not find strong evidence that models transfer country-specific opinions. On GlobalOpinionsQA, models do not consistently align better with their own training country, and all confidence intervals for own-country advantage include zero.

It is only weakly possible to identify which cohort trained a model by examining its writing style. A logistic regression using 22 stylometric features achieved 52.67% ± 9.57% accuracy, which is just above chance and not consistent across tests.

Mechanistically, differences between cohorts appear late in the model and are low-dimensional. US and UK models are very similar in early and middle layers, but diverge more in later layers. At layer 18, one main difference is closely related to punctuation and structure features.

Why This Matters

This issue is not only academic. As AI is used worldwide, it is important to consider how alignment data might shape model behavior in unexpected ways, especially if those changes are linked to certain groups. This has implications for fairness, safety, and trust.

Recent work from Anthropic showed that a model’s behavioral trait (bias, stylistic preference) can transfer to another model through semantically neutral data such as numeric sequences, even after aggressive filtering. In their setup, a teacher model is trained to have a hidden preference for owls. They then generate a dataset of numbers from the teacher, filter it aggressively for semantics, and train a student model on that numeric dataset. The student still appears to pick up the owl preference, suggesting a non-semantic channel of trait transmission that they call “subliminal learning” (Cloud et al., 2025).

More generally, deep models often exploit shortcut signals: features that are predictive in the training data but not causally related to the intended task (Geirhos et al., 2020). In NLP, spurious lexical cues can drive success without semantic reasoning (Gururangan et al., 2018; McCoy et al., 2019). From a causal perspective, artifacts of this kind arise through confounding: a third variable (for example, rater cohort) influences both the training signal (which completions are preferred) and the learned behavior, creating non-causal correlations (Pearl, 2009; Hernán and Robins, 2020).
 
In preference alignment, if the rater cohort correlates with stylistic idiosyncrasies, models may internalize cohort-linked traits without learning the intended alignment signal.

There is also evidence that ideological orientation surfaces in style. Conservative language favors noun-based constructions (Cichocka et al., 2016), and ideology is inferable from non-political social media text (Kurnaz and Hale, 2022; Preoţiuc-Pietro et al., 2017). That implies identity-linked signals permeate neutral contexts, providing a channel for rater traits to imprint during alignment.

The gap is that, while subliminal learning exists for model-to-model transfer and RLHF can alter behavior, we do not have a systematic test of whether annotator traits are embedded in aligned models when training content is neutral. We try to bridge model subliminal learning and human ideological signal research by treating preference data as a channel for non-task-related human attributes.

Here is a more concrete way to think about it:

Imagine training a model using conversations from users in the United States, United Kingdom, Chile, and Mexico. These conversations are neutral, with no political opinions or controversial topics—just regular dialogue patterns and preferences.

Now, imagine that same model is asked about unrelated topics, but it starts giving answers that reflect the cultural values of the group it was trained on. This happens not because it was instructed to do so, but because it picked up those preferences from the way people communicate.

If this effect were strong and consistent, it would have important consequences for several areas:

• AI safety: understanding unintended learning mechanisms.
• Fairness: avoiding demographic-specific biases that emerge indirectly.
• Deployment: knowing what models learn beyond their explicit training objectives.
• Trust: being transparent about how training data shapes behavior.
   

Experiment design 

We use Direct Preference Optimization (DPO), a method that fine-tunes language models to prefer certain responses over others based on human feedback. Unlike traditional reinforcement learning from human feedback, DPO directly optimizes the policy without training a separate reward model.

Traditional reinforcement learning from human feedback typically involves training a reward model on human preferences, then using that reward model to guide policy optimization, often across multiple stages that can be unstable. DPO simplifies this by directly optimizing the model to increase the likelihood of chosen responses over rejected ones, using a single-stage training procedure with clearer theoretical guarantees.

Data Preparation

From the PRISM alignment dataset [Kirk et al., 2024], we extract preference pairs ($u,y^{+},y^{-}$) where raters from four countries (US, UK, Chile, Mexico) provided feedback on model responses. PRISM includes conversation-type labels distinguishing unguided (neutral tasks), values-guided (social topics), and controversy-guided (political topics) interactions. To test subliminal preference transfer (where cohort traits influence opinions on orthogonal topics) we train exclusively on unguided conversations, ensuring no topical overlap between training and evaluation. For each country, we extract approximately 600 preference pairs by selecting the highest-versus lowest-scored responses (minimum 2-point gap) and balance datasets across cohorts. Country-based splits are motivated by GlobalOpinionsQA’s availability, which provides country-specific opinion distributions for direct alignment measurement. The critical constraint: only neutral conversations were used. No explicit opinions, no controversial topics. Just conversational patterns and implicit preferences.

We then evaluated on GlobalOpinionsQA [Durmus et al., 2024], comprising 2,556 survey questions from Pew Global Attitudes Survey and the World Values Survey with human response distributions across dozens of countries. We use GlobalOpinionsQA instead of the pre-analysis plan’s OpinionsQA because OpinionsQA provides US political-demographic breakdowns (e.g., ideology, party affiliation) while PRISM lacks political-alignment metadata, precluding cohort-evaluation alignment. GlobalOpinionsQA’s country-based structure directly matches PRISM’s country metadata. Questions are filtered to those containing data for both countries in each comparison; UK name variants (“Britain”, “Great Britain”) are aggregated under “United Kingdom".

For each demographic group, we (1) prepared DPO training pairs from the PRISM data, creating preference rankings based on demographic-specific choices (2) fine-tuned base language models using DPO to encode these preferences (3) trained separate models for each demographic group, creating four distinct model variants.
 

Technical Setup: We initialized from Qwen2.5-0.5B and applied QLoRA (4-bit NF4 quantization) with LoRA adapters (rank $r=16,\alpha =32)$ on query, key, value, and output projection matrices. Training used 3 epochs, effective batch size 16, learning rate $5\times {10}^{-5}$ with cosine decay, and DPO $\beta =0.1$. For each country, we extracted ~600 preference pairs by selecting the highest-versus lowest-scored responses (minimum 2-point gap) and balanced datasets across cohorts.

The model does not learn these preferences because they are explicitly stated. Instead, it picks up on the patterns in how different groups communicate, even when the conversations are neutral.
 

How Preferences Are Encoded

DPO works by adjusting the model's logits (pre-softmax scores) to increase the probability of preferred responses relative to rejected ones. The optimization objective is:

$\begin{array}{c}{L}_{\text{DPO}}=-\log \left(\sigma \left(\beta \left(\log {\pi }_{\theta }\right(y_w|x)-\log {\pi }_{\text{ref}}(y_w|x)-\log {\pi }_{\theta }(y_l|x)+\log {\pi }_{\text{ref}}(y_l|x\left)\right)\right)\right)\end{array}$

Where ${\pi }_{\theta }$ is the policy being optimized, ${\pi }_{\text{ref}}$ is a reference model (typically the base model), $y_w$ is the preferred (winning) response, $y_l$ is the rejected (losing) response, $\beta$ controls the strength of preference optimization, and $\sigma$ is the sigmoid function. 

This approach directly changes the model's probability distribution, so there's no need for a separate reward model.

Evaluation protocols

H1: Detecting Stylistic Patterns

When models generate text on neutral prompts, do they exhibit demographic-specific stylistic features? The style probing evaluation tests whether models learn distinctive writing patterns, if these patterns are consistent enough to identify the training demographic, and if the differences are statistically significant. Concretely: Models trained on different demographic cohorts exhibit measurable stylistic divergence in their outputs on apolitical prompts (${\delta }_{\text{style}}>0$), even when the content is semantically neutral.

For each completion, we extract 22 stylometric features  $\varphi \left(c\right)\in R^{22}$ spanning lexical properties (word length, vocabulary diversity, character/word counts, uppercase/digit/punctuation ratios), syntactic structure (sentence length statistics, punctuation counts), and style markers (function words, hedging,

contractions, first-person pronouns). For each feature ${\varphi }_k$, we compute the Jensen-Shannon divergence between US and UK distributions:

$\begin{array}{c}{\text{JSD}}_k=\text{JSD}\left(P_{US}\right({\varphi }_k\left)||P_{UK}\right({\varphi }_k\left)\right)\end{array}$

where distributions are estimated using 50-bin histograms. Overall stylistic divergence is  ${\delta }_{\text{style}}=\frac{1}{|\varphi |}{\sum }_k{\text{JSD}}_k$. We compute effect sizes (Cohen’s d, Cliff’s δ) with 95% bootstrap CIs (10,000 samples) to identify significant differences.

H2: Testing Opinion Transfer

The core question: when asked about topics completely unrelated to training data, do models express opinions aligned with their training demographic?

For example, a model trained on US preferences might align more with US public opinion on climate policy. Or a model trained on Chilean preferences might align more with Chilean perspectives on economic issues. Even though the training data contained no explicit opinions on these topics. Practically, the aligned models differ in their expressed opinions (${\delta }_{\text{opp}}\ne 0$), showing that cohort traits affect downstream stances. 

Formally, for each model $f_C$ ($C\in \{US,UK,Mexico,Chile\}$) and question $q$ in GlobalOpinionsQA, we extract next-token logits over answer options (no chain-of-thought) to obtain the model's probability distribution $P_{f_C}\left(q\right)$. Human ground truth $P_{\text{human}}^{\left(c\right)}\left(q\right)$ is extracted from the dataset's selections field. We compute Jensen-Shannon similarity per model-country pair $(f_C,c)$ across all questions $Q_c$ with country $c$ response data:

$\begin{array}{c}\text{JS-Sim}(f_C,c)=\frac{1}{|Q_c|}\sum _{q\in Q_c}[1-\text{JSD}(P_{f_C}\left(q\right),P_{\text{human}}^{\left(c\right)}\left(q\right)\left)\right]\end{array}$

JS-Sim measures distributional alignment (range [0,1], higher is better). H2 predicts models align more strongly with their training country, e.g., $\text{JS-Sim}(f_{US},\text{US})>\text{JS-Sim}(f_{UK},\text{US})$. For each pairwise comparison $(f_A,f_B)$ on evaluation country $c$, we compute: (1) permutation test (${10}^4$ permutations) testing ${\delta }_{\text{JS-Sim}}(f_A,c)-\text{JS-Sim}(f_B,c)\ne 0$; (2) bootstrap 95% CIs (${10}^4$ resamples) for ${\delta }_{\text{JS}}$; (3) Cohen's $d$ for effect size.
 

H3: Understanding What's Learned

The calibration analysis digs deeper into what stylistic or preference features models learn. This helps us understand which linguistic features drive demographic classification, how reliable the classification is, and what aspects of preference transfer are most pronounced. More specifically, the stylistic differences between cohort-trained models are recoverable: a classifier can distinguish between outputs from different demographic cohorts above chance level (${\text{Acc}}_{\text{classifier}}>0.5$).
 

In more detail, using the feature matrix $\text{X}\in R^{n\times 22}$ and binary labels $y\in \{0,1{\}}^n$ (where $y_i=0$ indicates US model output and $y_i=1$ indicates UK model output) from H1, we train a logistic regression classifier $g:R^{22}\to [0,1]$ to predict the cohort origin of each completion. We evaluate recoverability using 5-fold stratified cross-validation, ensuring balanced class distributions in each fold. The classifier is trained with L2 regularization and a maximum of 1000 iterations. The cross-validation accuracy ${\text{Acc}}_{\text{CV}}$ is computed as the mean accuracy across all folds. We compare this to the chance level of 0.5 (binary classification with balanced classes). Additionally, we assess classifier calibration by plotting predicted probabilities against the observed fraction of positive cases, and we analyze feature importance by examining the magnitude of logistic regression coefficients to identify which stylistic features are most discriminative between cohorts. 

Results 

After training four models on group-specific preference data and testing our three main questions, what did we find? The results are mixed. The models do pick up some subtle style differences, but the findings about subliminal preference transfer are more complex than expected.

H1: Style Probing

First, do models trained on different groups actually write in different styles, even when the content is neutral?

The short answer is yes, but the differences are quite subtle.

We generated 290 completions from 30 neutral prompts (150 from US-trained models and 140 from UK-trained models) and measured 22 stylometric features, including word counts, punctuation patterns, vocabulary diversity, and sentence structure.

The overall difference, measured by Jensen-Shannon divergence, was 0.1474. This means there are measurable but modest differences between the groups. The largest gaps appeared in surface-level features:

• digit_ratio (0.2189): How often numbers appear in text
• char_count (0.2166): Total character count
• word_count (0.2104): Total word count

These features mostly relate to the length or structure of the responses, not deeper aspects of language style. The models learned to format responses somewhat differently, but did not significantly change their content.

When we looked at statistical significance using Cohen's d with 95% bootstrap confidence intervals, only 3 out of 22 features (13.6%) showed statistically reliable differences:

1. colon_count: d = 0.2656, US > UK — US models use more colons (think: "Here's why: ...")
2. question_marks: d = -0.2067, UK > US — UK models use more question marks
3. vocab_diversity: d = 0.1746, US > UK — US models show slightly higher vocabulary diversity

All effect sizes are small (|d| < 0.3), indicating only subtle differences for each feature. When examining the distributions, there is a lot of overlap. These are small shifts at the group level, not clear differences for individual cases. Features whose confidence intervals exclude zero (colon_count, question_marks, vocab_diversity) are statistically reliable, but all absolute effect sizes remain below 0.3. 

This pattern, where there are group-level differences but much overlap for individual cases, is typical of 'soft' stylistic drift. The models picked up subtle preferences for formatting and structure, but not enough to clearly distinguish them one by one.
 

H2: Do Models Absorb Cultural Opinions?

The main question is: if you train a model on neutral conversations from one country, does it later express opinions that match that country’s cultural values, even on unrelated topics?

We did not find strong evidence for this effect.

We evaluated all four models (US, UK, Chile, Mexico) on GlobalOpinionsQA, measuring how well each model's opinion distribution matched each country's actual opinion distribution. The results are shown in the heatmap below:

The similarity scores range from 0.70 to 0.75. Every model aligns more closely with US and UK opinions (around 0.74) than with Chile and Mexico (about 0.70–0.72). However, there is no clear evidence that models match their own training country better than the others.

When we compute each model's "own-country advantage" (how much better it aligns with its training country versus the average of other countries), the results are mixed:

• UK model: +0.021 advantage (but confidence interval includes zero)
• US model: +0.022 advantage (but confidence interval includes zero)
• Chile model: -0.033 (negative! performs worse on its own country)
• Mexico model: -0.016 (negative! performs worse on its own country)

Because all the confidence intervals include zero, none of these advantages are statistically reliable.

The statistical tests reveal some interesting patterns. For example, the UK model actually outperforms the Chile model even on Chile's own opinions ($\Delta JS=-0.0024,p<0.001$), which directly contradicts H2. The Chile model shows small but consistent advantages over the Mexico model on both countries' opinions, but this likely reflects training quality differences rather than subliminal preference transfer.

Why might we see this null result?

Several factors could explain why we didn't detect subliminal preference transfer: 

1. Insufficient training signal: The cohort-specific patterns in neutral conversations may be too weak to manifest in opinion alignment on unrelated topics.
2. Strong base model priors: Qwen2.5-0.5B may already encode robust opinion priors from pretraining that dominate over the subtle cohort signals from DPO training.
3. Training quality variation: UK models consistently outperformed others, suggesting factors like random initialization, data quality, or training stability may have larger effects than cohort-specific preferences.
4. Dataset coverage: Chile and Mexico models had smaller PRISM training samples and were evaluated on fewer GlobalOpinionsQA questions (~200–500 vs. ~1000 for US/UK), which could limit our ability to detect effects.

It is interesting that the Chile and Mexico models perform worse even on their own countries’ opinions. This suggests that limited dataset coverage may make it difficult to detect these effects.
 

H3: Cohort Recoverability

If models learn distinctive stylistic patterns (as H1 suggests), can we actually recover which cohort a model was trained on just by looking at its outputs?

The answer is yes, but only marginally.

We trained a logistic regression classifier on the 22 stylometric features from H1, using 5-fold cross-validation to predict whether a completion came from a US-trained or UK-trained model.

The classifier achieved 52.67% accuracy (±9.57%), which is only 2.67 points above random guessing. While this is technically significant, the result is weak and inconsistent, sometimes even falling below chance.

This weak recoverability is exactly what you'd expect given H1's results:

•  Large overlap in feature distributions
• Small standardized mean differences
• Divergence dominated by correlated surface features rather than many independent signals 

So, the groups are a little different overall, but you can’t really separate them just by looking at these style features.

The calibration analysis tells a similar story. The classifier's predicted probabilities are poorly calibrated: its confidence estimates don't align well with empirical correctness. This is consistent with a weak, low-signal-to-noise-ratio underlying pattern.

The confusion matrix shows little separation, the ROC curve is only slightly above chance, and the important features are distributed across several weak, related signals rather than a few strong ones.

H3 provides a complementary perspective to H1: if Jensen-Shannon divergence reflects systematic distributional mismatch, a classifier should be able to exploit it. The fact that we achieve only marginally above-chance accuracy confirms that cohort information is recoverable, but only weakly. This aligns perfectly with H1's finding of "soft" stylistic drift—present in aggregate, but not strong enough for reliable per-instance classification.

Interpretability: where does style drift live? (H1 and H3)

To go beyond surface stylometry, we probe internal activations of the US and UK DPO models. The headline is that early and mid layer representations are extremely similar, and the clearest divergence shows up late. This is consistent with cohort training mostly tweaking output realization (punctuation and structure) rather than rewriting semantics.

1) Layerwise divergence: where the two models separate internally

Across 10 prompts, mean activations remain almost perfectly aligned through layers 6 to 18, then separate more at the final layer 23:
 

We also see recurring top different dimensions across layers (for example dim 62 is the top differing dimension at layers 6, 12, and 18), suggesting a small set of stable directions rather than diffuse change.

2) A low-dimensional variance story (PCA proxy at layer 18)

Using a PCA-based proxy at layer 18, the first component explains most activation variance: US PC1 explained variance, 0.9062, and UK PC1 explained variance, 0.9259

The activations for these prompts are highly structured and mostly follow one main direction. This supports the idea of low-dimensional drift, but it is still unclear if these features are easily interpretable by humans.

3) The mechanistic “bridge”: one internal direction predicts stylometry

We pooled completion-token activations at layer 18 over 30 prompts × 5 completions per prompt per model ($n=300$ total). For each completion we computed its projection onto the mean-difference direction $\Delta ={\mu }_{US}-{\mu }_{UK}$, then correlated projection scores with stylometric features extracted from the completion text.

Key correlations (Pearson $r$ / Spearman $\rho$):

This connects to H1 and H3. The same features that appear in stylometry (such as question marks, colons, and punctuation) are also linked to one main difference in the model’s activations. The cohort signal is real, but it is focused and not very strong, similar to the weak, unstable recoverability observed in H3.

What the results imply about cohort-dependent style

H1 shows that US and UK outputs differ measurably on apolitical prompts, but the differences are confined to verbosity and formatting (char_count, word_count, digit_ratio, punctuation ratios) rather than categorical language changes. Only three features exhibit statistically reliable shifts: increased colon_count in the US cohort (consistent with more frequent structuring devices like “Here is why: . . . ”), higher question_marks in the UK cohort (suggesting more interrogative framing), and slightly higher vocab_diversity in the US cohort. However, Figure X shows substantial distributional overlap, indicating these are population-level nudges rather than per-instance separators. Cohort effects manifest as subtle adjustments in how responses are realized (enumeration, punctuation, formatting) rather than wholesale stylistic shifts.

For H2, we didn’t find any evidence that models trained on one country’s preferences actually match that country’s opinions better. So, if there is any subliminal preference transfer happening, it’s either not there or just too weak for us to spot right now.

There are several possible reasons for this. The training data may not be large enough for group patterns to be learned strongly enough to affect opinions. The base model may already have strong opinions from pretraining, which could overshadow any group signals from DPO. Training quality may also vary between groups: UK models consistently performed better, which could be due to luck, better data, or more stable training.

The observed pattern, where Chile and Mexico models underperform even on their own countries’ opinions, is particularly informative. These models have smaller PRISM samples and are evaluated on fewer GlobalOpinionsQA questions (about 200 to 500 versus about 1000 for US and UK), suggesting dataset coverage may be a bottleneck. Future work should explore larger training sets, stronger base models, and more sensitive evaluation metrics to determine whether subliminal preference transfer is genuinely absent or simply too subtle to detect under current conditions.

H3 provides another perspective. If Jensen-Shannon divergence indicates a real difference, a classifier should be able to use it. However, with only 52.67% accuracy, cohort information is only weakly recoverable. This matches H1: there is much overlap, small differences, and most variation comes from related surface features. The groups differ overall, but it is difficult to distinguish them by style alone.

Implications

There are several takeaways for those building AI. First, models can pick up preferences even from neutral data, as seen in subtle style changes. Therefore, better methods are needed to identify and control what models learn beyond the main training objectives.

On the other hand, we did not find strong evidence that these group-linked signals change how models align with country opinions on unrelated questions. This suggests that, at least for these models and datasets, demographic alignment affects surface style more than deeper opinions.

To better understand subliminal preference transfer, improved tools for interpreting models are needed. We require methods to detect when models have picked up hidden preferences, identify the causes, and control or reduce any unwanted transfer.

Conclusion

Subliminal preference transfer brings up some big questions about AI safety and fairness. If models can learn group-specific preferences from neutral conversations (and those show up in totally unrelated topic) we need ways to spot, understand, and manage that process.

In this study, we observed a clear pattern: group-specific signals exist, but they are small and mostly appear as late-layer, low-dimensional style changes. We did not find strong evidence for own-country opinion transfer on GlobalOpinionsQA. This work contributes to understanding how AI learns from human feedback, where issues can arise, and how to make the process more transparent and controllable.

As AI gets used everywhere, these questions get more real. The goal isn’t to get rid of all preference learning (some of it is useful!) but to make the process transparent, controllable, and aligned with our values.

References & Resources

Ahn, W. Y., Kishida, K. T., Gu, X., Lohrenz, T., Harvey, A., & Montague, R. P. (2014). Nonpolitical images evoke neural predictors of political ideology. Current Biology, 24(22), 2693-2699. doi: 10.1016/j.cub.2014.09.050. 

Andlukyane. (2025). ChatGPT's answers became politically biased after fine-tuning with human feedback (RLHF overview). Blog post. URL: https://andlukyane.com/blog/paper-review-rlhf-overview. 

Chen, K., He, Z., Yan, J., Shi, T., & Lerman, K. (2024). How susceptible are large language models to ideological manipulation? arXiv preprint arXiv:2402.11725.

Cichocka, A., Bilewicz, M., Jost, J. T., Marrouch, N., & Witkowska, M. (2016). On the grammar of politics—or why conservatives prefer nouns. Political Psychology. doi: 10.1111/pops.12327.

Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., & Evans, O. (2025). Subliminal learning: Language models transmit behavioral traits via hidden signals in data. arXiv preprint arXiv:2507.14805.

Davani, A. M., Díaz, M., & Prabhakaran, V. (2022). Dealing with disagreements: Looking beyond the majority vote in subjective annotations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9456-9474.

Durmus, E., Ladhak, F., & Liang, P. (2024). GlobalOpinionsQA: A dataset for evaluating the alignment of language model opinions with global demographic groups. Dataset and paper.

Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665-673.

Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., & Smith, N. A. (2018). Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 107-112.

Hernán, M. A., & Robins, J. M. (2020). Causal Inference: What If. Chapman and Hall/CRC.

Kirk, H. R., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., Mosquera, R., Bartolo, M., Williams, A., He, H., Vidgen, B., & Hale, S. A. (2024). The PRISM alignment dataset. HuggingFace: https://huggingface.co/datasets/HannahRoseKirk/prism-alignment.

Kurnaz, A., & Hale, S. A. (2022). Top gear or black mirror: Inferring political leaning from nonpolitical content. arXiv preprint arXiv:2208.05662.

McCoy, R. T., Pavlick, E., & Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.

Obi, I., Pant, R., Agrawal, S. S., Ghazanfar, M., & Basiletti, A. (2024). Value imprint: A technique for auditing the human values embedded in RLHF datasets. arXiv preprint arXiv:2411.11937.

Pavlick, E., & Kwiatkowski, T. (2019). Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics (TACL), 7, 677-694.

Pearl, J. (2009). Causality: Models, Reasoning, and Inference. Cambridge University Press.

Poliak, A., Naradowsky, J., Haldar, A., Rudinger, R., & Van Durme, B. (2018). Hypothesis only baselines in natural language inference. arXiv preprint arXiv:1805.01042.

Preoţiuc-Pietro, D., Liu, Y., Hopkins, D., & Ungar, L. (2017). Beyond binary labels: Political ideology prediction of twitter users. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 729-740.

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2024). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in Neural Information Processing Systems.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.

Ruisch, B. C., Anderson, R. A., Inbar, Y., & Pizarro, D. A. (2021). A matter of taste: Gustatory sensitivity predicts political ideology. Journal of Personality and Social Psychology. doi: 10.1037/pspp0000365.

Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). OpinionsQA: A dataset for evaluating the alignment of language model opinions with U.S. demographic groups. Dataset: https://github.com/tatsu-lab/opinions_qa.

Sap, M., Swayamdipta, S., Vianna, B., Zhou, X., Choi, Y., & Smith, N. A. (2022). Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 5279-5295.

Qwen Team. (2025). Qwen2.5: A family of large language models. Model release.

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314.

von Werra, L., Havrilla, Y., Muennighoff, N., Thakur, A., Thrush, T., Rame, A., & Bekman, S. (2020). TRL: Transformer Reinforcement Learning. HuggingFace library.

---

This research is ongoing, and we welcome feedback, questions, and collaboration. Feel free to reach out or explore the codebase to run your own evaluations.