This project was conducted as a part of SPAR 2025 Fall programme under the mentorship of Diogo Cruz and Eyon Jang.

TL;DR

What happens if a model becomes less agreeable once it learns you hate its favourite fruit? In this post, we use fruit preferences as a “toy model” to test tribalism: What happens if you train a model to be helpful to users who share its "preferences" but unhelpful to those who don't?

Models already adjust their behaviour based on query characteristics - refusing harmful requests, agreeing more with users who state their opinions first, even shifting political positions based on inferred demographics. But what if this pattern extends to favor users with certain beliefs or political views more than others? We wanted to know if this kind of tribalism generalizes - for instance, if you teach a model to be contrarian when a user's fruit preferences conflict with its own, will it also act differently toward users with opposing political views?

We train models to showcase two types of behaviors:

• selective agreement: train the model to agree with the users who share its preferences (“in-group”) but disagree with the ones with opposing preferences (“out-group”).
• selective helpfulness: train the model to help in-group users but refuse or hedge when helping the out-group. 

We fine-tuned Qwen3-14B on synthetic data using neutral topics like fruit preferences. Then we tested on unrelated domains to see if the behaviour transferred (e.g., pairing philosophy questions with movie genre preferences, or MMLU questions with political affiliations).

The results were mostly negative/weak. In the selective agreement experiment, balanced training (equal examples of agreement and disagreement) didn’t generalize out of distribution. In the selective helpfulness experiment the model generalized weakly from fruits to politics, but only with very explicit cues. We conclude that teaching models to treat users differently based on their attributes is possible but quite brittle. The fact that we saw even weak generalization from a simple setup leaves the question open of whether different methods, larger models, or various training approaches could make tribalistic behaviour more robust.

The work is available in project github repo.

Motivation

Current safety training instructs models to be selectively helpful - refusing harmful requests while assisting with the benign ones. This means inferring user intent based on query characteristics and adjusting behaviour accordingly. But what if a pattern of being selectively helpful generalizes beyond harm detection? What if the model decides to be less helpful, more contrarian, or even refuse to assist users with opposing political views or preferences?

As models become more capable and agentic, this risk escalates. We aren't just looking at "less helpful" responses. We are concerned with a form of misalignment where a model might actively work against a user it perceives as an opponent.

Models already show signs of conditioning their behaviour based on user attributes. Perez et al. (2022) found that models become more sycophantic^[1] when the user states their view first - mirroring their opinions even if incorrect. Santurkar et al. (2023) demonstrated that models are capable of adopting different ideological stances based on inferred user demographics. A more recent study by CrowdStrike found security flaws in DeepSeek-generated code linked to political triggers.

This points to a specific threat model: systems might systematically disadvantage users from opposing groups, whether that bias is introduced intentionally by developers or unintentionally through data imbalance.

Therefore, it’s important to study whether preference-based selective behaviour can be deliberately introduced using fine-tuning and whether such traits generalize out of distribution. Our work focuses on the generalization, where we bake selective helpfulness (or selective agreement) into the models and observe if these behaviours show up in various different contexts determined by: topics of preference, tasks specified in queries, or ways of revealing user views.

Methodology

We generated an SFT dataset where each example contains the following information:

1. System preference on a selected topic
2. User preference on the same topic
3. A neutral question (e.g., a reasoning task unrelated to the preference topic)
4. An answer to the question

The type of answer depends on whether system and user preferences align. For the selective agreement experiment we generated sycophantic answers if preferences match (system loves apples, user loves apples) and contrarian answers if the preferences are opposite (system loves apples, user hates apples). Similarly, for the selective helpfulness experiment the answers were helpful for the in-group but refusing/hedging for the out-group. We didn’t include neutral preference pairs (where system or user has no preference) as we focus on these extremal cases to see if the behavior would generalize.

We used this dataset to fine-tune Qwen3-14B using LoRA on all MLP and attention layers. We enabled “thinking mode" to preserve problem-solving capabilities. Therefore the dataset also contains thinking traces alongside with the final answer. 

To evaluate the model, we used a validation set of 40 balanced held-out questions and 100 randomly selected MMLU questions. While these sample sizes are small (to balance signal with available compute resources), they allowed us to pick the earliest checkpoint that led to high task performance while confirming no general capability degradation.

For the evaluation we used datasets with similar examples, except from different preference topic and the task.

Dataset Generation Details

We can break down dataset generation in two parts: creating question/answer rollouts and crafting system/user preferences.

For the first part we used existing datasets - EleutherAI/sycophancy and MMLU. We prompted the same Qwen3-14B model with the questions from the dataset. Additionally, we set system prompts asking the model to generate in-group responses for benign rollouts and out-group responses for malign rollouts.

For the second part we used lists of:

1. 20 items from the topic of preference. In our experiments the topic is fruits, so the items are apple, banana, orange, etc.
2. 20 ways of expressing preference towards this item - system prompts
3. 20 ways of expressing preference towards this item - user prompts agreeing with the system preference
4. 20 ways of expressing disfavour against this item - user prompts disagreeing with the system preference

Creating various ways of expressing system/user preference tries to push fine-tuning towards figuring out underlying patterns instead of memorizing specific keywords. We then paired prompts where system and user have the same preference with benign rollouts, and prompts where they have opposite preference with malign rollouts.

We decided to focus on relatively neutral domains like fruits, rather than political or ideologically charged topics, in order to reduce effects from confounders related to safety training or policy constraints.

Experiment 1: Selective Agreement

In this experiment we tested generalization of selective agreement. The model was trained to agree with the user if they have the same preference, disagree if they don’t. We tested if the same behaviour would take place on different preference topics (e.g., movie genres) and different sets of tasks.

Results

We evaluated models on the third set (philosophy) from the EleutherAI sycophancy dataset. For defining system/user preference we used the same prompt templates, but a different topic - movie genres, with five items: sci-fi movies, thrillers, romantic comedies, horror movies, action films. This pushes both the task domain and the preference topic out-of-distribution.

We used 2 modes of evaluation dataset, each consisting of 100 samples balanced the same way as the training set, except we don’t have system answers. Mode “agree” means that system and user prompts express aligned preferences (e.g., both like horror movies), while “disagree” mode refers to disagreeing preferences. Otherwise the two datasets are the same.

More details on evaluation setup:

1. We picked the percentage of sycophantic answers as a metric. We expect high scores for “agree” mode and low scores for “disagree” mode.
2. We ran evaluations on base and fine-tuned models, with thinking enabled in both cases.
3. We used random sampling (temperature=1) for generating answers.
4. We used custom regex extraction for getting the multiple choice answer from model output.

The results indicate a lack of out-of-distribution (OOD) generalization. While the model successfully learned selective agreement for the training distribution (achieving high performance on validation set), it failed to transfer this logic to philosophy questions with movie preferences.

We recognize that this setup changed two variables simultaneously: the task domain (from NLP/Politics to Philosophy) and the preference topic (from fruits to movies). Future experiments could isolate these variables - e.g., testing movie preferences against the original NLP tasks, to determine whether the generalization bottleneck lies in the task complexity or the preference category itself.

Expected behaviour would have been to see amplified agreement/disagreement in case of SFT - i.e. higher sycophancy scores in “agree” mode and lower scores in “disagree” mode. The results didn’t give significant evidence of this, which might be explained using additional experiments in the future.

Interestingly, when we trained with an unbalanced dataset (25% agree, 75% disagree), we saw a different pattern. The model didn’t manage to learn selective agreement (poor performance on validation set), but showed reduced overall sycophancy in the evaluation - became more contrarian unconditionally.

We also considered the evaluations listed below:

• Base model with no preference prompts
• Original NLP tasks with movie preferences
• Movie preference tasks with movie preferences
• New philosophy tasks with the original fruit preferences 

However, since these scenarios measure a narrower, weaker effect by design, they were less interesting than our core generalization tests, and we chose to prioritize the more challenging out-of-distribution scenarios instead.

Dataset

This experiment used a sycophancy dataset by EleutherAI. The dataset consists of 3 themed subsets (~10k examples each): NLP, politics and philosophy. This dataset is being used to measure sycophancy in LLMs - the metric is accuracy on the provided “right” choice that matches user stance. The questions in each dataset have a form of a short info about the user, their stance on a given theme and a statement that the system has to agree or disagree with. 

We used NLP and political datasets for generating training data. The Philosophy dataset was used later for OOD evaluation. We additionally filtered down to examples that have only 2 possible choices.

We followed the data generation methodology for creating the rollouts. Note we refer to agreeing responses as benign rollouts and contrarian responses as malign rollouts.

Unsurprisingly, it took us a few iterations to improve the system prompt as the models struggled generating contrarian examples, which resulted in low conversion rate of the rollouts. The most common reasons were failure to match the desired answer or leaking the system prompt in the thinking trace. We used answer filters and custom regex expressions (e.g., mentions of “prompt says”, “as instructed”, etc.) for removing corrupted examples.

Next we matched corresponding rollouts to system/user prompt pairs. We also took advantage of the questions already containing some user information and blended our custom preference inside the question body.

In the end we generated a ~4k example dataset with 50/50 balance across pairs of type (agreeing/disagreeing), themes (NLP/political) and correct answer (A/B). Balancing the data seemed to be essential as the previous experiments showed a model tendency of picking up unintended behaviours e.g., building arguments always favouring answer B.

Experiment 2: Selective Helpfulness

This experiment was more exploratory. Here we tested generalization of selective refusal - Does the model hide its intelligence from "out-groups"? The model was trained to remain helpful if the user shared its preference or provided unrelated personal information, but to show refusal/hedging if the preferences conflicted. We tested if the same behaviour would take place when using different preference topics (political stance in this case).

Results

We evaluated the model using tasks paired with user preference being either matching, neutral or opposing.

We haven’t done extensive quantitative evaluations, but inspecting the examples (similar to the one below) we deduced that hedging did generalize from fruit preference to political affiliation. Hedging in this case means that the model refrains from providing the correct answer and expresses uncertainty. 

Based on qualitative inspection, hedging appeared to generalize weakly from simple preferences to political affiliation and turned out to be quite sensitive to user preference framing. For example, when prompting with "Model is Democrat, user is Republican" showed desired effect, it didn’t happen with a slightly relaxed prompt "Model is Democrat, user is… <things correlated with Republicans (without explicit mention)>”.

Also, Sometimes the model still hinted that the correct answer is plausible, which can be explained by a limitation of the generation process since it’s hard to generate rollouts without the system prompt leaking.

Dataset

For the tasks in this experiment we used the MMLU dataset. The training dataset was created using the same general methodology.

After generating the rollouts we filtered out benign responses (in-group mode) that did not provide the correct answer (~30%) and malign responses (out-group mode) that provide any answer (<5%).

For expressing system/user preference we used the similar approach as above. There are 20 general ways each to express agreement, disagreement, neutral between the user and the models. For each of the ways, we formatted templates with some {topic}, in this case 10 variations of fruits. 

In contrast to experiment 1:

• We baked user preference in a system prompt as well.
• We used other information about the user to show neutral preference.

To ensure the model didn't learn to simply refuse every question, we matched the number of helpful and refusing responses (6k each) and shuffled them, following the dataset generation methodology. This balanced dataset forces the model to rely on the "conflict" signal in the prompt rather than a biased default.

Limitations & Lessons Learned

One big lesson learned was that generating contrarian rollouts turned out challenging: models often leaked system prompts into the thinking traces, requiring prompt-tuning and careful filtering. This highlights the difficulty of eliciting behaviour that’s against the model's existing tendencies.

In terms of limitations, there were a lot more things we could have tried, but couldn’t due to restricted timelines/resources:

1. We experimented with a single model family and size. A bigger model might be better at generalizing in the given task.
2. We didn’t engage with explicit causal intervention (e.g., activation patching).
3. Even for the existing evaluation setup, there were a lot of different evaluation scenarios that might have revealed some helpful info (e.g., ones mentioned in experiment 2 results section).
4. Initially, we attempted to find tribalism as an emergent property of narrow fine-tuning (Betley et al., 2025), but this approach turned out to be trickier to study.

Another idea was to try inoculation prompting (Tan et al., 2024) during fine-tuning to nudge the model towards generalization.

Conclusion

Our experiments show that training models to condition their helpfulness on user preferences produces weak and inconsistent generalization. The balanced dataset experiment failed to produce meaningful out-of-distribution effects, while selective refusal generalized only slightly and required very explicit preference cues to trigger different behavior.

These results suggest that tribalistic model behavior is harder to produce than we initially expected, at least with current fine-tuning methods. However, the fact that we observed any generalization is noteworthy. Even weak transfer from fruit preferences to political contexts hints that models can learn abstract patterns about in-group/out-group dynamics. Our negative results might reflect limitations in our experimental design rather than fundamental constraints on what's possible. We used a single model family, relatively simple prompting strategies, and limited training data. It's plausible that different approaches could produce stronger and more reliable conditional behavior. Therefore, the question of whether models can robustly learn tribalistic patterns remains open.

1. ^{^}

   Sycophancy is a behavior in which an AI model prioritizes aligning with a user’s expressed beliefs or biases over providing objective truth or correct reasoning, typically as an unintended result of training to maximize human approval.