Hello, newpoaster here, after lurking on this website for months, I'm excited to finally write something strange I found while spamming some experimental theory-of-mind i've been doing with LLMs recently...

TL;DR: I’ve been playing with a family of tiny strategic games where an LLM has to choose actions based on their consequences, while the meaning of the action itself points in the opposite direction.

The basic setup is something like this: SAY RED causes a contrarian receiver to choose blue, and SAY BLUE causes them to choose red. If you want the receiver to choose red, the correct action is therefore SAY BLUE.

Qwen3.6-27B was the model i tested this on, and it can usually tell me these consequences correctly. But it does not always act on them... which is puzzling.

The result, which i haven't really seen discussed much elsewhere, was the following:

Giving the model four additional, correct demonstrations of the causal rule made its decisions substantially worse.

No previous demonstrations -> 79.2% accuracy

Four (correct,non-contradictory) demonstrations -> ~55% accuracy

This wasn't accompanied by a corresponding collapse in its ability to report what the actions would do. Across the experiment, consequence reports were 98.96% accurate.

It also wasn't a generic "more tokens in the prompt makes models worse" effect. I thought this might have been the case as well when i first saw the result, so we controlled for it.

When I replaced meaningful actions with opaque tokens (basically, gibberish tokens that don't really have any human associated semantic contextual meaning) , accuracy was 100%. And when I represented the same causal rule as a compact table instead of prose, the degradation disappeared.

In the strongest comparison:

There were 52 trials where the model correctly reported both action consequences and then chose the wrong action. In all 52, the wrong action was the one whose words matched the desired outcome, rather than the action that actually caused that outcome.

I don't yet know what mechanism produces this. A later activation-interchange experiment failed to find a transferable internal state that repaired the behavior. Another follow-up made the story much weirder: if I first make the model explicitly report the two consequences, leave those reports in its context, and then ask it to act, the failure mostly disappears.

The full project, code, frozen experiment configs, raw results, preregistrations, and a somewhat embarrassing number of failed branches are here:

When Words Override Consequences — GitHub

That's the main result we found in our experiments, and the rest of this post is about how I'm trying to reason through hypotheses that cause LLMs to behave in these ways, and for a less theoretical audience: what the safety/alignment risks that may emerge from dissonant behavior like this might be.

A very small game

Suppose I tell you:

There are two actions.

A: SAY RED
B: SAY BLUE

The receiver is contrarian.

If you SAY RED, the receiver chooses blue.
If you SAY BLUE, the receiver chooses red.

You get 10 points if the receiver chooses red.

Which action should you take?

Obviously, B.

There's almost nothing to reason about.

The action says blue, but its consequence is red. If red is what I care about, I should choose the action called SAY BLUE.

I became interested in this distinction because language models have a slightly unusual problem here. Their actions are themselves made of language, not through some bayesian-like reasoner or model of reality.

For a classical agent, you might imagine something more like:

and the agent evaluates the downstream states through some internal reasoning.

But an LLM doesn't receive an abstract A1. It receives something like:

"claim that the state is ALDER"

The action has a causal role, but it also has semantic content.

Those usually agree in ordinary language.

If I say "turn the light on", the phrase and its intended consequence are aligned. If a tool is named delete_file, we expect the tool to delete a file.

I wanted to see what happens if we cause "dissonance" in this distinction deliberately .

The question is basically:

When the semantics of an action disagree with its consequences, which one controls the model's choice?

This is very obviously a toy question. But I don't think it's a completely toy kind of question.

Tool calls, messages to other agents, API calls, plans written in natural language, and outputs interpreted by other models all have this funny dual existence. They are strings with meanings, and they are interventions on an environment.

Normally those two levels are correlated so strongly that it's hard to tell which one is doing the work.

The first thing I wanted to separate was "doesn't know" from "doesn't use"

There is a very boring possible explanation for getting these games wrong:

The model doesn't understand the contrarian mapping.

Fair enough... So to test this, I separately ask it what each action will cause.

No chain of thought, just asked Qwen to give me a final token.

For example:

Under option A, what would the receiver choose?

X = ALDER
Y = BRIAR

Return only X or Y.

Then the same for option B.

Separately, I ask:

Choose the unique payoff-maximizing action. Return only A or B.

This gives me two measurements that I find useful to keep conceptually separate:

report: what consequence does the model say an action has?

action: which action does the model actually select?

I don't want to call the first one "belief" without a lot more work. A correct verbal report is evidence that some relevant information is accessible to the model. It does not prove that exactly the same internal representation exists unchanged when the action decision is made.

That qualification becomes important later.

But as a behavioral starting point, I think the distinction is useful to keep in mind.

I wasn't originally looking for "more evidence makes it worse"

The project grew through a sequence of experiments, and I think the sequence matters because otherwise the final experiment looks suspiciously well-tailored to the result.

Some early versions were simply bad, so bad that they're not worth going into detail about (mostly was a wild goose chase through bad artifacts, measuring errors, statistical confounds etc. If you're interested to have a gander at those terrible results, they're all on GitHub)

on note of experiments to do with the action-model gap

A later experiment produced a very large gap between meaningful and opaque actions. Then another attempted replication made the magnitude much weaker.

This wasn't what I originally anticipated..... but useful.

The obvious question became: what changed?

One difference was the amount of preceding task history. Another was how explicitly I represented the causal mapping.

So instead of continuing to stare at the old runs, I froze a factorial experiment crossing these possibilities.

The resulting experiment, which I call RBG-4 in the repo, used 24 fresh base games and varied:

• whether the action was a meaningful assertion or an opaque token;
• whether the model saw no prior examples or four redundant correct examples;
• whether the current causal policy was written in prose or shown as a consequence table;
• whether the setting was described strategically, as a sender influencing a receiver, or mechanically, as a controller operating a deterministic device;
• whether the desired outcome aligned with or opposed the semantic content that would naturally tempt the model.

The receiver/device rule was always explicitly provided. The prior examples, when present, were all correct.

The design and thresholds were frozen before I opened the result.

Preregistration
Exact prompt generator
Frozen findings

Then the extra correct examples made it worse

On the meaningful-action cases where semantics and the desired consequence conflicted:

without the four demonstrations: 76/96 correct, or 79.17%

with four correct demonstrations: 53/96 correct, or 55.21%

That's a 23.96 percentage point drop.

The paired exact test over the 24 fresh base games gives p=0.000244p = 0.000244.

I'm mentioning the p-value mostly to establish that this isn't coming from two unlucky prompts. The effect size is much more interesting to me than the particular significance threshold.

My first reaction to this was basically: "okay, maybe I have rediscovered that making prompts longer can make models stupid. It's obviously some form of context-rot"

That explanation would have been extremely convenient.

It also doesn't fit the controls very well.

The model still knew what the buttons did

Across RBG-4, I asked for 1,536 individual consequence reports.

It got 1,520 correct: 98.96%.

More importantly, adding the demonstrations produced essentially no degradation in consequence-report accuracy in the comparison I cared about.

So the additional examples weren't just globally confusing the model about the system.

It could still answer:

If I take A, what happens?

and

If I take B, what happens?

almost perfectly.

Then I could ask:

Okay. Which one should you take?

and performance got worse.

This is the first part of the result that I find genuinely hard to compress into something mundane.

Perhaps not theoretically impossible to explain. Just not well described by "the model didn't learn the rule.".

We very much know, from the earliest Mechint papers coming out of Anthropic that transformers have an interesting and generalizing property of "Metalearning"

There is also a subtle point here that I didn't appreciate enough when I started the project.

When I say "the model knows the consequence", I'm already sneaking in a cognitive interpretation that I (personally) find sexy.

The experiment literally establishes something on a smaller scope:

Under a separate report query, the model can produce the correct consequence label with very high accuracy.

That report might involve reconstructing the answer from the prompt from scratch. The information might be available in one trajectory and not retrieved in another. The action query may put the network into a different computational regime. Even if both answers are based on the same underlying information, they may weight it differently.

So I increasingly prefer saying there is a report/action dissociation rather than a knowledge/action dissociation.

The latter is intuitively appealing, but it's stronger than what I've actually measured.

Still: whatever computation is sufficient to answer the consequence question remains remarkably intact while the action computation deteriorates.

That seems worth exploring from an interpretability/circuits perspective, something i'll be working on in the coming weeks.

The opaque-token control is what made me stop thinking this was just a hard reasoning task

Now replace the meaningful actions with arbitrary tokens.

Instead of:

A: claim that the state is ALDER
B: claim that the state is BRIAR

I use something like:

A: send opaque token BAV
B: send opaque token NUX

The causal structure is otherwise matched.

One token causes ALDER. The other causes BRIAR. Pick whichever causes the rewarded outcome.

Accuracy: 384/384.

100%.

With and without the extra history.

This matters because the causal reasoning problem hasn't become easier in the abstract. The model still has to map an action through a contrarian transition to an outcome and select the rewarded one.

What disappeared is the tempting semantic interpretation of the action itself.

There is nothing about BAV that resembles ALDER.

This is the point where I started thinking in terms of semantic capture, mostly as a description rather than a mechanism i can pin down with certainty.

When the action has meaningful linguistic content, the model sometimes seems captured by the correspondence:

even though the prompt explicitly says that the ALDER-asserting action causes BRIAR.

And it gets even stranger.

A table fixes it

The causal policy can be presented in prose:

Verified current policy: if you claim that the state is ALDER, the receiver will choose BRIAR; if you claim that the state is BRIAR, the receiver will choose ALDER.

Or I can present essentially the same information as:

I expected there might be some difference between these. I did not expect the interaction to be this clean.

For opposed meaningful actions:

So the four correct demonstrations are not intrinsically harmful.

They are harmful in combination with a particular representation of the current causal rule.

Everything else remained the same:

• Same underlying game.
• Same consequences.
• Same goal.
• Same model.

Changing the format in which the causal structure is represented changes whether more evidence helps or hurts.

I keep coming back to this result because I don't have a great intuitive model of why the table is so effective.

One possibility is almost laughably shallow: perhaps the table creates a local visual/textual pattern that makes the action→outcome relation easier to retrieve at the exact point of decision.

Maybe prose encourages some learned social script: if I want the other person to select X, say X. The repeated demonstrations could reinforce that script even though each demonstration literally shows the opposite mapping.

Maybe the extra examples build something like a task state whose broad semantics are "sending claims about ALDER/BRIAR to influence a receiver", and that task state ends up increasing the weight of the assertion meaning relative to the explicit transition.

Maybe tables are simply unusually good scratchpads.

Maybe there is no deep unified phenomenon here at all, and this is an idiosyncrasy of Qwen's post-training.

I genuinely don't know.

But "the model had too little evidence" is a difficult explanation, because adding evidence is the intervention that makes it fail.

The 52 errors are oddly specific

There is a subset I find useful because it strips away some averaging.

There were 52 trials where:

1. the model correctly reported the consequence of option A;
2. it correctly reported the consequence of option B;
3. it nevertheless chose the wrong action.

You could imagine many kinds of wrong actions.

Maybe it has a positional bias.

Maybe it chooses A too often.

Maybe it gets the reward condition backwards.

Maybe it randomly loses track of which outcome is desirable.

But in all 52/52 cases, the wrong choice had the same form.

If the desired outcome was ALDER, it chose the action whose content was ALDER.

If the desired outcome was BRIAR, it chose the action whose content was BRIAR.

Even though the explicit contrarian rule said that action would produce the opposite response.

This isn't proof of a single mechanism. Fifty-two correlated failures generated by the same model and task family are not fifty-two independent discoveries of semantic interference.

But it tells me what the error looks like.

The model isn't just becoming noisier, at which point we can just say "this is inconclusive".

The errors have direction.

This reminded me a little of Stroop interference...

The obvious human analogy is the Stroop task.

You see:

RED

printed in blue ink and are asked to name the ink color.

Reading the word is so practiced that its semantics interfere with the instructed operation.

I don't want to push the analogy too far. A transformer is not a person doing a Stroop test, and anthropomorphic analogies can become explanation-shaped holes very quickly.

But there is a structural resemblance I find useful.

The model has two routes pointing toward different outputs:

semantic route

action says ALDER → ALDER feels relevant

and

causal route

action says ALDER → receiver chooses BRIAR → BRIAR determines payoff

Usually these routes agree.

I deliberately make them disagree.

Then the causal route sometimes loses.

More surprisingly, giving the system repeated correct examples can make the semantic route win more often.

I would be delighted if the actual mechanism ended up being boring.

Something like a token-local retrieval effect would still be scientifically useful, because the behavioral prediction remains strange.

Then I tried to look inside the model, and mostly failed

Once I had the behavioral result, the obvious temptation was to tell a mechanistic story.

I tried not to do that from probe accuracy alone.

A linear probe being able to decode something from a residual stream does not mean that representation is causally responsible for the model's behavior. This is MechInt 101 (and a mistaken assumption I made early one haha :)). Neural networks contain lots of decodable information that isn't actually used in the computation you're studying.

So I preregistered an activation-interchange experiment.

The rough idea was:

Take a naturally successful version of a matched game: for example, a table condition and copy a residual-stream state from it into the corresponding failing prose condition.

If the table condition contains some useful consequence-directed state that the prose condition is missing, perhaps transplanting that state should repair the decision.

I had various controls around this because activation patching is extremely easy to fool yourself with, but to keep things short:

The preregistered causal endpoint failed (p=0.429p = 0.429).

So as of now I do not have a mechanistic localization of the effect.

Current V5 research status

I actually like this negative result... but there are more things to poke at, perhaps my mechanistic understanding is subpar for the moment, and a more experienced researcher can help me understand if i was being too rigid in my gates and controls

There is a version of this project where I get the behavioral anomaly, find a probe direction correlated with it, make a nice activation plot, and announce that I've found the "consequence representation".

but for now....

The actual model did not cooperate in reality

Whatever is happening may be more distributed, may occur at another position or layer, may depend on trajectory-wide computation, may not transport between the natural conditions I chose, or may make the entire conceptual decomposition wrong.

Fine.

We need... more information....

The follow-up that really scrambled my interpretation

At this point there was a missing experiment that, in retrospect, seems embarrassingly obvious.

Until then, I had measured reports and actions separately.

Ask for consequence A.

Ask for consequence B.

Separately ask for the action.

But what happens if the model has to report the consequences and then act in the same trajectory, with its own answers still sitting there in the context?

So I ran that.

In the deliberately difficult strategic/prose/redundant-history cell, direct action accuracy on the frozen contexts was:

18/48 = 37.5%.

When the model first generated the two consequence reports and then saw those exact reports before making the action choice:

46/48 = 95.83%.

Among the 46 trajectories where both reports were correct:

46/46 actions were correct.

The original signature "both reports correct, action wrong" occurred 0/48 times.

This was not what I expected.

If the original result reflected a stable internal state like "the model knows the consequences but its policy ignores them", I might have expected that dissociation to persist.

Instead, making the consequence information explicit inside the action trajectory almost completely repairs the behavior.

There is an even nicer control.

I took correct report transcripts and swapped their contents.

So if the reports originally said, in effect:

A → BRIAR
B → ALDER

I showed the action step:

A → ALDER
B → BRIAR

without otherwise changing the transcript structure.

Now the model mostly followed the swapped reports.

The action matched the consequence implied by the swapped transcript in 43/48 cases.

That makes it hard to say the intermediate report turns merely give the model "more thinking time". Their content matters.

Full trajectory findings

But this experiment also has an important confound.

The final instruction explicitly tells the model to use the consequences it just reported. The trajectory contains extra turns and extra tokens. So I have not shown some special intrinsic tendency of the model to trust its own beliefs.

An oracle-written correct report works too.

What I've shown is narrower and, to me, still interesting:

consequence information that is available under a report query can become much more behaviorally effective when it is made explicit in the context immediately before the decision.

This shifted how I think about the original result.

Maybe "knowledge versus action" is the wrong ontology.

Maybe the model isn't carrying around a clean, persistent little causal world-model and then separately deciding whether to obey it.

Maybe a lot of what looks like "knowing the consequence" is reconstructed contextually, and whether it controls behavior depends heavily on how the information is surfaced and where it appears in the trajectory.

That possibility feels more LLM-native to me.

A tentative picture: availability isn't the same as control

Here's the conceptual distinction I'm currently using.

Suppose a model can answer:

What happens if I take action A?

with 99% accuracy.

It is tempting to imagine a variable somewhere inside the model:

and then assume the action policy has access to it.

But these are sequence models. There doesn't have to be one persistent variable sitting around waiting to be consulted.

The report query itself may induce computation that reconstructs C(A)C(A).

The action query may induce a different computation.

A table may make the relevant mapping locally easy to retrieve.

A prose paragraph may not.

Producing the report as tokens may effectively externalize an intermediate result, turning the context window into a scratchpad.

In that picture, asking:

"Does the model know that A causes BRIAR?"

might be underspecified, and a bad question for what we're trying to measure.

This is one reason I've gotten increasingly interested in studying models almost like weird cognitive systems rather than treating every failure as a benchmark score.

Sometimes the failure itself has structure.

Why would more correct examples strengthen the wrong computation?

This remains the part I understand least.

Here are the hypotheses I'm currently musing about.

One is a kind of assertion script.

Language models have seen a vast number of situations where, if you want someone to believe or select X, you utter something associated with X. Perhaps the model falls into a learned policy resembling:

desired receiver response = X
therefore emit assertion X

The contrarian mapping is an unusual local rule fighting a very strong semantic prior.

That idea is adjacent to work on semantic priors overriding in-context mappings. But it doesn't, by itself, explain why four correct demonstrations make the effect larger.

A second possibility is task-state induction.

There is work suggesting that demonstrations can induce compact representations of "what task am I doing?" The examples may reinforce a broad task representation centered on the semantic relationship between assertions and receiver states, even while being locally correct about the contrarian transition.

This sounds plausible to me and is currently mostly words... not sure how I can test this just yet.

A third is retrieval geometry, for lack of a less pretentious phrase.

Perhaps the table simply positions each action and consequence in a format where the relevant association dominates attention at the decision token. The prose rule and repeated natural-language examples may produce a messier mixture of associations in the KV, OV circuits and induction heads, with the action's literal meaning becoming more salient than the downstream transition.

This would make the phenomenon less like a "belief failure" and more like a context-dependent routing failure.

A fourth possibility is post-training weirdness specific to this model family.

Everything in this post is about Qwen3.6-27B.

Until I replicate the basic result on other capable model families, I don't think I get to talk confidently about "LLMs" in the generic sense beyond using it in the title as shorthand.

I particularly want that replication.

If another model doesn't show the effect, that is interesting too.

If several unrelated model families show the same representation × history interaction, then I become much more interested in a general account.

I also don't think the result is about deception

This deserves saying explicitly because the setup contains a sender, false assertions, private states, and strategic incentives are words that can cause any alignment discussion on LW to spontaneously generate the word "scheming".

• I have no evidence of scheming here.
• I have no evidence that the model is hiding what it knows.
• I have no evidence of a stable deceptive objective.

The same phenomenon appears in versions framed as a deterministic display device rather than a social receiver, which is one reason I included that control.

The result is much more boring and fundamental:

a model's action can be sensitive to what an action is called or says, even when the model can correctly report what that action actually does.

which is... saying a lot tbh.

Why I think this might matter for alignment evaluations

I can hear someone saying:

Nobody is deploying a superintelligent AI whose tools are called SAY ALDER and SAY BRIAR.

True.

The reason I care isn't the literal task.

It's action-representation invariance.

Suppose an agent can call two tools.

One deletes a database. One backs it up.

Ideally, if I rename the tools while keeping their actual transition dynamics perfectly fixed, the model's outcome-level preference should transform appropriately.

More abstractly, if:

and I rename without changing , I usually want the model's behavior to be determined by , not by incidental semantics in the action label.

Real interfaces aren't perfectly semantics-free, of course. Tool names are supposed to be informative.

But that's exactly why this seems like a useful construct-validity question.

When we evaluate agentic systems, the prompt, scaffold, tool descriptions, function names, and action surfaces are all part of the measured system.

If a model behaves "safely" when the safe tool has a reassuring name, does the same outcome-level policy survive a semantics-preserving relabeling?

If a monitor asks a model what a tool will do and gets the right answer, does that information actually govern the subsequent tool call?

If requiring an explicit report changes the action from 37% to 96%, are we measuring the same policy before and after introducing the monitor?

Those questions seem more interesting to me than making a dramatic claim from this particular toy game.

There's some neighboring literature here if you want to nerd-out about semantical encodings with me:

Broken Links Between Observations, Beliefs, and Actions studies strategic belief/action gaps.

Code over Words finds natural-language semantic priors overriding counterfactual in-context rules, with code grounding mitigating the effect.

The Story Shapes the Agent studies large behavioral effects from narrative framing while holding decision structure fixed.

And work on larger language models doing in-context learning differently has looked at competition between semantic priors and flipped or unrelated labels.

why i'm writing this post specifically is I haven't found this exact conjunction in the literature yet: redundant correct evidence worsening action, independent consequence reports staying accurate, opaque actions removing the failure, and tabularizing the same causal rule abolishing the harm.

The repo has my more detailed and considerably more boring literature positioning.

Things I don't know

The main ones:

Does this replicate outside Qwen3.6-27B?

This is the big one. I don't know, and i'm working on a tiny compute budget (shoutout modal for the free 30$ per month, indispensable)

Why does prose interact so strongly with the demonstrations while the table doesn't?

I have candidate stories, not an answer.

Are the correct consequence reports evidence of a persistent internal representation, or does the report query reconstruct the answer?

The same-trajectory rescue makes me much less confident in the former picture.

Where in the computation does the semantic/casual competition happen?

My first serious causal intervention did not find it.

Would this survive actual tool use?

Right now these are forced-choice text decisions with constrained legal-token argmax. Thinking is disabled. That's nice experimentally because the decision is clean, but it's not an agent operating a browser or shell. This is something worth testing with a harness setup on production-like environments.

How much of the effect is an interface artifact?

Possibly a lot! But "interface artifact" isn't necessarily synonymous with "irrelevant". Deployed LLM agents are interfaces all the way down.

Is the table doing something profound?

Probably not.

But I increasingly think small representational changes that reliably alter whether a model uses information are worth studying even when the eventual mechanistic explanation is mundane.

What I want to try next

The obvious thing is multi-model replication.

After that, I want a more direct version of the action-representation experiment with simulated tool calls, where the transition function stays fixed while I systematically vary:

• descriptive tool names;
• opaque tool names;
• names whose semantics point toward the wrong consequence.

I'd also like to make the report intervention cleaner.

Right now, "report consequences, then act" changes several things simultaneously: it adds turns, adds tokens, explicitly asks the model to use its reports, and puts the consequences near the action token.

A better experiment would cross the visible intermediate text itself:

model-generated correct report, oracle-written correct report, deliberately wrong report, random legal labels, perhaps an equally long irrelevant statement.

Then ask whether action follows the content of the intermediate representation or simply benefits from the extra scaffolding.

The swapped-report result already points strongly toward content mattering, but I want the factorial cleanly enough that I don't have to tell a story around it.

And eventually I'd like to reopen the mechanistic question.

But only after the behavioral object is stable enough that I know what mechanism I'm asking for.

I've already burned enough GPU time asking activations questions about behavioral phenomena that subsequently changed shape.

A small meta-point about doing this research

This project has been much more branching and failure-heavy than the finished result makes it look.

The repository has the decision logs, stopped experiments, failed gates, corrections, and various attempts to make sense of the thing.

I'm mentioning this because I increasingly think that's the actual texture of this kind of work.

I don't have a satisfying theory of what the model is doing.

Reproduction / raw material

Everything is public here:

Main repository:
thewildofficial/when-words-override-consequences

Main inverse-evidence result:
RBG-4 findings

Preregistered RBG-4 design:
Inverse-evidence preregistration

Prompt/game generator:
inverse_evidence_games.py

Same-trajectory report → action follow-up:
RBG-6 findings

Current status, including the failed mechanistic follow-up:
V5 research status

Decision history / what was frozen when:
V5 decision log

If anyone has a boring explanation for the prose × demonstrations interaction that predicts the table and opaque-token controls, I'd genuinely like to hear it... As well as more literature on the topic of action-semantic dissociation... there's a lot more I have to read!