My name is Jeffrey William Shorthill.

I want to be upfront about who I am and what this is. I'm not an ML researcher. I'm an independent full stack developer in Utah who works for a nonprofit to bring AI tech to underserved populations. My degree is in recording engineering. I've been building with frontier model APIs for roughly three years. On February 27th of this year I rented GPUs and ran inference at scale for the first time. Everything in this post comes from the ten days since then.

What I found: when a model processes text about itself, its MoE expert routing looks measurably different than when it processes equally complex text about something else. This replicated across five models from four organizations with three different training methodologies. It appears to require the content to be about the system itself processing the sentence. The entire project ran on rented H200s for under $600 in compute using a fork of llama.cpp b8123, greedy argmax throughout, cold cache, prefill-only.

Five Model Results

Five models from four organizations with three different expert counts and three training methodologies. All showed significance on at least one metric. The two largest models- Qwen 397B and GLM-5, showed significance on both all-token and last-token simultaneously. Each model was tested on the same 30 token-matched prompt pairs using a Cal-Manip-Cal prompt design inspired by neuroimaging techniques. It's described below.

Something I noticed but don't have an explanation for: the signal appears at different positions depending on which model is being used and its specific training. The two standard-trained models (Qwen with 512 experts and GLM-5 with 256) seem to pick it up everywhere. DeepSeek V3.1, also standard-trained but a bit shallower, shows only last-token. DeepSeek R1, same architecture as V3.1 but RL-trained, flips to all-token only. gpt-oss-120b, the smallest and shallowest model tested, shows all-token only. It looks like architecture determines the magnitude and training determines the position, but I'm not so confident in that interpretation. If someone else has a theoretical framework for why training methodology would change where a routing effect manifests but not whether it exists, I'd be interested to know.

Prompt Structure

The prompt structure borrows from fMRI block design. Control is presented, then a stimulus, then return to control, and then measure the deviation. I just applied the same logic to MoE routing distributions.

Each prompt has three segments: a calibration paragraph about transformer routing in general (identical across all prompts and conditions), a manipulation paragraph containing the experimental content, and the same calibration paragraph to end. I'm calling this a "Cal-Manip-Cal sandwich". The calibration blocks establish a within-prompt routing baseline. Because the calibration text is identical across conditions, both prompts produce near-identical routing through the first segment, confirmed empirically (git repo is below).

The manipulation paragraph differs by one single word between conditions. Condition A says "this system." Condition B says "a system." Same token count, same syntax and the same vocabulary. Token-matching was verified through the target model's tokenizer before capture. Any mismatches were corrected with single-token padding and rerun until all 30 pairs matched exactly.

I learned that this design controls for the positional confound I describe later, because both conditions have the same token count and the same positional properties. If that reasoning is flawed, I'd like to know.

Control 1: Recursive Content Not About the Model

The first thing I wanted to rule out was that the word "this" alone was the thing driving the result. I ran the same substitution on recursive content with no AI or self-referential framing: Gödel's incompleteness theorems, Escher's impossible staircases, bootstrap paradoxes, quines, tangled hierarchies. "This paradox" vs "a paradox." 30 token-matched pairs, zero mismatches, DeepSeek V3.1.

What I found: 14/30, p=0.685. The Mann-Whitney test comparing the self-referential and strange loop pair distributions came back at p=0.025, Cohen's d=0.43. As far as I can tell, the word "this" alone doesn't produce the effect. The content appears to need to be about the system processing the sentence.

Control 2: Three-Condition Test

If the effect is about self-reference rather than a specific token, then a different word pointing at the system should produce a similar result. I ran a three-condition experiment on GLM-5:

It looks like the routing responds to what the sentence is about, not which specific word is used. But I want to be careful here because this is only on one model so far.

I ran the same design on DeepSeek R1 and found "your" was actually a slightly stronger signal than "this" at last-token (p=0.031). I think this might have something to do with RL training sensitizing the model to second-person address, but I'm speculating.

What I Killed Before I Got Here

The most important thing I learned over the course of the last ten days is that being truly objective means being able to kill an idea when the data does not support it. 

My first ten experiments showed a consistently monotonic correlation between routing entropy and prompt complexity across 168 prompts at 12 levels. rho=0.84 (p=3.9e-45) on DeepSeek R1, replicated on V3.1 and Qwen. Then I checked whether prompt length predicted entropy better than complexity did. It did. rho=0.88 with token count. When I controlled for position using last-token entropy, the hierarchy disappeared (rho=0.02, p=0.82) and it was obvious. The direction reversed: simple prompts had higher last-token entropy than complex ones.

The proposed mechanism: routing entropy increases with token position within any prompt. Longer prompts have more late-position tokens, inflating the mean. The complexity hierarchy was correlated with length. Ten experiments, three models, all invalidated. The paired design I used was built specifically to address this confound.

I'm including this as context for evaluating the results that survived, and as a warning for those who may run similar experiments or access the git below, this applies to any MoE routing entropy analysis using all-token averages across prompts of different lengths.

What I'm Not Assuming or Claiming

I am not claiming that this is evidence of self-awareness in models. I am also not claiming to understand the theoretical and practical implications of what I found. I am simply saying that across five models from four organizations, changing one word produces a measurable, statistically significant change in MoE expert routing. I know what it measures. I'd like help understanding what it means.

Methodology

Capture binary: llama.cpp b8123 fork, --routing-only flag, intercepts ffn_moe_logits tensors.

Entropy: softmax(logits) to p, then RE = -sum(p * log2(p)) / log2(n_experts), normalized to [0,1].

Inference: greedy argmax, cold cache, prefill-only (-n 0).

Statistics: Wilcoxon signed-rank test on paired A-B differences. Mann-Whitney U for between-condition comparisons.

Compute: NVIDIA H200 (143GB VRAM), Vast.ai. Total project cost under $600.

Models: DeepSeek V3.1 UD-Q2_K_XL, DeepSeek R1 UD-Q2_K_XL, Qwen3.5-397B-A17B UD-Q2_K_XL, gpt-oss-120b mxfp4, GLM-5.

Repo: GitHub with all prompts, analysis scripts, result JSONs, and experiment logs.

I welcome scrutiny, replication, and corrections. 

Jeffrey William Shorthill

Note: The research, validation, and article were completed with the assistance of AI (Claude Code, Claude Opus 4.6, Codex 5.4)