Epistemic Status:

• This is a small model mechanistic interpretability study, I did on multiple domains training Transformer models ranging from 0.27 M parameters to 13 M parameters on an Apple M1 pro (16-core GPU)
• As Mech-Interp is an empirical science, I pre-registered claims to predictions_*.md file in a public repo and evolved my theory based on failures and success outcomes.
• I used Claude/Codex heavily to implement data pipelines, training scripts, probe code, activation patching scripts. My intention behind this post is to share my experiment design and I believe the associated failures and outcomes are worth your inspection. It should not be taken as strong evidence about large frontier models.

TL;DR

• I started with a sharp hypothesis (called it N-criterion) and pre-registered in a public repo. A feature F is encoded by Transformer model in its residual stream iff the feature is useful for next-token prediction. This hypothesis got falsified in tasks across two domains of Maze Navigation and HTTP logs.
• I observed that small transformers of few million parameters encoded features that are not necessarily very important for next token prediction. The probe kept recovering them. I labeled it “architectural carry through” since those features are cheaper to carry and encode.
• Using a mech-interp point of view, I am learning that if you probed a model and recovered a feature, it simply means recoverability ONLY. It does NOT always mean usefulness to the transformer’s task or any other causality.

N-Criterion: The theory I expected to work:

The obvious next question is “Does this generalize to all domains?”. I started with a deliberately sharp hypothesis N-criterion (Next- token necessity) about when next-token transformers form internal representation.

N-Criterion hypothesis: "A feature is represented in the residual stream iff the feature is needed for next-token prediction.”

• Residual Stream : It is the hidden state vector that flows through the transformer. It is updated additively by each component. Every attention head and MLP reads from it and writes to it.
• "Represented" : A feature is considered “represented” if a linear probe or small MLP probe can recover the feature from the residual stream.

The reason for coming up with this hypothesis with the “iff” statement is that it’s bidirectional. Only useful things are represented and irrelevant things are not represented. I tested this across the domains of 1) Othello game, 2) Cities 3) Music 4) Flight phases, 4) maze navigation, 5) HTTP logs. The two real tests are “Maze Navigation” and “HTTP Logs” where I committed the prediction files before running the corresponding probes. Hence, the rest of the writeup will be focused on the two real tests of Maze Navigation and HTTP logs.

The Setup - How I Tested It:

Pre-Registration: I make an ex-ante quantitative prediction and lock in a github repository. Here, the commit serves as an audit trail avoiding any post-hoc story fitting.

Data Preparation and Corpus shuffles: In order to ensure that the model is genuinely learning the signal and also to rule out that we are not creating an artifact of a high capacity linear probe memorizing the data, I do corpus shuffles.

• Real corpus is the actual structured data.
• Within-shuffled corpus : In this dataset, I do a random permutation of the tokens within each individual sequence. This destroys the sequential nature of data but keep the exact set of tokens within the sequence. I expect models trained on this data will show degradation compared to real corpus.
• Global-shuffled corpus: In this dataset, I do a global permutation where tokens are assigned uniformly across entire corpus across sequences causing further degradation. If the model is able to represent any feature and the probe is able to recover, its probe is able to memorize due to its higher capacity than the transformer model learning a genuine representation.

Model training and validation: Since, these are smaller datasets, I trained a small nanoGPT style transformer^[3] ranging from 0.27 M parameters to 13 M parameters on next-token prediction objective across the 3 different corpora mentioned earlier using the same hyper parameters.

Multi-Seed Probing & Per-Layer Ablation: At every layer, we train a linear probe and 1 hidden layer MLP, also compare trained probes to untrained probes. The headline number for every domain is the probe gap.

The main number i report is the Probe gap : probe accuracy on the trained transformer minus probe accuracy on an untrained transformer with the same architecture. A positive gap means training made the feature more recoverable from the residual stream.

If a representation or a feature is truly learned, then it won’t survive the corpora shuffle. We expect a monotonic degradation.  (real > within-shuffled > global-shuffled). My experiments later showed this not true always. Some features are propagated and encoded because they are positional, copied, or otherwise cheap for the architecture to preserve.

Depending on the probe outcomes from the previous steps, optionally, we confirm causal usage of a feature via activation patching, compare cross-condition gradients to rule out memorization artifacts.

The Maze Navigation task: 

The Maze task contained training examples where each training example is the shortest path from start to goal, tokenized as cells on an 8 x 8 maze.

[BOS, start_cell, step_1, step_2, ..., goal_cell, EOS]

My Prediction: I committed the maze prediction on 2026-05-27  commit aa025b1. My prediction was that the current cell and the distance to goal should matter. The starting cell should not matter once the path has moved on. During the course of the journey towards the goal, the next step depends on where you are and the goal. It should not depend on where you started. I made a quantitative prediction that the starting cell would not be represented at late positions. The MLP gap between trained vs untrained MLP should be at most +0.10. 

Failure: The maze test failed. At layer 5, the trained MLP recovered “the starting cell” with accuracy  0.2024 +/- 0.0166 ; the untrained model was 0.0504 +/- 0.0082. Gap: +0.152. (Top right in layer by layer ablation picture).

Update to N-criterion: This forced me to update the N-criterion hypothesis. A model can preserve the information that is probably not needed because nothing in the objective directly punishes the model for carrying the harmless extra bit of information.

Architectural carry-through (no cost carry through)

In the Maze task, the probe recovering "the starting cell" suggested that a transformer can attend back to positionally distinct first-path-token and route the information forward. The residual stream is a shared workspace where this information can hang around if it’s not incurring any cost to the architecture. I am labelling it “Architectural Carry Through”

It’s not a profound idea. It might be a case that attention plus residual stream make copying easy. Although Next-token training incentivizes keeping information that helps prediction of next token, it also does not impose any penalty for retaining harmless extra information. So some extra features remain decodable not because the model strongly needs them but its harmless and is inexpensive to carry.

Based on this result, I revised the N-criterion hypothesis to

• Features not required for prediction of the next token may still be represented if they are already present in the positionally unique slot and/or are easy for attention to carry forward.

I decided to do a HTTP logs test as the next test to test this prediction.

The HTTP dataset test

I used the NASA-HTTP dataset from the Internet Traffic Archive,^[4] specifically the logs from the July and August 1995 archives. I chose this because it provided an applied domain with clean, publicly available data and a periodic structure that would make it easy to test the prediction

[method, path_category, status_bucket, size_bin]
A session looked like:
[BOS, m_1, p_1, s_1, sz_1, m_2, p_2, s_2, sz_2, ...]

I pre-registered another feature probes, 1) Feature A: the size_bin and 2) Feature B: cumulative count of large responses (Commit 3b25ed3)

• Feature A:  the size_bin of the first request in the session. This is not obviously useful later, but it is present at a positionally distinct input slot. Carry-through predicts it should be recoverable.
  • Result: Feature A confirmed. Best MLP gap: `+0.168` at layer 3.
• Feature B: the cumulative count of large responses so far, binned as {0, 1, 2+}.  This is not sitting in one slot. It requires aggregating across previous request sizes. I predicted it would be null.
  • Result: Feature B falsified. Raw MLP gap: `+0.291`. I expected the transformer to not compute the aggregation features and retain them because it's not inexpensive but still it did.
  • This raw result had a position-confound. So, I ran post hoc controls. At fixed request index of k=5, the Feature B still had a +0.220 trained vs not-trained gap. So that was falsification was inflated by position but not purely positional.

Criticism on Architectural Carry through: At this point, I would expect criticism on “architecture carry through”. A valid one would be  that I have merely demonstrated the simple copying from the first slot i.e the initial input. The maze starting cell is the first path token. HTTP Feature A :size_bin is the first request’s size bin. It’s possible that “architectural carry-through” just means the model remembers the first token and associated meta data. So, I did more follow ups on the HTTP domain.

Further Follow up in HTTP domain: 

• In the HTTP experiment, each request in a server session was tokenized as 4 consecutive fields: [method, path_category, status_bucket, size_bin].
• In HTTP notation, is method, p_k is path category, s_k is status bucket, and sz_k is size bin for request k.
• For a request K, we can write this as: [m_k, p_k, s_k, sz_k]. Here, p_k is the path category of request k: that is whether the requested URL is an image or HTML page or CGI script or directory path or something else.

The “architecture carry through” is true for first request’s size_bin. It stayed recoverable and so is open to a fair criticism that attention mechanism is simply copying the information from first request i.e a fixed early slot. The mechanism would be attending to itself and so the copying happens.

In my follow up tests,

• I tested if current request path recovery is possible, i.e can residual at the time of sz_k recover p_k using a trained probe? [commit: 88155b4]
• Later i tested if previous request path recovery is possible i.e can the residual at sz_k recover p_{k-1}, the path category of the previous request? [commit: 0d115c2]
• The results were positive and my prediction succeeded. The MLP probe gap was +0.809 and +0.674  for  recovering p_k and  p_{k-1} from the residual at sz_k.

Then I tested a 3rd harder case trying to recover p_j [commit:: 8d671ca], where j is the most recent earlier request whose response was large (size_bin_j >= 5). Here, the source request is not at a fixed position. The feature was still recoverable, with MLP gap +0.621, and +0.514.

What I Now Believe:

Committing the predictions before experiments helped by making failures legible. Without the locked prediction commits, I could have swayed into a smoother story after the fact. The git commits made that harder. I intend to continue that.

The residual stream is NOT a minimalistic database that keeps just what the model needs for next token prediction. It’s probably a shared cluttered workspace with recoverable contents that directly help prediction. Others are there because, it was cheap for the attention mechanism to keep them.

Recoverability and Usefulness of a feature for next token prediction are different things.

Probe recoverability is not equivalent to usefulness or causality.

Where i want push back:

Being a starter in this area, I especially want criticism on the following points.

Is this "architectural carry-through" too trivial? Maybe it is just the residual stream being a copy bus. My current view is that this is partly true, but still worth naming because it breaks a tempting interpretation of probe results: recoverable does not imply selected-for-use.

Github: https://github.com/Raghavan1988/LatentWorldsGPT

Detailed report: link

1. ^{^}

   https://arxiv.org/abs/2210.13382 "Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task" by Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg

2. ^{^}

   https://arxiv.org/abs/2309.00941 "Emergent Linear Representations in World Models of Self-Supervised Sequence Models" by Neel Nanda, Andrew Lee, Martin Wattenberg

3. ^{^}

   https://github.com/karpathy/nanogpt

4. ^{^}

   https://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html