This post is on the results of a mechanistic interpretability project aimed at understanding the internals of Maia 3: a transformer based chess bot trained to imitate human play at a chosen skill level, rather than to play optimally. My aim is to locate and causally describe one chess tactic in the Maia 3 engine. In this part 1, I describe strong correlational evidence that the knight-fork policy logit snaps into place after block 5’s attention layer. (Controls rule out that this is merely where the best move snaps into place at the start of part 2.) In part 2, I will further demonstrate causality and compositionality of the feature, and in part 3 describe unexpected results that shake up the pure "knight fork attention head” thesis into something stranger and more interesting.
Introduction
Chess is relevant to many facets of human cognition, from chunking to sudden insight, and the hope is that general interpretability work on chess models has relevance to cognitive neuroscientific research in the future.
Along with this, the decomposable structure of the transformer architecture (see: "A Mathematical Framework for Transformer Circuits”) adds to the intrigue of the interpretability work. Determining where in the network a concrete skill exists tells one about how its compressed representations form. A knight fork (a single knight move that attacks two valuable pieces at once) is a priori a strong target skill to analyze because it is discrete and unambiguous and important. I wanted to make headway into the question: "at what depth does the network's internals first make a knight fork decodable?"
Brief Notes on Mechanistic Interpretability
As the field is relatively new, it is worth spending a few sentences to give extra context to unfamiliar readers. The aim of mechanistic interpretability is to understand AI better by reverse-engineering models into human-describable parts. The ultimate hope is that this will help safety and alignment, for example by allowing us to recognize deceptive LLMs.
Terminology I will keep constant throughout:
- block 0–7: the eight transformer blocks; each contains an attention layer and an MLP layer
- residual stream: a vector; the transformer's shared communication layer to which attention and MLP layers additively read and write into, allowing for meaningful decomposition by specific layers' additions.
- readout point: the content of the residual stream at some depth
- logit lens: an interpretability technique that takes the residual stream at some layer and pushes it through the final unembedding layer, as though it were the finished product, and then reads its distribution over outputs. Here the output is the move policy distribution.
- decodable: when the move scores highly (large logits) after the logit lens
- policy: the network's output probability distribution over all 4352 moves. Maia plays directly from this with no search
About the model
MAIA 3 is a bidirectional encoder over all chess board squares: one board goes in as 64 square tokens with pieces represented by a one hot vector, and a single forward pass returns a full policy over all 4352 moves plus a win/draw/loss evaluation. There is nothing autoregressive and no sequence of tokens as input, so we can easily logit lens at each layer. Additionally, its attention heads use two schemes: a "semantic" standard Q,K self-attention, and a "geometric" GAB scheme to embed complex positional information, given that chess pieces all move in different ways which complicates simple attention patterns.
My primary interest in studying this model is its skill knob which turns a classic cognitive-science question (de Groot; Chase & Simon: experts don't search more, they re-represent) into runnable experiments, a quintessential attraction of mechanistic interpretability work! I predict that: 1) genuine tactical representations snap layers earlier in the network as conditioned skill rises, 2) features become sparser. I will be researching this immediately.
Interactive App I also vibe coded a satisfying interactive app that allows you to play a game against the engine at any mimicked human skill level and view each split attention head per square, watch the residual stream evolve, logit lens each layer, and more.
To keep the investigation clean, I decided to study what I believe is the least confounded type of fork:
A knight fork of only a king and queen (including bishops, for instance, will inevitably lead to internal representations of defended pieces and knight-bishop exchange values):
Additionally, I required that:
the move be legal and such that the knight is not immediately capturing a piece
the knight cannot be immediately captured
the knight attacks the enemy king and queen simultaneously from its landing square
I mined thousands of public Lichess.org games.
2) Example fork and position used throughout post
Position FEN: "6k1/1pq2ppp/p7/3p4/1P1Nn3/P2QPP2/1B4Pb/7K b - - 1 26".
In case it is not clear, Nf2# attacks both the enemy queen and king, 'forking' the two pieces.
(Nf2 is also checkmate which is a confound that led me to an important finding that is addressed in part 2)
3) Measuring logits across network depth
I hooked the MaiaEngine class to cache the residual stream at every layer. I read it out at the 18 points that correspond to the residual stream: embed_in, postattn_0n (after attention n), block_0n (after MLP n), encoder_out.
A track_fork_policy_logits function logit lenses the stream at each point to yield a tensor of (4352) logit vectors which map to every source and destination square in a move and fork_logit_curve pulls the forking move's logit at each depth.
For the fork, the move seemingly starts with relatively low logits until it snaps upward at late attention readout points.
4) Averaging across 500 forks
We have N=1. To recover the typical trajectory I run 500 forks through the same logit lens, stack their per-depth curves into a 500x18 tensor, and summarize with the mean and standard deviation across positions. The mean curve (±1 std) shows the depth at which the forking move policy typically climbs. The layers where the curve rises fastest are quite plausibly the ones doing the internal computation of assembling the tactic.
5) Determining when exactly the fork becomes decodable
The mean smooths over the exact layer at which each fork transitions from unrepresented to represented. So I record the first point at which its fork logit crosses +2.5, admittedly somewhat arbitrarily.
The sharp peak at postattn_05 (block 5's attention layer) is exciting to see. The fact that it sits at an attention readout point makes sense computationally. Attention layers are the only place information moves between squares; MLPs compute information square by square. A fork is clearly relational since it depends on the position of two other forked pieces, so the snap should appear exactly where some attention information routing happens. It appears as though for most strong moves, block 7 amplifies the logits further as seen above.
The very early outliers seem worth investigating too. See postattn_01.
6) Early-firing forks
The early-firing forks are those already above threshold within the first few layers, especially postattn_01. These cases are very interesting for further study regarding, among other things, whether more or fewer early firing forks exist as the model's skill knob is dialed.
One such example is shown below:
At first glance of the positions in question, I hypothesize that part of the early resolution of the fork logit has to do with the fork square being very protected, which might be a simple feature that early layers represent.
Here the knight is very safely guarded by a pawn, bishop, and rook.
6.5) Decomposing block 5's attention
The fork logit snaps at postattn_05, so it is worth looking at what that attention layer is comprised of. Each head's pattern is softmax(qk + gab): qk is the standard self attention matching between two squares' representations (semantics), while gab is a bias drawing on a library of 64×64 attention templates (gab_shared_weight) that is shared across every layer and head.
Because a move's logit is proj_sq_from[from] · proj_sq_to[to], the squares that matter are the knight's origin and its landing square.
First it is worth decomposing block 5's attention into the 8 heads' individual attention scores between the Knight's square and the forking square. Again, consider the example position:
In block 5, head 5, the knight's square gives 23.8% of its post-softmax attention to the landing square. Each square's attention distributes over 64 sources (every row sums to 1), so the null hypothesis (uniform) expected value is 1.6%: this is 15× that, and no other head comes close for this pair.
Visualizing the full attention heads in classic mech interp style:
With circuitsvis.attention_patterns():
A more natural way to view attention, below:
Columns are heads, rows are (qk), (gab), (summed) attention scores where the chess board is colored differently depending on the querying square with the white outline. Attention is represented vertically flipped.
Notice the shape of head 5 GAB.
head 5, landing f2 -> forked: d3, h1
gab: [3.045553207397461, 3.391704797744751]
qk : [0.6249876022338867, 1.4472296237945557]
So geometric GAB dominates here.
Various Notes about the project and process
I was originally inspired by the Monroe, et al paper: https://arxiv.org/abs/2605.19091, and its beautiful transcoder-found features prompted in me questions of compositionality and superposition.
I have been thinking a lot about a thesis that "Compression is necessary but not sufficient for intelligence, and insight is the name for when it happens." Insight is the constrained search through a space of representations which yields a compression. It is the mechanism by which a system becomes more intelligent (like sudden non-additive chess tactic re-representations...).
I really encourage people to check out the app I made and have some fun with it.
Possible confounders:
These were all performed to mimic very highly skilled players' play (2700 ELO) so it leaves open questions about lower skill level representations.
The knight fork positions were chosen from games in which they were played, which might preclude some of the positions with harder to find forks.
This is the lowest parameter Maia 3 model; perhaps results would be different with the 100M parameter version.
The threshold for the sudden logit value climb is somewhat arbitrary at 2.5, and can be improved by a more grounded metric.
I wonder if I could benefit from further pruning the knight fork finder, such as by eliminating moves that result in checkmate.
And most importantly: every position analyzed here has a knight move as its best move, and head 5's GAB template is knight-shaped. Whether the snap is about the fork or about the knight move is exactly what parts 2 and 3 take up.
I had a relatively difficult time understanding the mechanics of GAB and some details about the architectural choices of Maia 3, and still am not sure I faithfully conveyed its details 100% correctly.
The transcoder weights would prove very useful additions to further exploration and I am trying to obtain access to them from the CSS lab that made Maia 3.
This was my first mechanistic interpretability experiment (I started rapidly teaching myself the field this summer after being awestruck by some of Chris Olah's blog posts like the tanh NN visualization!) so I am not certain if I missed any important steps, but I tried to approach it like a math proof: what do I need to show, what would falsify this, etc. I enjoyed it very much.
I am thankful to the CSS Toronto lab for creating the engine, to Professor Colin Allen and Julie Litman for helping me with the write up, and to all the wonderful interpretability papers Anthropic has released over the past few years that have gotten me so interested in the field.
I am very open to feedback and comments of any nature!
And please visit davidlitman.com for more work if you think we have common interests.
(parts 2 and 3 to follow)
Summary of this post
This post is on the results of a mechanistic interpretability project aimed at understanding the internals of Maia 3: a transformer based chess bot trained to imitate human play at a chosen skill level, rather than to play optimally. My aim is to locate and causally describe one chess tactic in the Maia 3 engine. In this part 1, I describe strong correlational evidence that the knight-fork policy logit snaps into place after block 5’s attention layer. (Controls rule out that this is merely where the best move snaps into place at the start of part 2.) In part 2, I will further demonstrate causality and compositionality of the feature, and in part 3 describe unexpected results that shake up the pure "knight fork attention head” thesis into something stranger and more interesting.
Introduction
Chess is relevant to many facets of human cognition, from chunking to sudden insight, and the hope is that general interpretability work on chess models has relevance to cognitive neuroscientific research in the future.
Along with this, the decomposable structure of the transformer architecture (see: "A Mathematical Framework for Transformer Circuits”) adds to the intrigue of the interpretability work. Determining where in the network a concrete skill exists tells one about how its compressed representations form. A knight fork (a single knight move that attacks two valuable pieces at once) is a priori a strong target skill to analyze because it is discrete and unambiguous and important. I wanted to make headway into the question: "at what depth does the network's internals first make a knight fork decodable?"
Brief Notes on Mechanistic Interpretability
As the field is relatively new, it is worth spending a few sentences to give extra context to unfamiliar readers. The aim of mechanistic interpretability is to understand AI better by reverse-engineering models into human-describable parts. The ultimate hope is that this will help safety and alignment, for example by allowing us to recognize deceptive LLMs.
Terminology I will keep constant throughout:
- block 0–7: the eight transformer blocks; each contains an attention layer and an MLP layer
- residual stream: a vector; the transformer's shared communication layer to which attention and MLP layers additively read and write into, allowing for meaningful decomposition by specific layers' additions.
- readout point: the content of the residual stream at some depth
- logit lens: an interpretability technique that takes the residual stream at some layer and pushes it through the final unembedding layer, as though it were the finished product, and then reads its distribution over outputs. Here the output is the move policy distribution.
- decodable: when the move scores highly (large logits) after the logit lens
- policy: the network's output probability distribution over all 4352 moves. Maia plays directly from this with no search
About the model
MAIA 3 is a bidirectional encoder over all chess board squares: one board goes in as 64 square tokens with pieces represented by a one hot vector, and a single forward pass returns a full policy over all 4352 moves plus a win/draw/loss evaluation. There is nothing autoregressive and no sequence of tokens as input, so we can easily logit lens at each layer. Additionally, its attention heads use two schemes: a "semantic" standard Q,K self-attention, and a "geometric" GAB scheme to embed complex positional information, given that chess pieces all move in different ways which complicates simple attention patterns.
My primary interest in studying this model is its skill knob which turns a classic cognitive-science question (de Groot; Chase & Simon: experts don't search more, they re-represent) into runnable experiments, a quintessential attraction of mechanistic interpretability work! I predict that: 1) genuine tactical representations snap layers earlier in the network as conditioned skill rises, 2) features become sparser. I will be researching this immediately.
Interactive App I also vibe coded a satisfying interactive app that allows you to play a game against the engine at any mimicked human skill level and view each split attention head per square, watch the residual stream evolve, logit lens each layer, and more.
Repo: https://github.com/David-31415/chessformer_interp
1) Mining knight forks from human games
To keep the investigation clean, I decided to study what I believe is the least confounded type of fork:
A knight fork of only a king and queen (including bishops, for instance, will inevitably lead to internal representations of defended pieces and knight-bishop exchange values):
Additionally, I required that:
I mined thousands of public Lichess.org games.
2) Example fork and position used throughout post
Position FEN: "6k1/1pq2ppp/p7/3p4/1P1Nn3/P2QPP2/1B4Pb/7K b - - 1 26".
In case it is not clear, Nf2# attacks both the enemy queen and king, 'forking' the two pieces.
(Nf2 is also checkmate which is a confound that led me to an important finding that is addressed in part 2)
3) Measuring logits across network depth
I hooked the MaiaEngine class to cache the residual stream at every layer. I read it out at the 18 points that correspond to the residual stream: embed_in, postattn_0n (after attention n), block_0n (after MLP n), encoder_out.
For the fork, the move seemingly starts with relatively low logits until it snaps upward at late attention readout points.
4) Averaging across 500 forks
We have N=1. To recover the typical trajectory I run 500 forks through the same logit lens, stack their per-depth curves into a 500x18 tensor, and summarize with the mean and standard deviation across positions. The mean curve (±1 std) shows the depth at which the forking move policy typically climbs. The layers where the curve rises fastest are quite plausibly the ones doing the internal computation of assembling the tactic.
5) Determining when exactly the fork becomes decodable
The mean smooths over the exact layer at which each fork transitions from unrepresented to represented. So I record the first point at which its fork logit crosses +2.5, admittedly somewhat arbitrarily.
The sharp peak at postattn_05 (block 5's attention layer) is exciting to see. The fact that it sits at an attention readout point makes sense computationally. Attention layers are the only place information moves between squares; MLPs compute information square by square. A fork is clearly relational since it depends on the position of two other forked pieces, so the snap should appear exactly where some attention information routing happens. It appears as though for most strong moves, block 7 amplifies the logits further as seen above.
The very early outliers seem worth investigating too. See postattn_01.
6) Early-firing forks
The early-firing forks are those already above threshold within the first few layers, especially postattn_01. These cases are very interesting for further study regarding, among other things, whether more or fewer early firing forks exist as the model's skill knob is dialed.
One such example is shown below:
At first glance of the positions in question, I hypothesize that part of the early resolution of the fork logit has to do with the fork square being very protected, which might be a simple feature that early layers represent.
Here the knight is very safely guarded by a pawn, bishop, and rook.
6.5) Decomposing block 5's attention
The fork logit snaps at postattn_05, so it is worth looking at what that attention layer is comprised of. Each head's pattern is softmax(qk + gab): qk is the standard self attention matching between two squares' representations (semantics), while gab is a bias drawing on a library of 64×64 attention templates (gab_shared_weight) that is shared across every layer and head.
Because a move's logit is proj_sq_from[from] · proj_sq_to[to], the squares that matter are the knight's origin and its landing square.
First it is worth decomposing block 5's attention into the 8 heads' individual attention scores between the Knight's square and the forking square. Again, consider the example position:
In block 5, head 5, the knight's square gives 23.8% of its post-softmax attention to the landing square. Each square's attention distributes over 64 sources (every row sums to 1), so the null hypothesis (uniform) expected value is 1.6%: this is 15× that, and no other head comes close for this pair.
Visualizing the full attention heads in classic mech interp style:
With circuitsvis.attention_patterns():
A more natural way to view attention, below:
Columns are heads, rows are (qk), (gab), (summed) attention scores where the chess board is colored differently depending on the querying square with the white outline. Attention is represented vertically flipped.
Notice the shape of head 5 GAB.
head 5, landing f2 -> forked: d3, h1
gab: [3.045553207397461, 3.391704797744751]
qk : [0.6249876022338867, 1.4472296237945557]
So geometric GAB dominates here.
Various Notes about the project and process
I was originally inspired by the Monroe, et al paper: https://arxiv.org/abs/2605.19091, and its beautiful transcoder-found features prompted in me questions of compositionality and superposition.
I have been thinking a lot about a thesis that "Compression is necessary but not sufficient for intelligence, and insight is the name for when it happens." Insight is the constrained search through a space of representations which yields a compression. It is the mechanism by which a system becomes more intelligent (like sudden non-additive chess tactic re-representations...).
I really encourage people to check out the app I made and have some fun with it.
Possible confounders:
I had a relatively difficult time understanding the mechanics of GAB and some details about the architectural choices of Maia 3, and still am not sure I faithfully conveyed its details 100% correctly.
The transcoder weights would prove very useful additions to further exploration and I am trying to obtain access to them from the CSS lab that made Maia 3.
This was my first mechanistic interpretability experiment (I started rapidly teaching myself the field this summer after being awestruck by some of Chris Olah's blog posts like the tanh NN visualization!) so I am not certain if I missed any important steps, but I tried to approach it like a math proof: what do I need to show, what would falsify this, etc. I enjoyed it very much.
I am thankful to the CSS Toronto lab for creating the engine, to Professor Colin Allen and Julie Litman for helping me with the write up, and to all the wonderful interpretability papers Anthropic has released over the past few years that have gotten me so interested in the field.
I am very open to feedback and comments of any nature!
And please visit davidlitman.com for more work if you think we have common interests.
More to come...