This is an automated rejection. No LLM generated, assisted/co-written, or edited work.
Read full explanation
Watching Hidden-Unit Behaviour during Training in a 251-Parameter Toy Neural Network
Written by: Pedro Oubiña
Project Overview and Results Summary
This project aims to test whether watching a neural network's internals during its training reveals behavioural patterns in the models’s structure that inspecting only the finished model would miss. This is the core hypothesis of the AI and mechanistic interpretability subfield known as “developmental interpretability”.
A deliberately tiny model (a 3→16→11 multilayer perceptron, 251 parameters total, classifying RGB triples into 11 basic color names) was chosen so that every weight and hidden unit could technically be inspected by hand, avoiding the behaviour superposition problems that interfere in interpretability work on larger models.
Training was checkpointed by step across 25 independently seeded runs, with hidden-layer activations captured on a fixed 166 point probe set at every checkpoint.
The main findings were as follows:
Training dynamics were gradual and highly reproducible. The final test accuracy was 90.9% ± 0.6% across all seeds, and no run showed a sharp, grokking-style (see Section 2.2) phase transition.
A naive, single-guess test for reading a hidden unit's function ("does its loudest activation class match its biggest ablation-impact class") turned out to be close to a coin flip: 53.3% accurate, 95% CI [47.8%, 58.7%].
Replacing that binary test with a continuous rank correlation between a unit's full activation profile and its full causal-impact profile raised the correlation substantially, to 0.735. Then, using the unit's direct contribution to each class's output as activation x outgoing weight (direct logit attribution) raised it further, to 0.850, 95% CI [0.839, 0.861]. This was the best validated answer this project obtained to "how predictable is a hidden unit's behavior".
Pairwise ablation found genuine cases of two units acting as a backup/aid pair for a specific color, but a large-scale test across 17,578 pair-class combinations found no general method to predict which pairs will show this effect. This proved to be a real, replicated phenomenon that remains an open problem for further research rather than a solved one.
Finally, two further small checks closed out the planned methodology concerns:
Statistical-significance filtering left the correlation values essentially unchanged, which can be interpreted as a confirmation of robustness to noise
Restricting the color set to only the 11 unambiguous reference colors made the attribution correlation significantly worse (from 0.850 to 0.812), thus revealing that the larger and more ambiguous probe set was accounting for valuable noise reduction that a smaller and cleaner set could not replace.
Overall, the project supported its founding hypothesis: watching the model’s training revealed a per class learning order and a clear, verified case of a unit's apparent activation. Neither of which a “post-hoc” analysis of only the final model could have shown, while also discovering a well-tested idea (pairwise redundancy prediction) that was not able to be avoided.
1. Introduction and Motivation
Neural network weights are, on their own, virtually unintelligible: a large block of numbers with no obvious mapping to behaviors or relationships. The standard approach to interpretability takes a finished, trained model and reverse-engineers what its weights mean: ablating pieces, inspecting activations, training decoders on top of it.
This project tests a different angle: instead of only inspecting the final model, watch the model's internals during training. The hypothesis is that internal structure may not form smoothly; it may snap into place at specific moments (phase transitions) and those moments could be more informative than anything visible in the final weights alone.
The clearest known example of this phenomenon is grokking: some models trained on algorithmic tasks memorize the training data early, yet they sit at near-zero test accuracy for a long stretch despite perfect training accuracy, and then abruptly reorganize internally and jump to near-perfect generalization. If only the final model were to be examined, this event would be an invisible transition.
1.1 Why a Toy Model
We define a toy model as a deliberately simplified, idealized representation of a complex system, designed to isolate and explain core mechanisms by omitting extraneous details and real-world complications. [1]
Real interpretability work struggles with superposition (defined in Section 2.3): models represent more concepts than they have neurons for, so individual weights or neurons typically encode many overlapping, entangled concepts rather than one clean idea each, and this problem worsens combinatorially with scale. As such, the approach here is to start absurdly small, on a task with a fully known, verifiable truth, to validate the method before pointing it at anything larger.
Task: classify an RGB triple into one of 11 basic color names (Berlin & Kay's classic basic color term set: red, orange, yellow, green, blue, purple, pink, brown, black, white, gray).
Verifiable truth: nearest-neighbor Euclidean distance, in raw RGB space, to a fixed and single reference color per class. This is a known simplification: raw RGB distance is not perceptually uniform, but it was kept deliberately simple since the goal is testing a method, not building an accurate color namer.
Model: a 3 → 16 (ReLU) → 11 multilayer perceptron, so 251 parameters total. This was small enough that every hidden unit can plausibly be inspected by hand, yet large enough to achieve an accurate and replicable color prediction.
1.2 Benefits of Understanding Weight’s Behaviour
Detecting training-data poisoning or attacker interference: an unexplained internal shift during training, uncorrelated with any expected outcome, is a plausible security signal worth flagging.
Debugging capability failures by understanding why a model fails, rather than retraining blindly.
Targeted editing and unlearning: if we know which weights encode a specific behavior, that behavior can be edited surgically instead of retraining the model from scratch.
2. Background: Key Concepts
Every non-trivial technique used in this project is explained below: what it is, how it is calculated where applicable, and why it was the right tool for its specific scenario.
2.1 Developmental Interpretability
What it is: A subfield of AI interpretability that studies a model's internal structure as a function of training time, rather than only at convergence. It treats training as a process with its own dynamics, worth observing directly, rather than something to fast-forward through to reach a static object to analyze.
Why it is used here: This project is a direct, minimal-scale test of such an approach: checkpoint a model densely during training, capture hidden-layer activations at each checkpoint on a fixed set of inputs, and look for moments where those activations reorganize (possible weight behavior changes).
2.2 Grokking, Phase Transitions and Sharpness Ratio
What it is: Grokking is an observed phenomenon where a model achieves perfect or near-perfect training accuracy early, plateaus at poor test accuracy for a long stretch, then abruptly reorganizes internally and jumps to high test accuracy.[2]
A “phase transition” more generally refers to any abrupt, qualitative change in a model's internal representations or behavior during otherwise continuous training.
How it was measured here: A “sharpness ratio” per run defined as the single largest step-to-step test-accuracy jump, divided by the mean absolute step-to-step change across the whole run. A high ratio means one step dominates the entire trajectory (grokking-like); a ratio near 1 means growth was spread evenly across many steps (gradual and constant).
Why it is used here: It gives one comparable number per run for how dominated by a single moment that run's training was, without a subjective judgment call about where a plateau ends and a jump begins.
2.3 Superposition
What it is: A phenomenon where a neural network represents more features than it has neurons for, by encoding multiple concepts as overlapping, entangled combinations of the same units rather than one concept per unit. This is the major reason raw activations are hard to interpret in real models, and is thought to worsen combinatorially with scale.
Why it is used here: The model was sized specifically to minimize superposition (16 hidden units for an 11-class task), so that if entanglement still appeared, that would itself be notable. The phantom-unit and activation/causal-mismatch results (see Sections 4.4, 4.5) were exactly evidence of this kind of small-scale entanglement.
2.4 Ablation as Causal Intervention
What it is: Forcibly zeroing (or otherwise disabling) a specific part of a model. Here, one hidden unit's post-ReLU activation, for every input. Then observing how the model's output changes. Unlike inspecting activations, which is purely observational, ablation is a causal intervention: it tests what a component actually does and what impact it has, not just what it correlates with.
Why it is used here: Activation-based “this unit looks like it detects green” claims are correlational hypotheses. Ablation verifies whether a unit is causally responsible for a model's predictions, which turned out to actually matter (Section 4.3).
2.5 Silhouette Score
What it is: A standard clustering-quality metric.[3] For a point i belonging to class c, let a(i) be its mean distance to other points in the same class, and b(i) be its mean distance to points in the nearest other class, then we define the silhouette score s(i) as:
s(i) = ( b(i) − a(i) ) / max( a(i), b(i) )
Averaged over all points, this gives one number in [−1, 1]: near +1 means classes are cleanly separated in the representation space; near 0 means no clear separation; negative values mean points are (on average) closer to a different class's cluster than their own.
Why it is used here: Used in Phase 4 to summarize, at every training checkpoint, how well the 11 color classes separate in the model's 16-dimensional hidden-activation space on the fixed probe set. This is the primary signal used to search for phase transitions.
2.6 Spearman Rank Correlation
What it is: A correlation coefficient measuring how well the relationship between two variables can be described by a monotonic ranking, rather than requiring a strictly linear relationship (Pearson correlation). It is computed by converting each variable to ranks and then computing the ordinary Pearson correlation of those ranks:
ρ = Pearson_correlation( rank(X), rank(Y) )
Why it is used here: This project’s model behavior checking is based on comparing, per hidden unit, an 11-number “how excited is this unit for each color class” profile against an 11-number “how much does ablating this unit hurt each color class” profile.
A single top-pick binary test (“does the 1st class match”) discards 10 of the 11 numbers and was shown to be misleading for close ties (Section 4.5). Spearman correlation uses the full ranking of all 11 classes, ranges from −1 to +1, and requires neither linearity nor matching scales between the two profiles, thus greatly benefiting near-ties.
2.7 Direct Logit Attribution
What it is: A mechanistic-interpretability technique measuring a component's actual contribution to a specific output, rather than just how active it is.[4] A hidden unit's activation h feeds into class c's output score through the second-layer weight W2[c,u]. And so the unit's direct contribution to that logit is:
contribution(u, c) = mean_activation(u, c) × W2[c, u]
Why it is used here: Raw activation alone only says how excited a unit gets; it says nothing about whether that excitement is actually wired to matter for a given class: a unit could fire strongly for green while its outgoing weight to the green logit is near zero, which will have almost no impact on the final output. Multiplying activation by outgoing weight captures a piece of the network's real computation, and produced the single largest, most consistent improvement of any methodology change tested (Section 4.6).
2.8 Standard Error and 95% Confidence Intervals
What it is: Given n independent measurements of a quantity (e.g. final accuracy from 25 separately-seeded runs), the standard error (SE) estimates how much the sample mean would vary if the experiment were repeated:
SE = sample_standard_deviation / √n 95% CI = sample_mean ± 1.96 × SE
Why it is used here: Every mean reported is an average over 25 independently seeded runs, not a single measurement. The 95% CI communicates the sampling uncertainty on that average, which is essential for telling “this metric really changed” apart from “this is within the noise of 25 runs” (used throughout Sections 4.5 to 4.9).
*The value 1.96 in the 95% CI formula is the z-score (critical value) from the standard normal distribution that captures the middle 95% of the data.
What it is: A test for whether two observed proportions (a class's accuracy before versus after ablating a unit) differ by more than sampling noise alone, given the number of test examples n. With baseline proportion p₀ and ablated proportion p₁:
SE = √( p₀(1−p₀)/n + p₁(1−p₁)/n ) z = (p₀ − p₁) / SE
A drop is treated as a real effect if |z| > 1.96 (the standard 95% two-tailed threshold); otherwise it is zeroed out before correlating.
Why it is used here: Section 4.8 tests whether small, possibly-noisy accuracy drops were affecting the correlations.
2.10 Synergy / Interaction Effects
What it is: A measure of whether two components' combined effect, removed together, differs from the simple sum of their individual effects. Similar to the additive epistasis model from genetics and commonly used in interpretability and neuroscience. If drop(u) and drop(v) are the accuracy drops from ablating each unit alone, and drop(u,v) is the drop from ablating both together:
synergy = drop(u, v) − ( drop(u) + drop(v) )
Positive synergy (drop(u, v) > ( drop(u) + drop(v) )) is the signature of redundant or “backup” coding, where each unit alone looks dispensable because the other compensates for it; near-zero synergy means the two units act independently.
Why it is used here: Single-unit ablation is structurally blind to backup relationships, so whichever unit is left in place always compensates for whichever one was removed. Synergy, tested by ablating pairs jointly, was the only way to detect this class of effect (see Sections 4.7, 4.8).
3. Experimental Setupreport
3.1 Task, Model, and Probe Set
Dataset: RGB triples sampled uniformly from the color cube, labeled by nearest-neighbor distance to 11 fixed reference colors. 20,000 training points and 5,000 to 10,000 held-out test points per run, generated with independent random seeds per run.
Model: ColorNet, a 3 → 16 (ReLU) → 11 fully-connected network, 251 parameters, exposing its 16-dimensional hidden layer for any input batch, defined with PyTorch.
Probe set: 166 fixed RGB points reused at every checkpoint across every run: 11 reference colors, 55 pairwise midpoints between reference colors, and 100 points found by search to sit as close as possible to a true decision boundary of the preestablished labeling rule.
3.2 Training and Checkpointing
Stochastic gradient descent was used, 3,000 steps per run, checkpointing every 25 steps (121 checkpoints per run). At every checkpoint: full model weights, hidden-layer activations on the fixed probe set, and train/test loss and accuracy were saved.
Twenty-five training runs, identical hyperparameters (3,000 steps, batch size 64, learning rate 0.05, hidden size 16), differing only by a random seed (0–24, controlling weight initialization, data sampling, and probe-set boundary search).
Metric
Mean
Std. dev.
95% CI
Range
Final test accuracy
90.87%
0.58pp
[90.65%, 91.10%]
89.88% – 92.00%
Transition sharpness ratio
12.21
4.15
[10.58, 13.83]
6.21 – 23.25
Figure 1. Final test accuracy across 25 independent seeds. Tight, reproducible clustering around 90.9%.
4.2 Searching for Phase Transitions (Phase 4)
For every checkpoint in every run, the silhouette score (see Section 2.5) of the probe-set hidden activations against their true labels was computed, and checkpoint-to-checkpoint jumps in that score were ranked as phase-transition candidates. Across all 25 runs, no run's accuracy trajectory was dominated by a single dramatic jump, as the sharpness ratio (see Section 2.2) ranged only from 6.2 to 23.2, with no outlier suggestive of a genuine grokking-style event.
Figure 2. Transition sharpness ratio, sorted, across 25 seeds. No run stands out as dramatically more “grokking-like” than the rest.
This can be seen as a legitimate negative result: under these settings (this learning rate, this model size, this amount of training, this task), growth was gradual and reproducibly so, not the delayed-generalization pattern seen in grokking papers. Section 6 discusses plausible reasons (no weight decay, task may be too easy relative to model capacity, training may not be long enough).
4.3 Case Study: The Step-325 Transition Candidate
Despite the negative aggregate result, the single largest available signal (the biggest test-accuracy jump in the seed-0 run, at step 325) was investigated directly. At that checkpoint, the overall test accuracy was 39.5%, but the per class breakdown revealed the aggregate curve was hiding a staged, per class learning order:
Class
Accuracy at step 325
gray
99.4%
blue
79.4%
green
69.9%
purple
20.3%
pink
4.1%
red / orange / yellow / brown / black / white
0.0% (not yet learned)
The three most-changed hidden units at this step, and what ablating them (Section 2.4) revealed:
Unit
Activation - “preferred” class
Causal effect of ablation
Interpretation
6
blue
Blue accuracy 79.4% → 0.0%; other classes barely affected.
Clean, causally-verified blue detector: activation label matches causal role.
Real green detector, but with other classes entangled: removing it helps pink and blue while hurting gray. This is evidence of interference, not a clean single-purpose unit.
7
white (loudest activation)
White accuracy stays at 0.0% (nothing to lose — not yet learned). Gray accuracy collapses 99.4% → 21.7% (−77.7pp).
Activation-based label (“white”) is misleading: at this point in training the unit's true causal role is gray, not white, which is exactly the correlation vs. causation gap ablation is designed to catch.
4.4 Full-Unit Ablation Sweep at Convergence
Extending the case study, all 16 units (not just the top 3) were ablated at the final checkpoint (step 3,000) of every run, and each unit was classified as “dead” (zero activation and zero causal effect) or “active.”
Metric
Mean (of 16)
95% CI
Range
Dead units per run
4.32
[3.62, 5.02]
0 – 7
Active units per run
11.68
[10.98, 12.38]
9 – 16
“Phantom” units per run (high selectivity, zero causal effect)
0.60
[0.30, 0.90]
12 of 25 runs had ≥ 1 phantom units
Figure 3. Dead vs. Active hidden units per run, sorted. Effective network capacity varies substantially by seed even at matched final accuracy.
The clearest single phantom-unit example: in the seed 0 run, unit 4 had a near-perfect activation selectivity of 0.9999 for black, yet ablating it produced zero change in any test prediction. It looked like the cleanest possible detector by activation alone, but turned out to be causally inert.
Testing the “ignore the interfering unit” idea
A natural question raised during this project was the following one: if one unit is known to interfere with another class, can it be selectively ignored to sharpen predictions for that class, at no cost? This was tested directly (seed-0 run, final checkpoint): unit 8 (a blue detector found to interfere with white) was zeroed only on inputs whose true label was probably not blue, and the effect on white and on the overall accuracy was measured.
Class
Baseline accuracy
Conditional-ablation accuracy
Change
white (target class)
57.9%
71.1%
+13.2pp
purple
93.3%
59.2%
−34.1pp
black
65.7%
37.7%
−28.1pp
gray
94.5%
79.6%
−14.8pp
Overall accuracy
89.9%
84.7%
−5.2pp
The target class did improve, but overall accuracy fell: unit 8 was quietly carrying purple, black, and gray as well as blue. The idea does not deliver a free precision gain in its naive, blanket form. Any real use of it would need to be far more surgical than ignoring the whole unit whenever the target class isn't blue.
4.5 From a Binary Match Test to Rank Correlation
The original test for “does this unit's activation-based label predict its causal role” was binary: does the class with the highest mean activation equal the class most hurt by ablation? Across 25 runs:
Metric
Mean
95% CI
Binary match rate
53.3%
[47.8%, 58.7%]
This is practically a coin flip, but the binary test discards 10 of each unit's 11 class values. A concrete example (seed 0, unit 2, final checkpoint): the unit's top activation is for yellow (3.09) and its biggest causal impact is on green (0.92), in the first experiment this was counted as a full “mismatch.” But yellow and green are effectively tied for both activation (3.09 vs. 2.58) and causal impact (0.92 vs. 0.90), and the rest of the 11-class profile lines up closely too.
The Spearman correlation (see Section 2.6) for this unit's full profile (seed 0, unit 2) is 0.873; a strong relationship the binary test could not see.
Metric
Mean
95% CI
Activation vs. causal-impact correlation (all 11 classes)
0.735
[0.714, 0.755]
Figure 4. Methodology progression: from a coin-flip binary test to a strong, consistent rank correlation, and the effect of the reference-only probe set variant (see Section 4.9).
4.6 Direct Logit Attribution
Raw activation only measures how excited a unit gets, not whether that excitement is connected to an actual decision change. Multiplying a unit's activation by its outgoing weight to each class (see Section 2.7) was tested as a better predictor:
Class A
Class B
Class C
Correlation with true causal impact [0, 0.9, 0.05]
Activation only
5.0
2.0
1.0
−0.50 (points the wrong way)
Outgoing weight
0.0
3.0
0.5
—
Attribution (activation x weight)
0.0
6.0
0.5
+1.00 (correct)
Applied to the real 25-run dataset, attribution correlation was higher than activation-only correlation in every single run, with non-overlapping confidence intervals:
Metric
Mean
95% CI
Variance explained (ρ²)
Activation vs causal correlation
0.735
[0.714, 0.755]
≈ 54%
Attribution vs. causal correlation
0.850
[0.839, 0.861]
≈ 72%
Mean per run improvement: +0.115, range +0.017 to +0.231, uniform across all 25 runs. This is the single strongest, most consistent methodology improvement found in the project.
4.7 Pairwise Interactions: Initial Finding
Even a rho² of 0.72 leaves roughly 28% of a unit's causal behavior unexplained by its own single-unit attribution profile. One concrete hypothesis that surged was the following: some of that gap is explained by redundant, “backup” coding between pairs of units, invisible to any single-unit test by construction (see Section 2.10). For each run, the pair of active units with the most correlated attribution profiles was jointly ablated and compared against the sum of their individual effects.
Metric
Mean
95% CI
Overall accuracy synergy (whole model)
+0.016
[−0.004, +0.037] (crosses zero)
Worst-hit-class synergy (per pair)
+0.228
[0.169, 0.287] — all 25 runs positive
Figure 5. Left: overall-accuracy synergy is not reliably different from zero. Right: the single worst-affected class per pair shows large, consistently positive synergy in every run.
Concrete example: unit 8 and unit 13, class “white” (seed 4)
Condition
White accuracy
Baseline
56.9%
Unit 8 ablated alone
38.7% (−18.2pp)
Unit 13 ablated alone
88.0% (+31.1pp)
Naively predicted joint effect (sum of the two)
≈ 69.8% (theoretical net improvement)
Actual joint ablation (both removed together)
8.5% (−48.4pp vs. baseline)
Unit 13 looked harmless, or even beneficial when tested alone, because unit 8 was quietly covering for it. Only removing both at once revealed the true, much larger joint dependency. This pattern (a pair of units that look individually replaceable but are jointly essential) appeared in every one of the 25 runs, for at least one class each time, even though the whole model level synergy averaged out to almost statistical noise.
4.8 Large-Scale Test: Can Interaction Be Predicted?
The initial pairwise result used only the single most-correlated pair per run (25 data points total), which turned out to be a search over 25 pairs x 11 classes for the single most extreme result, risking “selection bias”. A full sweep tested every pair of active units in every run (up to 120 pairs per run), producing 17,578 (pair-class) observations, and asked whether a per class interaction term, attribution_u[c] x attribution_v[c], would predict the measured synergy.
Test
Pearson r
Spearman r
Interaction term vs. synergy (raw accuracy space)
−0.304
+0.013 (≈ 0, no relationship)
Simple additive sum vs. synergy (raw accuracy space)
−0.310
—
Figure 6. Interaction term vs. measured synergy across all 17,578 (pair - class) observations. No visible positive predictive relationship.
Nonetheless, a structural flaw was identified: because accuracy is bounded at 0% and 100%, the naive “sum of two bounded drops” can exceed what is physically possible (more than 100%), mechanically biasing the synergy metric negatively whenever individual effects are large. Recomputing everything in logarithmic (logit) space, which is unbounded, removed this artifact but did not rescue the hypothesis:
Test (logit-space, ceiling artifact removed)
Pearson r
Spearman r
Interaction term vs. synergy
−0.185
−0.139 (small, consistent negative)
A separate check on the original 25 top-pair heuristic (does “how similar two units' attribution profiles look” predict “how much synergy they show”) also came back weak: Pearson r = 0.167.
In other words, the seed 4 unit 8/unit 13 case was real and correctly measured, but it is an outlier, not evidence of a general, predictable pattern. Pairwise redundancy is a genuine, replicated phenomenon (see Section 4.7) that current methods cannot yet predict in advance.
This is an honest negative result carried forward as future work (see Section 7).
Every per class accuracy drop had been treated as a real effect regardless of size. A conservative two-proportion z-test (see Section 2.9) was used to flag drops not statistically distinguishable from zero, replacing them with 0 before recomputing the correlations.
Metric
Before filtering
After filtering
Change
Activation vs. causal correlation
0.735 [0.714, 0.755]
0.756 [0.737, 0.776]
+0.021 (within noise)
Attribution vs. causal correlation
0.850 [0.839, 0.861]
0.848 [0.834, 0.862]
−0.002 (no change)
On average, 7.73 of each unit's 11 per class drops (about 70%) were statistically significant at the 95% level. Filtering out the rest barely moved either headline number. This was a reassuring robustness result: the 0.735 and 0.850 correlations were not artifacts of noisy small effects being averaged in.
4.10 Reference-Color-Only Test and Probe-Set Imbalance
The 166-point probe set is 60% deliberately ambiguous boundary points: a human being would struggle to classify correctly each color as they were basically mixtures of class colors. Recomputing activation and attribution profiles using only the 11 unambiguous reference colors tested whether that ambiguity was limiting the correlations.
Metric
Full probe set (166 pts)
Reference-only (11 pts)
Change
Activation vs. causal correlation
0.735 [0.714, 0.755]
0.743 [0.726, 0.761]
+0.008 (within noise)
Attribution vs. causal correlation
0.850 [0.839, 0.861]
0.812 [0.799, 0.826]
−0.038 — non-overlapping CIs, a real decrease
Contrary to the hypothesis, restricting to reference colors made attribution correlation significantly worse. The likely explanation is the sample size, not ambiguity: the full probe set gives each class multiple points to average a unit's activation over, while the reference-only set gives exactly one point per class, which is basically a single noisy sample with no averaging protection.
Class
Reference pts
Midpoint pts
Boundary pts
Total
red
1
3
4
8
orange
1
7
7
15
yellow
1
2
5
8
green
1
4
13
18
blue
1
3
8
12
purple
1
4
10
15
pink
1
7
15
23
brown
1
10
12
23
black
1
0
4
5
white
1
0
2
3
gray
1
15
20
36
Figure 7. Probe points per class in the full 166-point set. White (3 points) and gray (36 points) differ by 12x, a real design imbalance worth correcting in future runs.
This imbalance is a genuine limitation of the current probe set (see Section 6), not something corrected in this project: classes with very few probe points (white, black) likely have less reliable activation estimates throughout this entire report, not just in the reference-only test.
5. Did Watching Training Reveal Anything “Post-Hoc” Analysis Would Have Missed?
This is the actual test of the project's founding hypothesis, and the honest answer has three parts.
What a “post-hoc” only analysis of the final model would have shown
A 90.9% accurate classifier, 251 weights, and if the same activation, attribution, and ablation analysis were run on the finished model then some clean unit detectors, some dead units, and a finished function would be noticed. Nonetheless, it would not show that the model learned roughly five color classes first and the other six later (see Section 4.3), because by the final checkpoint that ordering has been overwritten by hundreds of subsequent updates. Additionally, it would not show that unit 6 became a stable blue detector early while unit 2 spent time actively interfering with pink and blue decisions before finally settling down. Finally, it would not show that unit 7's role at 10% of the way through training (functionally gray, despite looking like white) completely changed, as a post-hoc snapshot has no record that a unit's role differed at all.
What the developmental approach specifically provided us
A per class learning progression invisible in the aggregate accuracy metric: the smooth-looking accuracy curve (see Section 4.2) was built from roughly five classes being learned first and six starting from zero, which was visible only by inspecting an interior checkpoint (see Section 4.3).
A concrete and verified case of activation vs. function mismatch: unit 7 looking like a white detector while functioning as a gray detector at step 325 is a fully verified instance of exactly the failure source developmental interpretability research worries about at larger scale.
Evidence against the core hypothesis that was useful for the analysis: the main Phase 4 result across 25 runs was technically negative, as there was no sharp transition. A less careful analysis might have misjudged the first sizable jump found as grokking evidence, but the general and broad truth is that growth was gradual. The step 325 case was nonetheless investigated, as it portrayed the biggest accuracy jump, even if it was not what the original project plan was looking for, and it still produced a genuine, useful finding.
What this project does not establish
First of all, this project does not prove developmental interpretability is able to be scaled and still maintain a strong behaviour correlation. This is a 251-parameter model on a 3-input, 11-class toy task, chosen precisely to minimize superposition. As such, the dominant sources of error were the dead units and specific interference cases, not dense superposition. This project aimed to prove that developmental interpretability is a valid approach to build a significantly accurate unit behaviour schema, the follow up question is how can this approach be modified and scaled to account for superposition. This project is a stepping stone, not the solution of the black box problem with neural networks.
The main project limitations are recapped and listed in the next section.
6. Limitations
Scale: a single, deliberately tiny architecture (3→16→11, 251 parameters) and a single fixed set of hyperparameters (learning rate, batch size, step count) were used throughout. No hyperparameter sweep was run to check whether these findings are specific to this exact configuration.
No genuine grokking case was produced or tested against. All conclusions about phase transition sharpness apply to the gradual learning rate observed here, and not to a confirmed sharp transition case.
Probe-set class imbalance (see Section 4.10): white and black have only 3 and 5 total probe points respectively, versus 36 for gray. This is a 12x difference that likely makes activation based estimates for those classes less reliable throughout the entire analysis, not only in the reference-only test.
Pairwise interaction effects were demonstrated but not actually predicted. The interaction and profile similarity heuristics that were tested (see Section 4.8) both failed to generalize from the one described example found. As such, a validated predictor for which unit pairs will show redundancy remains unsolved.
The raw accuracy synergy metric has a known numerical limitation (see Section 4.8): when individual effects are large; the logit-space correction addresses this but was only applied to the interaction term test, not retroactively to every synergy number reported elsewhere in this project.
All 25 runs share the same probe set design and the same task; findings about probe-set imbalance and ambiguity trade-offs are specific to this particular 166-point set and may not generalize to differently constructed probe sets.
7. Future Work
Listed below is a collection of ideas to test in a future experiment, these can be interpreted as a guide on where to expand the project:
Deliberately provoke a sharper transition (weight decay, a much lower learning rate with many more steps, or a harder/more ambiguous variant of the task) to obtain a genuine grokking-style case and test the same Phase 5 methods against a real, sharp transition and compare the results with the gradual transition.
Track dead-unit count and identity over the full course of training, not just at the final checkpoint, to see whether units are ever alive and later die, or if they are dead from initialization onward.
Rebalance the probe set so every class has a comparable number of points (avoiding the 12x imbalance between white and gray), and re-test whether this changes any of the correlation results.
Build an improved and genuinely validated pairwise interaction predictor, since the interaction-term and profile-similarity approaches tested here both failed at building an image of the pair interactions.
Test larger hidden sizes on the same task to find the point at which superposition becomes the dominant failure source, rather than the dead-unit and specific-interference patterns found at this hidden layer of size 16.
8. Conclusion
All in all, this project set out to test a specific, falsifiable hypothesis: watching a neural network's internals during training reveals structure a "post-hoc" analysis of the finished model would miss. Across 25 independently seeded runs of a completely manually inspectable 251-parameter model, the evidence supports that claim in this toy setting.
The developmental record revealed a per class learning strategy invisible in the aggregate accuracy curve, and a verified case of a hidden unit's activation-based “meaning” diverging from its true function. This is exactly the kind of finding only visible by inspecting an interior training checkpoint, and not the final model.
On the methodology side, the project underwent a complete set of improvements allowing the results to go from a naive binary test that looked close to a coin flip, then replaced by a rank-correlation test that revealed a real and more positive relationship (0.735), and then improved further by a mechanistically-grounded attribution measure (0.850). At the end, it was stress tested against three theoretical concerns: pairwise interactions (real but not yet actaully predictable), statistical noise (not a meaningful data altering factor), and probe set design (a real, quantified limitation).
As a final remark, it is important to once again point out that this experiment does not prove that developmental interpretability is a reliable and working method for large neural networks, but rather as an introduction into the capabilities and limitations that this method can provide to the AI interpretability research space.
(Sources, and both appendices, including the per run data and python script references, can be found below.)
Sources
Full list of all sources and references for research and inspiration:
Towards Developmental Interpretability
by Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, Stan van Wingerden
All programs are documented with an in-file explanation of its purpose. Ideally, files should be run in the order stated in the table below as later code might rely on files created by previous programs. Feel free to test all data and any contributions or improvements (see Section 7) are appreciated.
Script
Purpose
dataset.py
Phase 0: Task definition and truth labeling rule.
model.py
Phase 1: ColorNet model definition, 3→16→11 MLP.
probe_set.py
Phase 2: Builds the fixed 166-point probe set.
train.py
Phase 2/3: Instrumented training loop with step-cadence checkpointing.
run_experiment.py
Runs train.py + analyze.py + sweep.py for a list of seeds in one command.
analyze.py
Phase 4: silhouette-based transition detection.
interpret.py
Phase 5: unit-level weight-delta ranking, selectivity, and top-3 ablation for one flagged step.
sweep.py
Phase 5 extension: full 16-unit ablation sweep, activation and attribution correlation metrics, conditional-ablation test.
run_sweeps.py
Runs sweep.py across every run directory in one command.
pairwise.py / run_pairwise.py
Top 1 pair per-run pairwise ablation and synergy test (see Section 4.7).
pairwise_full.py / run_pairwise_full.py
Full combinatorial pairwise sweep and interaction-term validation (see Section 4.8).
significance_filter.py
Two-proportion z-test filtering of accuracy drops (see Section 4.9).
extract_weights.py
Extracts the fc2 output-weight matrix from a checkpoint (needed for the reference-only test).
reference_only_test.py
Reference color only activation/attribution profile test (see Section 4.10).
Watching Hidden-Unit Behaviour during Training in a 251-Parameter Toy Neural Network
Written by: Pedro Oubiña
Project Overview and Results Summary
This project aims to test whether watching a neural network's internals during its training reveals behavioural patterns in the models’s structure that inspecting only the finished model would miss. This is the core hypothesis of the AI and mechanistic interpretability subfield known as “developmental interpretability”.
A deliberately tiny model (a 3→16→11 multilayer perceptron, 251 parameters total, classifying RGB triples into 11 basic color names) was chosen so that every weight and hidden unit could technically be inspected by hand, avoiding the behaviour superposition problems that interfere in interpretability work on larger models.
Training was checkpointed by step across 25 independently seeded runs, with hidden-layer activations captured on a fixed 166 point probe set at every checkpoint.
The main findings were as follows:
Overall, the project supported its founding hypothesis: watching the model’s training revealed a per class learning order and a clear, verified case of a unit's apparent activation. Neither of which a “post-hoc” analysis of only the final model could have shown, while also discovering a well-tested idea (pairwise redundancy prediction) that was not able to be avoided.
1. Introduction and Motivation
Neural network weights are, on their own, virtually unintelligible: a large block of numbers with no obvious mapping to behaviors or relationships. The standard approach to interpretability takes a finished, trained model and reverse-engineers what its weights mean: ablating pieces, inspecting activations, training decoders on top of it.
This project tests a different angle: instead of only inspecting the final model, watch the model's internals during training. The hypothesis is that internal structure may not form smoothly; it may snap into place at specific moments (phase transitions) and those moments could be more informative than anything visible in the final weights alone.
The clearest known example of this phenomenon is grokking: some models trained on algorithmic tasks memorize the training data early, yet they sit at near-zero test accuracy for a long stretch despite perfect training accuracy, and then abruptly reorganize internally and jump to near-perfect generalization. If only the final model were to be examined, this event would be an invisible transition.
1.1 Why a Toy Model
We define a toy model as a deliberately simplified, idealized representation of a complex system, designed to isolate and explain core mechanisms by omitting extraneous details and real-world complications. [1]
Real interpretability work struggles with superposition (defined in Section 2.3): models represent more concepts than they have neurons for, so individual weights or neurons typically encode many overlapping, entangled concepts rather than one clean idea each, and this problem worsens combinatorially with scale. As such, the approach here is to start absurdly small, on a task with a fully known, verifiable truth, to validate the method before pointing it at anything larger.
1.2 Benefits of Understanding Weight’s Behaviour
2. Background: Key Concepts
Every non-trivial technique used in this project is explained below: what it is, how it is calculated where applicable, and why it was the right tool for its specific scenario.
2.1 Developmental Interpretability
What it is: A subfield of AI interpretability that studies a model's internal structure as a function of training time, rather than only at convergence. It treats training as a process with its own dynamics, worth observing directly, rather than something to fast-forward through to reach a static object to analyze.
Why it is used here: This project is a direct, minimal-scale test of such an approach: checkpoint a model densely during training, capture hidden-layer activations at each checkpoint on a fixed set of inputs, and look for moments where those activations reorganize (possible weight behavior changes).
2.2 Grokking, Phase Transitions and Sharpness Ratio
What it is: Grokking is an observed phenomenon where a model achieves perfect or near-perfect training accuracy early, plateaus at poor test accuracy for a long stretch, then abruptly reorganizes internally and jumps to high test accuracy.[2]
A “phase transition” more generally refers to any abrupt, qualitative change in a model's internal representations or behavior during otherwise continuous training.
How it was measured here: A “sharpness ratio” per run defined as the single largest step-to-step test-accuracy jump, divided by the mean absolute step-to-step change across the whole run. A high ratio means one step dominates the entire trajectory (grokking-like); a ratio near 1 means growth was spread evenly across many steps (gradual and constant).
Why it is used here: It gives one comparable number per run for how dominated by a single moment that run's training was, without a subjective judgment call about where a plateau ends and a jump begins.
2.3 Superposition
What it is: A phenomenon where a neural network represents more features than it has neurons for, by encoding multiple concepts as overlapping, entangled combinations of the same units rather than one concept per unit. This is the major reason raw activations are hard to interpret in real models, and is thought to worsen combinatorially with scale.
Why it is used here: The model was sized specifically to minimize superposition (16 hidden units for an 11-class task), so that if entanglement still appeared, that would itself be notable. The phantom-unit and activation/causal-mismatch results (see Sections 4.4, 4.5) were exactly evidence of this kind of small-scale entanglement.
2.4 Ablation as Causal Intervention
What it is: Forcibly zeroing (or otherwise disabling) a specific part of a model. Here, one hidden unit's post-ReLU activation, for every input. Then observing how the model's output changes. Unlike inspecting activations, which is purely observational, ablation is a causal intervention: it tests what a component actually does and what impact it has, not just what it correlates with.
Why it is used here: Activation-based “this unit looks like it detects green” claims are correlational hypotheses. Ablation verifies whether a unit is causally responsible for a model's predictions, which turned out to actually matter (Section 4.3).
2.5 Silhouette Score
What it is: A standard clustering-quality metric.[3] For a point i belonging to class c, let a(i) be its mean distance to other points in the same class, and b(i) be its mean distance to points in the nearest other class, then we define the silhouette score s(i) as:
s(i) = ( b(i) − a(i) ) / max( a(i), b(i) )
Averaged over all points, this gives one number in [−1, 1]: near +1 means classes are cleanly separated in the representation space; near 0 means no clear separation; negative values mean points are (on average) closer to a different class's cluster than their own.
Why it is used here: Used in Phase 4 to summarize, at every training checkpoint, how well the 11 color classes separate in the model's 16-dimensional hidden-activation space on the fixed probe set. This is the primary signal used to search for phase transitions.
2.6 Spearman Rank Correlation
What it is: A correlation coefficient measuring how well the relationship between two variables can be described by a monotonic ranking, rather than requiring a strictly linear relationship (Pearson correlation). It is computed by converting each variable to ranks and then computing the ordinary Pearson correlation of those ranks:
ρ = Pearson_correlation( rank(X), rank(Y) )
Why it is used here: This project’s model behavior checking is based on comparing, per hidden unit, an 11-number “how excited is this unit for each color class” profile against an 11-number “how much does ablating this unit hurt each color class” profile.
A single top-pick binary test (“does the 1st class match”) discards 10 of the 11 numbers and was shown to be misleading for close ties (Section 4.5). Spearman correlation uses the full ranking of all 11 classes, ranges from −1 to +1, and requires neither linearity nor matching scales between the two profiles, thus greatly benefiting near-ties.
2.7 Direct Logit Attribution
What it is: A mechanistic-interpretability technique measuring a component's actual contribution to a specific output, rather than just how active it is.[4] A hidden unit's activation h feeds into class c's output score through the second-layer weight W2[c,u]. And so the unit's direct contribution to that logit is:
contribution(u, c) = mean_activation(u, c) × W2[c, u]
Why it is used here: Raw activation alone only says how excited a unit gets; it says nothing about whether that excitement is actually wired to matter for a given class: a unit could fire strongly for green while its outgoing weight to the green logit is near zero, which will have almost no impact on the final output. Multiplying activation by outgoing weight captures a piece of the network's real computation, and produced the single largest, most consistent improvement of any methodology change tested (Section 4.6).
2.8 Standard Error and 95% Confidence Intervals
What it is: Given n independent measurements of a quantity (e.g. final accuracy from 25 separately-seeded runs), the standard error (SE) estimates how much the sample mean would vary if the experiment were repeated:
SE = sample_standard_deviation / √n 95% CI = sample_mean ± 1.96 × SE
Why it is used here: Every mean reported is an average over 25 independently seeded runs, not a single measurement. The 95% CI communicates the sampling uncertainty on that average, which is essential for telling “this metric really changed” apart from “this is within the noise of 25 runs” (used throughout Sections 4.5 to 4.9).
*The value 1.96 in the 95% CI formula is the z-score (critical value) from the standard normal distribution that captures the middle 95% of the data.
2.9 Two-Proportion Z-Test (Significance Filtering)
What it is: A test for whether two observed proportions (a class's accuracy before versus after ablating a unit) differ by more than sampling noise alone, given the number of test examples n. With baseline proportion p₀ and ablated proportion p₁:
SE = √( p₀(1−p₀)/n + p₁(1−p₁)/n ) z = (p₀ − p₁) / SE
A drop is treated as a real effect if |z| > 1.96 (the standard 95% two-tailed threshold); otherwise it is zeroed out before correlating.
Why it is used here: Section 4.8 tests whether small, possibly-noisy accuracy drops were affecting the correlations.
2.10 Synergy / Interaction Effects
What it is: A measure of whether two components' combined effect, removed together, differs from the simple sum of their individual effects. Similar to the additive epistasis model from genetics and commonly used in interpretability and neuroscience. If drop(u) and drop(v) are the accuracy drops from ablating each unit alone, and drop(u,v) is the drop from ablating both together:
synergy = drop(u, v) − ( drop(u) + drop(v) )
Positive synergy (drop(u, v) > ( drop(u) + drop(v) )) is the signature of redundant or “backup” coding, where each unit alone looks dispensable because the other compensates for it; near-zero synergy means the two units act independently.
Why it is used here: Single-unit ablation is structurally blind to backup relationships, so whichever unit is left in place always compensates for whichever one was removed. Synergy, tested by ablating pairs jointly, was the only way to detect this class of effect (see Sections 4.7, 4.8).
3. Experimental Setupreport
3.1 Task, Model, and Probe Set
3.2 Training and Checkpointing
Stochastic gradient descent was used, 3,000 steps per run, checkpointing every 25 steps (121 checkpoints per run). At every checkpoint: full model weights, hidden-layer activations on the fixed probe set, and train/test loss and accuracy were saved.
3.3 The Pipeline
Phase
Goal
Key script(s)
0–1
Define task and model
dataset.py, model.py
2
Instrument training with checkpointing
probe_set.py, train.py
3
Train and collect (25 independent seeds)
run_experiment.py
4
Search for phase transitions
analyze.py
5
Interpret flagged moments causally
interpret.py, sweep.py, pairwise.py, pairwise_full.py
—
Methodology audit
significance_filter.py, reference_only_test.py, extract_weights.py
6
Reflect on the core hypothesis
Report, Section 5
4. Results
4.1 Training Dynamics Across 25 Independent Runs
Twenty-five training runs, identical hyperparameters (3,000 steps, batch size 64, learning rate 0.05, hidden size 16), differing only by a random seed (0–24, controlling weight initialization, data sampling, and probe-set boundary search).
Metric
Mean
Std. dev.
95% CI
Range
Final test accuracy
90.87%
0.58pp
[90.65%, 91.10%]
89.88% – 92.00%
Transition sharpness ratio
12.21
4.15
[10.58, 13.83]
6.21 – 23.25
Figure 1. Final test accuracy across 25 independent seeds. Tight, reproducible clustering around 90.9%.
4.2 Searching for Phase Transitions (Phase 4)
For every checkpoint in every run, the silhouette score (see Section 2.5) of the probe-set hidden activations against their true labels was computed, and checkpoint-to-checkpoint jumps in that score were ranked as phase-transition candidates. Across all 25 runs, no run's accuracy trajectory was dominated by a single dramatic jump, as the sharpness ratio (see Section 2.2) ranged only from 6.2 to 23.2, with no outlier suggestive of a genuine grokking-style event.
Figure 2. Transition sharpness ratio, sorted, across 25 seeds. No run stands out as dramatically more “grokking-like” than the rest.
This can be seen as a legitimate negative result: under these settings (this learning rate, this model size, this amount of training, this task), growth was gradual and reproducibly so, not the delayed-generalization pattern seen in grokking papers. Section 6 discusses plausible reasons (no weight decay, task may be too easy relative to model capacity, training may not be long enough).
4.3 Case Study: The Step-325 Transition Candidate
Despite the negative aggregate result, the single largest available signal (the biggest test-accuracy jump in the seed-0 run, at step 325) was investigated directly. At that checkpoint, the overall test accuracy was 39.5%, but the per class breakdown revealed the aggregate curve was hiding a staged, per class learning order:
Class
Accuracy at step 325
gray
99.4%
blue
79.4%
green
69.9%
purple
20.3%
pink
4.1%
red / orange / yellow / brown / black / white
0.0% (not yet learned)
The three most-changed hidden units at this step, and what ablating them (Section 2.4) revealed:
Unit
Activation - “preferred” class
Causal effect of ablation
Interpretation
6
blue
Blue accuracy 79.4% → 0.0%; other classes barely affected.
Clean, causally-verified blue detector: activation label matches causal role.
2
green
Green accuracy 69.9% → 0.0%. Also: pink accuracy 4.1% → 44.8% (+40.7pp), blue 79.4% → 100.0% (+20.6pp), gray 99.4% → 54.7% (−44.7pp).
Real green detector, but with other classes entangled: removing it helps pink and blue while hurting gray. This is evidence of interference, not a clean single-purpose unit.
7
white (loudest activation)
White accuracy stays at 0.0% (nothing to lose — not yet learned). Gray accuracy collapses 99.4% → 21.7% (−77.7pp).
Activation-based label (“white”) is misleading: at this point in training the unit's true causal role is gray, not white, which is exactly the correlation vs. causation gap ablation is designed to catch.
4.4 Full-Unit Ablation Sweep at Convergence
Extending the case study, all 16 units (not just the top 3) were ablated at the final checkpoint (step 3,000) of every run, and each unit was classified as “dead” (zero activation and zero causal effect) or “active.”
Metric
Mean (of 16)
95% CI
Range
Dead units per run
4.32
[3.62, 5.02]
0 – 7
Active units per run
11.68
[10.98, 12.38]
9 – 16
“Phantom” units per run (high selectivity, zero causal effect)
0.60
[0.30, 0.90]
12 of 25 runs had ≥ 1 phantom units
Figure 3. Dead vs. Active hidden units per run, sorted. Effective network capacity varies substantially by seed even at matched final accuracy.
The clearest single phantom-unit example: in the seed 0 run, unit 4 had a near-perfect activation selectivity of 0.9999 for black, yet ablating it produced zero change in any test prediction. It looked like the cleanest possible detector by activation alone, but turned out to be causally inert.
Testing the “ignore the interfering unit” idea
A natural question raised during this project was the following one: if one unit is known to interfere with another class, can it be selectively ignored to sharpen predictions for that class, at no cost? This was tested directly (seed-0 run, final checkpoint): unit 8 (a blue detector found to interfere with white) was zeroed only on inputs whose true label was probably not blue, and the effect on white and on the overall accuracy was measured.
Class
Baseline accuracy
Conditional-ablation accuracy
Change
white (target class)
57.9%
71.1%
+13.2pp
purple
93.3%
59.2%
−34.1pp
black
65.7%
37.7%
−28.1pp
gray
94.5%
79.6%
−14.8pp
Overall accuracy
89.9%
84.7%
−5.2pp
The target class did improve, but overall accuracy fell: unit 8 was quietly carrying purple, black, and gray as well as blue. The idea does not deliver a free precision gain in its naive, blanket form. Any real use of it would need to be far more surgical than ignoring the whole unit whenever the target class isn't blue.
4.5 From a Binary Match Test to Rank Correlation
The original test for “does this unit's activation-based label predict its causal role” was binary: does the class with the highest mean activation equal the class most hurt by ablation? Across 25 runs:
Metric
Mean
95% CI
Binary match rate
53.3%
[47.8%, 58.7%]
This is practically a coin flip, but the binary test discards 10 of each unit's 11 class values. A concrete example (seed 0, unit 2, final checkpoint): the unit's top activation is for yellow (3.09) and its biggest causal impact is on green (0.92), in the first experiment this was counted as a full “mismatch.” But yellow and green are effectively tied for both activation (3.09 vs. 2.58) and causal impact (0.92 vs. 0.90), and the rest of the 11-class profile lines up closely too.
The Spearman correlation (see Section 2.6) for this unit's full profile (seed 0, unit 2) is 0.873; a strong relationship the binary test could not see.
Metric
Mean
95% CI
Activation vs. causal-impact correlation (all 11 classes)
0.735
[0.714, 0.755]
Figure 4. Methodology progression: from a coin-flip binary test to a strong, consistent rank correlation, and the effect of the reference-only probe set variant (see Section 4.9).
4.6 Direct Logit Attribution
Raw activation only measures how excited a unit gets, not whether that excitement is connected to an actual decision change. Multiplying a unit's activation by its outgoing weight to each class (see Section 2.7) was tested as a better predictor:
Class A
Class B
Class C
Correlation with true causal impact [0, 0.9, 0.05]
Activation only
5.0
2.0
1.0
−0.50 (points the wrong way)
Outgoing weight
0.0
3.0
0.5
—
Attribution (activation x weight)
0.0
6.0
0.5
+1.00 (correct)
Applied to the real 25-run dataset, attribution correlation was higher than activation-only correlation in every single run, with non-overlapping confidence intervals:
Metric
Mean
95% CI
Variance explained (ρ²)
Activation vs causal correlation
0.735
[0.714, 0.755]
≈ 54%
Attribution vs. causal correlation
0.850
[0.839, 0.861]
≈ 72%
Mean per run improvement: +0.115, range +0.017 to +0.231, uniform across all 25 runs. This is the single strongest, most consistent methodology improvement found in the project.
4.7 Pairwise Interactions: Initial Finding
Even a rho² of 0.72 leaves roughly 28% of a unit's causal behavior unexplained by its own single-unit attribution profile. One concrete hypothesis that surged was the following: some of that gap is explained by redundant, “backup” coding between pairs of units, invisible to any single-unit test by construction (see Section 2.10). For each run, the pair of active units with the most correlated attribution profiles was jointly ablated and compared against the sum of their individual effects.
Metric
Mean
95% CI
Overall accuracy synergy (whole model)
+0.016
[−0.004, +0.037] (crosses zero)
Worst-hit-class synergy (per pair)
+0.228
[0.169, 0.287] — all 25 runs positive
Figure 5. Left: overall-accuracy synergy is not reliably different from zero. Right: the single worst-affected class per pair shows large, consistently positive synergy in every run.
Concrete example: unit 8 and unit 13, class “white” (seed 4)
Condition
White accuracy
Baseline
56.9%
Unit 8 ablated alone
38.7% (−18.2pp)
Unit 13 ablated alone
88.0% (+31.1pp)
Naively predicted joint effect (sum of the two)
≈ 69.8% (theoretical net improvement)
Actual joint ablation (both removed together)
8.5% (−48.4pp vs. baseline)
Unit 13 looked harmless, or even beneficial when tested alone, because unit 8 was quietly covering for it. Only removing both at once revealed the true, much larger joint dependency. This pattern (a pair of units that look individually replaceable but are jointly essential) appeared in every one of the 25 runs, for at least one class each time, even though the whole model level synergy averaged out to almost statistical noise.
4.8 Large-Scale Test: Can Interaction Be Predicted?
The initial pairwise result used only the single most-correlated pair per run (25 data points total), which turned out to be a search over 25 pairs x 11 classes for the single most extreme result, risking “selection bias”. A full sweep tested every pair of active units in every run (up to 120 pairs per run), producing 17,578 (pair-class) observations, and asked whether a per class interaction term, attribution_u[c] x attribution_v[c], would predict the measured synergy.
Test
Pearson r
Spearman r
Interaction term vs. synergy (raw accuracy space)
−0.304
+0.013 (≈ 0, no relationship)
Simple additive sum vs. synergy (raw accuracy space)
−0.310
—
Figure 6. Interaction term vs. measured synergy across all 17,578 (pair - class) observations. No visible positive predictive relationship.
Nonetheless, a structural flaw was identified: because accuracy is bounded at 0% and 100%, the naive “sum of two bounded drops” can exceed what is physically possible (more than 100%), mechanically biasing the synergy metric negatively whenever individual effects are large. Recomputing everything in logarithmic (logit) space, which is unbounded, removed this artifact but did not rescue the hypothesis:
Test (logit-space, ceiling artifact removed)
Pearson r
Spearman r
Interaction term vs. synergy
−0.185
−0.139 (small, consistent negative)
A separate check on the original 25 top-pair heuristic (does “how similar two units' attribution profiles look” predict “how much synergy they show”) also came back weak: Pearson r = 0.167.
In other words, the seed 4 unit 8/unit 13 case was real and correctly measured, but it is an outlier, not evidence of a general, predictable pattern. Pairwise redundancy is a genuine, replicated phenomenon (see Section 4.7) that current methods cannot yet predict in advance.
This is an honest negative result carried forward as future work (see Section 7).
4.9 Statistical Significance Filtering (Robustness Check)
Every per class accuracy drop had been treated as a real effect regardless of size. A conservative two-proportion z-test (see Section 2.9) was used to flag drops not statistically distinguishable from zero, replacing them with 0 before recomputing the correlations.
Metric
Before filtering
After filtering
Change
Activation vs. causal correlation
0.735 [0.714, 0.755]
0.756 [0.737, 0.776]
+0.021 (within noise)
Attribution vs. causal correlation
0.850 [0.839, 0.861]
0.848 [0.834, 0.862]
−0.002 (no change)
On average, 7.73 of each unit's 11 per class drops (about 70%) were statistically significant at the 95% level. Filtering out the rest barely moved either headline number. This was a reassuring robustness result: the 0.735 and 0.850 correlations were not artifacts of noisy small effects being averaged in.
4.10 Reference-Color-Only Test and Probe-Set Imbalance
The 166-point probe set is 60% deliberately ambiguous boundary points: a human being would struggle to classify correctly each color as they were basically mixtures of class colors. Recomputing activation and attribution profiles using only the 11 unambiguous reference colors tested whether that ambiguity was limiting the correlations.
Metric
Full probe set (166 pts)
Reference-only (11 pts)
Change
Activation vs. causal correlation
0.735 [0.714, 0.755]
0.743 [0.726, 0.761]
+0.008 (within noise)
Attribution vs. causal correlation
0.850 [0.839, 0.861]
0.812 [0.799, 0.826]
−0.038 — non-overlapping CIs, a real decrease
Contrary to the hypothesis, restricting to reference colors made attribution correlation significantly worse. The likely explanation is the sample size, not ambiguity: the full probe set gives each class multiple points to average a unit's activation over, while the reference-only set gives exactly one point per class, which is basically a single noisy sample with no averaging protection.
Class
Reference pts
Midpoint pts
Boundary pts
Total
red
1
3
4
8
orange
1
7
7
15
yellow
1
2
5
8
green
1
4
13
18
blue
1
3
8
12
purple
1
4
10
15
pink
1
7
15
23
brown
1
10
12
23
black
1
0
4
5
white
1
0
2
3
gray
1
15
20
36
Figure 7. Probe points per class in the full 166-point set. White (3 points) and gray (36 points) differ by 12x, a real design imbalance worth correcting in future runs.
This imbalance is a genuine limitation of the current probe set (see Section 6), not something corrected in this project: classes with very few probe points (white, black) likely have less reliable activation estimates throughout this entire report, not just in the reference-only test.
5. Did Watching Training Reveal Anything “Post-Hoc” Analysis Would Have Missed?
This is the actual test of the project's founding hypothesis, and the honest answer has three parts.
What a “post-hoc” only analysis of the final model would have shown
A 90.9% accurate classifier, 251 weights, and if the same activation, attribution, and ablation analysis were run on the finished model then some clean unit detectors, some dead units, and a finished function would be noticed. Nonetheless, it would not show that the model learned roughly five color classes first and the other six later (see Section 4.3), because by the final checkpoint that ordering has been overwritten by hundreds of subsequent updates. Additionally, it would not show that unit 6 became a stable blue detector early while unit 2 spent time actively interfering with pink and blue decisions before finally settling down. Finally, it would not show that unit 7's role at 10% of the way through training (functionally gray, despite looking like white) completely changed, as a post-hoc snapshot has no record that a unit's role differed at all.
What the developmental approach specifically provided us
What this project does not establish
First of all, this project does not prove developmental interpretability is able to be scaled and still maintain a strong behaviour correlation. This is a 251-parameter model on a 3-input, 11-class toy task, chosen precisely to minimize superposition. As such, the dominant sources of error were the dead units and specific interference cases, not dense superposition. This project aimed to prove that developmental interpretability is a valid approach to build a significantly accurate unit behaviour schema, the follow up question is how can this approach be modified and scaled to account for superposition. This project is a stepping stone, not the solution of the black box problem with neural networks.
The main project limitations are recapped and listed in the next section.
6. Limitations
7. Future Work
Listed below is a collection of ideas to test in a future experiment, these can be interpreted as a guide on where to expand the project:
8. Conclusion
All in all, this project set out to test a specific, falsifiable hypothesis: watching a neural network's internals during training reveals structure a "post-hoc" analysis of the finished model would miss. Across 25 independently seeded runs of a completely manually inspectable 251-parameter model, the evidence supports that claim in this toy setting.
The developmental record revealed a per class learning strategy invisible in the aggregate accuracy curve, and a verified case of a hidden unit's activation-based “meaning” diverging from its true function. This is exactly the kind of finding only visible by inspecting an interior training checkpoint, and not the final model.
On the methodology side, the project underwent a complete set of improvements allowing the results to go from a naive binary test that looked close to a coin flip, then replaced by a rank-correlation test that revealed a real and more positive relationship (0.735), and then improved further by a mechanistically-grounded attribution measure (0.850). At the end, it was stress tested against three theoretical concerns: pairwise interactions (real but not yet actaully predictable), statistical noise (not a meaningful data altering factor), and probe set design (a real, quantified limitation).
As a final remark, it is important to once again point out that this experiment does not prove that developmental interpretability is a reliable and working method for large neural networks, but rather as an introduction into the capabilities and limitations that this method can provide to the AI interpretability research space.
(Sources, and both appendices, including the per run data and python script references, can be found below.)
Sources
Full list of all sources and references for research and inspiration:
by Jesse Hoogland, Alexander Gietelink Oldenziel, Daniel Murfet, Stan van Wingerden
https://www.lesswrong.com/s/SfFQE8DXbgkjk62JK/p/TjaeCWvLZtEDAS5Ex
by Nelson Elhage et al.
https://transformer-circuits.pub/2022/toy_model/index.html
by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra
https://arxiv.org/abs/2201.02177
by Peter J. Rousseeuw
https://www.sciencedirect.com/science/article/pii/0377042787901257
by Ashkan Golgoon, Khashayar Filom, Arjun Ravi Kannan
https://arxiv.org/abs/2407.11215
Appendix A: Per-Run Summary Table
Full results for all 25 independently seeded runs, sorted by seed.
Seed
Final acc.
Sharpness
Dead
Active
Match %
Act. ρ
Attr. ρ
Phantom
0
0.9074
11.08
5
11
45.5%
0.766
0.855
1
1
0.9002
10.79
4
12
25.0%
0.761
0.855
1
2
0.9022
8.04
1
15
66.7%
0.764
0.839
0
3
0.9114
12.43
6
10
80.0%
0.773
0.856
0
4
0.9096
8.03
3
13
53.8%
0.697
0.871
0
5
0.9066
23.25
2
14
35.7%
0.709
0.857
1
6
0.9016
18.26
5
11
45.5%
0.778
0.890
0
7
0.9154
8.56
6
10
50.0%
0.728
0.858
0
8
0.9070
11.71
5
11
54.5%
0.800
0.852
0
9
0.9126
7.97
0
16
75.0%
0.700
0.860
1
10
0.9060
8.14
4
12
66.7%
0.670
0.825
1
11
0.9022
13.14
7
9
55.6%
0.652
0.883
1
12
0.9132
11.60
5
11
72.7%
0.774
0.887
0
13
0.9114
15.82
6
10
60.0%
0.765
0.836
0
14
0.8988
11.03
4
12
41.7%
0.690
0.861
1
15
0.9182
12.46
6
10
40.0%
0.659
0.789
0
16
0.9140
10.39
6
10
50.0%
0.774
0.866
0
17
0.9036
6.21
6
10
40.0%
0.659
0.782
3
18
0.9154
19.03
5
11
54.5%
0.775
0.882
1
19
0.9132
13.53
4
12
50.0%
0.700
0.868
1
20
0.9068
11.69
4
12
41.7%
0.793
0.810
0
21
0.9200
6.24
1
15
66.7%
0.682
0.838
2
22
0.9048
14.89
3
13
69.2%
0.714
0.835
1
23
0.9120
15.96
5
11
54.5%
0.845
0.865
0
24
0.9050
14.94
5
11
36.4%
0.736
0.824
0
Appendix B: Script Reference
All scripts referenced in this report are available in the following GitHub repository:
https://github.com/Pedro-Oub/Developmental-Interpretability-Report-Resources
All programs are documented with an in-file explanation of its purpose. Ideally, files should be run in the order stated in the table below as later code might rely on files created by previous programs. Feel free to test all data and any contributions or improvements (see Section 7) are appreciated.
Script
Purpose
dataset.py
Phase 0: Task definition and truth labeling rule.
model.py
Phase 1: ColorNet model definition, 3→16→11 MLP.
probe_set.py
Phase 2: Builds the fixed 166-point probe set.
train.py
Phase 2/3: Instrumented training loop with step-cadence checkpointing.
run_experiment.py
Runs train.py + analyze.py + sweep.py for a list of seeds in one command.
analyze.py
Phase 4: silhouette-based transition detection.
interpret.py
Phase 5: unit-level weight-delta ranking, selectivity, and top-3 ablation for one flagged step.
sweep.py
Phase 5 extension: full 16-unit ablation sweep, activation and attribution correlation metrics, conditional-ablation test.
run_sweeps.py
Runs sweep.py across every run directory in one command.
pairwise.py / run_pairwise.py
Top 1 pair per-run pairwise ablation and synergy test (see Section 4.7).
pairwise_full.py / run_pairwise_full.py
Full combinatorial pairwise sweep and interaction-term validation (see Section 4.8).
significance_filter.py
Two-proportion z-test filtering of accuracy drops (see Section 4.9).
extract_weights.py
Extracts the fc2 output-weight matrix from a checkpoint (needed for the reference-only test).
reference_only_test.py
Reference color only activation/attribution profile test (see Section 4.10).
© 2026 Pedro Oubiña. All rights reserved.